web-eval-agent vs Playwright MCP (ExecuteAutomation)

A side-by-side comparison of two Browser Automation servers — tools, transport, auth, maintenance, and copy-paste config for each.

	web-eval-agent Autonomous browser agent that evaluates and debugs your web app end-to-end from your IDE. Unverified stdio (local) API key Python	Playwright MCP (ExecuteAutomation) Popular community Playwright + API testing MCP server with codegen, screenshots, and device emulation. Verified stdio (local) No auth TypeScript
Category	Browser Automation	Browser Automation
Language	Python	TypeScript
Transport	stdio (local)	stdio (local)
Auth	API key	None
GitHub stars	1.2k	4.5k
Last commit	4 months ago	1 month ago
Verified
Actively maintained
Install	`curl -LSf https://operative.sh/install.sh -o install.sh && bash install.sh && rm install.sh`	`npx -y @executeautomation/playwright-mcp-server`
Repo	Open	Open

Verdict

Pick web-eval-agent if you want the official, vendor-maintained server.

Pick Playwright MCP (ExecuteAutomation) if you prefer the more popular, battle-tested option, and you want zero-setup with no API key.

Add web-eval-agent

Install / run

curl -LSf https://operative.sh/install.sh -o install.sh && bash install.sh && rm install.sh

Paste into ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "web-eval-agent": {
      "command": "uvx",
      "args": [
        "--refresh-package",
        "webEvalAgent",
        "--from",
        "git+https://github.com/Operative-Sh/web-eval-agent.git",
        "webEvalAgent"
      ],
      "env": {
        "OPERATIVE_API_KEY": "<YOUR_KEY>"
      }
    }
  }
}

Requires `uv` (the Python package runner). Install it from https://docs.astral.sh/uv/ if `uvx` is not found.

Add Playwright MCP (ExecuteAutomation)

Install / run

npx -y @executeautomation/playwright-mcp-server

Paste into ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "playwright-mcp-executeautomation": {
      "command": "npx",
      "args": [
        "-y",
        "@executeautomation/playwright-mcp-server"
      ]
    }
  }
}

FAQ

web-eval-agent or Playwright MCP (ExecuteAutomation) — which is better?

Pick web-eval-agent if you want the official, vendor-maintained server. Pick Playwright MCP (ExecuteAutomation) if you prefer the more popular, battle-tested option, and you want zero-setup with no API key.

Can I use both web-eval-agent and Playwright MCP (ExecuteAutomation)?

Yes — MCP clients let you enable multiple servers at once. Add both configs to your client's mcpServers and use whichever tool fits the task.

Do web-eval-agent and Playwright MCP (ExecuteAutomation) work with Claude, Cursor and Windsurf?

Both do. Copy the per-client config below into Claude Desktop, Cursor, or Windsurf.

View web-eval-agent →View Playwright MCP (ExecuteAutomation) →More comparisons →