HomeProjectsArticlesPK BoardAbout
Explore Projects

Midscene.js

Active
GitHub TypeScript MIT

Description

AI-powered vision-driven UI automation that lets you describe actions in natural language instead of writing selectors, supporting browser and mobile platforms

Key Features

  • AI-powered vision-driven UI automation using natural language descriptions instead of selectors
  • Cross-platform support: Web (Puppeteer/Playwright), Android (adb), and iOS (WebDriverAgent)
  • Pure vision localization — no DOM dependency for element interaction, reducing token cost
  • Supports multiple vision-language models: Qwen3.x, Doubao-Seed-2.0, GLM-4.6V, UI-TARS
  • Three API layers: Interaction API, Data Extraction API, and Utility API (aiAssert, aiLocate, aiWaitFor)
  • MCP integration exposing Midscene Agent actions as tools for upper-layer agent orchestration

Use Cases

💡 Automating web UI testing with natural language test scripts across browsers
💡 Building cross-platform mobile automation for Android and iOS app testing
💡 Creating AI agent workflows that interact with any user interface visually
💡 Prototyping UI automation without writing fragile CSS/XPath selectors

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (14.8k stars)
  • • Permissive open-source license (MIT)
  • • Established track record (2 years in production)

Tags

browser-automation ui-testing vision playwright natural-language typescript

Categories

🌐 Browser Agent

Quick Start

npm install @midscene/web && npx @midscene/web playground
Visit GitHub Visit Website

Related Projects

MCP Playwright

5.6k · TypeScript
Stale B

Playwright Model Context Protocol server for automating browsers and APIs in Claude Desktop, Cline, Cursor IDE and other AI coding tools

mcpplaywrightbrowser-automation +2
  • · MCP server enabling LLMs to interact with real browser environments via Playwright for web automation, scraping, and testing
  • · 143 real device presets for accurate emulation including iPhone, iPad, Pixel, Galaxy and desktop browsers
  • · Automatic browser binary installation — Chromium, Firefox, and WebKit downloaded on first use with no manual setup
View full details → View on GitHub

HyperAgent

1.6k · TypeScript
Stale C

HyperAgent is a Playwright-based AI browser automation framework offering high-level APIs like page.ai(), page.perform(), and page.extract(). It features built-in MCP client support and action caching, enabling AI agents to browse, interact, and extract data using natural language.

browser-automationplaywrightai-agent +3
  • · Natural language AI commands drive browser automation, replacing brittle scripts
  • · Three-tier API system: page.ai(), page.perform(), page.extract() for simple to complex scenarios
  • · Built-in MCP client connects to tools like Composio for full workflow orchestration
View on GitHub

AIPex

1.2k · TypeScript
Active A

AI browser automation assistant as a Chrome extension, privacy-first with MCP support, alternative to Claude Chrome and Manus Browser Operator

browser-automationchrome-extensionmcp +2
  • · Zero-migration browser automation — runs in your existing Chrome/Edge browser, no new app to install
  • · Privacy-first BYOK model — your data never leaves your machine, bring your own API key
  • · MCP protocol support — 30+ browser automation tools accessible via Model Context Protocol
View on GitHub

Magnitude

4.1k · TypeScript
Stale B

An open-source, vision-first browser agent that drives web automation through visual understanding, supporting complex web interaction tasks for QA testing and workflow automation.

vision-firstbrowser-automationweb-agent +2
  • · Vision-first architecture: visually grounded LLM specifies pixel coordinates for true DOM-independent generalization
  • · Navigate: sees and understands any interface to plan out actions
  • · Interact: executes precise actions using mouse and keyboard control
View on GitHub

Related Articles

Browser AgentWeb 自动化Playwright

Browser Agents in Practice: Architecture and Pitfalls of AI-Controlled Browsers

Breaking down three abstraction layers for browser automation—from raw Playwright to structured extraction—with production patterns, runnable code, and common pitfalls.

Read more →