MCP server that uses Microsoft OmniParser to visually parse the current screen and then drive GUI automation (Windows-focused, but cross-platform possible).
https://github.com/NON906/omniparser-autogui-mcpStop fighting with brittle UI selectors and hardcoded coordinates when automating desktop applications. This MCP server brings Microsoft's OmniParser directly into your AI workflow, letting you describe GUI elements in plain English while it handles the visual recognition and automation.
Traditional GUI automation breaks every time an interface changes. You're constantly updating selectors, tweaking coordinates, and maintaining fragile scripts. But what if your AI assistant could actually see your screen and interact with applications the same way you do?
That's exactly what this MCP server delivers. Instead of hunting down CSS selectors or accessibility IDs, you simply tell Claude "click the Save button" or "fill in the username field" and it visually identifies and interacts with those elements in real-time.
Works with Any Application: Legacy desktop apps, custom software, games - if you can see it, this can automate it. No APIs required.
Natural Language Targeting: Describe elements as you would to a human colleague. "The blue Submit button in the bottom right" works better than memorizing element IDs.
Resilient to UI Changes: Minor interface updates won't break your automation since it recognizes elements visually rather than relying on brittle selectors.
Cross-Application Workflows: Seamlessly automate across multiple applications in a single conversation with Claude, even when those apps have no integration APIs.
Remote Processing: Run the computationally intensive OmniParser processing on a separate machine while keeping the lightweight automation client local.
Legacy System Integration: That ancient ERP system your company refuses to upgrade? Now you can automate data entry and extraction without reverse-engineering proprietary APIs.
Cross-Platform Desktop Testing: Validate application behavior across different operating systems by describing test scenarios in natural language rather than maintaining separate automation scripts.
Document Processing Workflows: Automatically extract data from PDF viewers, image editing tools, or any application displaying documents by describing what information you need.
Gaming and Entertainment: Automate repetitive tasks in games or creative software where traditional automation tools fall short.
Multi-Application Data Pipelines: Pull data from one desktop application, process it, and input it into another without writing custom integrations.
Setting up visual GUI automation traditionally requires computer vision expertise and complex image processing pipelines. This MCP server handles all that complexity behind a simple interface that works with any MCP-compatible AI client.
The server runs locally (with optional remote processing for performance), takes screenshots, processes them through OmniParser's visual understanding models, and executes the requested actions through PyAutoGUI. You get enterprise-grade computer vision capabilities without the enterprise-grade complexity.
Configure it once in your MCP client, and you can start automating any desktop application immediately. Target specific windows to avoid interference, or work across your entire desktop environment - the choice is yours.
Quick Setup Example:
{
"mcpServers": {
"omniparser_autogui_mcp": {
"command": "uv",
"args": ["--directory", "/path/to/omniparser-autogui-mcp", "run", "omniparser-autogui-mcp"],
"env": {
"OCR_LANG": "en",
"TARGET_WINDOW_NAME": "My Application"
}
}
}
}
This MCP server turns your AI assistant into a visual automation powerhouse that works with any application, understands context like a human would, and adapts to interface changes automatically. Perfect for developers who need reliable desktop automation without the traditional maintenance overhead.