dsh-vision-toolkit

When you run DeepSeek or another text-only model in DeepSeek Harness (DSH), familiar problems appear quickly: the model cannot see a screenshot, generic image descriptions miss the point, buttons have no usable coordinates, and a rebuilt page may look “close enough” without any way to measure the remaining difference.

DSH Vision Toolkit packages agent-vision-toolkit as a native DSH plugin. It helps an agent do more than describe an image: the agent can read, locate, crop, trace, rebuild, and verify visual work around the task at hand.

Install and use it immediately. The default setup includes a free Gemma 4 vision service and requires no API key. Cropping, pixel diffing, color analysis, foreground extraction, SVG tracing, and HTML screenshots run locally without spending vision API requests.

dsh plugin --profile web add @anionex/dsh-vision-toolkit

Upstream toolkit: Anionex/agent-vision-toolkit · Project website: agent-vision.anionex.me

Recent updates

• 2026-08-16 · Windows Python: Added Microsoft Store Python support, fixing first-time isolated-runtime setup failures for affected Windows users.
• 2026-08-16 · Better free vision: Switched the default model to Gemma 4, improving the no-key image-understanding path.
• 2026-08-16 · Image paste: Text-only routes now switch to a (Vision Toolkit) variant and keep a workspace path, fixing blocked pastes and images that could not be reused later.
• 2026-08-16 · Higher free quotas: Raised per-client, global, and burst limits to 100/day, 400/day, and 20/minute, reducing avoidable rate-limit failures while the shared capacity is lightly used.
• 2026-08-16 · Real model test: Added a full image-request test in Settings, fixing the false confidence caused by a successful /models request to a model that still cannot process images.

Problems it solves

See it in action

Paste an image directly into DSH

Paste an image into the conversation. A text-only model can switch to its Vision Toolkit variant and inspect the image in the context of the user's question.

Screenshot to editable page

Left: the reference screenshot. Right: an editable HTML/CSS result. The result can continue into screenshot rendering and pixel comparison instead of ending as an image description.

Sketch to working interface

Left: a hand-drawn reference. Right: the working interface reconstructed from it.

Turn “looks close” into a verifiable result

The repository includes a reproducible UI-restoration example. The first implementation differs from the reference by 6.04%. After the highlighted regions are corrected, the final 1200 × 720 render reaches 0% pixel difference.

Highlights

• Free by default. New installations use the built-in Gemma 4 service without requiring another account or API key.
• Focused on the current task. The agent sends the reason it needs to inspect the image, so the result emphasizes useful evidence instead of producing a generic caption.
• Outputs you can keep working with. Coordinates, OCR, transparent PNGs, SVGs, screenshots, heatmaps, and JSON can feed directly into the next step.
• Built for UI and screenshot work. Reference analysis, element location, asset extraction, HTML rendering, and pixel comparison form one continuous workflow.
• Local where possible. Crop, trace, pixel diff, color, foreground, and HTML screenshot operations do not need a remote vision model.
• The same capabilities in Web and Headless. Web users can preview and download artifacts; Headless runs still receive replayable structured results and workspace paths.

Quick start: three steps

1. Install

dsh plugin --profile web add @anionex/dsh-vision-toolkit

You can install it into a Headless Profile too:

dsh plugin --profile headless add @anionex/dsh-vision-toolkit

2. Restart and check it

Restart a running Web Profile, then open Settings → Vision Toolkit. The free provider is already configured; run Test vision model to confirm it is reachable.

The first start prepares an isolated runtime, so it needs access to the Python package cache or the network. A normal installation does not require an agent-vision-toolkit source checkout or a local path setting.

3. Paste an image and describe the outcome you want

Paste a screenshot into the conversation or place an image in the session workspace, then invoke /vision-tools. For example:

Inspect this screenshot. Explain the error and tell me what to fix first.
Find the login button in the top-right corner, return original pixel coordinates, and make a boxed preview.
Crop this icon and convert it to SVG.
Rebuild the page from reference.png. After each pass, render it and run a pixel diff until the major differences are gone.

Common workflows

Toolbox

The plugin provides 10 tools that can be called independently or composed into a workflow:

Coordinates always use original-image pixels in x1,y1,x2,y2 form, so grounding output can feed directly into cropping, tracing, or later automation.

For a long HTML document, pass fullPage=true. The requested width and height remain the layout viewport, while the resulting PNG covers the complete document and reports pageHeight in CSS pixels.

How it works

The plugin keeps image understanding and deterministic local image processing in one Agent workflow. Expand the flow below for the implementation boundary.

flowchart LR
    Image["Screenshot or local HTML"] --> Skill["vision-tools Skill"]
    Skill --> Agent["Text agent selects a task"]
    Agent --> Vision["Use a vision model when image understanding is needed"]
    Agent --> Local["Run crop, SVG, and pixel work locally"]
    Vision --> Result["Answer, OCR, coordinates"]
    Local --> Artifact["PNG, SVG, heatmap, JSON"]
    Result --> Session["Continue reasoning and acting"]
    Artifact --> Session

Configuration and limits

Built-in free service

The default setup uses:

Base URL: https://vision.anionex.me/v1
Model:    gemma-4-26b-a4b-it
API Key:  no user configuration required

This is a shared zero-configuration entry point, not an unlimited private endpoint. Current limits are:

The limits protect shared capacity and prevent unusually large images from monopolizing memory or request time. When a limit is reached, the service returns a readable reason and error code. Rate-limit responses also include Retry-After instead of collapsing into an unexplained model failure.

Bring your own vision model

For higher quotas, private endpoints, or another model, change the provider in Settings → Vision Toolkit and store the API key as a DSH Credential. Settings stores the Credential reference and never reads the saved secret back into the browser.

You can also configure a Profile patch:

- id: vision-toolkit
  config:
    provider:
      baseUrl: https://api.example.com/v1
      credential: MY_VISION_KEY
      model: your-vision-model
      protocol: openai

OpenAI Chat Completions-compatible endpoints and Anthropic Messages are supported. The Web Settings panel exposes the full provider, runtime, timeout, image-limit, and image-input-variant configuration.

Requirements

• A DeepSeek Harness Web or Headless Profile.
• Node.js ^22.19.0 or >=24.0.0.
• Python 3.11+; the plugin prepares an isolated environment by default.
• Only vision_html_screenshot requires Chrome, Chromium, or Edge.
• Inputs must be PNG, JPEG, GIF, or WebP files in the session workspace or an explicitly allowed directory.

dsh plugin --profile web update @anionex/dsh-vision-toolkit
dsh plugin --profile web remove @anionex/dsh-vision-toolkit

- id: vision-toolkit
  disabled: true

Plugin updates

In Settings → Vision Toolkit, Check for updates queries the Profile's npm registry. For a direct registry installation, Update and restart installs only the exact version you confirmed, verifies it, and restarts an explicitly opted-in POSIX Web process on a fixed --port. Local/workspace/file/git/URL installs, Windows, dynamic ports, read-only Profiles, and manager-owned processes remain check-only.

The updater revalidates the Profile before mutation, snapshots the original manifest and lockfile, and holds a token-owned cross-process lock. The current Web process exits only after the restart helper confirms that the backup is readable and the lock handoff succeeded. When the Profile was already operational, the replacement must report both the target plugin version and a ready runtime; failed replacements restore the original manifest/lockfile and rebuild dependencies with a frozen lockfile before retrying the previous exact version. If automatic recovery itself fails, the backup and lock are preserved and their paths are written to $DSH_HOME/logs/vision-toolkit-restart.log. Detached restart requires DSH_VISION_TOOLKIT_ALLOW_DETACHED_RESTART=1; unsaved Settings or API-key input blocks installation.

Troubleshooting

Project status and limitations

The current release focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification. It is not a video, audio, or camera-input system and does not automatically click GUI controls. Interactive box editing, remote service clusters, model voting, and cross-session visual caches are also outside the current scope.

Development and community

pnpm install --frozen-lockfile --trust-lockfile
pnpm run verify:portable
pnpm run build
pnpm test
TSX_TSCONFIG_PATH=tsconfig.json pnpm dlx tsx scripts/ui-restoration-example.ts --check

• Read CONTRIBUTING.md before contributing.
• Use GitHub Issues for bugs, focused feature requests, and usage questions; see SUPPORT.md for channel guidance.
• Report vulnerabilities privately through SECURITY.md.
• See CHANGELOG.md for releases and FUNDING.md for sponsorship details.
• Visit upstream agent-vision-toolkit for the general toolkit, cross-agent integrations, and visual-task playbooks.

agent-vision-toolkit was created by Anionex. This repository maintains its native DeepSeek Harness integration.

License

The plugin is available under the MIT License. The packaged upstream snapshot retains its original MIT license in vendor/agent-vision-toolkit/LICENSE.