ScreenMind

**Captures your screen → Analyzes with Gemma 4 → Builds a searchable AI memory**
**100% local. 100% private. Zero cloud dependencies.**
[](https://pypi.org/project/screenmind/) [](https://github.com/ayushh0110/ScreenMind/actions/workflows/ci.yml) [](https://codecov.io/gh/ayushh0110/ScreenMind) [](https://python.org) [](https://ai.google.dev/gemma) [](https://github.com/ggerganov/llama.cpp) [](LICENSE) [](MCP_SETUP.md) [](https://screenmind.pages.dev)
[**Website**](https://screenmind.pages.dev) · [**Features**](#-features) · [**Comparison**](#-how-screenmind-compares) · [**Gemma 4 Deep Dive**](#-how-gemma-4-is-used) · [**Quick Start**](#-quick-start) · [**Architecture**](#-architecture) · [**Agent Platform**](#-agent-platform) · [**MCP**](#-mcp-server-claude--cursor--vs-code) · [**API**](#-api-reference)
 | Agents | |:---:| |  |
**💬 Chat in Action** — *Ask anything about your screen history* 

Microsoft showed the world wants screen-aware AI with Recall. But Recall stores data in plaintext, sends telemetry, and was met with massive privacy backlash. ScreenMind is the open-source, privacy-first alternative — every screenshot analyzed, every insight generated, every search result — all computed locally using Gemma 4's multimodal capabilities.

It's not just a screen recorder. It's an AI memory you can talk to, search through, and build automations on top of.

✨ Features

🧠 Core Intelligence

• 📸 Smart Capture — Content-change detection, not a fixed timer. Captures when your screen actually changes.
• 🔬 Gemma 4 Vision Analysis — Every screenshot analyzed: app detection, activity categorization, mood, scene description, spatial layout regions.
• 🔍 Hybrid Search — Semantic embeddings (MiniLM) + FTS5 keyword search. Find anything by meaning, not just keywords.
• 💬 Chat with Memory — Conversational RAG with follow-up support. Ask "what did Alex say on Discord?" → get the actual message.
• 🧠 Model Hub — In-app model download with variant selection (Q4_0, Q8_0, BF16, etc.). Supports Gemma 4 E2B, E4B, and 12B. Chat and Summary are locked with witty brain animations until the model is ready — then auto-unlock. No terminal needed.
• 🎙️ Voice Memos — Hold Ctrl+Shift+V → Gemma 4's native audio encoder transcribes. Screenshot captured alongside. Audio always saved, even if transcription fails.
• 🎤 Meeting Transcription — Auto-detects Zoom/Teams/Meet, records audio, transcribes, generates structured summaries.
• 📊 Analytics Dashboard — Category breakdown, top apps, hourly heatmap, meeting stats, focus metrics.
• ⏪ Day Rewind — Timelapse playback of your entire day with play/pause/scrub/speed controls.
• 🚀 Desktop Launcher — Splash screen with auto-open dashboard. Desktop shortcut auto-created on first run.
• 🔄 Run at Startup — Cross-platform auto-start (Windows Registry, macOS LaunchAgent, Linux XDG). Toggle from Settings or CLI.

⚡ Performance

• Three Analysis Modes — Accurate (~76s, deep thinking + layout), Balanced (~40s, thinking), or Fast (~12s, no thinking). You choose.
• Per-App pHash Cache — 3-tier caching with app-aware staleness. Communication apps refresh faster than IDEs. Significantly fewer inference calls.
• Chat-First GPU Priority — Chat cancels in-flight analysis instantly. GPU freed in <1s.
• Parallel Pipeline — Dev context detection and embedding generation run concurrently (~2-3s savings per frame).
• Auto-Pause Heavy Apps — Games, video editors, 3D software detected → capture pauses automatically.

🔒 Privacy & Security

• 100% Local — All data stays on your machine. Zero network calls after initial model download. No telemetry. Ever.
• Sensitive Data Filter — Auto-redacts credit cards, SSNs, API keys, passwords before storage.
• Encryption at Rest — AES encryption for screenshots (Fernet + OS keyring).
• Dashboard PIN Lock — Session-based auth with configurable auto-lock timeout.
• Incognito Mode — One-click pause. Nothing recorded.

⌨️ System-Wide Hotkeys

All hotkeys customizable from Settings.

📊 How ScreenMind Compares

TL;DR: ScreenMind is the only option that's fully MIT open-source, runs on any hardware (including a $150 GPU), handles vision + audio + reasoning with a single local model, and lets you actually chat with your screen memory.

🧠 How Gemma 4 Is Used

Gemma 4 is not a bolt-on — it's architecturally load-bearing. ScreenMind uses all three modalities (E2B shown as default; E4B and 12B also supported):

1. Vision — Screenshot Analysis

Every screenshot is sent to Gemma 4 with OCR context. It returns structured JSON: - App name, activity category, summary, detailed context - Mood classification, confidence score - Rich scene description (every visible element inventoried) - Layout regions (sidebar, chat area, toolbar boundaries)

Three modes (benchmarked on GTX 1650 4GB — scales dramatically with better GPUs): - Accurate — single call with thinking (~76s). Best layout detection. - Balanced — thinking enabled, analysis-only (~40s). Richer descriptions than Fast. - Fast — no-thinking prefill trick (~12s). Layout via OCR clustering instead.

2. Audio — Voice Memos & Meeting Transcription

Gemma 4's encoder-free architecture handles audio natively across all model sizes. ScreenMind uses it for: - Voice memo transcription (hold hotkey → speak → release) - Meeting transcription (15s chunks, map-reduce summarization for long meetings)

No Whisper dependency. One model handles everything.

3. Reasoning — Summaries, Chat, Agents

• Daily summaries with deep reasoning (think=True)
• Chat answers grounded in actual screen data (text-first RAG with vision fallback)
• Agent execution — Gemma processes markdown agent prompts with injected screen data

Supported Models

E2B is the recommended default — it checks all the boxes:

If you have more VRAM, E4B offers richer analysis. The 12B model gives the best text reasoning for complex prompts.

Models are sourced from ggml-org (official llama.cpp-compatible GGUFs) and stored in ~/.screenmind/models/.

🚀 Quick Start

Requirements: Python 3.10+ · GPU recommended (4GB+ VRAM) · ~2GB disk for model (E2B Q4_0)

1️⃣ Install

pip install screenmind

2️⃣ Run

screenmind              # Normal start (foreground)
screenmind --launch     # Splash screen + auto-open dashboard
screenmind --background # Run silently without console window

3️⃣ Open → http://127.0.0.1:7777

On first run, ScreenMind will: - Prompt to install AI packages (~2.5GB one-time download) - Auto-detect your GPU and download llama-server if not found (CUDA/CPU auto-selected) - Open the Model Hub — pick your model + quantization variant, download with progress tracking right in the UI - Chat and Summary stay locked (🧠💤 "I need my brain to think!") until the model is ready, then auto-unlock - Start llama-server in background (or adopt an already-running external server) - Show the welcome screen to set up an optional PIN - Create ~/.screenmind/ for data storage, ~/.screenmind/models/ for GGUF files - Auto-create a desktop shortcut

screenmind --install-startup    # Register to start at system login
screenmind --uninstall-startup  # Remove from system startup
screenmind --install-shortcut   # Create desktop shortcut
screenmind --version            # Show version
screenmind --help               # Show all options

git clone https://github.com/ayushh0110/ScreenMind.git
cd ScreenMind

python -m venv venv
venv\Scripts\activate        # Windows
# source venv/bin/activate   # macOS/Linux

pip install -e ".[ai,dev]"

cp .env.example .env
# Edit capture interval, blocked apps, hotkeys, etc.

🏗️ Architecture

For a full deep-dive into threading, caching, search internals, and the privacy pipeline, see ARCHITECTURE.md.

┌─────────────────────────────────────────────────────────────────────┐
│                          ScreenMind                                  │
│                                                                     │
│  ┌────────────┐    ┌──────────────┐    ┌─────────────────────────┐ │
│  │  Capture   │───▶│  Async Queue │───▶│    Analysis Worker      │ │
│  │  Worker    │    │  (max: 100)  │    │                         │ │
│  │            │    └──────────────┘    │  ┌───────────────────┐  │ │
│  │ • Screen   │                        │  │  Per-App pHash    │  │ │
│  │ • Window   │                        │  │  Cache (3-tier)   │  │ │
│  │ • Dedup    │                        │  └───────────────────┘  │ │
│  │ • A11y     │                        │           │             │ │
│  │ • Privacy  │                        │           ▼             │ │
│  └────────────┘                        │  ┌───────────────────┐  │ │
│                                        │  │   EasyOCR         │  │ │
│  ┌────────────┐                        │  │   (text extract)  │  │ │
│  │   Audio    │                        │  └───────────────────┘  │ │
│  │   Worker   │                        │           │             │ │
│  │            │                        │           ▼             │ │
│  │ • Meeting  │                        │  ┌───────────────────┐  │ │
│  │   detect   │                        │  │   Gemma 4 E2B     │  │ │
│  │ • Record   │                        │  │   (via llama.cpp) │  │ │
│  │ • Transcr. │                        │  │   Vision + Audio  │  │ │
│  └────────────┘                        │  └───────────────────┘  │ │
│                                        │           │             │ │
│  ┌────────────┐                        │           ▼             │ │
│  │   Agent    │                        │  ┌───────────────────┐  │ │
│  │  Scheduler │                        │  │  Layout Analyzer  │  │ │
│  │            │                        │  │  (spatial OCR)    │  │ │
│  │ • .md AI   │                        │  └───────────────────┘  │ │
│  │ • .py code │                        │           │             │ │
│  └────────────┘                        │           ▼             │ │
│                                        │  ┌───────────────────┐  │ │
│                                        │  │  MiniLM-L6-v2     │  │ │
│                                        │  │  (embeddings)     │  │ │
│                                        │  └───────────────────┘  │ │
│                                        └─────────────────────────┘ │
│                                                    │               │
│                                                    ▼               │
│                                        ┌───────────────────┐       │
│                                        │   SQLite (WAL)    │       │
│                                        │   + FTS5 index    │       │
│                                        └─────────┬─────────┘       │
│                                                  │                 │
│  ┌───────────────────────────────────────────────┘                 │
│  │                                                                 │
│  ▼                                                                 │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                    FastAPI REST Server                         │ │
│  │  /timeline · /search · /chat · /stats · /agents · /mcp       │ │
│  │                                                               │ │
│  │  ┌───────────────────────────────────────────────────────┐   │ │
│  │  │           Web Dashboard (Vanilla JS SPA)               │   │ │
│  │  │  Timeline · Chat · Search · Analytics · Agents · Settings │ │
│  │  └───────────────────────────────────────────────────────┘   │ │
│  └───────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Multi-Model AI Pipeline

Screenshot → EasyOCR (text) → Gemma 4 E2B (understanding) → MiniLM (embeddings) → SQLite + FTS5
                                     ↑
                              OCR text fed as context
                              (Gemma sees image + reads text)

Four AI models working in concert, with Gemma 4 as the brain: 1. EasyOCR — extracts raw screen text 2. Gemma 4 E2B — understands what you're doing (vision + reasoning) 3. MiniLM-L6-v2 — generates semantic vectors for natural language search 4. FTS5 — indexes text for instant keyword search

🤖 Agent Platform

ScreenMind includes a full agent/plugin system. Build any automation on top of your screen data.

Two Modes

Markdown Agent Example

---
name: Daily Focus Report
schedule: every 6h
data: timeline, apps, mood
output: local, obsidian
---

Analyze my screen activity and generate a focus report:
- How many hours of deep work vs shallow work?
- What were my main distractions?
- Give me a focus score out of 10.

Drop this file in ~/.screenmind/agents/ — it runs automatically.

Python Plugin SDK

from screenmind.screenmind_sdk import ScreenMindSDK

sdk = ScreenMindSDK("my-tracker")

# Get today's activities filtered by app
activities = sdk.get_activities(app="Chrome", limit=20)

# Persistent state across runs
last_count = sdk.load_state("url_count", 0)
urls = sdk.get_urls_visited()
sdk.save_state("url_count", len(urls))

# Ask Gemma (GPU-safe — waits for idle)
insight = sdk.ask_gemma(f"Summarize these URLs: {urls}")
print(insight)

Data Selectors (Frontmatter)

Markdown agents declare what data they need:

Data injection auto-scales to your model's context window.

4 Agents Ship Built-In

• daily-journal.md — First-person journal entry from your day
• focus-report.md — Focus score, deep work hours, distractions
• meeting-actions.md — Extract action items from meeting transcripts
• code-changelog.md — Summarize coding activity (commits, files, repos)

🔌 MCP Server (Claude / Cursor / VS Code)

ScreenMind exposes your screen history to any MCP-compatible AI tool:

python -m screenmind.mcp_server  # stdio transport

Claude Desktop config (~/.claude/claude_desktop_config.json):

{
  "mcpServers": {
    "screenmind": {
      "command": "python",
      "args": ["C:/path/to/screenmind/mcp_server.py"]
    }
  }
}

Tools Available

📡 API Reference

Full Swagger docs at http://127.0.0.1:7777/docs

Key Endpoints

⚙️ Configuration

🔧 Tech Stack

🐧 Wayland Support

ScreenMind auto-detects Wayland sessions and uses compositor-native capture:

Install grim (recommended for wlroots compositors):

# Arch
sudo pacman -S grim

# Ubuntu / Debian (if available)
sudo apt install grim

# Fedora
sudo dnf install grim

GNOME / KDE Wayland: Best-effort only. Screenshots use the XDG Desktop Portal, which prompts for permission on each capture — not viable for continuous background recording. For full functionality, use an X11 session or a wlroots-based compositor with grim.

Optional (for portal fallback): python3-gi / python-gobject system package.

📁 Project Structure

screenmind/
├── screenmind/                    # Entry point — starts all services
├── config.py                  # Pydantic settings (env + runtime overrides)
├── setup_llama.py             # Auto-detect + install llama-server
├── launcher.py                # Splash screen launcher (tkinter)
├── startup.py                 # Cross-platform auto-start registration
├── requirements.txt           # Full Python dependencies
├── requirements-test.txt      # Lightweight CI deps (no PyTorch)
│   ├── main.py              # MCP server for Claude/Cursor/VS Code
├── screenmind_sdk.py          # SDK for Python plugin agents
│
│   ├── assets/                    # Bundled resources
│   ├── logo.png               # App logo
│   └── favicon.ico            # Browser favicon
│
│   ├── capture/                   # Screenshot capture layer
│   ├── screen.py              # Capture facade (mss / Wayland backend)
│   ├── wayland.py             # Wayland backend (grim / XDG Portal)
│   ├── window.py              # Active window detection
│   ├── dedup.py               # Perceptual hash deduplication
│   ├── hotkey.py              # Global hotkeys (bookmark, pause, voice)
│   └── voice_recorder.py      # Mic recording for voice memos
│
│   ├── engine/                    # AI & intelligence layer
│   ├── analyzer.py            # Gemma 4 vision analysis (dual mode)
│   ├── llm_client.py          # llama-server client (chat, vision, audio)
│   ├── model_manager.py       # Server lifecycle, model download/switch
│   ├── embedder.py            # MiniLM semantic embeddings
│   ├── ocr.py                 # EasyOCR text extraction
│   ├── layout_analyzer.py     # Spatial OCR organization
│   ├── dev_context.py         # Git repo/branch/diff detection
│   ├── a11y_extractor.py      # Accessibility API text extraction
│   └── agent_runner.py        # Agent scheduling & execution
│
│   ├── workers/                   # Background processing
│   ├── capture_worker.py      # Smart capture loop + privacy filtering
│   ├── analysis_worker.py     # OCR → Gemma → Layout → Embed → Store
│   └── audio_worker.py        # Meeting detection & transcription
│
│   ├── storage/                   # Data persistence
│   ├── database.py            # SQLite + FTS5 + migrations
│   └── models.py              # Pydantic data models
│
│   ├── privacy/                   # Privacy & security
│   ├── encryption.py          # Fernet AES encryption at rest
│   └── data_filter.py         # Sensitive data redaction
│
│   ├── platform_support/          # Cross-platform abstraction
│   ├── windows.py             # Win32 + UI Automation
│   ├── macos.py               # AppKit + AXUIElement
│   └── linux.py               # xdotool + AT-SPI
│
│   ├── integrations/              # External connections
│   ├── obsidian.py            # Vault markdown export
│   ├── notion.py              # Notion API export
│   ├── webhooks.py            # HTTP webhooks (HMAC, retry)
│   └── smart_notify.py        # Distraction/break notifications
│
│   ├── api/                       # REST API + dashboard
│   ├── server.py              # FastAPI app + auth middleware
│   ├── dependencies.py        # Shared state for routes
│   ├── routes/                # 16 route modules
│   └── static/                # Web dashboard (HTML + CSS + JS)
│
├── default_agents/            # 4 built-in agents
│   ├── daily-journal.md
│   ├── focus-report.md
│   ├── meeting-actions.md
│   └── code-changelog.md
│
├── tests/                     # pytest test suite (29 modules)
│   ├── conftest.py            # Shared fixtures
│   └── test_*.py              # Unit + integration tests
│
└── docs/
    └── BUILD_YOUR_OWN_AGENT.md

🛡️ Error Handling & Resilience

🎨 Dashboard

The web dashboard at http://127.0.0.1:7777 features:

• Timeline — Browse activities by date with thumbnails, AI summaries, category badges
• Chat — Conversational AI with screen memory. Ask anything about your history. Locked with 🧠💤 brain animation until model is ready.
• Search — Semantic + keyword hybrid search with OCR highlighting on screenshots
• Analytics — Category charts, top apps, hourly heatmap, meeting stats
• Rewind — Timelapse player with play/pause/scrub/speed controls
• Memos — Voice memo list with audio player
• Agents — Create, edit, run, and monitor agents
• Settings — Model Hub (download/switch models with progress), Shortcuts, Capture, AI, Audio, Privacy, Automation, Integrations, Storage

Dark glassmorphism UI. No build step. Instant load.

🧪 Development

Run the test suite:

# Fast (lightweight deps — same as CI, ~2 min install)
pip install -r requirements-test.txt
pytest --cov=. --cov-report=term-missing -q

# Full (includes ML models — sentence-transformers, easyocr)
pip install -r requirements.txt
pip install pytest pytest-asyncio pytest-cov
pytest --cov=. --cov-report=term-missing -q

CI runs automatically on push/PR via GitHub Actions using the lightweight deps.

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines. High-impact areas:

• 🍎 macOS/Linux testing — platform adapters exist, need real hardware testing
• 🐳 Docker container — one-command setup
• 🧩 Community agent registry — share agents between users
• 🌐 Browser extension — richer URL/tab context
• 📤 Export formats — Markdown, CSV, JSON

⭐ Show Your Support

If you find ScreenMind useful, please consider:

• ⭐ Star this repo — it helps others discover the project
• 🍴 Fork it — build your own agents and features
• 🐛 Report issues — help us improve
• 📣 Share it — tell others about privacy-first AI

📝 License

MIT License — see LICENSE for details.