effGen

What is effGen  •  Install  •  Quick Start  •  Features  •  Presets  •  Tools  •  Prompts
Models  •  Examples  •  Deploy  •  Dev Experience  •  Security  •  News  •  Citation

🤔 What is effGen?

effGen transforms Small Language Models into powerful AI agents. While most frameworks assume a massive LLM, effGen is optimized from the ground up for efficient, smaller models — delivering fast, capable agents without the compute overhead — while still supporting all major cloud providers when you want them.

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator, PythonREPL

# Load a small but mighty model
model = load_model("Qwen/Qwen2.5-1.5B-Instruct", quantization="4bit")

# Create an agent with tools
config = AgentConfig(
    name="math_agent",
    model=model,
    tools=[Calculator(), PythonREPL()],
)
agent = Agent(config=config)

# Run a computation
result = agent.run("What is 24344 * 334?")
print(f"Answer: {result.output}")

📰 News & Updates

⚡ Installation

Requires Python 3.11 or newer. Tested on Python 3.11, 3.12, 3.13 and 3.14.

pip install effgen            # from PyPI (recommended)

pip install effgen[all]                       # step 1: gets torch + the rest
pip install flash-attn --no-build-isolation   # step 2: reuses the torch from step 1

git clone https://github.com/ctrl-gaurav/effGen.git
cd effGen

./install.sh            # quick install
./install.sh --full     # full install (includes vLLM + dev tools)
pip install -e .        # manual editable install

See docs/installation.md for the full guide.

🚀 Quick Start

# Run a task
effgen run "What is the capital of France?"

# Interactive chat
effgen chat

# Start the API server
effgen serve --port 8000

# List presets · check health · wizard
effgen presets
effgen health
effgen

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

model = load_model("Qwen/Qwen2.5-1.5B-Instruct",
                   quantization="4bit")

agent = Agent(config=AgentConfig(
    name="calculator_agent",
    model=model,
    tools=[Calculator()],
    system_prompt="You are a helpful math assistant.",
))
result = agent.run("Calculate 15% tip on $85.50")
print(result.output)

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

# Native Metal GPU, unified memory, no CPU-GPU transfer
model = load_model("LiquidAI/LFM2.5-1.2B-Instruct-MLX-8bit", engine="mlx")

agent = Agent(config=AgentConfig(name="mlx_agent", model=model, tools=[Calculator()]))
result = agent.run("What is sqrt(144) + 2^10?")
print(result.output)

✨ Features

from effgen.models import load_model

model = load_model(
    "Qwen/Qwen2.5-7B-Instruct",
    provider="openai_compatible",
    base_url="http://127.0.0.1:8000/v1",
)

effgen code "add a --dry-run flag to the importer"   # diffs first, writes on your word
effgen models browse --vision --min-context 128000 --sort price-out
effgen battle "Explain gradient clipping" -m groq:llama-3.1-8b-instant,gemini:gemini-3.1-flash-lite
effgen top                                           # terminal mission control

from effgen import PIIGuardrail, get_guardrail_preset

# Redaction that covers the labeled clinical identifiers, plus site-specific patterns.
g = PIIGuardrail(action="redact", custom_patterns=[(r"MRN[:#]\s*\d+", "[MRN REDACTED]")])
print(g.check("Jane Doe  DOB: 1980-02-14  MRN: 55123").modified_content)
# "[NAME REDACTED]  DOB: [DOB REDACTED]  MRN: [MRN REDACTED]"

chain = get_guardrail_preset("phi")   # redaction + fail-closed strict mode

effgen batch --input tickets.jsonl --output out.jsonl -m groq:llama-3.1-8b-instant --schema schema.json
effgen eval --suite cases.jsonl -m groq:llama-3.1-8b-instant --fail-under 0.9   # exit 1 if it drops
effgen compare --models "groq:llama-3.1-8b-instant,gemini:gemini-3.1-flash-lite" --suite cases.jsonl --optimize cost
effgen run "What was Q3 revenue?" --file report.pdf -m groq:llama-3.1-8b-instant

from effgen import create_agent, LegalDomain

# Grounded research: sources/citations come from the URLs the tools retrieved.
agent = create_agent("research", "openai:gpt-5-nano")
r = agent.run("What is the capital of France? Cite a source.")
print(r.text)                  # "...Paris (Source: https://en.wikipedia.org/wiki/Paris)."
print(r.sources)               # ['https://en.wikipedia.org/wiki/Paris']
print(r.metadata["cost_usd"], r.metadata["latency_ms"])

# A knowledge domain becomes a runnable agent in one call.
legal = LegalDomain().to_agent("openai:gpt-5-nano")
print(legal.run("What does an NDA confidentiality clause protect?").text)

effgen run --json -q "What is 25 * 17?" | jq .output   # pure-JSON stdout for CI
effgen models status                                    # physical GPU memory; which card is free

from effgen.observability import get_logger
log = get_logger(__name__)
log.event("model.call.started", provider="cerebras", model="gpt-oss-120b", cached_tokens=0)
# → {"ts": "2026-05-23T...", "level": "INFO", "event": "model.call.started", ...}

# Run a 30-second load test (JSON report prints to stdout by default)
effgen loadtest --concurrency 10 --duration 30 --scenario fixed

# Or write the report to a file with --output
effgen loadtest --concurrency 10 --duration 30 --output report.json

# Integrate with Alertmanager
cp docs/observability/alert_rules.yaml /etc/prometheus/rules/effgen.yaml

from effgen import image_from, audio_from
from effgen.presets import create_agent
from effgen import load_model

model = load_model("gemini-3.1-flash-lite", provider="gemini")
agent = create_agent("multimodal", model)

# Image question — pass media through inputs=
img = image_from("https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png")
result = agent.run("What is in this image?", inputs=[img])
print(result.output)

# Audio transcription
aud = audio_from("/tmp/clip.mp3")
result = agent.run("Transcribe and summarize.", inputs=[aud])

effgen run --preset multimodal "Describe this image" --image /tmp/photo.jpg
python -c "from effgen.models.capabilities import Capability; print(Capability.vision)"

effgen prompts list
effgen prompts list --domain research
effgen prompts show research.literature_review.v1.cot
effgen prompts eval --domain coding --live --model gpt-oss-120b
effgen prompts playground

from effgen.prompts.library import registry

p = registry.get("data.sql_from_nl.v1")
sql_prompt = p.template(
    schema_ddl="CREATE TABLE orders (id INT, customer TEXT, total FLOAT, created_at DATE)",
    question="Total revenue per customer this month",
    dialect="postgresql",
)

🎯 Agent Presets

Nine ready-made agent configurations. Each one wires up a model, a tool set and a system prompt in a single call.

math · research · coding · general · rag · minimal · multimodal · notify · media

from effgen import load_model
from effgen.presets import create_agent

model = load_model("Qwen/Qwen2.5-3B-Instruct", quantization="4bit")

# One-line agent creation
math_agent = create_agent("math", model)        # Calculator + PythonREPL
research_agent = create_agent("research", model) # WebSearch + URLFetch + Wikipedia + academic
coding_agent = create_agent("coding", model)     # CodeExecutor + PythonREPL + FileOps + Bash
general_agent = create_agent("general", model)   # Broad built-in tool suite
rag_agent = create_agent("rag", model, knowledge_base="./docs/")  # RAG pipeline
minimal_agent = create_agent("minimal", model)   # Direct inference, no tools

# CLI preset support
effgen run --preset math "What is sqrt(144)?"
effgen run --preset research "Tell me about quantum computing"

🛠️ Built-in Tools (66)

Sixty-six tools ship in the box, from a calculator to sandboxed code execution to a full RAG pipeline. Any typed Python function becomes a tool with @tool.

📝 Prompt Library

Thirty-five reusable prompt templates across 8 domains, each with a golden evaluation test and CLI access. Browse the full gallery.

effgen prompts list                            # browse all 35 templates
effgen prompts show research.paper_summary.v1  # inspect a template
effgen prompts eval                            # run golden eval (no model needed)
effgen prompts playground                      # interactive REPL

from effgen.prompts.library import registry

# Get and render a template
p = registry.get("coding.code_review.v1")
prompt = p.template(code="def add(a, b): return a + b", language="python")

# Search templates
cot_prompts = registry.search(variant="cot")
sql_prompts = registry.search(domain="data")

🤖 Multi-Model Support

Nine cloud providers, four local engines, and any server that speaks the OpenAI protocol — point at it with base_url= and effGen drives it like a first-class backend.

from effgen import load_model, Agent
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

# Any of the 9 cloud providers
model = load_model("llama-3.1-8b-instant", provider="groq")          # Groq
# model = load_model("meta-llama/Llama-3.3-70B-Instruct-Turbo", provider="together")
# model = load_model("Qwen/Qwen2.5-72B-Instruct", provider="hf")

agent = Agent(config=AgentConfig(name="agent", model=model, tools=[Calculator()]))
result = agent.run("What is (17 * 23) + sqrt(144)?")
print(result.output)  # → 403

effgen doctor   # see which provider API keys are configured

📚 Examples

python examples/basic/qa_agent.py                  # Q&A agent (no tools)
python examples/basic/calculator_agent.py          # Math: Calculator + PythonREPL
python examples/tools/multi_tool_agent.py          # Simple multi-tool
python examples/tools/advanced_multi_tool_agent.py # 5 tools + fallback chains
python examples/tools/file_operations_agent.py     # File read/write/search
python examples/tools/coding_agent.py              # Code execution + iteration
python examples/advanced/conversational_agent.py   # Multi-turn memory
python examples/advanced/advanced_streaming_agent.py # Streaming w/ callbacks
python examples/advanced/data_processing_agent.py  # JSON & data pipelines
python examples/advanced/multi_agent_pipeline.py   # Multi-agent orchestration
python examples/advanced/error_recovery_agent.py   # Error-handling patterns

python examples/basic/basic_agent.py               # Basic (Transformers)
python examples/basic/basic_agent_vllm.py          # Basic (vLLM, 5-10× faster)
python examples/plugins_presets/preset_agents.py   # Ready-to-use presets
python examples/plugins_presets/plugin_example.py  # Custom tool plugins
python examples/web_retrieval/web_agent.py         # Web search agent
python examples/web_retrieval/retrieval_agent.py   # RAG retrieval
python examples/web_retrieval/weather_agent.py     # Weather (Open-Meteo, free)
python examples/web_retrieval/streaming_agent.py   # Simple streaming
python examples/web_retrieval/memory_agent.py      # Simple multi-turn memory

python examples/basic/chat_gui_mlx.py       # MLX streaming chat (:7860)
python examples/basic/agent_viz_mlx.py      # Reasoning + code editor (:7860)
python examples/basic/tool_builder_gui.py   # Build custom tools (:7863)
python examples/basic/tool_tester_gui.py    # Browse/test all 66 tools (:7864)

python examples/basic/basic_agent_mlx.py            # Basic MLX agent + calculator
python examples/basic/chat_gui_mlx.py --autoload    # Chat GUI, auto model load
python examples/basic/agent_viz_mlx.py --autoload   # Visualizer, auto model load

📊 See examples/compatibility_matrix.md for model compatibility across all agents.

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator, WebSearch, PythonREPL

model = load_model("Qwen/Qwen2.5-3B-Instruct")
config = AgentConfig(
    name="research_agent",
    model=model,
    tools=[Calculator(), WebSearch(), PythonREPL()],
    system_prompt="You are a research assistant.",
)
agent = Agent(config=config)
result = agent.run("Search for the population of Tokyo and calculate what percentage it is of Japan's total population")

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

model = load_model("Qwen/Qwen2.5-3B-Instruct", quantization="4bit")
agent = Agent(config=AgentConfig(
    name="stream_demo", model=model,
    tools=[Calculator()], enable_streaming=True,
))
for token in agent.stream("What is 2 + 2?"):
    print(token, end="", flush=True)

agent = Agent(config=AgentConfig(
    name="memory_demo", model=model,
    tools=[], enable_memory=True,
))
agent.run("My name is Alice and I'm working on quantum computing.")
result = agent.run("What's my name and what am I working on?")
# → "Your name is Alice and you're working on quantum computing."

from effgen.tools.builtin import Retrieval

retrieval_tool = Retrieval(knowledge_base_path="./docs")
config = AgentConfig(name="qa_agent", model=model, tools=[retrieval_tool])
agent = Agent(config=config)
result = agent.run("What does the documentation say about configuration?")

🚀 Deployment

Deployment recipes for every major target, each with a working manifest in the repo.

docker build -f deploy/docker/Dockerfile -t effgen .
docker run -p 8000:8000 --env-file .env effgen
curl http://localhost:8000/health

helm lint deploy/k8s/helm/effgen/
helm install effgen deploy/k8s/helm/effgen/

cd deploy/aws_lambda
sam build && sam deploy --guided

cd deploy/cloudflare
wrangler deploy   # staging: wrangler deploy --env staging

🔷 Developer Experience

A VS Code extension, Jupyter magics, shell completion and a live dashboard.

cd tools/vscode-effgen
npm ci && npm run compile

%load_ext effgen.jupyter
%effgen_chat "What is 17 * 23?"
%%effgen_agent general
Summarise the top HackerNews
stories and rank by interest.
%effgen_metrics

EFFGEN_DEV_MODE=1 effgen serve --port 8000
open http://localhost:8000/dashboard

🔒 Security

Sandboxed execution, guardrails for PII and prompt injection, SSRF and path confinement, secret scanning and a signed supply chain.

pip install pre-commit && pre-commit install

EFFGEN_SANDBOX_BACKEND=off effgen run ...   # a loud warning is emitted

export EFFGEN_OIDC_ISSUER=https://your-tenant.auth0.com/
export EFFGEN_OIDC_CLIENT_ID=your-client-id
export EFFGEN_OIDC_JWKS_URI=https://your-tenant.auth0.com/.well-known/jwks.json
effgen serve --port 8000

📖 Citation

If you use effGen in your research, please cite our paper:

@software{srivastava2026effgen,
      title={effGen: Enabling Small Language Models as Capable Autonomous Agents},
      author={Gaurav Srivastava and Aafiya Hussain and Chi Wang and Yingyan Celine Lin and Xuan Wang},
      year={2026},
      eprint={2602.00887},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.00887},
}

⭐ Star History

🔗 Links & License

Licensed under the **Apache License 2.0** — see [LICENSE](LICENSE) for details.



**Made with ❤️ for the AI community**