duckle

Pipelines you own. Author locally, deploy to your servers.

Duckle is an open-source ETL platform for teams who want their pipelines running on their own infrastructure. Author on your laptop with a canvas, Python or SQL, then ship the same file to a server: duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit trail. Every pipeline is one file in git, so it outlives whoever wrote it. It compiles to SQL on DuckDB and uses every core you give the box, so a bigger instance is a faster pipeline: 96 million rows out of Postgres to Parquet in 39.9s. No vendor cloud. No per-row billing. No lock-in.

_{Duckle is an independent open-source project by SlothFlowLabs. It builds on the DuckDB engine but is not part of, affiliated with, or endorsed by DuckDB Labs or MotherDuck.}

_{Star Duckle if it looks useful. It genuinely helps other data engineers find the project.}

Where Duckle runs

The laptop is where you author. The work runs wherever you deploy it, on the same file.

Nothing here depends on a person's machine being switched on:

• Pipelines are plain files in git. Review them in a pull request, roll them back, and let them outlive whoever wrote them. There is no proprietary repository and no exported binary artifact.
• The console has roles and an audit log, so more than one person can operate it and you can see who did what.
• Secrets are not in the pipeline file. They resolve from the environment or an encrypted per-workspace store at run time.

Working recipes for AWS (EC2, ECS, EKS), Azure (VM, Container Apps, AKS) and Google Cloud (Compute Engine, GKE), with manifests and the mistakes worth avoiding, are at duckle.org/deploy. Three things worth knowing before you start:

• The console refuses to start on a non-loopback bind without a credential. Pass --token, set DUCKLE_CONSOLE_TOKEN, or create accounts with duckle-runner console add-user.
• The scheduler runs in serve, not in the editor. Start the editor with schedules armed and it now says so rather than leaving you to wonder why nothing fired.
• GET /healthz needs no credential and answers ok, so a Kubernetes probe or a load balancer can check liveness without holding a token. Every other route is authenticated, so pointing a probe anywhere else reports the pod unhealthy forever.

Deploying a pipeline to a running server

POST /api/deploy lands a pipeline on a server from wherever it was authored, with the schedule it should eventually run on:

curl -X POST https://duckle.internal/api/deploy \
  -H "Authorization: Bearer $DUCKLE_TOKEN" \
  -d '{"name":"orders-load",
       "pipeline": '"$(cat orders-load.json)"',
       "schedule":{"intervalMinutes":30}}'

Two things are deliberate. The schedule arrives disabled, so a cadence someone set while testing on a laptop cannot start firing the moment it reaches production; enabling it is a separate call. And deploying needs admin while enabling needs operator, because a deployed pipeline runs shell and SQL on that host: shipping the code and starting it are two acts, and the audit log records both with the name of whoever did them.

Scaling it

duckle-runner serve is an ordinary service. Run it on EC2, EKS, a VM or a container next to everything else you operate, and scale it the way you scale any service:

• More cores. The engine is parallel and uses every core on the box by default, so a bigger instance is a faster pipeline with no change to the pipeline. Bound it with DUCKLE_THREADS when you would rather it did not take the whole machine.
• More RAM. Set memoryLimitMb per stage, or a workspace default, and spill to disk past it.
• More pipelines at once. DUCKLE_MAX_CONCURRENT_RUNS raises how many run together; it ships at 1 so an unattended server stays predictable until you decide otherwise.
• More machines. duckle-runner work drains a queued batch from as many workers as you start, on as many hosts as you like, each claiming its items under a lock so nothing runs twice.
• Bigger than any box. Turn on pushdown and the query runs verbatim inside Postgres, Oracle, SQL Server or Snowflake. The warehouse does the scan; Duckle keeps the scheduling, lineage, data quality and alerting.

Measured, rather than asserted:

• 96,000,000 rows out of live Postgres to Parquet in 39.9s (details)
• Oracle extract at 65.0s, against 68.6s for python-oracledb with pyarrow on the same machine (details)

The one thing Duckle does not do is split a single query across a cluster the way a distributed warehouse does. When you need that, push the work down into the system that has it and let Duckle orchestrate around it.

Quick links

What is Duckle?

A visual data pipeline studio that runs on your laptop. Drag sources, transforms, validators, and sinks onto a canvas. Wire them together. Press Run. Duckle compiles the graph to SQL and executes it through a real columnar engine, with live previews, generated SQL on every node, and zero hidden state.

In short: a free, open-source, single-engine alternative to hosted, per-row-priced ETL platforms like Fivetran and Airbyte - one pipeline for ingest, transform, and load that runs anywhere, and can also run dbt on DuckDB inside the same tool.

Three things make Duckle different from the heavyweights and the toy ETL tools:

1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs locally - no API key, no telemetry, no cloud round-trip.
2. 366 components ready at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
3. A self-contained binary you can audit. ~65 MB download. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

What's new in v0.6.1

Talend jobs import straight into the canvas, and credentials are masked more carefully in exported SQL.

• Import a Talend job from the editor. A Talend button sits in the project sidebar next to New Pipeline and New Folder, and the same action is in the editor's ⋯ menu as Import Talend job.... Either one picks a .item job, translates it, and opens it as a new pipeline tab, laid out on the canvas at the coordinates the job was drawn with. Measured on a real 44-job corpus: all 44 parse and 211 of 216 nodes map, the only refusal being a site-specific custom component. Nothing is written to your workspace until you save, so a job that translates badly costs a closed tab.
• The import report says what still needs a person. A node count on its own would suggest a working pipeline, so the report leads with how many components actually translated, then lists everything unresolved. Encrypted Studio passwords arrive as ${ENV:...} placeholders instead of guesses. Connections stored outside the job file are named, so you can fill them in or point the node at a saved connection. tMap outputs computed by Java are listed column by column with the expression to rewrite as SQL. A component with no Duckle equivalent is imported as a labelled placeholder, so the shape of the job survives rather than quietly losing a step.
• Convert a whole repository at once, from the terminal. Importing through a file dialog is the right shape for trying Duckle and the wrong shape for leaving another tool, because nobody has one job - they have several hundred in a checkout. duckle-runner import <dir> walks the tree, converts every job it finds, and mirrors the folder layout under --out so two jobs that share a name cannot overwrite each other. Measured on a real 125-file corpus: 42 files hold a job and 83 do not (routines, contexts and SQL templates share the extension), all 42 convert with none failing, and exactly one component across the whole corpus has no equivalent - a site-specific custom one. Everything else still to resolve is credentials that were never in the job files to begin with: 119 encrypted passwords and 75 connections defined outside the job. The closing tally lists unmapped components by how often they appear, which is both the answer to "is this migration viable" and the shortest path to finishing it. --json for a script, --strict to fail a CI job.
• A file that was never a job is skipped, not counted. The extension is shared by routines, contexts and SQL pattern templates. Parsing those yields an empty pipeline, and counting an empty pipeline as a converted job inflates the only number anyone reads - on the corpus above it turned 42 real conversions into a reported 82. They are now reported separately and no empty pipeline is written. The same rule applies in the other direction: a routine is Java source whose javadoc breaks any XML reader, so it is skipped rather than reported as a failed job, while a file that declares itself a job and then will not parse is still a failure.
• Credentials are masked on token boundaries in exported SQL. Redaction replaced a credential value wherever it appeared, so a password that was also a substring of an ordinary identifier corrupted the statement around it: a password of prod rewrote production_report.parquet as ${DUCKLE_PASSWORD}uction_report.parquet. The secret itself was always protected; the damage was to everything else, and it mattered most when reading the Plan or SQL view to debug. Matching is now delimiter-aware, so LOAD postgres is left intact while a one-character password is still masked in password=p'. Deliberately no minimum length: a short password is still a password.

What's new in v0.6.0

A multimodal AI data store, an importer for legacy visual ETL jobs, a chat model you choose, and two geometry transforms that finally have the second input they always needed.

• Pixeltable, read and write (#223). src.pixeltable reads a table, optionally filtered by a Pixeltable expression, a column subset and a limit; snk.pixeltable inserts into an existing table or creates one from the incoming rows. Versioned reads work by passing myapp.media:3. The exchange runs over Parquet on both legs - Pixeltable exports, Duckle ingests with read_parquet, and on the way back Duckle writes Parquet that Table.insert takes directly - so no rows are serialised one at a time. Pixeltable is a Python library, so the desktop app provisions a private Python for it with uv on first use; nothing is installed into your own environment.
• Clip and Erase can now be wired up (#217, #218). Both shipped in v0.5.9 as two-layer overlays, and the engine required the second layer, but the palette declared only one input - so the node could be placed and configured and never run. Both now offer a second input labelled clip layer / erase layer, like Spatial Join. Behaviour is unchanged: the second layer is still dissolved with ST_Union_Agg before the operation, attributes of the input layer are preserved, and features left with nothing are dropped. Thanks to @OmarMustaafa for reporting it twice with screenshots. A test now pins this contract for every component whose builder needs a second input, checked by removing a port and confirming it fails.
• Choose the assistant's model, from 14 (#223 adjacent). The setup step installed one hardcoded 1.5B model, which is right on a laptop and wrong on a workstation with a GPU. The catalogue now spans 469 MB to 9.9 GB - Qwen2.5 Coder 0.5B through 14B, Qwen3, Llama 3.2, Phi-3.5 Mini, Mistral 7B, Gemma 2 9B and DeepSeek Coder V2 Lite - each with its real download size and an honest note on what it needs. Every entry was checked to resolve before being offered, so the picker cannot hand you a file that 404s halfway through a multi-gigabyte download.
• Import jobs from legacy visual ETL tools. Reads the XML those jobs are stored as and produces a Duckle pipeline. Measured on a real 44-job corpus: all 44 parse and 211 of 216 nodes map, the only refusal being a site-specific custom component. Encrypted passwords become ${ENV:...} placeholders rather than guesses, connections that live outside the job file are reported rather than silently half-imported, and anything with no equivalent is imported as a labelled placeholder so the shape of the job survives instead of quietly losing a step.
• CI for your pipeline repo. duckle-runner is now published as a release asset, and there are ready workflows for GitHub Actions and GitLab CI under docs/ci/. They gate every push on duckle-runner validate, which compiles pipelines to SQL without opening a source, writing a sink, or needing credentials or a network. This is the check that catches a column renamed in one commit and still referenced by another - they merge cleanly, and nothing else notices.
• Node ids no longer collide. Adding a folder or duplicating an item minted an id from the clock alone, so two of the same kind created in the same millisecond could share one. Both now use the same timestamp-plus-random scheme as everything else, which is what makes pipeline JSON safe to merge across branches.

Full notes: see the v0.6.0 release.

What's new in v0.5.10

Power mode, context layering, and an Oracle extract that is now faster than python-oracledb with pyarrow on the same table.

• Power mode (Settings -> Power mode). Two throughput settings per workspace. Pipelines at once caps how many run together; the placeholder shows the machine's core count. Spill folder points DuckDB's spill files at a bigger or faster disk. Only the lever with a measurement behind it is offered: independent pipelines scaled about 3.8x across 8 concurrent processes on a 20-core box, while splitting a single pipeline across processes measured slower (72ms to 123ms at 8-way), so there is deliberately no option for it. Each concurrent run gets its own memory limit and its own DuckDB process, so N at once needs roughly N times the memory, and the panel says so.
• Scheduled runs have a ceiling. Every schedule that came due in the same tick fired at once, so ten due at midnight meant ten pipelines each sized for the whole machine. They are now bounded, by power mode where it is set and by a sane default otherwise. The headless duckle serve honours the same setting, so desktop and server agree.
• Contexts can be layered (#204). A context can declare a Layer; higher layers override lower ones. A shared base plus a per-environment override is now expressible directly: give the base layer 0 and the environment a higher number, and the override applies quietly. Previously all contexts merged flat in repo order, so every intended override looked like a collision and had to be resolved by hand. Only two contexts on the same layer defining the same name are still reported, because nothing there says which should win. Workspaces that set no layers merge exactly as before.
• Oracle extracts beat python-oracledb (#221). Three changes, each measured on a 1,466,723-row x 236-column table with the same query and SNAPPY on both sides:
• Unconstrained NUMBER columns are now measured before the write instead of typed after it. Those columns have no declared width, so they used to travel as text and be typed by a pass over the finished Parquet - which is exactly the pass a direct write skips, meaning one such column forced a whole second pass over every column. The ambiguous columns are now read on their own first (about 2s for 4 of 236), their real widths pin the schema, and the file is written once. Both reads share one snapshot via a read-only transaction, so they cannot disagree.
• The driver no longer rebuilds an owned row per fetch. ResultSet<Row> reconstructs every value in the row; on this table that was 346 million reconstructions, measured at 11.5s of a 42.7s fetch against a 31.2s floor.
• Scaled NUMBER values no longer allocate a string per cell while being rescaled - 88 million allocations per run on this shape.

Together: 100.7s to 65.0s in the shape reported on #221, against 68.6s for python-oracledb with pyarrow doing the same job on the same machine. With column types already pinned it is about 56.7s. Output was verified against pyarrow's: identical row counts, and equal sums, hashes, ranges and null counts across every column type. Worth noting that python-oracledb maps an unconstrained NUMBER to DOUBLE, which cannot hold the 24 significant digits one test column carries; Duckle types it exactly, so the comparison is not quite like for like and not in our favour. - A direct Parquet write no longer produces string columns (#221). With Write directly from the source enabled on a table containing any bare NUMBER, those columns were written as text while the run reported success. The source now declines the shortcut when a column cannot be typed before the write, and says so in the run log. Anyone who enabled that toggle on v0.5.9 against such a table should re-check the output. - Concurrent runs no longer fight over spill files. DuckDB's default spill location is already per-run, but setting a shared spill folder made every run share one - which reads as a flaky run rather than a bug: across three trials of four concurrent spilling queries, a shared folder lost 3 of 12 runs to a segfault or a delete failure, private folders lost 0 of 12. Each run now spills into its own subfolder.

Full notes: see the v0.5.10 release.

Meet Duckie - the local AI pipeline assistant

Describe what you need. Duckie writes the pipeline.

The sidebar on the right is Duckie AI Assistant - powered by Qwen 2.5 Coder 1.5B running through llama.cpp, downloaded once (~1.1 GB) and then run entirely on your CPU. Ask in plain English; Duckie streams back a valid Duckle pipeline definition. One click drops it onto the canvas, ready to inspect, tweak, and run.

Why Duckle is different

Benchmark

20M-row CSV into DuckDB

The most common job in data engineering: load a 20M-row CSV into DuckDB. One identical 2.49 GB file (20M rows of TPC-H lineitem, 16 typed columns), every tool measured at its best configuration, wall-clock time to land the data as a table.

How it was measured

• Machine: Intel Core i7-13650HX (14C / 20T), 24 GB RAM, NVMe SSD, Windows 11, DuckDB 1.5.4. Duckle, dlt and ingestr ran here.
• Best config per tool: dlt used the Arrow plus parquet loader path; Talend and Informatica used their bulk output connectors at max config (their default row-by-row sinks were 5-7x slower). Nothing was left on a slow default to pad the gap.
• Talend and Informatica ran on a separate 8 GB VPS, so per-tool peak RAM was not captured for those two.
• Airbyte (source-file to destination-duckdb) is scaled from measured 2M and 5M runs at a steady ~18k rows/s, and it also needs an always-on ~8 GB platform just to start.
• Wall-clock time, peak working-set of the whole process tree.

Why Duckle is this fast: its 15.69s sits right on top of raw DuckDB's own load floor (~16s to fully parse and write all 20M typed rows into an on-disk table). Duckle wraps the engine with pipelines, connectors, and a UI, then gets out of its way. That is the entire design goal. A read-only scan or aggregate over the same CSV is far faster still; this benchmark measures the heavier "materialize it as a table" job that every ETL tool here performs.

96M rows, Postgres to Parquet

A second, harder job against a live database: full-refresh extract of 95,988,640 rows of TPC-H lineitem (14 GB in Postgres 16) out to Parquet.

Run it yourself. The harness is in this repo at benchmarks/pg-to-parquet. ./bench.sh all brings up Postgres, generates the data at any scale factor, and times every tool you have installed. No timing is recorded until the output has been reopened and checked for the right row count and the right sum(l_orderkey), so a tool that writes a fast but wrong file gets a failure rather than a number.

Read it with these caveats

• The DuckDB floor is not a competitor. It is raw postgres_scanner plus COPY TO: no scheduling, no typing, no incremental state, no UI. It is there to show how much of the clock is the machine reading Postgres. Duckle landing 11% under it is the honest framing, not "Duckle beats DuckDB".
• ingestr has no Parquet destination and writes a DuckDB file, so its output size is not like-for-like. Its time is.
• Compression is not normalised. Duckle wrote zstd, the others snappy.
• Airbyte and Meltano are absent. Airbyte has no local Parquet destination and has only run against an earlier synthetic dataset; Meltano was not wired up. Neither is claimed here.

Hardware, per-run numbers and the two measurement traps that produced wrong figures on the first attempt are written up in RESULTS.md.

Status

Duckle is in public beta. The visual designer, the DuckDB execution engine, the scheduler, the cloud connectors, and the Duckie AI assistant all work today and are covered by 170+ integration tests across Linux, macOS, and Windows. The catalog is still growing and APIs may evolve before 1.0, but the day-to-day surface is stable enough for real work.

Scope, stated plainly: Duckle runs as a service on hardware you provision, and uses all of it. What it does not do is split one query across a cluster, so when a job outgrows the largest instance you want to pay for, push the work down into the source system or point the output at a warehouse, object store or lakehouse. It will not pretend to be a cluster.

The component palette ships 384 nodes so the roadmap is visible in the product itself:

• 366 available runs on the DuckDB engine today
• 3 preview is configurable in the designer (drag, wire, set properties); execution is being wired engine-by-engine
• 15 planned is reserved in the palette but not yet executable - see docs/roadmap.md

Screenshots

Real pipelines, built and run in Duckle - not mockups.

_{A 5M-row pipeline: a CSV, a Parquet file, a DuckDB table, and a SQLite table enriched through one visual Map (3-way join), no SQL.}

_{Left: the visual Map editor - main plus lookups, per-output expressions, an inline filter. Right: Parallelize fanning out aggregate, window, and top-N branches.}

_{One run, many branches: 16 nodes finish in a few seconds. Concurrency auto-detects from CPU cores; branches write to Parquet, CSV, DuckDB, and SQLite at once.}

_{Left: DuckLake CDC change-feed mirrored via upsert + delete propagation (100k rows). Right: watermark incremental load over 5M rows, advancing state only on a fully successful run.}

Capabilities

Duckle is not a CSV tool with extras. It reads a broad set of formats and sources, ships a deep transform library, and writes to files, databases, object storage, vector DBs, message buses, and email.

Sources (103 available)

For CSV / TSV sources, the Schema panel accepts an optional per-column Format (a strptime token string such as %d/%m/%Y) on Date and Timestamp columns. Several date columns can each parse a different layout in one read - the column is read as text and re-parsed with its own format, working around DuckDB's single global date format. A value that does not match its format becomes null rather than failing the run. Set a Date or Timestamp column's Format to excel to convert Excel day-serials correctly. CSV sources also surface ignoreErrors (skip unparseable rows) and nullPadding (pad short rows with nulls) toggles in the GUI.

For JSON sources, a Format selector picks how the file is read (auto / array / JSON Lines / object), and a skip malformed records toggle drops records that fail to parse instead of failing the run.

Transforms (130 available)

All 6 AI transforms ship today. Three need a model API (LLM, Classify, Embeddings) and ride the apiKey-in-props pattern; three are pure-local (Chunk, PII Redact, Dedupe).

Data quality (27 available)

Validators split their input: passing rows continue on the main port, failures route to a reject port you can sink, count, or inspect.

Custom code (6 available)

Sinks (64 available)

Database sinks support an optional dead-letter (validate-before-insert) step: rows that do not match the declared column types are split off to a dead-letter file (parquet / csv / json) and only the clean rows are inserted.

Control flow (18 available)

| Try / Catch | Install a fallback sub-pipeline if the wrapped stage fails | | Retry | Per-stage retry policy (configure on Advanced tab) | | Log Message | Emit an info log line ({rows} = upstream count), pass rows through (ctl.log) | | Warn | Emit a warning log line, pass rows through (ctl.warn) | | Die / Fail | Stop the run with a message: always, only when the input has rows, or only when empty (ctl.die) | | Schedule | Cron / interval / file-watch triggers via the orchestration crate |

A sub-pipeline runs under its own name, so its run log lands in logs/<child>/ and an xf.incremental watermark inside it is saved to state/<child>/<node>.json. Two different children driven by the same For Each therefore keep separate marks.

Set For Each -> Item key column to separate the ITERATIONS too. The child then runs as <child>@<value>, so loading 400 tables through one sub-pipeline keeps 400 watermarks in state/<child>@<table>/<node>.json instead of one. Leave it blank and every row shares a single mark, which silently skips rows when each row is a different table. It is never inferred from the row's position, because that would move every watermark the moment the driving query is reordered.

Set For Each -> Dispatch to Queue for workers and the rows are written to batches/<id>.ndjson instead of being run, one JSON line per row carrying the child reference and that row's substitutions. Nothing runs until a worker picks the batch up, so the run that queued it reports how many items are waiting rather than pretending they loaded. A batch is a file in the workspace like everything else here: no queue server, no database, no network service.

Queueing also reports whether the items are actually safe to spread out. Both "400 items each loading their own table" and "400 items appending to one file" look identical on the canvas - one sink node with a variable in the path - and only the first survives being run at once. So each item's variables are put into the child and the resulting targets are named with the same function that builds the workspace catalog, before anything picks the batch up:

duckle: 400 item(s) write to targets nothing else in the batch writes, so they
        are safe to run at the same time
duckle: heads up - 1 target(s) are written by more than one item (400 items
        write /lake/everything.parquet). Workers run items at the same time, so
        these will collide unless the sink is an upsert or the target is
        append-safe

It warns rather than refuses: appending many items into one table is a real thing to want, and only you know whether that sink is safe for it. Items whose child cannot be read are counted and reported, so a partial check never reads as a clean one.

Then run workers against it:

duckle-runner work --workspace /path/to/workspace     # drain every batch
duckle-runner work --batch fe-20260816T101112123      # just this one
duckle-runner work --once                             # one item, then exit

Start it on several machines pointed at one workspace and they share the batch. Each item is claimed with the same OS lock a pipeline run uses, so no two workers take the same one, and a worker that is killed mid-item leaves nothing to clean up: the kernel drops the lock and the item becomes claimable again. There is no lease, no heartbeat and no timeout, because there is nothing to expire. Progress is appended to batches/<id>.ledger.ndjson, so re-running a worker resumes rather than repeats.

Items run at least once, not exactly once. The ledger is written after an item succeeds, so a worker that finishes an item and then dies leaves it looking undone and another worker repeats it. That is the honest trade for having no transactional store - the alternative loses items instead of repeating them, and a lost load is worse. Make the child idempotent (an upsert sink rather than an append) and a repeat costs time, not correctness. A failed item stays claimable and is retried on a later pass, with the failure kept in the ledger.

The console has a Batches view: progress per batch, how many items are running right now, how many failed, and the recent attempts with the worker that ran each one. "Running" is answered by asking the run lock rather than by trusting a heartbeat, so a worker that died is not counted as running and there is no lease that could have gone stale. Retry failed clears the recorded failures so those items are claimable again, keeping the successes so a retry never repeats finished work.

Before running anything, a worker proves the lock actually excludes on that filesystem: it takes a lock and asks a second process whether it can take the same one. Some shared filesystems tell every caller it has the lock - NFS with no lock daemon is the classic case - and on one of those every worker would claim every item and each item would run once per worker, silently, with no error anywhere. A worker refuses to start there. Check it yourself with duckle-runner work --check; --no-check overrides, knowing the above. A test that could not be run is only a warning, because failing to prove exclusion is not the same as having disproved it.

Measured on one machine: three workers against a twelve-item batch took four items each, with no item run twice. Several machines against one shared filesystem is the design intent and is not yet measured, so treat it as untested until it is. scripts/measure-multi-host-batch.sh is the measurement: point it at a shared workspace and two or more hosts and it counts duplicate executions, failing if there are any.

Advanced settings (per-node)

Every node has an Advanced tab with fields the engine honours at run time:

Orchestration and workspace

Clean data before it reaches your AI

Models inherit the quality of their inputs. RAG indexes, embedding stores, and training sets quietly accumulate duplicates, nulls, malformed rows, mixed encodings, and inconsistent schemas. Duckle is built to scrub that data before it lands in a vector store:

• Deduplicate with exact Distinct, Uniqueness, and Fuzzy Deduplicate (Jaro-Winkler / Levenshtein); use Record Match to find near-duplicate pairs with a similarity score
• Semantic dedupe with xf.ai.dedupe over a precomputed embedding column
• Profile + describe every column up front (Column Profile, Describe, Histogram) so issues surface before they reach a model
• Validate and filter malformed, empty, or out-of-range records and route failures to a reject port
• Normalize types, encodings, casing, and null handling across messy sources (Standardize, Cast, regex / string transforms)
• Redact PII (emails, phones, SSNs, credit cards) via xf.ai.pii before embedding
• Chunk + embed long text via xf.ai.chunk -> xf.ai.embed for RAG indexing
• Classify rows with an LLM (xf.ai.classify constrains the model to one of N user-supplied categories)
• Retrieve with both halves of hybrid search, locally, no model API required: Vector Similarity Search (cosine / L2 / inner product) and Full-Text Search (BM25)
• Land it in your store - pgvector ships, and Pinecone, Qdrant, Weaviate, Milvus all have working sinks that POST batches through each vendor's HTTP API

Engines

Duckle ships a thin shell and installs its engines on first launch.

First-launch extension pre-fetch

When the installer downloads the DuckDB CLI it also pre-fetches the extensions Duckle uses, with per-extension progress, so the first time you touch a Postgres source or an Iceberg table there is no surprise network hop mid-pipeline:

httpfs (S3 / GCS / HTTP), azure (Azure Blob native), sqlite, postgres, mysql, excel, iceberg, delta, ducklake, vss, fts.

spatial is lazy-loaded (~50 MB GDAL bundle) - it installs on first use of a geospatial source/sink to keep the initial download small.

Download / Install

Pick the binary for your OS from the latest release:

The single-file binary above is all you need for Build Pipeline too: the headless runner is embedded into the app at build time, and exporting a pipeline produces ONE self-contained executable (the engine, the DuckDB CLI, any needed extensions, and the resolved pipeline are all inside that one file). Copy that single file to your server and run or schedule it - no separate runner download required.

One command, nothing installed: it scaffolds sample data and a pipeline, compiles it to SQL, runs it on DuckDB, and shows you the rows.

uvx duckle quickstart

Let an agent do it

Paste this into Claude Code, Cursor, or Codex:

Run uvx duckle quickstart to build my first pipeline and run it

Nothing to install first. The agent fetches Duckle and the DuckDB engine on demand, runs a real pipeline, and shows you the rows.

CLI only (CI, cron, containers)

If you do not want the desktop studio, install just the headless runner. It is about 20 MB rather than ~100 MB, has no GUI dependency, and is what a build step actually needs.

pip install duckle

That is the whole install. It brings the DuckDB CLI with it (via the duckdb-cli package published by the DuckDB Foundation), so there is nothing else to fetch and it works offline. Wheels ship for Linux, macOS and Windows on x86-64 and arm64.

It also gives you a Python API, where pipelines are built as code and executed by DuckDB rather than by Python:

import duckle
from duckle import col

(duckle.read_csv("orders.csv")
    .where(col.amount >= 20)
    .derive(total="round(amount * 1.2, 2)")
    .write_parquet("out.parquet")
    .run())

Python expressions compile to vectorized SQL at plan time, so no rows pass through the interpreter. See the PyPI page for the full API.

The same package provides the duckle command-line runner for CI, cron, and containers - it bundles the headless runner and the MCP server per platform:

pip install duckle          # or run ad hoc, no install: uvx duckle --help

Pipelines execute as SQL on the DuckDB CLI, so the runner needs a duckdb on PATH or DUCKLE_DUCKDB_BIN set (pip install duckdb-cli is the quickest route). Validation does not:

duckle validate                 # compile-check every pipeline under ./pipelines
duckle validate --json          # machine-readable, for a CI step
duckle --pipeline my.json       # run one

validate opens no source and writes no sink, so it needs no engine, no credentials and no network. Exit codes are stable: 0 clean, 1 a real finding (a pipeline failed or did not compile), 2 the runner could not start (bad usage, unreadable file, missing engine).

The binary is ~55-78 MB depending on platform (it embeds the headless runner and the bundled MCP server). On first launch you'll be guided through downloading two engines into your app-data directory:

App-data location: - Windows: %APPDATA%\io.duckle.app\engines\ - macOS: ~/Library/Application Support/io.duckle.app/engines/ - Linux: ~/.config/io.duckle.app/engines/

Delete the engines/ folder if you ever want to force a fresh install.

Quickstart (60 seconds)

1. Download the binary for your OS (see Download / Install above) - or build from source.
2. Launch it. First run shows the setup modal: - Click Install on DuckDB (required, takes ~30 s). - Optionally click Install on Duckie AI Assistant (~1.1 GB, takes 5-10 min on average broadband).
3. Pick a workspace folder. Pipelines, connections, context variables, and routines live there as plain files.
4. Build a pipeline two ways: - Drag + wire: drag a CSV source in, point it at samples/orders.csv, hit Autodetect schema. Drag a Filter, wire it up. Drag a Parquet sink with an output path. Press Run, watch the nodes light up. - Ask Duckie: click the Sparkles icon (top-right of the toolbar), type "read orders.csv, filter where status = 'paid', write to paid.parquet". When Duckie streams back a pipeline, click Insert into canvas.
5. Inspect. Click any node to see its generated SQL in the Plan tab and a live row sample in the Preview tab.

That's a real, native ETL pipeline built and run in under a minute. CSV is just the easiest first node; swap in Parquet, JSON, S3, Snowflake, MongoDB, or Stripe the same way.

Run your first pipeline

A worked example using the bundled samples/orders.csv data.

1. Add a source

• Open the Components sidebar (left). Click Sources -> Files -> CSV.
• Drag it onto the canvas.
• In the right-side Properties panel:
• Path: browse to samples/orders.csv
• Click Autodetect schema - the Schema tab fills in column types from the file, the Preview tab shows the first 20 rows.

2. Add a transform

• Components -> Transforms -> Rows -> Filter. Drag onto canvas.
• Wire the CSV source's main output port to the Filter's main input.
• In Properties:
• Predicate: status = 'paid' (you can write raw SQL or use the visual builder)
• Filter has two output ports: pass (rows matching) and reject (rows that don't).

3. Add a sink

• Components -> Sinks -> Files -> Parquet.
• Wire Filter's pass port to the Parquet sink.
• Path: paid_orders.parquet. Write mode: overwrite. Compression: zstd.

4. Run it

• Press Run in the toolbar. Nodes light up in execution order; row counts appear under each.
• Open the Output tab (bottom panel) to see per-stage timing.
• Click any node to inspect generated SQL in Plan + sampled rows in Preview.

5. Iterate

• Add a Group By before the sink to aggregate. Re-run. Sub-second on small data.
• Cancel mid-run with the Stop button - the DuckDB process is killed cleanly.
• Save your work: Cmd/Ctrl-S writes a JSON pipeline file to your workspace folder.

How to use Duckle

A wider tour of the workflow.

Recipes and examples

Ready-to-adapt patterns. Each one is a few nodes you wire on the canvas (or ask Duckie to sketch).

CSV cleanup

"Read orders.csv, drop nulls, deduplicate by order_id, write to orders_clean.parquet"

src.csv -> qa.not_null -> qa.uniqueness -> snk.parquet

Set qa.not_null to the columns that must be present; set qa.uniqueness to order_id. Rejected rows go to a snk.csv on the reject port for inspection.

Postgres -> Snowflake nightly load

"Read all rows from Postgres events, upsert into Snowflake table analytics.events on event_id"

src.postgres -> snk.snowflake (mode=upsert, conflict=event_id)

Attach a ctl.schedule with cron 0 2 * * * to run nightly at 02:00.

S3 -> partitioned Parquet

"Read all .json.gz files in s3://logs/2026/*/*.json.gz, parse, write Hive-partitioned by event_date"

src.s3 (glob, autodetect json.gz)
  -> xf.derive (event_date = CAST(ts AS DATE))
  -> snk.parquet (path=out/, partitionBy=event_date, mode=overwrite_or_ignore)

RAG ingestion

"Chunk our docs, embed with OpenAI, dedupe near-identicals, store in pgvector"

src.s3 (markdown files)
  -> xf.ai.chunk (chunkSize=1500, overlap=150)
  -> xf.ai.pii (redact)
  -> xf.ai.embed (model=text-embedding-3-small, baseUrl=https://api.openai.com)
  -> xf.ai.dedupe (threshold=0.95)
  -> snk.pgvector (table=docs)

Slack channel digest

"Pull yesterday's Slack messages from #support, classify by sentiment, email a summary"

src.slack (channels.history with oldest=yesterday)
  -> xf.ai.classify (categories=positive,negative,neutral)
  -> xf.aggregate (group by sentiment, count)
  -> snk.email (to=oncall@..., subject=Daily Support Digest)

Webhook -> S3 archive

"Receive 100 webhooks, archive each one as JSON in S3"

src.webhook (port=8080, maxRequests=100, timeoutMs=300000)
  -> snk.s3 (path=s3://archive/events/, format=jsonl, partitionBy=event_date)

Git commit-log analytics

"Build a dashboard of who's been committing what in the last 30 days"

src.git (mode=log, maxRows=10000)
  -> xf.filter (date > current_date - INTERVAL '30 days')
  -> xf.aggregate (group by author_email, count)
  -> snk.csv (path=author-stats.csv)

More examples live in samples/ - drop the pipeline files into a workspace and open them.

Git integration (GitHub + GitLab)

Push, pull, branch, and watch CI from inside Duckle. No terminal required.

Click the Git icon in the topbar to open the workspace Git panel. Built-in integration with GitHub and GitLab, on the system git CLI (no FFI, no embedded git library):

Workflow. Workspaces are plain folders (see Workspace and Git flow) - any standard Git workflow works:

Create / clone -> open in Duckle -> edit pipelines -> commit + push -> 
PR / MR -> CI runs your pipeline tests -> merge -> pull

You can do the entire push / pull / merge loop without leaving Duckle. Heavy operations (interactive rebase, conflict resolution, log archaeology) still live in your terminal or external Git tool - the panel is designed for the everyday flow, not as a full Git replacement.

Provider detection. The remote URL host determines which CI API the badge polls:

The badge uses the same PAT you saved for pushes - no separate auth step.

Workspace and Git flow

A workspace is a folder you pick on first launch. Everything you build lives there as plain text:

my-workspace/
  pipelines/
    orders_etl.pipeline.json     # the node graph
    nightly_load.pipeline.json
  connections/
    prod-postgres.connection.json # saved DB credentials (encrypted)
    snowflake-analytics.connection.json
  contexts/
    dev.context.json              # variables for dev environment
    prod.context.json
  routines/
    cleanse-addresses.sql         # reusable SQL snippets
  documents/
    runbook.md                    # plain-Markdown docs
  schedules.json                  # all scheduled runs in this workspace
  run-history/
    orders_etl/                   # one folder per pipeline
      2026-05-25T14-30-00.json    # one file per run

Git-friendly by design. Every file is human-readable JSON or Markdown. Standard workflows work:

git init my-workspace && cd my-workspace
git add . && git commit -m "Initial pipelines"

# Pull a teammate's update
git pull --rebase

# Push your changes
git push

# Branch for a risky migration
git checkout -b feature/upsert-mode
# ...edit pipelines in Duckle...
git diff       # readable JSON diffs
git push -u origin feature/upsert-mode
# open PR / MR

Sensitive values in connections get encrypted with a workspace-local key (workspace/.duckle/keys/). Don't commit that file - add **/.duckle/keys/ to .gitignore. The connection JSON files themselves only hold the ciphertext, which is safe.

Schedules and triggers

Pipelines can run on cron, fixed interval, or file-watch triggers. Configure these in the Schedule panel (toolbar -> Schedule icon), not as graph nodes.

Schedules persist to workspace/schedules.json and execute via the in-process scheduler crate. They survive app restarts but require Duckle to be running.

For headless / always-on schedules that run when Duckle is closed, build the pipeline into a standalone file and let the operating system's own scheduler run it - see Server deployment below.

Server deployment (Build Pipeline)

The in-app scheduler runs only while Duckle is open. To run a pipeline on a server with no desktop app, Build Pipeline turns it into ONE self-contained executable - the equivalent of a standalone "Job".

Right-click a pipeline (in the project tree or on the canvas) and choose Build Pipeline. The output is a single file named after the pipeline (orders_etl.exe on Windows, orders_etl on macOS / Linux) that embeds everything it needs:

• the headless execution engine,
• the DuckDB CLI,
• only the DuckDB extensions that pipeline's components actually use,
• the resolved pipeline (context variables substituted, routines inlined),
• its secrets (see below).

On first run it self-extracts to a temp cache and uses its own embedded DuckDB, so the server needs nothing installed - no Duckle, no DuckDB. There is no folder to copy, no run.sh, and no separate runner download. A CSV-to-CSV pipeline builds to about 28 MB; only the extensions a pipeline uses are bundled, so the file stays lean.

./orders_etl            # or orders_etl.exe on Windows

The process exits 0 on success and non-zero on failure, and writes the same NDJSON run logs under logs/ (Splunk / Dynatrace friendly).

Build options

Schedule it with whatever the server already has - point the OS scheduler straight at the file:

# Linux cron - run every day at 02:00
0 2 * * * /opt/duckle/orders_etl >> /var/log/orders_etl.log 2>&1

On Windows use Task Scheduler; on macOS a launchd plist; on Linux a systemd timer. Full examples in docs/current/scheduler.md.

Run against an existing workspace - the same embedded headless runner can also execute a pipeline JSON directly, resolving context the way the app does:

duckle-runner --pipeline /path/to/pipeline.json [--workspace /path/to/workspace] [--duckdb /path/to/duckdb]

Web panel (remote management console)

To run and monitor pipelines on a server with a browser instead of the desktop app, start the built-in web panel - it is part of the same duckle-runner binary, so there is nothing extra to install:

duckle-runner serve --port 8080 --workspace /path/to/workspace

Open http://localhost:8080. The panel has six views:

• Overview - every pipeline with its last status, duration and next scheduled run, and a Run button.
• Runs - run history across every pipeline (status, duration, rows, errors) with expandable per-pipeline run logs and optional auto-refresh.
• Schedules - an editable cron or interval schedule per pipeline, showing what is running now and what is due next.
• Catalog - everything the workspace reads and writes, who owns it, and what is written but never read. See Workspace catalog.
• Batches - work queued for workers: progress, what is running now, what failed, and a retry for the failures.
• Audit - who signed in, what they changed and who was turned away. Admin only, and shown only to admins.

Runs execute in-process through the same engine, are written to the same run history (<workspace>/runs/) and logs (<workspace>/logs/), and a built-in scheduler triggers any pipeline whose schedule has elapsed - so the server itself runs your schedules, no OS cron needed.

Sign-in and roles

The console can run any pipeline in the workspace, and a pipeline can run shell and SQL, so reaching it is equivalent to running code on the host. On 127.0.0.1 (the default) it is open, because anyone who can reach it is already on the machine. Any other --host refuses to start without a credential:

DUCKLE_CONSOLE_TOKEN=<secret> duckle-runner serve --host 0.0.0.0 --port 8080

For more than one person, give each their own token and role. The token is printed once and stored only as an Argon2id hash in <workspace>/.duckle/console-users.json:

duckle-runner console add-user reporting --role viewer
duckle-runner console add-user ops       --role operator
duckle-runner console list

A browser exchanges the token for a session cookie, so the browser never stores the credential; an API client sends Authorization: Bearer <token>. Every state-changing request, and every refusal, is appended to <workspace>/logs/audit.ndjson with who, what, when and the outcome. The same accounts and roles cover duckle-runner web.

Read it back from the Audit view, or from a terminal with no server running:

duckle-runner audit                                  # newest first, 50 by default
duckle-runner audit --outcome denied                 # who reached for what they do not have
duckle-runner audit --actor ops --action schedule    # one person, one family of actions
duckle-runner audit --limit 500 --json               # for a collector

allowed means the caller was permitted to proceed, not that the work then succeeded - run history answers that. Reads are not recorded, so a dashboard polling every few seconds does not bury the entries worth seeing. A page says when older entries exist beyond it, and a line that will not parse is counted rather than silently skipped.

Still put it behind a reverse proxy if you need TLS.

Migrating a repository of legacy jobs

The editor imports one job at a time, which is how you try Duckle. This is how you leave another tool: point it at a checkout and convert everything.

duckle-runner import ./legacy-jobs                    # convert the tree into ./imported
duckle-runner import ./legacy-jobs --out ./pipelines  # somewhere else
duckle-runner import ./legacy-jobs --json             # for a migration script
duckle-runner import ./legacy-jobs --strict           # CI gate, exits 1 if anything needs a person

The folder layout is mirrored rather than flattened, because two jobs in different folders routinely share a name and flattening would silently drop one. Files that are not jobs - routines, contexts, SQL templates - are reported separately rather than counted as conversions, and no empty pipeline is written for them.

The closing tally is the number to decide on. It says how many jobs came across clean, how many need a person, and which components have no equivalent yet, sorted by how often they appear. On a real 125-file corpus that list had a single entry, a site-specific custom component: coverage is the head of the distribution, so a corpus usually converts far better than a raw component count suggests. What remains is almost always credentials that were never in the job files - encrypted passwords become ${ENV:...} placeholders and connections defined outside the job are named so you can point them at a saved connection.

Workspace catalog (what reads and writes what)

Every other lineage view in Duckle answers about one pipeline. The catalog answers about the whole workspace, by joining pipelines through the assets they name: two pipelines that read and write the same table are connected whether or not anyone drew a line between them.

duckle-runner catalog lint                                        # CI gate, exits 1 on findings
duckle-runner catalog diff main                                   # what this branch does to the graph
duckle-runner catalog build                                       # scan every pipeline
duckle-runner catalog assets                                      # every table, file, topic and endpoint
duckle-runner catalog impact postgres://db:5432/sales.public.orders
duckle-runner catalog orphans                                     # written here, read by nobody
duckle-runner catalog owners                                      # what nobody has claimed

impact is the blast radius: the pipelines that read an asset, the assets they write, everything downstream of those, and how many hops away each is. Assets that could not be named are counted on every answer rather than dropped, so a partial graph never looks complete. An asset name nothing in the workspace uses exits non-zero, including under --json, so a mistyped name in a CI gate fails instead of reporting an empty blast radius.

build walks the whole workspace, skipping Duckle's own folders (runs, logs, connections, .duckle), so pipelines kept in subfolders are included. The saved graph records what it was built from, so it knows when the pipelines have moved on: the CLI rebuilds on read rather than answering from a stale graph, and the console says "pipelines have changed since this was built" instead of quietly presenting an old blast radius as current. The console reports it rather than rebuilding, because reading the catalog is a viewer action and rebuilding writes a file - Rescan is the operator's button. The check is stat only, so it costs nothing to make on every read; a same-length edit inside the same millisecond would slip through, which is the price of not hashing every pipeline on every read. Asset names never carry a credential: a mongodb://user:pass@host uri or an ODBC connection string is reduced to the address before it becomes a name, which also keeps the name stable when the password is rotated.

Add <workspace>/owners.json and it also tells you who to notify. Rules are globs and the first match wins, so a narrow rule above a broad one carves out an exception:

{
  "assets": [
    { "match": "/lake/raw/pii_*", "owner": "Privacy Office", "contact": "[email protected]",
      "description": "Landing zone for regulated source tables.", "tags": ["raw", "pii"] },
    { "match": "/lake/raw/*",     "owner": "Data Platform",  "contact": "[email protected]" }
  ],
  "pipelines": [{ "match": "*-ingest-*", "owner": "Ingest Squad" }],
  "terms": { "active customer": "Ordered in the last 90 days." }
}

The same file carries the human half of the catalog: an optional description and tags per rule, and a workspace terms glossary for the words three teams would otherwise each define differently. Every one of those is optional, so an owners.json written before they existed still loads unchanged. They live here rather than in a file of their own because they are authored, reviewed and committed alongside ownership, and a second file would drift from this one.

Every run records which assets it read and wrote, with row counts, under the same names the graph uses - so the catalog can answer the first question anyone actually asks of an entry: is this current? A table nobody has written for three weeks is the interesting one, and no amount of structure reveals that. Only successful runs count towards freshness: a failed run may have written nothing, or half of something, and showing either as the last write would make a broken load look like a fresh table.

Assets also carry the columns the pipelines declare, unioned across every node that touches them - a pipeline reading three columns of a table another writes twenty to does not make the table three columns wide. They come from the schema a node already carries, so building the graph still opens no source and needs no credentials. No declared columns means none are known, which the catalog does not confuse with the asset having none.

In the desktop app this is the Data Catalog screen (Home -> Govern): search every asset by name, owner, tag or column; see who writes it, who reads it, what columns are declared and when it was last written; and set the owner, description and tags without leaving the app. Saving writes a rule for that exact name above any wildcard covering it, so describing one file never re-describes its neighbours. Read live schema opens the source on demand through a node that already reads it, so it authenticates the way the pipeline does - it is never done just because a screen was opened.

catalog lint is the gate for a CI job: it exits 1 when it finds something. It reports ownership rules that match nothing - almost always a typo or a renamed asset, and a failure that is otherwise silent, because the team the rule names simply never gets told about anything - patterns that will not compile (they own nothing, safely and invisibly), and nodes the graph could not name. Unowned assets are reported but only fail under --strict: most workspaces have a long tail nobody will ever claim, and failing CI over it on day one is how a useful check gets deleted from the pipeline instead of acted on.

catalog diff <rev> answers what a change does to the graph, which is the question a review of a data platform actually asks: which assets appear, which disappear, and - the one that matters most - which are still there but have lost every pipeline that wrote them. A deleted asset is loud, because something errors. An asset nothing writes any more is silent: no error, no missing file, the table simply stops moving and whoever reads it finds out weeks later. The revision is read straight from git's object store, so nothing is checked out and it is safe to run on a dirty worktree.

The console's Catalog view now shows the same facts as the desktop screen - description, tags, columns and freshness - because both are assembled by one function in the engine rather than two that would drift. The same answers are available over MCP as workspace_impact.

Alerting (tell someone when a run fails)

snk.email and snk.rest are pipeline nodes - they need wiring into every pipeline and cannot fire when a pipeline dies before reaching them. <workspace>/alerts.json watches the runs themselves, for both the desktop scheduler and the server:

{
  "rules": [
    { "match": "nightly-*", "channel": "webhook", "url": "${ENV:SLACK_WEBHOOK}", "cooldownMinutes": 15 },
    { "match": "*", "channel": "email", "smtpHost": "smtp.example.com",
      "from": "[email protected]", "to": ["[email protected]"] }
  ]
}

The webhook payload carries a text field as well as structured fields, so Slack, Teams and Discord render it directly. Three behaviours are deliberate:

• Repeat suppression. A five-minute schedule that starts failing would otherwise send 288 messages a day. cooldownMinutes (default 15) bounds it per pipeline and event.
• The all-clear. A success after a failure is its own recovery event and ignores the cooldown, so nobody is left thinking an outage is still running. Ordinary successes are silent unless you add "on": ["success"].
• It never breaks a run. Delivery happens after the run is recorded, is time-bounded, and an unreachable channel is logged rather than raised.

A schedule whose pipeline file has been renamed or deleted also raises an alert, instead of silently doing nothing.

MCP server (connect Claude or any LLM to Duckle)

Duckle ships its own Model Context Protocol server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor, or any other LLM agent) can drive Duckle directly: browse the full component catalog and per-component property schemas, generate a pipeline straight into a working directory you choose, validate it (compile without running), run it headlessly, read existing pipelines and their run logs, build a standalone artifact, and manage saved connections.

Connect with nothing installed

If you have uv, one line connects any MCP client. Nothing is installed, no engine to configure: uv fetches the package and the DuckDB engine into a throwaway environment and the server finds it there.

claude mcp add duckle -- uvx duckle mcp

For Claude Desktop, Cursor, or any other client, the same thing as config:

{ "mcpServers": { "duckle": { "command": "uvx", "args": ["duckle", "mcp"] } } }

uvx duckle mcp works because the package and the command are both named duckle, so there is no --from to remember. If you would rather install it, pip install duckle puts duckle on PATH and the same duckle mcp command applies.

Then ask the agent something like "use duckle to list the available components". It can discover a real connector rather than guess one, compile-check a pipeline with validate_pipeline before anything executes, run it, and hand back column-level lineage. What it produces is the same JSON the canvas opens, so you can see what it built.

Connect in one click

The MCP server is also bundled inside the app - there is nothing extra to install. In the designer, click Connect to Claude in the top bar to open the connector popup, then pick your client:

• Connect to Claude Code - registers the duckle server for you (runs claude mcp add under the hood).
• Add to Claude Desktop / Add to Cursor - writes the duckle entry into that client's config, with the resolved engine paths filled in (both the Microsoft Store / MSIX and standalone Claude Desktop layouts are handled).
• Or copy the command / config for any other MCP client.

Restart the AI client, then try "Use duckle to list the available components" to confirm the connection.

Manual / headless

For a build-from-source or server setup, point any client at the duckle-mcp binary directly. It speaks JSON-RPC over stdio and reuses the DuckDB engine in-process (no GUI, no Node runtime).

cargo build -p duckle-mcp --release      # target/release/duckle-mcp
claude mcp add duckle -- /path/to/duckle-mcp

For Claude Desktop and other clients, add it to mcpServers:

{
  "mcpServers": {
    "duckle": {
      "command": "/path/to/duckle-mcp",
      "env": {
        "DUCKLE_DUCKDB_BIN": "/path/to/duckdb",
        "DUCKLE_RUNNER_BIN": "/path/to/duckle-runner"
      }
    }
  }
}

Tools: list_components, get_component_schema, create_pipeline, validate_pipeline, run_pipeline, list_pipelines, read_pipeline, read_run_logs, build_pipeline, list_connections, create_connection. run_pipeline / build_pipeline need a DuckDB binary (DUCKLE_DUCKDB_BIN); build_pipeline also needs duckle-runner (DUCKLE_RUNNER_BIN). Full guide: docs/current/mcp.md.

Connection management

Saved connections become DuckDB secrets at runtime so credentials never leak into the pipeline JSON.

Connections live in workspace/connections/ as JSON. The token/password field is encrypted with the workspace key; the rest is plain text.

To use a connection in a pipeline, the Properties panel of any compatible source/sink shows a Connection dropdown - pick one and the fields auto-fill. The list is filtered to connections of a matching kind, so a REST connection is not offered on a JDBC node.

A REST connection is the exception to auto-fill, because it exists to be shared by many nodes that each send a different request: put the vendor's headers and token on the connection once, and rotating a key is a single edit. Headers are merged per key at run time, and the node wins on a key it sets itself; the node's own url and request body are never overwritten. A node with no URL of its own inherits the connection's.

The Copy SQL / Export SQL output is display-only and never executed. Secret values (passwords, tokens, keys, connection strings) are replaced with named placeholders such as ${DUCKLE_PASSWORD}, so the exported script stays valid and is safe to share - substitute the real value at run time. To emit the real credentials instead (so the script runs unchanged), set the environment variable DUCKLE_EXPORT_INCLUDE_SECRETS=1; the output then contains live secrets and should be handled accordingly.

Context variables

Bind any field to a context variable that resolves at run time. Useful for dev vs prod, per-environment paths, secrets injected from CI, etc.

In a context file (workspace/contexts/prod.context.json):

{
  "name": "prod",
  "vars": {
    "DB_HOST": "db.internal.acme.com",
    "S3_BUCKET": "acme-prod-data",
    "BATCH_SIZE": "10000"
  }
}

In the Properties panel of any node, switch a field from Manual to Context and pick DB_HOST. Or inline-reference one with ${DB_HOST} in a string field.

Pick the active context from the topbar's Context dropdown. Switch contexts and re-run without editing the pipeline.

Build from source

Prerequisites

• Rust (stable)
• Node.js 18+ and npm
• cargo-tauri CLI: cargo install tauri-cli --version "^2"
• Platform webview dependencies per the Tauri prerequisites. WebView2 is preinstalled on Windows 10 and 11.

Clone and install

git clone https://github.com/slothflowlabs/duckle
cd duckle
npm --prefix frontend install

Run in development (hot-reloading frontend plus the native shell):

cargo tauri dev

Build a release binary:

# The --features custom-protocol flag is required: without it, tauri-codegen
# embeds the dev URL instead of the bundled frontend.
cargo build --release --manifest-path apps/desktop/Cargo.toml --features custom-protocol

Outputs land in target/release/duckle (or duckle.exe). The engine is not statically linked: DuckDB downloads at first launch, which is why the build is fast and the binary is tiny.

Run the tests:

cargo test                                                          # workspace unit + plan tests
DUCKLE_DUCKDB_BIN=/path/to/duckdb cargo test -p duckle-duckdb-engine # full integration suite

Architecture

duckle/
  apps/desktop/         Tauri 2 shell: Tauri commands, engine installer, llama runtime, window
  frontend/             React 19 + Vite + TypeScript: the designer UI + chat panel
  crates/
    duckdb-engine/      Compiles the node graph to SQL and drives the DuckDB CLI
    slothdb-engine/     SlothDB adapter
    scheduler/          Cron / interval / file-watch triggers
    metadata/           Schema and type model
    plugin-sdk/         Connector / inspector traits
    connectors/         Source and sink connectors
    runtime, workflow-engine, transform-engine, stream-engine, execution-core

• The frontend (React with @xyflow/react) is the visual designer; it talks to the Rust core over Tauri commands.
• duckdb-engine topologically sorts the graph, lowers each node into SQL, and executes by shelling out to the downloaded DuckDB CLI. Non-sink nodes materialize as tables so later stages can reference them; sinks become COPY ... TO statements; cancel kills the process. No statically linked database, so the binary stays small.
• Duckie is a llama-server subprocess on 127.0.0.1 exposing an OpenAI-compatible chat-completions API. The chat panel streams from it via SSE. The model is sandboxed: no fs, no net, no tools - it can only emit text.
• Everything persists to the workspace folder you choose, as plain JSON and Markdown files.

Configuration

A few knobs you can set without touching code.

Performance tips

A few patterns that consistently produce sub-second runs at small / medium data scale, and tractable runs at warehouse scale.

FAQ

Troubleshooting

If you see something not listed, please open an issue with steps to reproduce + the relevant log line.

CI / CD

Duckle's CI pipeline runs on both GitHub and GitLab - the project mirrors to both. Push / pull-request / merge-request / tag events all trigger builds.

What each pipeline does:

1. Frontend - npm ci + npm run build (type-check + bundle)
2. Rust test matrix - cargo test --workspace on Linux + macOS + Windows
3. Live-service integration tests - PostgreSQL + MySQL + MinIO services spun up via Docker, real connector code runs against them
4. Desktop release-build smoke check - cargo build --release --features custom-protocol then grep the binary for the embedded frontend JS chunk (catches the v0.0.7-class "binary loads devUrl" bug at PR time)
5. Format + clippy - informational (does not block merge)
6. On tag: build the Duckle binary on all three OSes, upload as release assets

See .github/workflows/ and .gitlab-ci.yml for the exact steps. The two pipelines are kept feature-equivalent so contributors can fork to either platform.

Releasing a new version

Nothing regenerates this README, the hero / flow SVGs, or the download links automatically - they are hand-maintained, so they drift unless each release updates them. Treat the README as a release artifact: walk this checklist every time before tagging.

# 0. Update the README in the SAME commit as the version bump:
#    - bump every vX.Y.Z reference (the Download / Install link, badges)
#    - refresh capability tables for any new sources/transforms/sinks
#    - add/replace screenshots in docs/assets for shipped features
#    - re-check the hero/flow SVG wording if positioning changed
# 1. Bump version in apps/desktop/tauri.conf.json
# 2. Commit (README + version together)
git commit -am "Release: bump to vX.Y.Z"
# 3. Tag + push
git tag vX.Y.Z
git push origin main vX.Y.Z
# Both GitHub Actions and GitLab CI pick up the tag and build the
# release artifacts automatically. Once green, the draft release on
# GitHub gets the binaries uploaded; un-draft + mark Latest with:
gh release edit vX.Y.Z --draft=false --latest

Roadmap

A complete planned-component breakdown lives in docs/roadmap.md. Highlights:

• [ ] Multi-shard Kinesis and Pulsar streaming (Pulsar blocked on protoc at build time)
• [ ] Apache ORC read / write (blocked on the Arrow version conflict between orc-rust and our workspace pin)
• [x] SFTP source (shipped - russh + russh-sftp on the ring backend, password / key auth, host-fingerprint pin)
• [ ] OAuth-heavy SaaS (Google Sheets, Excel Online, full Salesforce OAuth, Gmail / O365 IMAP)
• [ ] Embedded Python / Rust code stages (current code.* family: SQL, Shell, JavaScript, WebAssembly all ship)
• [ ] Hosted documentation site
• [ ] Plugin marketplace via the connector SDK
• [ ] In-process Native engine - a Rust streaming / incremental executor as an alternative to shelling out to the DuckDB CLI

Contributing

Contributions, issues, and ideas are welcome. Duckle is young and there is a lot of green field. Open an issue to discuss a change before a large PR, match the existing code style, and keep changes focused. Run cargo test and npm --prefix frontend run build before submitting. See CONTRIBUTING.md.

Contributors

Thanks goes to these wonderful people who contribute to Duckle (emoji key):

mits 🚇 ⚠️	Christian 🤔 ⚠️ 💻	gmacc00 🚇 ⚠️ 💻	Stéphane Heckel 🚇 ⚠️ 💻	Steven Snowball 🚇 ⚠️ 💻	Suffian0610 🚇 ⚠️ 💻	add944 🚇 ⚠️ 💻
KNP-BI 🚇 ⚠️ 💻	Richard Wesley 🚇 ⚠️ 💻	micha9ski 🚇 ⚠️ 💻

This project follows the all-contributors specification. Contributions of any kind - code, docs, design, bug reports, ideas - are welcome and recognized here. Comment on any issue or PR with @all-contributors please add @name for code, doc and the bot opens a PR adding them.

License

Licensed under either of MIT or Apache-2.0 at your option.