Task Queue Management
Enqueue and dequeue tasks with metadata, tags, priorities, and ETAs. Higher-priority work claims first.
Distributed task queue · Rust core, multi-language clients
Durable task queues for distributed systems.
Enqueue, dispatch, and observe background work across a fleet of workers. A Rust-fast core with client libraries for Rust, Python, and Java, a live operations dashboard, and an MCP server for agent-driven workflows — all from one binary.
App
Install from
Download the macOS App — Free!
brew install chopflowOne install — broker, CLI, and MCP server. The macOS app bundles all of it in a single .dmg. Workers ship separately.
Queued
12
Running
04
Completed
2,184
Failed
01
Workers
06
TaskStatusAttemptWorker
train_model · 9f4a2cRunning
index_batch · e10c8dCompleted
export_report · 40bb71Queued
sync_records · af1182Failed
Core
Rust-fast core
Clients
Multi-language clients
Observe
Live ops dashboard
Rust·gRPC·tonic·axum·SQLite (WAL)·Tauri·React·TanStack Query·framer-motion·Java·Python·MCP·
Built for control
The primitives distributed work actually needs.
Clear task state, explicit resource ownership, and failure semantics you can reason about.
Worker Dispatch
Assign tasks to workers by tag and resource availability, with bounded per-worker concurrency. Pull model: workers fetch work, the broker never pushes — backpressure is implicit.
Acknowledgments & Retries
Workers ack success or failure; failures retry with exponential backoff and a computed ETA, then dead-letter.
Resource Tracking
CPU, GPU, and memory allocation and live utilization per worker.
Multi-language Interface
Client libraries for Rust, Python, and Java — Celery-like @task and AsyncResult on the Python side.
Metrics & Observability
Queue length, latency, throughput, and resource utilization, live (≤2s poll).
MCP Server
Drive the queue from any MCP-compatible agent — Claude Desktop, Cursor, or your own. Enqueue, inspect, schedule, and run LLM tasks with a synchronous wait, no glue code.
Priority Queues
Every task carries a priority. Ready work is claimed priority DESC, eta ASC, so urgent jobs jump the line without separate queues.
Cron & Schedules
Cron and one-shot ETA schedules with overlap policies. Recurring jobs are first-class — materialize, dispatch, and observe them like any task.
Task lifecycle
State transitions, not guesswork.
Every task moves through an explicit lifecycle. Claims, retries, cancellation, and dead-lettering remain visible from submission to terminal state.
enqueue
Client submits payload, metadata, tags, priority, and ETA.
Queued
Broker persists the task until it becomes eligible.
FetchTasks
A matching worker pulls and claims available work.
Running
Lease and worker ownership stay observable.
AcknowledgeTask
Worker reports a result or structured failure.
Completed
Result and terminal metadata are persisted.
Terminal statesretry · computed ETACancelleddead-letter
Multi-language client
Familiar where it counts.
Define work in the language your team already knows. ChopFlow handles durable state, scheduling, dispatch, and observation underneath.
tasks.pyPython
from chopflow import task, Client
@task(tags=["ml"], resources={"gpu": 1}, ack_late=True)
def train_model(dataset, hyperparams):
return {"accuracy": 0.95}
result = train_model.delay("imagenet", {"lr": 0.001})
output = result.get(timeout=3600, propagate=False)tasks.rsRust
use chopflow::{task, Client};
#[task(tags = ["ml"], resources = {"gpu": 1})]
async fn train_model(dataset: &str, hyperparams: Hyperparams) -> Result<Accuracy> {
Ok(Accuracy { value: 0.95 })
}
let result = train_model.delay("imagenet", &hyperparams);
let output = result.get().await?;chopflow_broker start --host 127.0.0.1 --port 8000 --http-port 8080 --open
chopflow_worker start -b http://localhost:8000 -t default -r cpu:1One system, focused crates.
Default storage: SQLite in WAL mode · Transports: gRPC via tonic + HTTP/JSON via axum
coreShared task models, state transitions, and protocol types.
brokerPersistence, scheduling, leases, retries, and APIs.
workerPull-based execution, heartbeats, and acknowledgments.
cliLocal startup, configuration, and operator commands.
dashboardEmbedded live console and native Tauri shell.
mcpModel Context Protocol server — agents drive the queue.
Embedded operations
One binary, a live console.
The broker embeds the dashboard, so there is no separate frontend deployment to operate. The same interface also ships as a native macOS app through Tauri.
Inspect queues, worker capacity, task attempts, timing, and terminal failures from one consistent operational view.
chopflow_broker — zsh
FAQ
Questions, answered.
The practical details before you run ChopFlow in production.
What language is ChopFlow written in, and what can talk to it?
The core broker and workers are written in Rust (2021 edition). Client libraries are available for Rust, Python, and Java, with a Celery-like
@task decorator and AsyncResult for Python. Workers and the CLI speak gRPC to the broker; browsers and external tooling speak HTTP/JSON.How does ChopFlow survive a broker restart?
Tasks persist in SQLite (or an in-memory backend for testing). On boot the broker reconciles in-flight work: any task left
Running from a previous run is reset to Queued, since its worker is gone. No silently dropped work.Do I need to deploy a separate frontend?
No. The dashboard is embedded in the broker binary via
rust-embed and served on the HTTP port — one process, no Node build, no separate deploy. The same UI also ships as a native macOS app through Tauri.What happens when a task fails or times out?
Workers acknowledge each task with a success or structured failure. Failed tasks follow configurable retry policies, and a timeout monitor reclaims work whose lease has expired. Tasks that exhaust retries are dead-lettered so they stay visible instead of vanishing.
Can I route work to specific workers?
Yes. Tasks carry tags and resource requirements; workers register with tags (e.g.
gpu,ml) and capacity (CPU, GPU, memory). The dispatcher only hands a task to a worker whose tags match and whose resources are available.How is this different from Temporal?
Different category. Temporal is a durable workflow engine — it orchestrates long-running, multi-step processes via event-sourced replay. ChopFlow is a task queue in the Celery / Ray / Dask lineage: discrete units of work, a pull-based dispatch model, leases, retries, and dead-lettering. If you're scheduling background jobs across a worker fleet and observing them live, that's ChopFlow's shape; if you need saga-style workflows with checkpointed replay, Temporal is the right tool.
Is ChopFlow production-ready or research-only?
Both. It's built for production applications and for distributed-systems research: the
experiment directory ships reproducible configs and analysis scripts. See the GitHub repo for current status and the roadmap.Start flowing in minutes.
$ chopflow broker start --http-port 8080 --open