Quickstart

Install ChopFlow, start a broker, register a worker, and enqueue your first task — in under two minutes.

1. Install ChopFlow#

One install gives you a single chopflow command — the broker, CLI, and MCP server behind subcommands. Pick whichever path fits.

Option A — From a package manager#

From crates.io with Cargo:

$ cargo install chopflow

Or with Homebrew:

$ brew install chopflow

Option B — Build from source#

Requires Rust (2021 edition) and Cargo. Builds the whole workspace into optimized binaries — no Node toolchain needed to run.

$ git clone https://github.com/ricardoleal20/ChopFlow.git
$ cd ChopFlow
$ cargo build --release
   Compiling chopflow-core v0.1.6
   Compiling chopflow-proto v0.1.6
   Compiling chopflow-broker v0.1.6
   Compiling chopflow-cli v0.1.6
   Compiling chopflow-mcp v0.1.6
   Compiling chopflow v0.1.6
    Finished release [optimized] target(s) in 42.8s

The unified chopflow binary lands in target/release/ (alongside chopflow_worker). Add that directory to your PATH (or reference it directly) so the commands below work as written. See Installation for the full binary list and prerequisites.

2. Start the broker#

The broker persists tasks to SQLite, serves gRPC on port 8000 and HTTP on port 8080, and opens the dashboard in your browser.

$ ./target/release/chopflow broker start \
    --host 127.0.0.1 --port 8000 --http-port 8080 --open
Persistence: sqlite (wal mode at ./chopflow.db)
Reconciled in-flight tasks: 0
ChopFlow gRPC on 127.0.0.1:8000
ChopFlow HTTP/ on 127.0.0.1:8080
Opened dashboard in browser

3. Start a worker#

Workers poll the broker, claim matching tasks, and execute them. Tags route work; resources declare capacity.

$ ./target/release/chopflow_worker start \
    --broker http://localhost:8000 \
    --tags gpu,ml --resources gpu:1,cpu:4
Worker registered: worker-7a3f (tags: gpu,ml)
Polling broker at http://localhost:8000 (interval: 2s)

4. Enqueue a task#

Define a task payload in task.json and enqueue it via the CLI. The broker assigns an ID and makes it eligible for dispatch.

$ ./target/release/chopflow enqueue \
    -f task.json --name training --tags gpu,ml
Enqueued: training · 9f4a2c (attempt: 0, eta: now)

Installation

Build ChopFlow from source using Rust and Cargo. The whole workspace compiles to a single set of binaries — no Node toolchain is required for a fresh clone.

Prerequisites#

• Rust (2021 edition) — install via rustup.
• Cargo — ships with Rust.
• Python 3.8+ — only if you intend to use the Python client.
• Node + pnpm — only if you will edit the embedded dashboard UI (the built bundle is committed).

Install#

Two paths to get the binaries onto your PATH.

Package manager#

From crates.io:

$ cargo install chopflow

Or with Homebrew:

$ brew install chopflow

Both install the unified chopflow command. The Python client/worker SDK is on PyPI: pip install chopflow.

Build from source#

$ git clone https://github.com/ricardoleal20/ChopFlow.git
$ cd ChopFlow
$ cargo build --release

This produces the following binaries in target/release/:

Storage backends#

The broker persists tasks with SQLite by default (a chopflow.db file in the working directory). Pass --storage memory for an ephemeral, in-memory backend useful for tests and demos. See Storage Backends.

Task Lifecycle

Every task moves through an explicit lifecycle from submission to terminal state. There are no hidden transitions — each state is persisted and visible in the dashboard.

The flow#

A task advances through six stages, driven by the client, the broker, and the worker:

State transitions#

The TaskStatus field on a task encodes its entire lifecycle. The broker never pushes work — workers call FetchTasks and the broker atomically transitions matching Queued tasks to Running. This pull model means backpressure is implicit: if no worker fetches, work simply waits.

On failure, a task with remaining retries transitions back to Queued with an incremented retry count and a computed ETA (see Retry & Dead-Letter). When retries are exhausted, it moves to DeadLettered.

See Task States & Statuses for the full state reference.

Task States & Statuses

ChopFlow tasks move through seven states. Each state is represented by a colored status dot paired with a text label — never color alone.

State reference#

Scheduling & Recurrence

A Schedule is a recurring or one-shot task template that the broker ticker materializes into Tasks on a background ticker (one tick per second).

Schedule kinds#

Overlap policies#

Each schedule carries an overlap policy that governs what happens when a fire is due but a previous run for the same schedule is still in flight (Queued or Running):

Time semantics#

Missed cron runs are not backfilled. On broker startup, enabled cron schedules have their next_fire recomputed from now, so the ticker only fires future matches. A disabled one-shot with a past ETA stays disabled until re-enabled.

Task template#

Every schedule carries a TaskTemplate — the name, payload, tags, resources, and max_retries that each materialized task inherits. The dashboard's "Run now" action enqueues a one-off task from the template without disturbing the schedule's overlap accounting.

Manage schedules via the HTTP API or the CLI schedule subcommand.

Retry & Dead-Letter

Failed tasks retry with a configurable backoff strategy and a computed ETA. After the maximum number of retries is exhausted, tasks are dead-lettered.

RetryPolicy#

The RetryPolicy enum (in core/src/retry.rs) defines three strategies:

Defaults#

The exponential_backoff(max_retries) constructor produces a sensible default policy:

ExponentialBackoff {
    initial_delay_seconds: 5,
    max_delay_seconds: 300,   // 5 minutes
    max_retries,
    jitter: 0.1,
}

Next retry time#

next_retry_time(retry_count) computes when a failed task becomes eligible again. It returns None when the retry count exceeds max_retries — at that point the task transitions to DeadLettered and is available for inspection and replay.

The computed ETA is stored on the task, so a retried task simply sits in Queued until its ETA arrives and a matching worker fetches it.

Priority & Ordering

Every task carries an integer priority. The broker dispatches the highest-priority ready tasks first, so urgent work jumps ahead of a backlog of batch jobs. The default priority is 0, which preserves FIFO ordering.

The priority field#

Task.priority is a signed 32-bit integer (core/src/task.rs). Higher numbers are claimed before lower ones. The default is 0; negative values are allowed and dispatch last. Priority is set at enqueue time and rides along on a schedule's TaskTemplate.

let task = Task::new("resize_image", payload, tags)
    .with_priority(5);

How claim_ready orders#

When a worker fetches tasks, the broker's claim_ready selects ready tasks (matching tags, ETA elapsed, resources satisfied) and orders them by:

The same ordering applies to both storage backends. The in-memory backend sorts a candidate scan; SQLite uses ORDER BY priority DESC, eta_ms, enqueue_ms, id backed by a tasks_priority index on (status, priority).

Setting priority#

Priority is accepted everywhere a task is enqueued — the CLI (--priority), the HTTP POST /api/tasks body, the gRPC EnqueueTask request, the MCP enqueue_task tool, and the Java/Python client builders. Omitting it defaults to 0.

SQLite migration#

Existing SQLite databases gain a priority INTEGER NOT NULL DEFAULT 0 column via an idempotent ALTER TABLE on broker startup; old rows default to priority 0. No manual migration step is required.

Resources & Tags

Tags route tasks to matching workers. Resources declare per-worker capacity for CPU, GPU, and memory — the dispatcher only hands a task to a worker that can satisfy it.

Tags#

A task carries a list of tags (e.g. gpu, ml). A worker registers with the tags it can handle. The broker only returns a task to a worker via FetchTasks when the worker's tags match the task's tags. This is how you keep GPU work on GPU workers and CI work on CI workers.

ResourceRequirements#

A task declares the resources it needs as a map of resource name to amount (e.g. {"gpu": 1, "cpu": 4}). The ResourceRequirements type (core/src/resources.rs) is the task-side half of the contract.

pub struct ResourceRequirements {
    pub resources: HashMap<String, u32>,
}

impl ResourceRequirements {
    pub fn can_be_satisfied_by(&self, availability: &ResourceAvailability) -> bool {
        for (resource, required) in &self.resources {
            match availability.available.get(resource) {
                Some(available) if *required <= *available => continue,
                _ => return false,
            }
        }
        true
    }
}

ResourceAvailability#

A worker declares ResourceAvailability with two maps: total (capacity) and available (currently free). When a worker claims a task, the broker decrements available; when the task is acked, it is returned. This prevents overcommitment — a worker never receives more work than it can run.

Matching, end to end#

When a worker calls FetchTasks, the broker matches queued, ETA-eligible tasks against that worker:

1. The task's tags must be a subset of the worker's tags.
2. The task's ResourceRequirements must be satisfiable by the worker's current ResourceAvailability.
3. The task's ETA must have arrived.

Matching tasks are atomically transitioned Queued → Running and handed to the worker, up to max_tasks.

Checkpointed Pipelines

A task may declare its pipeline stages. A context-aware handler persists a checkpoint per stage, and a retried task resumes from the last completed stage — Temporal-style durable execution without a workflow engine.

Declaring stages#

Stages are declared at enqueue time. The CLI takes a comma-separated list; the HTTP body, the gRPC EnqueueTaskRequest, and the MCP enqueue_task tool all accept the same stages field.

$ chopflow enqueue -f task.json -n rag.ingest -g rag \
    --stages "chunk,embed,index" --idempotency-key "doc-42"
Broker assigned task ID: 9f4a2c1e-...

The stages may also live as a top-level "stages" array in the task JSON file — the --stages flag wins when both are present. Stage names are free-form; declaring them makes the pipeline visible to the dashboard and gives the worker the ordered list to checkpoint against.

Checkpointing from a handler#

Plain handlers registered with registry.register(name, fn) keep working unchanged. New context-aware handlers register with register_ctx and receive a TaskCtx that can persist per-stage state:

use chopflow_worker::TaskCtx;
use serde_json::{json, Value};

// async fn(TaskCtx, Value) -> Result<Value, String>
async fn ingest(ctx: TaskCtx, payload: Value) -> Result<Value, String> {
    // Skip a stage a previous attempt already finished.
    if ctx.stage_checkpoint("chunk").is_none() {
        let chunks = chunk_document(&payload);
        ctx.checkpoint("chunk", json!({ "chunks": chunks }))
            .await?;  // persisted (upsert) on the broker
    }
    // embed / index — checkpoint progress so a retry resumes mid-pipeline
}

registry.register_ctx("rag.ingest", ingest);

ctx.checkpoint(stage, payload) serializes the payload and upserts it keyed by (task_id, stage) — re-saving a stage overwrites it, so each stage has at most one checkpoint. ctx.stage_checkpoint(stage) and ctx.completed_stages() read what previous attempts recorded.

Resume on retry#

When a task fails mid-pipeline and retries, the worker re-fetches the task's checkpoints (a dedicated GetCheckpoints gRPC) before invoking the handler. The handler sees the stages that already completed and skips them, so only the unfinished tail runs again. Checkpoints are rows in the task store — with SQLite they survive broker restarts too.

Reading checkpoints back#

Only the single-task view pays for the checkpoint read: GET /api/tasks/:id includes them, the list endpoint stays cheap. recorded_at is Unix epoch milliseconds.

{
  "id": "9f4a2c1e-...",
  "name": "rag.ingest",
  "status": "running",
  "stages": ["chunk", "embed", "index"],
  "idempotency_key": "doc-42",
  "checkpoints": [
    {
      "task_id": "9f4a2c1e-...",
      "stage": "chunk",
      "payload": "{\"chunks\":[\"...\"],\"chunk_size\":64}",
      "recorded_at": 1758172800000
    }
  ]
}

The task drawer in the dashboard renders the declared stages as a timeline — completed stages with their checkpoint timestamp, the in-flight stage pulsing, pending stages muted. chopflow status get <id> prints stages, the idempotency key, and each checkpoint (stage (recorded_at): payload-preview).

Rate-Limit Resources

Worker resources are either static slots or replenishing token buckets. A replenishing resource models a provider quota — tokens return with time, not on release — so quota exhaustion becomes backpressure instead of a burst of failures.

Declaration syntax#

Each --resources entry is name:capacity (static) or name:capacity@refill_amount/period_secs (replenishing):

$ chopflow_worker start --broker http://localhost:8000 \
    --tags demo --resources "cpu:4,llm.rpm:60@60/60"

Token-bucket semantics#

A replenishing resource is a lazy token bucket — no background timers:

• Allocate consumes. Claiming a task that requires the resource takes tokens from the bucket.
• Release does not restore. When the task is acked, the tokens are not returned — a rate-limited call has been spent either way.
• Refill is time-based and lazy. Tokens accrue at refill_amount / period_secs and are credited (capped at capacity) the next time the bucket is consulted.

Task-side requirements stay a plain map: a task requiring {"llm.rpm": 1} is only claimed once some worker's bucket holds at least one token, and otherwise simply waits in Queued.

Idempotent Submits

Give a submit an idempotency key and it becomes safe to retry: the broker returns the existing task instead of creating a duplicate — atomically, even under concurrent submits.

Submitting with a key#

Pass idempotency_key in the POST /api/tasks body (or --idempotency-key on the CLI, or the field of the same name in the task JSON file — the flag wins):

POST /api/tasks
{
  "name": "rag.ingest",
  "payload": { "document": "..." },
  "stages": ["chunk", "embed", "index"],
  "idempotency_key": "doc-42"
}

The first submit creates the task:

{ "task_id": "9f4a2c1e-...", "deduplicated": false }

Submitting the same key again — from any client, at any time, in any task status — returns the same task:

{ "task_id": "9f4a2c1e-...", "deduplicated": true }

Atomic under concurrency#

The lookup-and-insert pair runs under a broker-wide submit lock, so two clients racing the same key resolve to a single created task — exactly one wins the 201, everyone else gets the existing task back. The key rides on the task (visible as idempotency_key in GET /api/tasks/:id, the CLI status output, and the dashboard's task drawer) and is unique across the store.

Core (Rust library)

The chopflow-core crate provides shared task models, state transitions, retry/schedule primitives, and the storage trait used by the broker, worker, and CLI.

Modules#

Public re-exports#

The crate root re-exports the primary types so downstream crates can write use chopflow_core::Task rather than reaching into modules:

pub use dispatcher::Dispatcher;
pub use queue::Queue;
pub use resources::ResourceRequirements;
pub use retry::RetryPolicy;
pub use schedule::{Schedule, ScheduleKind, OverlapPolicy, TaskTemplate};
pub use storage::{Storage, TaskFilter, StatusCounts, InMemoryStorage, SqliteStorage};
pub use task::Task;

Task builder#

Tasks are constructed with a fluent builder:

let task = Task::new("train_model".into(), json!({"lr": 0.001}))
    .with_tags(["ml", "gpu"])
    .with_eta(Utc::now() + Duration::minutes(5))
    .with_max_retries(5)
    .with_resource("gpu", 1)
    .with_resource("cpu", 4);

Broker

The chopflow-broker crate is the server: it persists tasks, runs the schedule ticker, manages leases and retries, and serves both gRPC (for workers/CLI) and HTTP/JSON + the embedded dashboard (for browsers).

One shared state#

Both the gRPC service and the HTTP layer read and write a single BrokerState backed by one Storage implementation. The dashboard sees live updates from workers and vice versa — there is no second source of truth.

Ports and protocols#

Startup sequence#

1. Initialize tracing (tracing_subscriber::fmt::init()).
2. Build the storage backend (SQLite or in-memory).
3. Reconcile in-flight tasks: any task left Running is reset to Queued.
4. Construct BrokerState and the ChopFlowBrokerService; spawn the timeout monitor.
5. Bind the HTTP listener and spawn the Axum server.
6. Serve gRPC via tonic (blocks until shutdown).

With --open, the broker waits ~300ms for the HTTP listener to bind, then opens the dashboard URL in the platform's default browser.

Flags#

Worker

The chopflow-worker crate is a pull-based executor: it registers with the broker, polls for matching work, runs a handler, and acknowledges the result.

Registration#

On start, the worker calls RegisterWorker with its address, tags, and resource capacity. The broker returns a worker ID used for all subsequent calls.

$ ./target/release/chopflow_worker start \
    --broker http://localhost:8000 \
    --tags gpu,ml --resources gpu:1,cpu:4

Poll loop#

The worker repeatedly calls FetchTasks with its worker ID and a max_tasks bound. The broker matches queued, ETA-eligible tasks against the worker's tags and available resources, atomically transitions them to Running, and returns them. If no work matches, the worker sleeps for its poll interval and tries again.

Acknowledgment#

After executing a handler, the worker calls AcknowledgeTask with success: true (and a JSON result) or success: false (and an error message). The broker records the outcome and either persists the result or applies the retry policy.

Flags#

CLI

The chopflow command's enqueue / status / schedule subcommands are the operator's interface to a running broker. (The standalone chopflow_cli binary is equivalent — same flags, same crate — but the unified chopflow command is the recommended entry point.)

enqueue#

Submit a task from a JSON payload file:

$ ./target/release/chopflow enqueue \
    --task task.json --name training --tags gpu,ml --eta 2026-09-04T15:00:00Z --priority 5

status#

$ ./target/release/chopflow status --id 9f4a2c
$ ./target/release/chopflow status --all

schedule#

Manage recurring and one-shot schedules (see Scheduling & Recurrence for semantics):

$ ./target/release/chopflow schedule create \
    --name nightly-build --task build --cron "0 9 * * *" \
    --tags ci --resources cpu:4 --overlap skip
$ ./target/release/chopflow schedule create \
    --name one-off-report --task report \
    --eta 2026-09-04T14:30:00Z --overlap allow
$ ./target/release/chopflow schedule list
$ ./target/release/chopflow schedule delete <schedule-id>

Embedded Dashboard

The broker embeds a live operations dashboard — no separate frontend deployment required. Serve it from the same binary with --open.

The dashboard is a React + Vite + Tailwind SPA in broker/ui/ (TypeScript, TanStack Query, framer-motion). The built bundle in broker/ui/dist/ is embedded into the broker binary via rust-embed, so a fresh clone runs the UI with no Node toolchain. The HTTP handler serves the SPA from any non-API path, falling back to index.html for client-side routing.

Views#

Data freshness#

The SPA polls the HTTP API (see HTTP API Reference) every 2 seconds. There is no separate data plane — the dashboard is a first-class client of the same /api endpoints any external tooling would use, reading the same live BrokerState as the gRPC workers.

Iterative development#

$ pnpm --dir broker/ui install     # first time only
$ pnpm --dir broker/ui dev         # http://localhost:5173, hot reload

The Vite dev server proxies /api to http://localhost:8080, so it runs against a live broker. After changes, rebuild the embedded bundle:

$ pnpm --dir broker/ui build
$ touch broker/src/http.rs   # force rust-embed to re-read ui/dist
$ cargo build

Demos

The chopflow-demos crate ships a drop-in demo worker and a seeding tool so cargo run produces a live dashboard end-to-end.

The demo worker is structurally identical to worker/src/main.rs — the only difference is its handler registry. It registers four showcase handlers (plus echo and a default fallback):

One-command run#

$ bash demos/run.sh

This builds the workspace, starts an in-memory broker with --open (dashboard at http://localhost:8080), starts a demo worker (--tags demo,ml --resources cpu:4), then seeds:

• One task of each handler type — the flaky_handler one with max_retries: 5 so you can watch it retry.
• A */2 * * * * cron schedule running batch_compute with overlap skip.
• A one-shot simulate_pipeline schedule 5 minutes out with overlap allow.

Ctrl+C stops the broker and worker. You'll see four tasks flow through distinct lifecycles, the flaky handler visibly retrying, and two schedules ticking in the Schedules view.

Seeding a running broker#

$ cargo run -p chopflow_demos --bin chopflow_demo_seed -- [broker_base_url]
# broker_base_url defaults to http://localhost:8080

Drive a demo from each client#

The demo worker already registers resize_image and batch_compute, so once a broker and the demo worker are running you only need a producer to drive them. Because every client speaks the same gRPC contract, that producer can be Rust, Python, or Java — pick whichever client you ship in. The two demos below are enqueued from each current client.

Demo 1 — resize_image#

Generates a synthetic gradient PNG of width × height and resizes it to half dimensions (image-rs, nearest-neighbour). Returns input_dims, output_dims, and output_pixels. Payload defaults to 64×64 when omitted.

use chopflow::chop_flow_broker_client::ChopFlowBrokerClient;
use chopflow::EnqueueTaskRequest;
use tonic::Request;
use std::collections::HashMap;

let mut client = ChopFlowBrokerClient::connect("http://localhost:8000").await?;
let resp = client.enqueue_task(Request::new(EnqueueTaskRequest {
    name: "resize_image".into(),
    payload: r#"{"width":128,"height":128}"#.into(),
    tags: vec!["image".into()],
    eta: None,
    max_retries: 3,
    resources: HashMap::from[("cpu".into(), 2)],
    priority: 0,
})).await?;
let task_id = resp.into_inner().task_id;

from chopflow import ChopFlowClient

with ChopFlowClient.connect("localhost:8000") as client:
    result = (
        client.enqueue("resize_image")
        .payload({"width": 128, "height": 128})
        .tags("image")
        .max_retries(3)
        .enqueue()
    )
    task = result.get(timeout=60)
    print(task.status_name, task.result)
    # COMPLETED {"status":"ok","input_dims":[128,128],"output_dims":[64,64],"output_pixels":4096}

import java.time.Duration;
import java.util.Map;
import dev.ricardoleal20.chopflow.client.ChopFlowClient;

try (ChopFlowClient client = ChopFlowClient.connect("localhost:8000")) {
  var result = client.enqueue("resize_image")
      .payload(Map.of("width", 128, "height", 128))
      .tags("image")
      .maxRetries(3)
      .enqueue();
  var task = result.get(Duration.ofSeconds(60));
  System.out.println(task.getStatus());   // COMPLETED
  System.out.println(task.getResult());   // {"status":"ok","input_dims":[128,128],...}
}

Demo 2 — batch_compute#

CPU-bound n × n f64 matrix multiply (nalgebra), with n clamped to 2–256. Returns matrix_size, elapsed_ms, and a checksum. Enqueue it at priority: 5 alongside lower-priority work to watch ordering and resource contention play out on the dashboard.

let resp = client.enqueue_task(Request::new(EnqueueTaskRequest {
    name: "batch_compute".into(),
    payload: r#"{"n":128}"#.into(),
    tags: vec!["demo".into()],
    eta: None,
    max_retries: 2,
    resources: HashMap::from[("cpu".into(), 4)],
    priority: 5,
})).await?;

with ChopFlowClient.connect("localhost:8000") as client:
    result = (
        client.enqueue("batch_compute")
        .payload({"n": 128})
        .tags("demo")
        .priority(5)
        .max_retries(2)
        .enqueue()
    )
    task = result.get(timeout=120)
    print(task.status_name, task.result)
    # COMPLETED {"status":"ok","matrix_size":128,"elapsed_ms":18,"checksum":...}

try (ChopFlowClient client = ChopFlowClient.connect("localhost:8000")) {
  var result = client.enqueue("batch_compute")
      .payload(Map.of("n", 128))
      .tags("demo")
      .priority(5)
      .maxRetries(2)
      .enqueue();
  var task = result.get(Duration.ofSeconds(120));
  System.out.println(task.getResult());   // {"status":"ok","matrix_size":128,...}
}

RAG Demo

The rag.ingest demo puts every pipeline feature on one screen: a checkpointed chunk → embed → index flow, a replenishing llm.rpm token bucket gating dispatch, and an idempotency key deduplicating the submit.

Run it#

$ bash demos/rag.sh
   ... cargo build output ...
ChopFlow gRPC  on 127.0.0.1:8000
ChopFlow HTTP/ on 127.0.0.1:8080
   ... worker startup + the four standard demo-task submits ...
POST /tasks -> 201 Created
POST /tasks -> 200 OK
Idempotent submit: both rag.ingest submits returned the same task id 9f4a2c1e-... (deduplicated: true)
Seed complete. Open the dashboard at http://localhost:8080 — tasks flowing + 2 schedules ticking.
open http://localhost:8080 — watch the rag.ingest task's checkpoints grow chunk -> embed -> index

The script builds the workspace, starts an in-memory broker, starts the demo worker with --resources "cpu:4,llm.rpm:60@60/60" (a static CPU pool plus a replenishing 60-per-minute LLM quota), and seeds the demo tasks. Ctrl+C stops everything.

The rag.ingest pipeline#

The task declares stages: ["chunk", "embed", "index"], requires {"llm.rpm": 1}, and carries the idempotency key rag-demo-doc-1 — submitted twice by the seed to prove the dedup. The handler is registered with register_ctx and checkpoints each stage:

The finished task returns {"chunks":4,"dim":16,"indexed":true,"resumed_from":null} for the sample document (resumed_from names the stage a retry resumed from, null on a clean first run).

What to watch in the dashboard#

• Pipeline stages timeline — open the rag.ingest task's drawer; as checkpoints land, chunk → embed → index fill in with timestamps (the embed stage even shows a 3/4-style progress note while it walks the chunk list).
• Deduplication — exactly one rag.ingest task despite the double submit; the drawer also shows its idempotency key.
• Rate-limit gating — the task sits in Queued until the worker's llm.rpm bucket has a token; it waits for quota rather than failing.

LLM Worker

The chopflow-llm-worker crate is a specialized worker that executes LLM inference tasks against an OpenAI-compatible endpoint — turning ChopFlow into an LLM-job orchestrator an agent can drive through the MCP.

It uses the same pull-based protocol as the generic worker (register → heartbeat → FetchTasks → AcknowledgeTask), but with two differences: handlers are async (each inference call is network I/O), and it runs a bounded concurrency pool so a single worker pipelines multiple in-flight requests rather than blocking on one at a time.

Task names#

The worker subscribes to the llm tag by default and dispatches by task name:

Unknown task names fall back to llm.complete (treating prompt as the input).

Configuration#

Run#

$ OPENAI_API_KEY=sk-... cargo run -p chopflow-llm-worker -- \
    --broker http://localhost:8000 --concurrency 8

With a worker running, an agent (or the CLI) can enqueue an llm.complete task tagged llm; the worker picks it up, calls the model, and acks the result. The MCP run_llm_task tool does this end-to-end and returns the model's text — see run_llm_task.

Rust Client

Rust is the first-class client language. Speak gRPC to the broker with the tonic client generated from proto/proto/chopflow.proto, or embed chopflow-core in-process for tests.

gRPC client#

The proto compiles to a ChopFlowBrokerClient under the chopflow module. This is the same client the worker and CLI use internally:

use chopflow::chop_flow_broker_client::ChopFlowBrokerClient;
use chopflow::{EnqueueTaskRequest, GetTaskStatusRequest};
use tonic::Request;

let mut client = ChopFlowBrokerClient::connect("http://localhost:8000").await?;

let resp = client.enqueue_task(Request::new(EnqueueTaskRequest {
    name: "resize_image".into(),
    payload: r#"{"path":"img.png","w":128}"#.into(),
    tags: vec!["gpu".into()],
    eta: None,
    max_retries: 3,
    resources: HashMap::from[("cpu".into(), 4)],
    priority: 5,
})).await?;

let task_id = resp.into_inner().task_id;
let status = client
    .get_task_status(Request::new(GetTaskStatusRequest { task_id }))
    .await?
    .into_inner()
    .task
    .unwrap()
    .status;

In-process (core)#

For tests or single-process embedding, use chopflow-core directly against an InMemoryStorage — no broker process needed:

use chopflow_core::storage::InMemoryStorage;
use chopflow_core::task::{Task, TaskStatus};

let storage = InMemoryStorage::new();
let mut task = Task::new("echo", serde_json::json!({"msg":"hi"}))
    .with_tags(vec!["demo".into()]);
task.status = TaskStatus::Queued;
storage.insert(task.clone()).await?;

Python Client

The Python client brings Celery-like ergonomics — a @task decorator, .delay(), and an AsyncResult handle — speaking gRPC to the broker.

Ergonomics#

The API surface, mirroring the Celery idiom Python developers already know:

from chopflow import task, Client

client = Client("http://localhost:8000")

@task(client, tags=["ml"], resources={"gpu": 1}, max_retries=3)
def train_model(dataset, hyperparams):
    # Your training code here
    return {"accuracy": 0.95}

# Enqueue for asynchronous execution
result = train_model.delay("imagenet", {"lr": 0.001})

# Block until the task reaches a terminal state
output = result.get(timeout=3600)
print(output)  # {"accuracy": 0.95}

Under the hood, .delay() calls EnqueueTask and .get() polls GetTaskStatus until the task is Completed, Failed, or DeadLettered.

Java Client

The Java client offers a typed task interface over the same gRPC contract — register handlers, submit tasks, and await results.

Ergonomics#

The API surface — a typed @ChopTask annotation and a future-backed result handle, speaking gRPC via the generated stub:

ChopFlowClient client = ChopFlowClient.connect("localhost:8000");

@ChopTask(tags = {"ml"}, resources = {"gpu:1"}, maxRetries = 3)
public Map<String, Object> trainModel(String dataset, Map<String, Object> hyperparams) {
    // Your training code here
    return Map.of("accuracy", 0.95);
}

// Enqueue for asynchronous execution
AsyncResult<Map<String, Object>> result =
    client.submit("trainModel", "imagenet", Map.of("lr", 0.001));

// Block until the task reaches a terminal state
Map<String, Object> output = result.get(3600, TimeUnit.SECONDS);

Under the hood, submit calls EnqueueTask and get polls GetTaskStatus until the task reaches a terminal state.

HTTP API Reference

The broker exposes a JSON HTTP API on the configured --http-port. All endpoints are prefixed with /api except the health probe.

Endpoints#

API — Stats

Cluster-wide statistics — a single call the dashboard polls every 2 seconds.

GET /api/stats#

Returns aggregate counts derived from storage and the live worker registry, plus the broker's own environment identity. No query parameters.

{
  "queue_length": 12,
  "tasks_processing": 2,
  "tasks_completed": 184,
  "tasks_failed": 7,
  "active_workers": 3,
  "total_tasks": 203,
  "schedules": 2,
  "env": "prod",
  "region": "us-east-1"
}

API — Tasks

Task lifecycle endpoints: list with filters, fetch one, enqueue, and cancel.

GET /api/tasks#

Query params: status (one of created|queued|running|completed|failed|dead-lettered|cancelled), limit, offset. Returns a page plus the unfiltered total.

{
  "tasks": [
    {
      "id": "9f4a2c1e-...",
      "name": "resize_image",
      "payload": {"path": "img.png", "w": 128},
      "tags": ["gpu"],
      "status": "running",
      "retry_count": 0,
      "max_retries": 3,
      "enqueue_time": 1725446400000,
      "eta": null,
      "resources": {"cpu": 4},
      "result": null,
      "schedule_id": null,
      "priority": 0
    }
  ],
  "total": 203
}

GET /api/tasks/:id#

Returns a single task by UUID. 404 if not found, 400 on a malformed ID.

POST /api/tasks#

Enqueue a task. The broker sets status = queued and persists it; matching workers pick it up on their next FetchTasks.

{
  "name": "resize_image",
  "payload": {"path": "img.png", "w": 128},
  "tags": ["gpu"],
  "max_retries": 3,
  "resources": {"cpu": 4},
  "priority": 5
}

{ "task_id": "9f4a2c1e-..." }

POST /api/tasks/:id/cancel#

Cancels a non-terminal task. Returns {"success": false} if the task is already terminal (Completed, Failed, DeadLettered, Cancelled) or not found. If the task was Running, the broker also releases it from the assigned worker.

{ "success": true }

API — Schedules

Recurring and one-shot schedule endpoints: list, create, update, and delete. See Scheduling & Recurrence for semantics.

GET /api/schedules#

Returns every schedule (enabled and disabled). Each carries a kind discriminated by type: {"type":"cron","cron":"0 9 * * *"} or {"type":"oneshot","eta":"2026-09-04T14:30:00Z"}.

POST /api/schedules#

Create a schedule. The broker validates the cron expression / ETA and computes next_fire. overlap_policy defaults to skip.

{
  "name": "nightly-build",
  "task_template": {
    "name": "build",
    "payload": "{}",
    "tags": ["ci"],
    "resources": {"cpu": 4},
    "max_retries": 3,
    "priority": 0
  },
  "kind": {"type": "cron", "cron": "0 9 * * *"},
  "overlap_policy": "skip"
}

{ "schedule_id": "a1b2c3d4-..." }

PATCH /api/schedules/:id#

Partial update. All fields optional: enabled (pause/resume), overlap_policy, cron (replaces the kind and recomputes next_fire). Returns the updated schedule.

DELETE /api/schedules/:id#

Delete a schedule by ID. Returns {"success": true}. Deleting a schedule does not affect tasks it already materialized.

API — Workers

Worker registry, capacity, and liveness endpoints.

GET /api/workers#

Returns every registered worker with its tags, resource meters, assigned task count, and liveness (based on heartbeat age).

[
  {
    "id": "b7e1f0a2-...",
    "address": "10.0.0.4:50051",
    "tags": ["gpu", "ml"],
    "alive": true,
    "assigned_tasks": 2,
    "resources_total": {"cpu": 8, "gpu": 1},
    "resources_available": {"cpu": 4, "gpu": 0},
    "last_heartbeat": 1725446412000
  }
]

GET /healthz#

Liveness/readiness probe. Returns 200 OK with body ok. Use it for load-balancer health checks and container orchestrators — it does not report broker internals.

API — Environments

Read-only fleet catalog and the broker's own identity. The dashboard polls this to render its environment switcher. See Environments for the concepts and configuration.

GET /api/environments#

Returns the broker's own identity (current) and the full fleet catalog (environments) loaded from environments.yml. The current entry always carries an empty http_url: the serving broker is same-origin to its own dashboard, so selecting current must not retarget the API. No query parameters.

{
  "current": {
    "name": "prod",
    "region": "us-east-1",
    "grpc_url": "http://broker.prod:8000",
    "http_url": ""
  },
  "environments": [
    {
      "name": "local",
      "region": "default",
      "grpc_url": "http://localhost:8000",
      "http_url": "http://localhost:8080"
    },
    {
      "name": "prod",
      "region": "us-east-1",
      "grpc_url": "http://broker.prod:8000",
      "http_url": "http://broker.prod:8080"
    }
  ]
}

gRPC Reference

The broker serves a gRPC API on the configured --port (default 8000) via tonic. This is the wire protocol workers and clients speak; the HTTP API is a thin facade over the same BrokerState.

The service is defined in proto/proto/chopflow.proto (package chopflow). Generate a client in any language with gRPC tooling.

Service methods#

Key messages#

Task carries id, name, JSON payload, tags, enqueue_time/eta, retry_count/max_retries, status, a resources map, result (empty until terminal), schedule_id (set when materialized by a schedule), and priority (default 0; higher is claimed first — see Priority & Ordering). TaskStatus is the enum: CREATED=0, QUEUED=1, RUNNING=2, COMPLETED=3, FAILED=4, DEADLETTERED=5, CANCELLED=6.

MCP Server

A Model Context Protocol server that exposes the ChopFlow broker through all three MCP primitives — tools, resources, and prompts — so an assistant like Claude Desktop, Cursor, or Cherry can enqueue tasks, run LLM jobs, inspect their lifecycle, and manage schedules with no glue code.

The chopflow-mcp crate is a thin wrapper over the broker's HTTP/JSON API. It adds no new broker surface: every tool is a 1:1 wrapper over an HTTP route. The server speaks JSON-RPC over stdio using the official Rust SDK (rmcp); tool input schemas are derived from typed param structs (schemars::JsonSchema), so clients see rich, typed arguments with no manual schema work.

Install & run#

The MCP server is the chopflow mcp subcommand. Point it at a running broker's HTTP port (the broker must already be serving HTTP on :8080):

$ chopflow mcp --broker http://127.0.0.1:8080

Building from source? The standalone crate works too:

$ cargo run -p chopflow-mcp -- --broker http://127.0.0.1:8080

Configuration#

The broker URL is set once at server launch — it is not a per-tool argument. Precedence (highest first):

Logs go to stderr only — stdout is reserved for the JSON-RPC protocol, so piping the server into a client never corrupts the stream.

Tools#

Thirteen tools. Eleven wrap a broker HTTP route 1:1; two are higher-level helpers for LLM workflows. All tool results are the broker's JSON response body returned as text.

get_stats · list_workers · list_schedules

Take no parameters. Call them with an empty arguments object.

list_tasks

get_task · cancel_task · get_schedule · delete_schedule

enqueue_task

{
  "name": "resize_image",
  "payload": {"width": 128, "height": 128},
  "tags": ["image"],
  "max_retries": 2,
  "resources": {"cpu": 1},
  "priority": 5
}

create_schedule

task_template accepts the same shape as enqueue_task: name (required), payload (required), tags, resources, max_retries, priority.

kind is a tagged union mirroring the broker's schedule type:

{
  "name": "nightly-build",
  "task_template": {
    "name": "batch_compute",
    "payload": {"n": 256},
    "tags": ["ci"],
    "resources": {"cpu": 4}
  },
  "kind": {"type": "cron", "cron": "0 9 * * *"},
  "overlap_policy": "skip"
}

update_schedule

Unspecified fields are left as-is. This is a partial update (PATCH semantics).

wait_for_task

Polls get_task roughly twice per second until the task reaches a terminal status (completed, failed, dead-lettered, or cancelled), then returns the full task JSON. Useful for getting a synchronous-style answer from an async task.

run_llm_task

Phase 2. Enqueues an llm.complete task (routed to an LLM worker via the llm tag), waits for it to finish, and returns the final task JSON — with the model's text in result.text. This is the agent-friendly entry point to ChopFlow's LLM orchestration: an agent gets a synchronous answer back from an async LLM job.

Resources#

Live, readable cluster state. An agent reads these to understand the system without firing tools blindly — resources/read fetches fresh data from the broker at read time. Returned as JSON text content.

Prompts#

Ready-made agent workflows. prompts/get returns a seeded user message describing a multi-step task, parameterized by prompt arguments. The agent then carries it out using the tools above.

Errors#

A non-2xx broker response surfaces as a tool error (isError: true) carrying the broker's {"error": "..."} body. Transport failures (broker unreachable, response read failure) return a descriptive error string instead. There is no retry inside the server — the caller decides what to do.

Wire into a client#

The server is launched as a child process by the MCP client. For Claude Desktop / Cursor, add it to claude_desktop_config.json — the unified chopflow command takes mcp as its first arg:

{
  "mcpServers": {
    "chopflow": {
      "command": "chopflow",
      "args": ["mcp", "--broker", "http://127.0.0.1:8080"]
    }
  }
}

If the client only accepts a command (no args), set the broker URL via the environment instead:

CHOPFLOW_HTTP_URL=http://broker:8080 chopflow mcp

(The standalone chopflow-mcp binary is equivalent if you built the crate directly — same flags, no mcp subcommand arg.)

Smoke test by hand#

The server speaks plain JSON-RPC over stdio, so you can drive it without an MCP client. Start the broker, then pipe an initialize + notifications/initialized + a tools/call into it:

{
  printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"t","version":"0"}}}'
  printf '%s\n' '{"jsonrpc":"2.0","method":"notifications/initialized"}'
  printf '%s\n' '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"enqueue_task","arguments":{"name":"echo","payload":{"hello":"world"}}}}'
  sleep 1
} | cargo run -q -p chopflow-mcp

Use "method":"tools/list" to enumerate all 13 tools and their input schemas, "resources/list" for the four resources, and "prompts/list" for the four prompts.

Storage — SQLite

Default storage backend. SQLite in WAL mode for durable persistence with minimal operational overhead.

SqliteStorage holds a single rusqlite connection behind a std::sync::Mutex, with the SQLite library bundled in (no system dependency). All methods are async; synchronous database work is offloaded to tokio::task::spawn_blocking so the runtime is never blocked.

Configuration#

Restart reconciliation#

On open, SqliteStorage::reconcile walks every Running task and resets it to Queued. After a broker restart the workers that claimed those tasks are gone, so leaving them Running would orphan them. Re-queuing makes them eligible for the next live worker to claim.

Storage — In-Memory

In-memory storage backend for testing and demos. Not durable — all state is lost on restart.

InMemoryStorage is a HashMap behind a tokio::Mutex. It implements the same Storage trait as SqliteStorage, so it is a drop-in for ephemeral runs. Select it with --storage memory.

$ ./target/release/chopflow broker start \
    --storage memory --http-port 8080 --open

Because both backends implement the same trait, you can also construct an InMemoryStorage directly in a Rust test to exercise dispatch, retry, and scheduling logic with no I/O — see Rust Client → In-process.

Environments

An environment is a named, region-tagged broker deployment. Declare a small fleet catalog and the dashboard can hop between brokers at runtime; the CLI resolves --env <name> to the right gRPC address. Workers are unchanged — they stay single-homed to one broker.

ChopFlow is a single-binary system: one broker process owns its storage and its workers. The environments feature does not federate brokers or share state between them. It is a lightweight operations layer — an identity tag per broker plus a read-only catalog — so a human operating several brokers (local, staging, prod) can point the same dashboard and CLI at each one without editing URLs by hand.

Broker identity#

Every broker advertises an env and region, set with flags on broker start. These appear on /api/stats and as the current entry of /api/environments, so the dashboard always shows which broker it is talking to.

$ ./target/release/chopflow broker start \
    --env prod --region us-east-1 --http-port 8080 --open

The fleet catalog#

The catalog is a single YAML file listing every known environment. It is optional: a broker with no catalog still runs and advertises itself — its --env/--region become the lone "current" entry. Add entries only when you operate more than one broker and want the dashboard to switch between them.

environments:
  - name: local
    region: default
    grpc_url: http://localhost:8000
    http_url: http://localhost:8080
  - name: prod
    region: us-east-1
    grpc_url: http://broker.prod.internal:8000
    http_url: http://broker.prod.internal:8080

The catalog is served read-only at GET /api/environments (see API — Environments). It is reloaded on broker start; there is no hot reload.

Dashboard switcher#

The TopBar cluster selector lists every catalog entry. Picking one retargets the dashboard's reactive API base to that broker's http_url and invalidates all queries, so the whole UI re-fetches from the newly selected broker. The current entry always has an empty http_url — the serving broker is same-origin to its own dashboard, so selecting "current" never retargets to a possibly-wrong host. A broker whose --env is not in any catalog still synthesizes a current entry so the selector has something to display.

CLI --env#

The CLI verbs (enqueue, status, schedule) accept --env <name>, which resolves the broker gRPC URL from the catalog and overrides --broker. Pair it with --env-config to point at a non-default catalog. Unknown names and entries missing grpc_url error out with a helpful message listing the known environments.

$ ./target/release/chopflow status --env prod --all
$ ./target/release/chopflow enqueue -f task.json --env prod

Workers stay single-homed#

Environments add no worker-side concept. A worker still connects to exactly one broker gRPC endpoint and is unaware of the catalog. To move a worker between environments, point it at a different --broker address — there is no federation, no cross-broker task migration, and no shared storage. This keeps the dispatch path exactly as described in Architecture.

Architecture

ChopFlow is a single-binary system with a pull-based dispatch model, durable persistence, and an embedded dashboard. Seven workspace crates, one shared broker state.

The seven crates#

Shared BrokerState#

The broker holds a single BrokerState — a shared handle to the Storage and the Dispatcher. Both the gRPC service (:8000, for workers and clients) and the HTTP API + dashboard (:8080) read and write the same live state. A task enqueued from the web UI is immediately visible to a gRPC worker's next FetchTasks, and vice versa — no sync step, no separate data plane.

Pull-based dispatch#

The broker never pushes tasks. A worker calls FetchTasks on its own poll loop; the broker atomically claims matching ready tasks (Queued → Running). A slow or saturated worker simply stops pulling, so backpressure is implicit — there is no flow-control signaling to implement or misconfigure.

Why one binary#

The broker, the dashboard, and the HTTP API ship in a single binary so an operator deploys one process and gets a live, observable cluster. The dashboard is embedded via rust-embed (no separate frontend host); storage defaults to a local SQLite file (no external database to provision). Workers and the CLI are separate binaries that talk to the broker over gRPC.

macOS App

ChopFlow ships a small macOS desktop app that supervises your local stack: it starts and watches the broker, optionally runs the persistent MCP gateway, and gives you tray shortcuts and a Settings screen. The dashboard itself is the same embedded UI the broker serves.

Download & install

Grab the latest .dmg from the GitHub Releases page (the landing page's Download the macOS App — Free! button points there). The app bundles its own chopflow CLI sidecar, so no extra install is required. Since it is not notarized yet, first launch uses right-click → Open on macOS.

First launch

On a fresh install the Welcome wizard walks through four steps before the dashboard loads:

1. Environment — start the local broker or connect to a remote one you already run.
2. MCP — optionally enable the persistent MCP gateway and decide whether it requires its own access token.
3. Security — optionally protect the broker: turn the switch on and create the first labelled token (shown once).
4. Learn — links to the Rust / Python / Java worker guides.

“Finish setup” shows the loading rail, then the dashboard.

Menu bar (tray)

The app lives in the macOS menu bar: Open · Local broker (Start/Stop) · MCP (Start/Stop + copy endpoint) · Remote brokers (live status, click to switch) · Quit. With the window focused, the native ChopFlow menu offers Settings… (⌘,).

Settings

Data privacy Everything is local: connections and tokens live in the app-data folder (~/Library/Application Support/io.chopflow.ops), the broker database is a local SQLite file, and nothing phones home.

Broker API Tokens

Start the broker with --api-token to require a Bearer token on every /api/* request. /healthz stays open so adopt-probes and load balancers keep working.

CLI

The flag is repeatable: each occurrence adds an equally valid credential. A bare --api-token auto-generates one (printed at startup); CHOPFLOW_API_TOKEN is honored when no flag is given; no flag at all means no auth.

# one explicit token
chopflow broker start --api-token my-secret

# several tokens (one per client) + auto-generated
chopflow broker start --api-token alpha --api-token beta

# token from the environment
CHOPFLOW_API_TOKEN=sekret chopflow broker start

Requests without a matching Authorization: Bearer <token> get 401; /api/stats and /api/environments report auth_required: true so clients know.

From the desktop app

Wizard (first run): the Security step asks whether to protect the broker; turning it on creates the first labelled token (identifier + value, shown once).

Settings → Security: a switch turns authorization on/off (non-destructive — it never creates or deletes tokens by itself), and below it you add tokens (label + value, with Generate) and revoke them individually. Protection is enforced the moment a token exists and the switch is on.

From config / files

Vending a fleet catalog is done via config/environments.yml; tokens are not stored there. Paste a generated token into the Token (optional) field when adding a remote in the app, and remote gates are per-broker.

MCP Access Tokens

The chopflow mcp --http gateway is an AI-facing endpoint. By default anyone who can reach it can use it. Give it its own access token with --access-token — independent from the broker's API tokens.

CLI

# gateway serving the broker, requiring a token to reach /mcp
chopflow mcp --http 127.0.0.1:8810 --broker http://127.0.0.1:8080 \
  --access-token mcp-secret \
  --api-token <broker-token-if-protected>

Clients present the token via Authorization: Bearer <token> or ?access_token=/?code= URL params (both are in the MCP streamable-HTTP spec). Missing/invalid values return 401.

From the desktop app

Wizard (first run): the MCP step can enable the gateway and toggle Requires token to connect — turning it on generates the gateway token (shown once).

Settings → MCP: toggle the gateway, then use the Access token section: the on/off switch Requires token to connect (non-destructive, keeps the stored value), plus Set / Generate / Regenerate / Remove and Show/Hide.

Note --api-token on the MCP command authenticates the gateway to the broker; --access-token gates who can reach the gateway itself. They are orthogonal.

FAQ

Frequently asked questions about ChopFlow — the system, the model, and where it fits.

What language is ChopFlow written in, and what can talk to it?#

The core broker and workers are written in Rust (2021 edition). Rust is the first-class client language; Python and Java clients are also available. Workers and the CLI speak gRPC to the broker; browsers and external tooling speak HTTP/JSON against the same live BrokerState.

How does ChopFlow survive a broker restart?#

Tasks persist in SQLite (or the in-memory backend for tests). On boot the broker reconciles in-flight work: any task left Running from a previous run is reset to Queued, since its worker is gone. No silently dropped work — see Storage — SQLite → Restart reconciliation.

Do I need to deploy a separate frontend?#

No. The dashboard is embedded in the broker binary via rust-embed and served on the HTTP port — one process, no Node build required to run, no separate deploy. The same UI also ships as a native macOS app through Tauri.

What happens when a task fails or times out?#

Workers acknowledge each task with a success or structured failure. Failed tasks follow a configurable RetryPolicy (fixed or exponential backoff). Tasks that exhaust their retries are DeadLettered so they stay visible instead of vanishing — see Retry & Dead-Letter.

Can I route work to specific workers?#

Yes. Tasks carry tags and resource requirements; workers register with tags (e.g. gpu,ml) and capacity (CPU, GPU, memory). The dispatcher only hands a task to a worker whose tags match and whose resources are available — see Resources & Tags.

How is this different from Temporal?#

Different category. Temporal is a durable workflow engine — it orchestrates long-running, multi-step processes via event-sourced replay. ChopFlow is a task queue in the Celery / Ray / Dask lineage: discrete units of work, a pull-based dispatch model, leases, retries, and dead-lettering. If you're scheduling background jobs across a worker fleet and observing them live, that's ChopFlow's shape; if you need saga-style workflows with checkpointed replay, Temporal is the right tool.

Is ChopFlow production-ready or research-only?#

Both. It's built for production applications and for distributed-systems research. The experiment directory (reproducible configs and analysis scripts) is planned but not yet landed; the demos crate is the current way to exercise the system end-to-end.

Are missed cron runs backfilled?#

No. On startup, enabled cron schedules have next_fire recomputed from now, so the ticker only fires future matches. A disabled one-shot with a past ETA stays disabled until re-enabled. See Scheduling & Recurrence.

Why pull-based dispatch?#

The broker never pushes tasks. A worker calls FetchTasks on its own poll loop, so a slow or saturated worker simply stops pulling — backpressure is implicit, with no flow-control signaling to implement or misconfigure.