Skip to content

Concurrency

Limit how many instances of a task can run simultaneously.

Task-Level Concurrency

Control the maximum number of concurrent executions for a specific task:

const processVideo = conductor.createTask(
  {
    name: "process-video",
    concurrency: 3, // Soft max of 3 videos at once
    groupConcurrency: 1, // Soft max of 1 per invocation group
  },
  { invocable: true },
  async (event, ctx) => {
    // Heavy video processing
    await encodeVideo(event.payload.videoId);
  }
);

When the limit is reached, additional executions remain queued until capacity becomes available.

How It Works

Postgres Conductor evaluates active executions when claiming work. Task and group limits are coordinated with FOR UPDATE SKIP LOCKED; limits are intentionally soft across concurrent workers. Grouped candidates whose group is full do not consume task-level capacity, so another available group can be claimed in the same batch.

This happens entirely in Postgres - no external coordination needed.

Concurrency vs Worker Concurrency

Task-level concurrency (this page):

  • Limits concurrent executions per task type
  • Set on individual tasks with concurrency option
  • Applies across all workers

Worker concurrency (see Worker Configuration):

  • Controls how many tasks a single worker processes at once
  • Set on worker/queue with config: { concurrency }
  • Independent per worker instance

Scale a queue by running more replicas of the same codebase. Every orchestrator polling a queue must run the same task set; separate services need separate queues (see One Queue, One Codebase).

Child invocations use the group option supplied to ctx.invokeAndWait. Dynamic cron schedules accept group alongside cron, and each subsequent cron execution preserves it.

What's Next?

Group concurrency

group may be supplied when invoking a task. groupConcurrency limits active executions within each (queue, task, group) scope; ungrouped invocations bypass that limit. Task and group limits compose and are intentionally soft across concurrent workers.

A claim skips the pending executions of full groups one by one, so a very large backlog in a single group slows every claim of that task until the backlog drains. Spread bulk work across groups, or move it to a separate task or queue.