Parallel Pipelines: Exploiting Independence in Course Generation

Course authoring has a dependency graph. You cannot write learning outcomes without a curriculum to attach them to, and you cannot write a lesson without a plan that specifies what it should teach. But once planning is done, the actual content work — lessons, exercises, quizzes, capstone — is embarrassingly parallel. The Stay Learning pipeline is built around this observation.

The Dependency Graph

The pipeline has five sequential stages, each producing an artefact that the next stage consumes:

graph LR
  A[audience] --> B[curriculum]
  B --> C[outcomes]
  C --> D[assessment]
  D --> E[lesson plans]
  E --> F{{wave}}
  F --> G[lesson content]
  F --> H[exercises]
  F --> I[quizzes]
  F --> J[capstone]

These sequential stages are called exclusive — one agent runs at a time, and it updates course.yaml in the same turn to record the new status. Each stage gates on the previous one: if the upstream artefact has status missing, the stage refuses to run. If the upstream is stale, the stage warns but can proceed (useful during iterative edits).

Status propagation enforces correctness: writing any artefact marks all downstream artefacts stale. If you rewrite the curriculum, outcomes through lesson plans all become stale, and the pipeline will re-run them before allowing wave work.

The Wave Pattern

Once every lesson plan in the course reaches status current, four types of work become available simultaneously:

• Lesson content — one Markdown file per lesson
• Exercises — one exercise file per lesson
• Quizzes — one quiz file per lesson
• Capstone project — one per course

These are siblings derived from the plan. They don’t depend on each other. A lesson’s exercises don’t need to read the lesson’s content; both are generated from the same plan.

For a 4-module course with 12 lessons, the wave contains 37 independent tasks: 12 content + 12 exercises + 12 quizzes + 1 capstone. All 37 can run concurrently.

A wave in flight

Here is the orchestrator dispatching a real wave for a Linux and Kubernetes networking course — a 2-module, 5-lesson course, which comes to 16 delegations in a single batch:

Every line after “Dispatching the wave” is a separate agent working at the same time as the others. One project-designer writes project.yaml. Five lesson-writer tasks write m01-l01 through m02-l02 content. Five exercise-generator tasks and five quiz-generator tasks do the same for their own artefacts. Nothing in that list is waiting on anything else in it.

The detail worth noticing is what each task is doing on the right-hand side: several are independently reading course-content.instructions.md or loading the course-state skill. They are not coordinating that between themselves — each task loads the same contract on its own, which is exactly why they can be trusted to write to disk without talking to each other.

The “Hook warning” annotations are the validator firing after individual writes, scoped to the task that produced them.

Why It’s Safe

Parallelism is only safe if concurrent tasks cannot corrupt each other’s state. The wave guarantees this through file isolation and a shared-state protocol:

1. Wave tasks write to different files. Each task owns exactly one output file. Lesson content for m02-l03 writes to m02-working-with-images/lessons/l03-layer-caching.md. Exercises for the same lesson write to a separate exercises file. There is no file that two wave tasks both write to.

2. Nothing shared is written during the wave. Three files are shared across the course: course.yaml, glossary.yaml, and .state/run-log.md. During a wave, no individual task touches these. Each task reports its result back, and the orchestrator applies the updates to shared state after collecting all results.

3. Each task reads immutable input. The lesson plan is the input to every wave task. Plans are locked (current status) before the wave begins. No task modifies a plan during the wave — the exclusive stages already handled that.

4. Validation runs after each individual write. The validate.py script (triggered by a PostToolUse hook) checks schema compliance after every file write. A broken output is caught immediately, scoped to the single task that produced it.

Exclusive vs Wave: The Core Distinction

The two modes differ in how they handle shared state:

This is the only reason the wave is safe. If wave tasks updated course.yaml directly, you’d have concurrent writes to a shared file with no locking. By deferring shared-state updates to the orchestrator (which runs alone, after the wave completes), the architecture avoids the problem entirely.

Practical Impact

Throughput. A 12-lesson course runs 37 wave tasks instead of sequencing them. The wall-clock time for the wave is bounded by the slowest single task, not the sum of all tasks.

Quality gates remain sequential. You cannot skip planning to get parallelism. The exclusive chain ensures that by the time the wave runs, every lesson has a reviewed plan with explicit outcomes, Bloom levels, and assessment alignment. The parallelism is a reward for completing the planning discipline, not a shortcut around it.

Failure isolation. If exercise generation for lesson 7 fails validation, only that task re-runs. The other 36 tasks’ outputs are unaffected. Compare this to a monolithic generation pass where a failure midway through forces a full restart.

Staleness detection. If someone edits a lesson plan after the wave, downstream artefacts derived from it are marked stale. The pipeline knows exactly which tasks to re-run — only those whose plan changed — rather than regenerating the entire course.

The Shape of the Pipeline

Put together, a full course run looks like this:

1. Five exclusive stages run in sequence (~5 agent turns)
2. The orchestrator checks all plans are current
3. The wave fans out to N×3 + 1 tasks (N lessons × 3 content types + 1 capstone)
4. The orchestrator collects results and updates shared state
5. Final validation runs in --strict mode

The sequential phase is where the intellectual design happens — deciding what to teach, in what order, to what depth. The wave phase is where volume happens — producing the actual files that learners will read. The architecture matches the work: serialize the decisions, parallelize the production.