Stay Learning

· Last updated on

Two Runtimes, One Pipeline: Copilot Agents vs. Mastra

Comparing this repo's VS Code Copilot agent pipeline against its Mastra port — what moved from prose to code, what that bought, and what it cost.

Stay Learning exists twice. This repo builds the course-authoring pipeline entirely from VS Code Copilot customization files: agents, prompts, and skills under .github/, driven by a person in Copilot Chat. A second implementation, stay-learning-mastra, rebuilds the identical pipeline on Mastra — TypeScript workflows and a command line, with no chat panel involved.

Same stages. Same output contract — courses/<slug>/ as YAML and Markdown that git can diff and the same Astro viewer can render, unmodified, from either version. What differs is everything about how a course gets from “topic” to “files on disk”: where the sequencing lives, how approval survives between steps, how staleness is known, and what happens when a lesson fails validation.

This post is about that difference, not about which pipeline writes a better lesson. Both use the same instructional-design contract — backward design, Bloom-levelled outcomes, cognitive load budgets — described in What Makes a Good Course. This is about the runtime underneath it.

The same seven stages, two engines

The stage list — audience, curriculum, outcomes, assessment, module plans, then the parallel wave of lesson prose, exercises, quizzes, and capstone — is identical between the two. What changed is what carries the sequencing, what an approval is made of, and what happens after validation finds a defect.

flowchart TD
    O1[Orchestrator agent] --> S1[Sequential stages]
    S1 --> G1{Approve in chat?}
    G1 -->|yes| W1[Wave: subagent delegations]
    W1 --> V1[validate.py, PostToolUse hook]

This repo: the orchestrator agent reads course.yaml and delegates. Approval rides on conversational continuity — asked and answered in the same session.

flowchart TD
    O2[createCourse workflow] --> G2{{Gate 1: gates.json}}
    G2 --> S2[planModules workflow]
    S2 --> G3{{Gate 2}}
    G3 --> W2[runWave, concurrency-capped]
    W2 --> V2[TypeScript validators]
    V2 --> R2[repair workflow]

stay-learning-mastra: sequencing is a workflow graph, not an agent’s instructions. Each gate is a decision recorded to disk, because the three commands that make up a run are three separate processes with no shared memory between them.

Orchestration: an agent that reads state, or a workflow that encodes it

This repo’s course-orchestrator.agent.md is the only user-invocable agent. It reads course.yaml, decides which stage is missing or stale, and delegates to the responsible subagent. The sequencing lives in that agent’s instructions — a model has to read them correctly, every time, to sequence the pipeline correctly.

Mastra’s README states the change plainly:

The orchestrator is gone. It was an agent that read course.yaml and delegated; here the sequencing is a workflow, so it is code rather than instructions a model has to follow.

createCourse, planModules, runWave, and repairCourse are typed Mastra workflows. Each step’s inputs and outputs are declared, and the order of steps is a property of the workflow graph, not a rule a model needs to remember to obey. Nothing here claims the underlying agents got smarter — the lesson writer and the curriculum designer are still prompted models, doing the same reasoning. What moved is which layer is responsible for not skipping a step.

Gates: conversational continuity, or a signed ledger

In this repo, a course review happens inside the chat: the orchestrator asks whether the curriculum looks right, the person answers, and the answer stays live in that session’s context for as long as the conversation continues. There’s no separate record of the decision — the conversation is the record.

Mastra can’t do that, because a course run there is three separate CLI invocations across three separate processes: course new, then (after a person looks at the output) course plan, then course write. An approval given to the first process is gone by the time the third one starts, unless it’s written down. So it is:

{
  "version": 1,
  "gates": {
    "design": {
      "approved": true,
      "at": "2026-08-10T17:53:02.924Z",
      "covered": {
        "audience.yaml": "3d9aaa963e02973c",
        "curriculum.yaml": "1f7d2a7a33440e75",
        "outcomes.yaml": "ebf84a3a08b877e4",
        "assessment.yaml": "9be466acf83006eb"
      }
    }
  }
}

The interesting part isn’t that the decision is stored — it’s that the hash of every file it covered is stored alongside it, in gates.ts. An approval stops counting the moment any covered file’s hash no longer matches — approving a curriculum and then hand-editing it doesn’t leave a stale approval standing in for a document that no longer exists. plan refuses to run without the design gate holding; write refuses without both.

This repo has no equivalent failure mode to guard against, because there’s no process boundary for an approval to fall across. That’s a property of chat-driven, single-session orchestration, not a gap somebody forgot to close.

Staleness: a field you update, or a question you ask

This repo stores status directly: course.yaml carries current | stale | missing per artefact, and every agent that writes a file is responsible for marking everything downstream stale. It works as long as every agent keeps the flag honest.

Mastra doesn’t store the flag at all. status in the CLI output is computed by rehashing each artefact’s recorded inputs and comparing:

npm run course -- status how-http-requests-travel
#   stale  modules/m01.../l02....md  <- curriculum.yaml
#   up to date

A stored flag can disagree with the disk in either direction, and a lost flip is invisible until something downstream breaks for no apparent reason. A derived one can’t drift, because it isn’t a memory — it’s recomputed every time it’s asked. It also only invalidates on the direct inputs that actually changed content, not on anything downstream of a stage that ran again. Correcting a typo in curriculum.yaml that leaves outcomes byte-identical doesn’t cascade to every lesson; this repo’s stored-flag model would mark the whole tree stale on any write to an upstream file, whether or not the content that mattered actually moved.

The wave: a delegated batch, or a capped workflow step

Both pipelines exploit the same insight — once every lesson plan exists, lesson content, exercises, quizzes, and the capstone don’t depend on each other, so they can run at once. Read more on why that’s safe.

This repo’s orchestrator dispatches the wave as one batch of subagent delegations in a single chat turn — for a 5-lesson course, 16 delegations in flight together, bounded by nothing but what the chat session can hold. Mastra’s runWave workflow caps concurrency explicitly via STAY_WAVE_CONCURRENCY (default 4 lessons in flight), and the shared files — course.yaml, glossary.yaml, .state/ — are written once, afterward, by the step that gathers results, rather than by any of the wave agents themselves.

One planning difference is worth calling out because it changes what “safe” means for the terminology budget. This repo plans one lesson at a time; a term’s cost against the module’s budget is checked against what’s already been spent. Mastra’s modulePlanner plans an entire module in one call, so a term can’t be introduced twice by two lessons that couldn’t see each other — the budget is allocated, not audited after the fact.

Model choice: whatever backs your subscription, or a role you pin

No agent file in this repo names a model. That’s deliberate — Copilot Chat calls whatever backend your subscription points at, and pinning a vendor string in an agent file would break portability for anyone on a different plan. The tradeoff is explicit in this repo’s own architecture writeup: recommending Opus for planning-heavy stages lives in documentation a person can override, not in a file that would force it.

Mastra can’t be provider-agnostic in the same way, because it calls the Anthropic API directly and needs an explicit model string and your own key. It narrows the choice instead of avoiding it — one file, models.ts, maps a role (analysis, planning, writing, judging) to a model, so curriculum design runs on a stronger model than quiz-distractor generation by default, and moving providers is one edit in one file:

const DEFAULTS: Record<ModelRole, string> = {
  analysis: 'anthropic/claude-sonnet-4-6',
  planning: 'anthropic/claude-opus-4-7',
  writing: 'anthropic/claude-sonnet-4-6',
  judging: 'anthropic/claude-sonnet-4-6',
}

STAY_MODEL_<ROLE> overrides one role from the environment; STAY_MODEL moves everything at once. It’s a different answer to the same problem — this repo optimizes for “runs on whatever you already have open,” Mastra optimizes for “the cost of the expensive stage is a deliberate choice, and swapping providers is a one-line diff.”

Reference documents: a skill the model chooses to load, or a file the prompt already names

Both pipelines run on the same ten instructional-design documents — backward design, Bloom’s taxonomy, cognitive load, assessment and example and capstone design, retrieval and spacing, plus the machine-readable bloom-verbs.yaml and house-style.yaml. What differs is who decides which of them sit in front of a given stage, and when that decision gets made. In this repo they are packaged as skills under .github/skills/ — instructional-design/ and course-state/, each a SKILL.md with references/ and assets/ behind it — and agent files ask for them in prose. outcomes-designer.agent.md says to load “the course-state skill for the schema and write procedure, and the instructional-design skill for outcome rules and Bloom levelling,” and that instruction is the whole mechanism. The model reads it, decides at inference time whether it needs the taxonomy tables, and pulls them in if it does. Progressive disclosure: the summary is always present, the depth is one read away.

The Mastra port kept all ten documents verbatim — 1,109 lines, added whole in the first commit, sitting at src/mastra/prompts/reference/ — and moved the decision to construction time. referenceText() in style.ts reads a document off the filesystem, and reference() in index.ts wraps each one in a <reference name="…">…</reference> block. A stage declares its own set on a single line inside its instructions template:

${reference('assessment-design.md', 'blooms-taxonomy.md')}

The most load-bearing reason for that is not about the agents at all: these files are shared with the validators. bloom-verbs.yaml and house-style.yaml are parsed by TypeScript in the same style.ts to build the verb tables and prose limits the output is then checked against, and the file’s own comment says that is deliberate — “they are data rather than prose so that the agent instructions and the validator cannot drift apart: the same file supplies the verb tables an outcomes designer is told to use and the tables its output is checked against.” A skill is something an agent loads. This is something the agent and the checker both load, so it has to be a plain file at a path both of them know.

Two smaller reasons follow from the port’s shape. These agents have no tools — the COMMON preamble states they are single-shot structured-output calls that return data and nothing else — so progressive disclosure would need a read tool and an extra turn to use it, and it would introduce a failure mode a fixed list doesn’t have: the agent decides it doesn’t need blooms-taxonomy.md and levels an item wrong. And the scorers depend on determinism: they exist to answer whether “a change to a prompt or a reference document made the output better — a question about the method, not about one file,” which only holds if a stage’s prompt is a pure function of the stage plus its inputs. Model-chosen loading makes two runs of the same stage non-comparable, and sits badly beside the hash-based staleness and resume machinery above.

None of which is free. The skill version keeps the default context smaller — SKILL.md is a summary with the long references behind it, and a stage that never needs the capstone rubric never pays for it — and a skill is reusable across agents and editable without a rebuild. Adding an eleventh reference document here means dropping a Markdown file into references/ and adding a row to the table in SKILL.md. In Mastra it means first extending the Reference union in index.ts, because the argument to reference() is typed. That is the same trade the rest of the port makes: the compiler will now tell you when a stage asks for a document that doesn’t exist, and in exchange, adding a document is a code change.

Input moderation: a person in the loop, or a guardrail before the first write

The most recent thing Mastra grew is at the other end of the pipeline from everything above — an input guardrail, 54 lines in moderation.ts plus 26 lines of workflow wiring. It is not a bespoke blocklist. It is Mastra’s own ModerationProcessor from @mastra/core/processors, configured with strategy: 'block' and running on modelFor('analysis') — the same role-based model config described above, so a provider swap moves the moderator along with everything else. The source is explicit about why that off-the-shelf piece: it “is built to sit in an agent’s input pipeline, scoring a message against content categories and aborting the call if one crosses the threshold.”

What’s interesting is that it isn’t attached to an agent’s input pipeline. It’s called directly, against a message assembled from the brief, from a moderate step wedged between intake and scaffold in the createCourse chain:

  .then(intake)
  .then(moderate)
  .then(scaffold)

scaffold is the first step that touches disk, and the placement is chosen for exactly that: “so a flagged brief never becomes a directory, and never reaches an agent’s prompt either,” rather than firing inside whichever agent first happens to see the brief’s text. Every field — title, topic, audience, time available, industry, and the constraints joined together — is checked as one message rather than field by field, also deliberately, “so the moderator can weigh them in context: a phrase that reads as a slur in isolation can be a legitimate topic (‘the history of ~’) once the rest of the brief is visible alongside it.” A flag raises a TripWire, caught and rethrown as Course brief rejected by input moderation: …. Empty and whitespace-only fields are dropped, and a brief with nothing in it at all returns without a model call — that short-circuit is the one behavior the test pins, because it is what keeps intake suspending on a partial brief cheap rather than one call away from a model.

This repo has no equivalent step, and doesn’t obviously need one in the same place. The brief is typed by a person into a chat session that is already running inside Copilot’s own platform-level content filtering, and there is a person watching every turn who can see a bad brief and stop it before anything is written. The unattended CLI has no such person: a scheduled or CI run takes its brief from a file, an issue body, or an HTTP payload, with nobody reading it on the way in. That is the case the guardrail exists for, and it is the same thesis as everything else here — Mastra optimizes for the pipeline surviving without a person watching it, and an input guardrail is part of what “no person watching” costs.

It is also the only place either pipeline looks at the input. Everything else both versions do about quality happens on the way out, after a model has already written something.

Validation and judgment: one deterministic script, or a validator plus a scorer

Both pipelines refuse to let a model check its own work — see Mechanical Verification for why. This repo’s validate.py runs automatically after every file write via a PostToolUse hook and returns PASS/WARN/FAIL. Mastra’s validator suite is a TypeScript rewrite of the same checks, invoked explicitly with course validate, and tested against a fixture course specifically to confirm it reaches the same verdicts as the Python original.

What Mastra adds has no equivalent in this repo at all: a scorer layer, sitting above the validators, for the judgments that counting can’t settle.

A validator answers a question with a fact; a scorer answers one with an opinion. The dividing line is whether counting settles it.

distractor-quality checks whether each wrong answer in a multiple-choice item actually embodies the misconception its why_wrong field claims. outcome-coverage asks the model to quote the passage that teaches each outcome — asking for a quote instead of a verdict forces it to find the passage first, rather than rubber-stamping an outcome the lesson never actually reaches.

Scorers are registered with Mastra but deliberately not wired into the wave — they don’t gate anything, and they’re not run per-course. They exist to answer a different question: did a change to a prompt or a reference file make the method better, evaluated across a set of courses, not whether any single course passes.

Repair: report and stop, or hand it back and retry

This repo’s validator reports defects; nothing after that is automatic. A person, or the next agent turn in the chat, decides what to do about them.

Mastra’s course repair closes that loop for leaf artefacts — prose, exercises, quizzes, the capstone — by handing each defect back to the agent that wrote the file, with the defect list attached, and checking whether the count went down. It stops on any of three conditions: nothing left to fix, a round that didn’t reduce the count, or a two-round cap (STAY_REPAIR_ROUNDS). It deliberately never touches audience, curriculum, outcomes, assessment, or a lesson plan — those were approved at a gate, and silently rewriting one would turn that approval into a signature on a document that has since changed underneath it.

What only one version can do

Only in the Mastra port:

  • Run unattended — CI, a cron job, a server — because the pipeline is a CLI, not a chat session with a person driving it.
  • Resume a suspended run from a different process, potentially days later, because gate state and workflow progress live in a LibSQL-backed run store, not in a conversation’s context.
  • Cap concurrency and repair rounds with an environment variable, with no prompt to edit.
  • Automatically repair mechanically-detected defects in a bounded loop.
  • Refuse a course brief at the door with an input guardrail, before the first step that touches disk, so flagged text never becomes a directory or reaches an agent’s prompt.
  • Score output with a model acting as judge, decoupled from both generation and from gating, to evaluate whether a prompt change actually helped.
  • Enforce artefact shape with Zod schemas shared between the agents that generate structured output and the validators that check it — one schema, not a description in prose plus a separate structural check.
  • Inspect a workflow as a graph in Mastra Studio, step by step, including a suspended gate’s payload.

Only in this repo’s Copilot version:

  • Start working the moment the repo is open in VS Code with Copilot Chat — no npm install, no database file, no API key or billing of your own to configure.
  • Stay genuinely provider-agnostic: the same agent files run against whatever model your Copilot subscription points at, with nothing to change if that changes.
  • Let a person redirect an agent mid-turn, in full conversational context, rather than approving a JSON ledger from a CLI flag with no back-and-forth.
  • Carry a single pipeline run across one continuous session without deliberately splitting it into separately resumable commands.
  • Change orchestration behavior by editing Markdown, with no build step, no types to satisfy, and no second runtime to install.
  • Let the model decide at inference time which reference documents a stage actually needs, and add a new one by dropping a Markdown file into a skill — no union type to widen, no rebuild.

Prose you can edit, or code you can test

Both pipelines still prompt models with natural-language instructions — porting to Mastra didn’t replace the lesson writer’s judgment with a schema. What moved is the scaffolding around that judgment. This repo’s scaffolding is Markdown: readable, editable by anyone who can write English, and enforced only by the after-the-fact validator hook. Mastra’s scaffolding is TypeScript: sequencing, gates, and schemas are compiled and type-checked, and the project ships a test suite (npm test, npm run typecheck) that asserts its validator agrees with the original’s on a fixture course — a claim this repo’s Markdown agents have no mechanism to make about themselves.

Neither is strictly the better architecture. They’re optimized for different failure modes. This repo optimizes for a person steering every decision, in one session, with nothing to install. Mastra optimizes for the pipeline surviving without a person watching it, and for being able to prove, mechanically, that a change to a prompt didn’t quietly break something downstream.

If you’re prototyping what a course-authoring pipeline should even contain, or you want to sit in the loop for every judgment call, Copilot Chat driving Markdown agents is the shorter path to a working system. If you need the pipeline to run without you — overnight, in CI, resumable after a crash — or you need to measure whether a change to a reference file actually improved the output, that’s what the Mastra port is for. Stay Learning built both, because the two questions are different questions.