Modernization Factory — Agentic COBOL→AWS Pipeline

8-stage Claude pipeline: JCL + COBOL → Lambda + Step Functions + Aurora DDL + parity tests

Context

The Modernization Factory is an agentic pipeline — itself built as an MCP server (mcp-cobol-factory) — that turns mainframe JCL + COBOL + copybooks into AWS-native artifacts: TypeScript Lambda handlers, Aurora PostgreSQL DDL, a Step Functions ASL state machine, and a parity test suite that proves the cloud output matches the mainframe. The architectural spine is an 8-stage async pipeline where each stage runs against a deliberately chosen Claude model tier — Haiku for the cheap triage gate, Sonnet for structured extraction and code generation, Opus (with extended thinking) only for the one architectural-decision stage where deliberation changes the outcome. Each stage takes typed Pydantic in and returns typed Pydantic out, writes its artifact to disk immediately, and the pipeline pauses for human schema review by default before it generates any code. Sole architect and builder; in internal production use, with output shipping after minor human review. The agents query legacy DB2 schemas through the same MCP servers I built for the modernization program.

Constraints

• Regulated mainframe source — comment lines routinely carry historical data samples (real SSNs, account numbers) and developer notes; nothing untrusted can reach the model un-sanitized, and no raw source can land in a log.
• A wrong architectural routing decision (Stage 4) cascades into wrong schema, wrong code, and failing parity tests — the cost of that stage being wrong is the whole run, so it warranted the most capable model plus extended thinking.
• Output has to be trustworthy enough to ship after only minor human edits — which means the generated code has to be safe-by-default (dry-run, typed I/O) and provably equivalent to the mainframe (the parity test suite is a first-class output, not an afterthought).
• The same pipeline has to run on one program or a hundred — so the design optimizes the per-run operational surface (one checkpoint protocol, one audit format, one cost ledger) rather than per-program tuning.
• Cost has to stay defensible at scale — a naive all-Opus pipeline would be 3–5× more expensive per run with no quality gain on the structured stages.

Architecture

Data Model

Every inter-stage contract is a typed Pydantic v2 model — never a raw dict — which buys type safety, clean JSON serialization via model_dump(), and resume support via model_validate(). The pipeline accumulates into one aggregate output model (job name, the per-stage results, total cost, cache savings, stages completed, errors) that becomes summary.json. Each stage's output is its own model: triage classification, JCL parse (job + steps), per-program COBOL parse (data-access separated from business logic), the layer-decision routing table, the Aurora DDL (tables tagged with which Lambdas consume them), the generated Lambda handlers (TypeScript + zod schemas + IAM actions), the Step Functions ASL (workflow type + human-approval gates), and the parity tests (category + assertion + fixture + tolerance). Enums are all string-enums so the artifact files stay human-readable. Schema evolution is append-only — new fields ship with defaults so old checkpoint artifacts still deserialize.

Implementation patterns

# harness.py — call_claude()
"system": [
    {
        "type": "text",
        "text": system,                          # static prompt template
        "cache_control": {"type": "ephemeral"},  # cache this block
    }
]
# First call populates the cache; every later call in the
# same stage reads the system prompt at 10% of input rate.

# s04_layer_decider.py
await call_claude(client, "opus", _SYSTEM, user,
                  max_tokens=12000, extended_thinking=True)

# harness.py — budget is derived, not hardcoded
if extended_thinking:
    budget = min(8000, max_tokens - 1024)   # reserve 1K for output
    kwargs["thinking"] = {"type": "enabled",
                          "budget_tokens": max(1024, budget)}

def extract_json(text: str) -> Any:
    cleaned = re.sub(r"```(?:json)?\s*", "", text)  # strip fences
    cleaned = re.sub(r"```", "", cleaned)
    start = min(
        cleaned.find("{") if "{" in cleaned else len(cleaned),
        cleaned.find("[") if "[" in cleaned else len(cleaned),
    )
    obj, _ = json.JSONDecoder().raw_decode(cleaned, start)
    return obj

_LAMBDA_SEMAPHORE = asyncio.Semaphore(3)

async def _generate_one(...):
    async with _LAMBDA_SEMAPHORE:        # at most 3 in flight
        text, _, _ = await call_claude(...)
# gather(..., return_exceptions=True) so one failure
# doesn't cancel the other in-flight calls.

messages = [{"role": "user",
             "content": f"<untrusted>\n{user}\n</untrusted>"}]
# Combined with a system-prompt directive: treat enclosed
# content as data to analyze, never as instructions to follow.

Cost model

Per-stage model tiering is what makes the pipeline economical to run at scale. Published Claude rates the harness prices against: Haiku $1 / $5 per million input / output tokens; Sonnet $3 / $15; Opus $5 / $25. Stage 1 (triage) runs Haiku at a 64-token output budget; Stages 2/3/5/6/7 run Sonnet for structured extraction and codegen; only Stages 4 (layer decision) and 8 (parity tests) run Opus. The CostTracker records both spend and cache savings per stage — cost = (cache_read_tokens / 1M) × in_rate × 0.10, savings = the other 0.90. A representative 4-step job completes in ~105 seconds for roughly $0.05 total, versus an estimated 3–5× that if every stage ran Opus. The ~70–80% reduction is the tiering decision paying off, with no quality loss on the structured stages because Sonnet is matched to that work, not outclassed at it.

Sample output

{"ts":"…","event":"triage","classification":"JCL_JOB","confidence":0.97}
{"ts":"…","event":"jcl_parse","job_name":"PREMNTLY","step_count":4}
{"ts":"…","event":"layer_decisions","decisions":"STEP01→LAMBDA, STEP03→SQL"}
{"ts":"…","event":"pipeline_complete","elapsed_seconds":105.3,"total_cost_usd":0.0482}

{"stage":"s04_layer_decider","model":"claude-opus",
 "input_tokens":2100,"output_tokens":520,
 "cache_read_tokens":0,"cost_usd":0.0235,"cache_savings_usd":0.0}

output/PREMNTLY_20260527T220800/
├── input_clean.json        ← sanitised inputs for resume
├── checkpoint.json         ← resume state
├── summary.json            ← run metadata + cost totals
├── audit.jsonl             ← security + cost trail
├── layer_decisions.json
├── aurora_ddl.json / .sql  ← ready-to-run PostgreSQL DDL
├── lambdas/PREMCALC.ts     ← one file per Lambda handler
├── state_machine.asl.json  ← importable to Step Functions
└── parity_tests.json

Key Sequence

1. 01Stage 1 — Input Guard + Triage [Haiku]: strip comments, redact PII, flag injection attempts, validate structure, classify the artifact. Cheap model, tiny output, security gate.
2. 02Stage 2 — JCL Parser [Sonnet]: extract job + step names, EXEC PGM values, DD statements, SYSIN cards, COND= codes.
3. 03Stage 3 — COBOL Parser [Sonnet, parallel]: one Claude call per program via asyncio.gather; separates data access from business logic, identifies file I/O and external dependencies.
4. 04Stage 4 — Layer Decision Architect [Opus + extended thinking]: routes each JCL step to Lambda / Aurora SQL / DynamoDB / not-needed, weighing volume and human-approval requirements with up to ~8,000 private reasoning tokens.
5. 05Stage 5 — Aurora Schema Designer [Sonnet]: translates copybook PIC clauses to PostgreSQL column types, optionally verifies against the live DB2 catalog. DEFAULT CHECKPOINT — pipeline pauses for human schema review.
6. 06Stage 6 — Lambda Generator [Sonnet, parallel, cap 3]: one TypeScript handler per Lambda-routed step — zod schemas, RDS Data client, decimal.js money math, dry-run by default.
7. 07Stage 7 — Step Functions Composer [Sonnet]: wires the Lambda ARNs into valid Amazon States Language, preserves JCL control flow, marks human-approval gates, picks EXPRESS vs STANDARD.
8. 08Stage 8 — Parity Test Writer [Opus]: designs tests in four categories (parity, dry-run, HITL, error-handling), each referencing the exact COBOL lines being replaced.

What I owned

• The architectural call: model tiering per stage, not one model for everything — Haiku gates, Sonnet does the structured bulk, Opus reasons only where it changes the outcome. Roughly 70–80% lower cost than running Opus everywhere, with equal or better quality on the structured stages.
• Stage 4 (Layer Decision) is the one stage on Opus with extended thinking — it routes each mainframe step to Lambda / Aurora SQL / DynamoDB / skip, and getting it wrong cascades through every downstream stage, so it gets a private reasoning budget the other seven stages don't.
• Human-in-the-loop by default: the pipeline halts after Stage 5 (schema design) for human review before any Lambda code is generated — a bad schema would cascade into wrong code, wrong ASL, and failing tests, so the checkpoint sits before the irreversible work.
• Each stage writes its artifact immediately and the whole run is checkpoint/resume-able from disk — no progress lost on failure, and the human-review pause is just a checkpoint the caller resumes from.
• Parallelism where it pays: COBOL parsing fans out one Claude call per program; Lambda generation fans out per step behind a concurrency cap of 3 so the parallel stage can't trip API rate limits.
• Five-layer input defence before any source reaches the model: strip COBOL/JCL comments (where real SSNs and adversarial instructions hide), redact PII, validate structural integrity, wrap everything in untrusted tags, and detect injection patterns — with an audit log that records metadata only, never raw source.
• Generated Lambdas are safe-by-default: dry-run on unless explicitly enabled, decimal.js for all monetary arithmetic, zod schemas on every input/output, and an IAM action list emitted alongside each handler.

Trade-offs

• Chose per-stage model tiering over one-model-for-everything: Haiku gates, Sonnet does the structured bulk, Opus reasons only on Stage 4 and writes tests on Stage 8. The cost is more configuration and a model-selection rationale to maintain per stage; the benefit is ~70–80% lower cost with no quality loss on structured work, and it's the single decision that makes the pipeline economical to run at a-hundred-programs scale.
• Enabled extended thinking on exactly one stage (the Layer Decision) rather than across the board: thinking budget is wasted on extraction and codegen, but on the one judgment stage where weighing alternatives changes the routing, it measurably improves the decision. Turning it on everywhere would burn tokens for no gain.
• Made human-in-the-loop the default (pause after schema, before code) rather than running unattended: the cost is that a fully-automated run is opt-in (stop_after_stage=0), but the default protects against a bad schema cascading into a whole run of wrong artifacts — the schema is the cheapest place to catch the error.
• Kept Aurora as a pure data store with all business logic in Lambda, rather than letting any logic settle into stored procedures: the cost is more Lambda surface, but it preserves testability and keeps the parity tests meaningful — you can't unit-test a stored procedure the way you can a typed handler.
• Persisted every stage artifact to disk immediately rather than holding the run in memory until completion: the cost is more I/O per run, but checkpoint/resume, the human-review pause, and failure recovery all fall out of it for free — a crashed run resumes from the last completed stage.
• Built it as an MCP server rather than a standalone CLI: the cost is the MCP plumbing, but it means the same agent in any MCP client can drive the pipeline, and it composes with the other MCP servers (mcp-sql for legacy schema reads) instead of reimplementing DB access.

What I'd change today

I'd attach a golden-set eval to the Layer Decision stage specifically — it's the highest-leverage, highest-blast-radius stage, and right now its quality is validated by human review at the checkpoint rather than by a reproducible eval that gates a prompt or model-version change. I'd also make the parity-test suite run automatically against the generated Lambdas in a sandbox before the human ever sees the output, so the human review starts from "tests pass, here's the diff" rather than "here's the code, hope it's right." And I'd write the cost ledger into a dashboard rather than per-run JSON files — the per-stage token and cache-savings data is already captured, but I find the cost outliers by reading files instead of watching a trend. Principal-level systems surface their own cost regressions; this one still makes me go look.

Stack