Situation
On the Enterprise Reinsurance migration, the first sketch I produced for the cloud ETL was the design every senior engineer would have drawn: one Glue job per table, three stages each — 23 tables, ~69 jobs. Familiar shape. Easy to assign. Easy to defend.
Task
Lead Engineer accountable for the migration pattern across an 18-engineer program with $billions in active treaty exposure on the line. The architectural choice would propagate into operations, testing, and on-call for years.
Action
I spent close to two weeks shipping toward the per-table design — staffing it, sketching the Step Functions orchestration, writing the IAM roles. The signal that I was on the wrong path showed up slowly: every conversation about the design ended with someone (often me) saying “yeah, it'll be a lot to operate, but that's the cost.” When I finally noticed I had said that sentence four times in a week, I stopped and asked the question I should have asked on day one — why would the number of jobs equal the number of tables? The tables didn't differ in shape that mattered to the pipeline. The transformations were parameterisable. I scrapped what we had and rebuilt it as 3 generalized Glue jobs (extract, transform, load) driven by a metadata table — one pattern, every source.
Result
23 tables migrated through 3 jobs instead of ~69 — fewer surfaces to test, version, and operate; one bug fix, not 23. What I'd do differently isn't the final design — it's the two-week tax. The principal-level move is to treat the words “that's the cost” as a tripwire. If you're justifying the cost of your own design more than once, you're not designing — you're defending. I now interrupt myself the first time, not the fourth.