How do I run database migrations safely in CI/CD?

Short answer: the real mistake is tightly coupling the schema change to the deploy while old code is still live — the fix is to decouple backward-compatible (expand/contract) migrations from the code deploy.

Short answer

Let me name the problem: either old code meets the new schema and breaks, or a big ALTER locks the table and times out the pipeline. Both stem from the same root — coupling. I covered the live-table side of the same transition — changing the schema with dual writes and no data loss — in a separate record; the question here is how to spread that transition across CI/CD steps.

Why

1. Old code is forced to meet the new schema. During the deploy there’s a window where live code talks to a schema it wasn’t written for; the moment it doesn’t recognize a column, or can’t find one it expects, it starts erroring.

2. A big, locking ALTER doesn’t fit inside a deploy step. Adding a column to a multi-million-row table takes minutes; if that work sits inside the deploy step, the pipeline times out and the table stays locked the whole time.

3. The root cause is coupling. As long as the schema change and the code deploy are pinned to the same moment, both failures are inevitable; the fix isn’t to patch symptoms one by one but to loosen that coupling.

What to do

1. Adopt the expand/contract discipline. Don’t add a column and deploy the code that needs it in the same release; never do it in the same release that uses a rename/drop. First add the column nullable, then deploy code that “writes to both old and new,” and finally remove the old in a separate, later release.

2. Take backfill out of the request path and the deploy step. Do the work of filling the big table in a separate, throttled step — not inside the deploy step’s transaction. Otherwise a single massive UPDATE both locks and times out.

3. Make the migration its own pipeline stage, free of timeout pressure. Move the schema change to a stage separate from app deploy. For big/locking changes, run them out of band with online-DDL tools like pt-online-schema-change / gh-ost; they add the column without locking the table.

4. Write them idempotent and reversible, and gate the deploy on the migration. Make migrations safe to re-run and reversible. Gate the deploy on the migration’s success; but leave destructive steps (drop, rename) for a follow-up release after the new code is fully rolled out.

Bottom line: I’d make expand/contract a rule, decouple migrations from app deploy, and run big/locking changes off the critical path with online tools. Never do a schema change “mid-deploy while old code is live” — roll it out gradually and backward-compatible; then both the pipeline timeout and the old-code breakage disappear.