How do I run database migrations safely in CI/CD?

Short answer: the real mistake is tightly coupling the schema change to the deploy while old code is still live — the fix is to decouple backward-compatible (expand/contract) migrations from the code deploy.

Let me name the problem: either old code meets the new schema and breaks, or a big ALTER locks the table and times out the pipeline. Both stem from the same root — coupling.

1. Adopt the expand/contract discipline. Don’t add a column and deploy the code that needs it in the same release; never do it in the same release that uses a rename/drop. First add the column nullable, then deploy code that “writes to both old and new,” and finally remove the old in a separate, later release.
2. Take backfill out of the request path and the deploy step. Do the work of filling the big table in a separate, throttled step — not inside the deploy step’s transaction. Otherwise a single massive UPDATE both locks and times out.
3. Make the migration its own pipeline stage, free of timeout pressure. Move the schema change to a stage separate from app deploy. For big/locking changes, run them out of band with online-DDL tools like pt-online-schema-change / gh-ost; they add the column without locking the table.
4. Write them idempotent and reversible, and gate the deploy on the migration. Make migrations safe to re-run and reversible. Gate the deploy on the migration’s success; but leave destructive steps (drop, rename) for a follow-up release after the new code is fully rolled out.

Bottom line: I’d make expand/contract a rule, decouple migrations from app deploy, and run big/locking changes off the critical path with online tools. Never do a schema change “mid-deploy while old code is live” — roll it out gradually and backward-compatible; then both the pipeline timeout and the old-code breakage disappear.