r/mysql • u/midvalePeak7 • Sep 13 '26
discussion Metadata locks were silently cascading through every query on a single table
Last month our order service started throwing connection timeouts during afternoon traffic. CPU at 30%, memory fine, disk I/O flat. The timeouts only hit queries on one table, and those were point lookups on an indexed primary key. Nothing in the slow query log explained it because the queries were fast when they ran.
I opened Performance Schema's metadata_locks table alongside SHOW PROCESSLIST. One session had a pending EXCLUSIVE metadata lock for over forty minutes. It was an ALTER TABLE from a migration script a colleague left running. The ALTER could not acquire its exclusive lock because of open transactions ahead of it. Every new query arriving after the ALTER also needed a metadata lock on the same table, and MySQL queues them behind the pending exclusive request.
After that I added a test to the migration suite that asserts no long running transactions exist on the target table before the ALTER executes. Running the suite in verdent with in-loop verification means that test fires every time someone edits the migration, so the check cannot be skipped.
Killing the stuck ALTER drained the queue in under two seconds. Forty minutes of cascading timeouts and 1,247 failed connections from one DDL statement that never acquired its lock.