MySQL fails to start after auto-upgrading to 8.0.30.
Most of my VPS run on Ubuntu (20.04/22.04), and a couple of weeks ago, one of my customers had an issue.
The system is - by default - set to unattended upgrades. One of the upgrades it did was upgrading MySQL
server. This had no effect, until the VPS was rebooted. Now it will not start anymore.
We restored from backup, took a backup from 8.0.29, dropped the database, recreated it and did a restore.
When disaster strikes...This morning I had the same issue. 😳
2022-08-28T05:59:41.270083Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.20.04.2) starting as process 814 2022-08-28T05:59:41.280784Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2022-08-28T05:59:41.426273Z 1 [ERROR] [MY-012962] [InnoDB] The redo log file ./#innodb_redo/#ib_redo7 size 2449408 is not a multiple of innodb_page_size 2022-08-28T05:59:41.426359Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error. 2022-08-28T05:59:41.808322Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2022-08-28T05:59:41.808629Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. 2022-08-28T05:59:41.808673Z 0 [ERROR] [MY-010119] [Server] Aborting 2022-08-28T05:59:41.809287Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.20.04.2) (Ubuntu).
So I restored my backup from last night, only to have the same issue reappearing.
OMG, the unattended upgrade happened on July 29th, a month ago. I do not have a backup, nor could I restore it, this server runs PowerDNS.
Not to put a blame on...
The information I found back in July was that this was caused by running MySQL on top of ZFS on Ubuntu.
A bit of a strange conclusion. And since all my VPS run on ProxMox on ZFS volumes, not something I can easily remedy.
Looking at the release notes - kind of - on the MySQL dev site, I read under
InnoDB that it now supports dynamic configuration of redo logging.
But much more to the point is the single sentence almost at the end.
As is generally required for any upgrade, this change requires a clean shutdown before upgrading.
This is much more likely the culprit.
Not all is lost...
On StackOverflow I found an article with a possible solution.
The file should be a multiple of
innodb_page_size, by default that would be
16384. So how much too small is my file?
2449408 % 16384 = 8192
Let's create a zeroed-file of that size and add it to the logfile.
dd if=/dev/zero bs=1 count=8192 of=./zeros # Append zeroes to invalid file cat zeros >> /var/lib/mysql/#innodb_redo/#ib_redo0 # Restart MySQL systemctl restart mysql.service
And luckily for me this worked as advertised.