This happened while upgrading my Raspberry Pi mail server from Jessie to Buster. The upgrade went with hiccups of course, I won’t be covering that here though, maybe later when I have quality time to write again.
During the upgrade, at some point package installations began to complain not being able to access their databases. Aha! MySQL is the culprit! Easy peasy fix right?
Sure enough, MySQL wasn’t running. I issued sudo service mysql restart, /etc/init.d/mysql restart and countless others. I must have tried every troubleshooting article on the net. None started MySQL. Reboot? Didn’t help either. Something about upgrading to MySQL Server 5.5 borked MySQL from starting.
Syslog showed MySQL mumbling something about checking if there was another MySQL server running? Nope! Also that it couldn’t find its mysqld.sock socket. Uh oh … that’s bad.
I disabled AppArmor, thinking that had blocked MySQL from starting. Nope! Ayyy, I have to go back and enable it now.
I checked which configuration MySQL 5.5 was using with sudo mysqld –help –verbose (it has several different folders it checks for the existence of my.cnf). Once MySQL has started the command sudo mysqladmin variables -uroot -p lets you see what values it has picked up from it’s configuration file wherever it found its my.cnf file.
By default MySQL checks the following my.cnf locations in the order listed:
Using the sudo mysqld command, looking at the log, I was able to see which of those my.cnf files is in use.
Removing obsolete conffile /etc/mysql/conf.d/.keepme … Obsolete conffile /etc/mysql/my.cnf has been modified by you.
- Saving as /etc/mysql/my.cnf.dpkg-bak …
- update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode
- Moving /etc/mysql/my.cnf.dpkg-bak to /etc/mysql/my.cnf.migrated
- Using your previous configuration through /etc/mysql/my.cnf.migrated
- update-alternatives: using /etc/mysql/my.cnf.migrated to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Finally! The last line tells me it’s using the /etc/mysql/my.cnf configuration file.
I check that my.cnf to see where the socket file is supposed to be. The peculiar thing is if I run mysqld directly on the command line in the foreground with sudo mysqld it runs perfectly fine. The socket file mysqld.sock is created by itself when it starts running (which is how it’s supposed to work) and deleted when it shuts down. MySQL is listed in the process list as it should be too.
When I attempt to run it with sudo service mysql start I don’t see a socket file created. Nothing was happening at all. Running sudo service mysql status to get status information from mysqld shows it was active (exited) meaning there is a process for it, but it had already quit running. If I stop it using sudo service mysql stop the status shows inactive (dead). Even curiouser.
So, socket configuration is correct. MySQL will run with that configuration. Now it’s just the matter of starting and stopping it as a service.
sudo service makes use of the /etc/init.d/mysql script to do the work of managing the mysql service. Ah, something interesting to see how it does that, troubleshooting.
Opening the mysql script shows runtime commands the script uses. It calls a command mysqld_safe which in turns launches mysqld in the background.
I hit up locate mysqld_safe to see where it is to troubleshoot further. Confusion ensues … locate doesn’t find anything, which mysqld_safe doesn’t find it either! No wonder MySQL never launches with sudo service mysql start !!! Baaaad news, how do I reinstall the missing mysqld_safe? I almost went into a tailspin at this point.
Well, no need to despair, google to the rescue. I searched for mysqld_safe and found it was an obsolete command, no longer included much less installed with new MySQL packages.
Ok, but what do I replace it with? Another almost tailspin moment … Recall that I was able to run MySQL in the foreground using mysqld? I edited /etc/init.d/mysql to call mysqld instead of mysqld_safe and hope for the best not to manage to bork it futzing around with system service scripts.
I ran sudo service mysql start and voila … MySQL came to life again to my relief. It only took me from 8am to 4pm to figure it out, and 1 hour to write this down 🙂
So, if you find this post in the wee hours trying to fix the same issue, you’re welcome.