MySQL Woes
We’ve been having some trouble with our MySQL server running on the HPUX machine. It’s been crashing multiple times for the last few days. When it crashes it becomes completely unresponsive. No clients can connect and any clients that were connected don’t respond. I tried keeping a few windows open to monitor the processlist and extended-status but they didn’t help a whole lot when the server crashed (all values seemed normal when the crashes occurred). I was even unable to shutdown the server. I tried running the script to stop the server but the server wouldn’t respond to any kill commands. Basically the only thing I could do was kill the process with a “kill -9″ (NEVER A GOOD THING).
I’ve been watching the logs and I haven’t found anything out of the ordinary. Nothing significant in the error log. Nothing in the binary log and nothing in the query log file. I assumed it was one particular query that was killing the server but the last queries in the log files never matched. I checked the system log files and couldn’t find anything suspicious except for one thing. The server was restarted a few days ago for unknown reasons. I put in a ticket to engineering inquiring why the server had been restarted (assuming they’d made some changes). I got a response back indicating the server had crashed. I suspected the database might have some corruption problems due to the kill -9’s but it wasn’t until I learned the server had crashed that I thought corruptions could be the cause of the problem.
I decided to take the server down and run a myisamchk. It found quite a few errors. I ran it again with the –recover option. Just to be safe I then ran a CHECK TABLE [table name] EXTENDED on all the tables (MyISAM and Innodb) to verify they all had an “OK” status. They did. Hopefully this solved the problem. We’ll have to see how things go tomorrow.