Today I woke up because the Nagios used by our isp sent me a sms: something does not work well on one of the association servers. According to the sms, the webserver was not responding. Ok, the webserver is never the cause of the problem, something else is going wrong.
Logged in over ssh, at least this works well. ps told me that the nightly database backup is hanging around, a script which fills some tables is also waiting. Mysql shell does not really work. Disc space? Ok, enough free space.
Shutting down webserver and database, killed all scripts hanging around and restarted the db. After a while, I got an error mail:
warning : Table is marked as crashed and last repair failed
warning : Size of indexfile is: 281307136 Should be: 267612160
error : Found 5174283 keys of 5174282
error : Corrupt
Ok, Let's repair the beast. myisamchk seems to work his way through the table, until I got this message:
myisamchk: warning: 301 records have been removed
So this wannabe database just deleted 301 datasets for whatever reason. Nice to see how data integrity is not handled at all.
I'm happy at all that the new platform is coming around, betatest just started yesterday. It's time to kick Mysql in the ass and use a real database.