Growing up in the world of linux uptime was always considered a good thing. On IRC every once in a while someone would post an uptime. Everyone else in that channel would then check their uptime and if it was greater or close they would post it in the channel. Most of these systems were home linux boxes used for compiling random programs or maybe hosting a webserver for experimenting. It was fun to see how long we could keep them running for. Since those days I have come to realize that high uptimes are a bad thing.
Keeping a server up for months or even years means that you aren’t maintaining it. It hasn’t been kept up to date with new kernels that have fixes for security holes. It doesn’t have new packages or new tools that can help it run more efficiently and have features that can make using it easier. It’s also not up to date with new servers that are being deployed which means that people logging into your server with a high uptime have to adjust themselves to the older software and possible missing tools.
Hardware fails, colos lose power, network connections, and sometimes catch on fire. If you’re entire system depends on a single server, say a mysql master. It’s going to fail. I know there are mysql servers out there that have been up for years. Those are going to fail. It’s inevitable. If you’re system is not designed to withstand the failure of a master it should be fixed. Jeremy Cole and I gave a tutorial at the 2006 MySQL User Conference about MySQL replication and failover. See Jeremy’s blog for links to the presentation and photos.
“But I can’t take down my master to fix it?” It’s much better to do a planned downtime than it is to get paged at 3am because the master died and the whole site is down. Take some time. Plan to take down the master and fix the system. It will be worth it in the end. If your manager says no to a planned downtime to make your website fault tolerant. Find a new job. Preferably at Yahoo!
By building a system that can handle the failure of a master it’s much easier to upgrade MySQL so you can take advantage of all the new nifty features.