Cause of 503s Possibly Established
Posted by Cat333Pokémon on February 21, 2010 at 4:30 PM
I have found the pattern: 503s occur daily at 4:00 PM PDT. In UTC (the time the server uses), this is 12:00 midnight. Because servers often have cron jobs that run at this time, I started moving vBulletin cron jobs like cleanup and birthday notices to 9:00 AM UTC (about the quietest part of the day). A week or so ago, FreezeWarp recommended I run all of them one by one to find out whether any of them caused a forum blackout. None of them caused outages and the errors continued promptly at midnight. Just today, I thought about server jobs and why only the forum goes down.
Then it hit me: the forum has a very large database. None of the other sites have one anywhere near its size, thus they only are down for about 10-15 seconds. The daily backups cause the downtime! Hopefully, we'll be able to move these to around 11:00 AM UTC, so they occur after the forum cron jobs and before the activity begins going back up.
So, the 503s were actually something good, just not running at the right time. With that, I should point out that the forum will be going down daily for scheduled maintenance at midnight UTC. (I know it's a terrible time. We're working on moving it!)