Thanks for the most constructive post in this whole discussion. Hacker News is a much better place to talk about solutions than just whining.
At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
That's our next step is automating our fail over. We currently use DNS Made Easy and while you can do that with them it's tougher. We are switching to Dyn, and their Layer 2 failure detection will put our failover in that 6 second range.
DNS based failover in practice will not give you 6 second failover, unlike BGP. Too much of DNS is broken and out of your control.
(The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)
Exactly. We announce different subnets for each continent or region, and all the sites within that region announce the same subnet. When one site ceases announcing the shared subnet, BGP usually converges in just more than 5 seconds. I was astonished the first time I saw it. It really makes you appreciate the solid engineering behind the core routers.
It's nontrivial to determine exactly when to drop the announcement. And be careful, because if you are too eager to drop the announcement, you may do it in more than one site at a time.
At first we used DNS with short timeouts, but those timeouts are only advisory and are ignored by some implementations. We would see most traffic tail off within 10 minutes for a one minute timeout, but it took many hours for all the traffic to migrate over to the new DNS. The folklore on using less than one minute for a DNS timeout is that a huge percentage of implementations ignore sub-minute timeout. Funny how much of the Internet's operation is passed along as folklore and not really known for sure.
Thanks for asking. Hacker News should be about sharing best practices and making the Internet a more reliable place.
At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.