Why is my site down? Yola Unplanned Outage

Edit Subject

Hi All

I have a further update for you on the situation with your published sites.

As the various people from our company (including our CEO) have already shared with you: the quality and reliability of our service and particularly the availability of our hosted sites is our number 1 priority at all times. To this end we have already invested very heavily in our infrastructure and continue to make upgrades and improvements all the time.

However, what we experienced today was the largest DoS attack we or our service provider have ever encountered. There is no question that this is criminal and malicious. All the many defenses that we have already put in place for attacks of this nature failed under the sheer weight of the attack.

Because of our commitment to a stable and reliable service, we are immediately investing in a massive upgrade that should allow published sites to be back online within 24 hours. It should then give us significantly added protection for any further attacks of this size or even bigger.

Our company values demand that we maintain an open and honest line of communication with our customers at all times and this is what we have tried to do on this thread, and through our other communication channels.

As always we deeply regret the damage that this is doing to our business and yours, and will do all in our power to set things to rights.

Watch this thread for the latest information.

Hi everyone,

I'm Jonathan, the Operations Team Leader at Yola, and I've been awake since 4am when I was alerted to this problem. We don't take this sort of problem lightly at Yola - keeping published sites up and running is literally the prime concern of my team, above everything else. Bearnard, our Platform Engineer, has been up since 3am trying to mitigate the problem as much as he can.

Unfortunately, there is only so much we can do to rectify the issue. This is a massive-scale Denial of Service attack on the Boston infrastructure, which is affecting three major ISPs in the area. We're working closely with their technicians to isolate and firewall off the attackers, but until we can do this safely, we are unfortunately cut off from the internet at large.

The nature of this problem is that a very large number of machines out there are being silently (and unknowingly) manipulated by a few controllers, and being made to flood our networks with bad data, causing the infrastructure to struggle and fail. Most of the attacking machines are desktop machines like your own, which have been overtaken by malware and forced to perform these attacks in the background, without the knowledge of their owners. As such, it's very difficult to block the real attackers, because they use these "zombie" machines to do what they want.

We'll keep you posted with every bit of information we have. Please, sit tight, and hold thumbs for us that we can get this issue under control as soon as we can - we hate your sites being down at least as much as you do, I promise!

Cheers,
-Jonathan
Yola Operations Engineer