Why is my site down? Yola Unplanned Outage

  • 48
  • Question
  • Updated 8 years ago
  • Answered
  • (Edited)
Archived

This conversation was archived and is no longer visible to community members. The community moderator provided the following reason for archiving: Archiving as this issue has now been solved.

We are currently experiencing technical difficulties with our upstream provider and will be back up as soon as possible. Our engineers are working as fast and hard as they can to get you back into your sites.

We apologize for the inconvenience and thank you for your patience as we get this sorted out.

~ Sanja
Photo of Sanja

Sanja

  • 10698 Posts
  • 495 Reply Likes

Posted 9 years ago

  • 48
Photo of Vinny Lingham

Vinny Lingham

  • 12 Posts
  • 3 Reply Likes
Official Response
To all our loyal users:

We apologise for the current outage. Yola's track record of hosting websites is over 99.99% for the past 3 years that we have been in business. We strive to deliver the best service to both paid and free customers. The current denial of service attack is affecting us and numerous other companies on that operate servers at our hosting provider. We apologize for any inconvenience that this DDOS attack may having on our userbase.

We are constantly upgrading our infrastructure to the highest possible standards available and will continue to do so. Please note that this is one of the most high priority items on my desk at the moment and is getting high priority attention from every available resource at both our company and our service provider.

Please be patient while we resolve this issue - again, on behalf of Yola, I apologize for any inconvenience experienced.

Regards
Vinny Lingham
CEO, Yola
Photo of Jonathan Hitchcock

Jonathan Hitchcock

  • 37 Posts
  • 16 Reply Likes
Official Response
Hi everyone,

I'm Jonathan, the Operations Team Leader at Yola, and I've been awake since 4am when I was alerted to this problem. We don't take this sort of problem lightly at Yola - keeping published sites up and running is literally the prime concern of my team, above everything else. Bearnard, our Platform Engineer, has been up since 3am trying to mitigate the problem as much as he can.

Unfortunately, there is only so much we can do to rectify the issue. This is a massive-scale Denial of Service attack on the Boston infrastructure, which is affecting three major ISPs in the area. We're working closely with their technicians to isolate and firewall off the attackers, but until we can do this safely, we are unfortunately cut off from the internet at large.

The nature of this problem is that a very large number of machines out there are being silently (and unknowingly) manipulated by a few controllers, and being made to flood our networks with bad data, causing the infrastructure to struggle and fail. Most of the attacking machines are desktop machines like your own, which have been overtaken by malware and forced to perform these attacks in the background, without the knowledge of their owners. As such, it's very difficult to block the real attackers, because they use these "zombie" machines to do what they want.

We'll keep you posted with every bit of information we have. Please, sit tight, and hold thumbs for us that we can get this issue under control as soon as we can - we hate your sites being down at least as much as you do, I promise!

Cheers,
-Jonathan
Yola Operations Engineer
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
Official Response
Hi All

I have a further update for you on the situation with your published sites.

As the various people from our company (including our CEO) have already shared with you: the quality and reliability of our service and particularly the availability of our hosted sites is our number 1 priority at all times. To this end we have already invested very heavily in our infrastructure and continue to make upgrades and improvements all the time.

However, what we experienced today was the largest DoS attack we or our service provider have ever encountered. There is no question that this is criminal and malicious. All the many defenses that we have already put in place for attacks of this nature failed under the sheer weight of the attack.

Because of our commitment to a stable and reliable service, we are immediately investing in a massive upgrade that should allow published sites to be back online within 24 hours. It should then give us significantly added protection for any further attacks of this size or even bigger.

Our company values demand that we maintain an open and honest line of communication with our customers at all times and this is what we have tried to do on this thread, and through our other communication channels.

As always we deeply regret the damage that this is doing to our business and yours, and will do all in our power to set things to rights.

Watch this thread for the latest information.