One of our database servers had emergency work done on it by our upstream provider and has not yet been resolved. We are working to get that back up. While underway performance of our website is intermittent.
While this emergency work is being done the following actions will not be possible on our website:
There may be other issues as well. We are working hard on trying to get things back up as quickly as possible. Sorry for any inconvenience.
There is a DDOS attack happening in the Eastern US seaboard right now (see http://finance.yahoo.com/news/theres-major-problem-internet-thats-130316780.html for more details) that is affecting our EventEspresso and EventSmart properties. If you are having trouble accessing our sites it is because of this attack. We’re awaiting the progress of engineers working on this stopping this attack.
The initial work being done on the eventespresso.com database is complete. There might be another short downtime period later today.
We’re doing some preliminary work to prepare for a database upgrade on eventespresso.com. Initial downtime should be fairly short. More updates will come if this takes longer than expected.
Eventespresso.com is going down briefly for some scheduled maintenance. We’ll be keeping this maintenance window as short as possible. Watch this space for news.
We spent the majority of today putting in place some measures to prevent DDOS attacks in the future and implemented some rate-limiting on the api endpoints for our plugin update engine. While doing this, we also noticed an inefficient query that compounded the server problems and we fixed that.
Long story short, automatic updates have been re-enabled. However, there is the possibility that we may still have to fine tune some of the measures we’ve put in place. If any of the following happen to you..
…please contact us and we’ll look into things and see if there’s a problem with some preventative measures we put in place on our server.
Again, thanks for the patience during this experience folks!
Hi folks, as the developer responsible for finding out the cause of our recent server interruptions I just wanted to give you an update on what we’ve discovered and what we’re doing to hopefully prevent this from happening again.
As you are no doubt aware from some of the high profile sites that have gone down over the years, no matter how big or small you are in the internet worlds, there are always people out there who think its okay to cause harm to others on the internets (for whatever lame reason they may manufacture).
What we discovered, is that there was a ddos attack against eventespresso.com and the attack vector they used was the api we have in place for recording uxip stats and our plugin update engine (used for autoupdates of plugins, and key verification).
Temporarily, we have added site-wide rate limiting to the api. This means that for many of our clients, auto updates and uxip stats (if you opted in) will not function as normal. This will not impact your EE code hosted on your site. You just won’t be able to auto update until we get the fix in place.
Within the next 48 hours, we will be implementing a more permanent solution that will hopefully prevent this from happening again, we’ll update here and on our blog once that is in place and you’ll be able to resume normal auto-updates again.
Please Note: The changes we’ve put in place temporarily should not affect your ability to interact with EventEspresso.com and download plugin files from your account or post on the forums. Again, all this will affect (temporarily) is the ability to do one-click updates via your site dashboard.
Thanks for your patience while we’ve been trying to resolve this. The timing of this attack was unfortunate as much of our team (including myself) was traveling back home from WordCamp San Francisco this weekend and we were unable to address the problem sufficiently until we arrived home.
Thanks for your patience folks. Just one of those things we do to kEEp things runnEEng 🙂