The Journal Of A Linux Sysadmin

EV1Servers on Fire!

| Comments

EV1Servers, now part of ThePlanet after last year merger, experienced Saturday afternoon a major downtime on their Huston H1 datacenter: electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding their electrical equipment room. No one was injured and apparently no server damaged or lost. Still they were not allowed to bring up their backup power equipment resulting in (all the datacenter servers being down of course):

  • approximately 9,000 servers and 7,500 clients were affected
  • their own management servers, EV1Severs domain management, and SSL management; also their client management and communication tool ServerCommand was in the same situation;
  • nameservers and the H1 resolvers: and were down also (affecting other clients that might had those configured). Also the ones with the domain hosted on ev1 infrastructure, even if they had their server up were not functional because of dns problems.

At this time about 90% of the servers are up, but they are still working to bring up the rest.

So far, The Planet has been managing the situation as a whole quite well, by addressing the issues in a transparent and also timely manner. ThePlanet/Ev1 support team published updates during this period on their status page and also on their forum; also phone greetings recordings reported the status of the problem.