Well that was ugly.
Today at 4:40pm Eastern we experienced a wholesale loss of services from Media Temple, my expensive hosting provider, and that complete and total failure of redundant website hosting, FTP, email and everything else that I am virtually made of demonstrates just how awful and tender and fragile technology is on the internet and how poor planning can come around and chomp even a good company on the behind.
Here is the official notification from Media Temple that took over an
hour to get published online. It should have also been mass mailed to
every customer but it was not. Media Temple are located in California
so that explains the time difference between me and them. “Availability
Issues” is putting it mildly:
(mt) Media Temple Operations
Availability issues on Friday, July 28th 1:00pm
At approximately 1:00pm today, (mt) Media Temple’s (LA-IDC2) data
center experienced another power issue. Unfortunately, information from
the building management at the Garland Building is sparse, vague at
best, and lacking important detail. It is being speculated that one of
the buildings back up generators has caught fire; however this
information is hearsay and can not be confirmed. (mt) Media Temple
staff has responded to this issue promptly and with the utmost urgency
by fully staffing the company and answering customer phone calls and
support tickets.At the time of this writing, all power systems have
been restored and engineers are currently bringing customer servers
back up and repairing file systems that have become temporarily
corrupted.
This incident is clearly an unacceptable situation for our company and
our customers. (mt) Media Temple has decided to accelerate its plans to
move all customers in (IDC-LA2) to a new data center which has
undergone exhaustive testing to insure such power issues will not be a
problem in the future.
Our President and CEO, Demian Sellfors, is making himself personally
available to call customers back who wish to discuss these issues in
more detail. We encourage any customer who wishes to receive a call
back from Mr. Sellfors to please indicate so inside this ticket.
Shortly, he will be calling you back to discuss these matters with you.
I pay a lot of money for the Media Temple Dedicated-Virtual server and
when this blog and all my sites and email were still down at 6:10pm
Eastern, I called Media Temple. I was told the problem was being
addressed and my services should be restored “soon.”
“Soon” is a hard word to swallow when all your eggs are in the dropped
Media Temple basket.
We fully went back online around 7:22pm.
We’ll see if we stay alive or
if we pop on and off again as things simmer down or bubble up again.
I’m sorry for being offline so long.
If everything seems to disappear again in the future and you are
wondering what happened — or whatever — you are welcome to use my
Gmail address “dboles” to tell me what’s up or to ask me what happened.
For a moment earlier today I thought my Verizon DSL connectivity was
having internet routing issues, but now we know that wasn’t the case.
We were dead and down.
Hard.
It makes you a little sick to your stomach.