It was the best of times. It was the worst of times.
I was in Orlando Florida, having just spent a Bickford-family-record of 1 gerzillion bucks for for a set of two day passes to Disney World, and a 1 day pass to Universal Studios. “Sure, I may wind up in debtor’s prison with expenditures like this” I thought, ”but at least the kids will have the time of their lives…and I’ll finally get to see Space Mountain!”
So, early on Thursday morning, we jumped out of bed and drove, shuttled, and monorailed our way to the Magic Kingdom. And it really was magical: random chorus lines debarked from street cars to perform impromptu musical numbers; fireworks went off in the distance at random intervals; lines were short, and I’d even succeeded in grabbing my FastPass to ride Space Mountain—the ride that had eluded me on every trip I’d taken to Disneyland since I was a wee lad of 8. It was a really terrific day.
Then, as I headed over to Space Mountain to slide into the express line with my FastPass, my cell phone rang. It was Mark, back at the office. “Hey… I just got in and the web site seems to be down.”
“Well, try restarting the server”, I said, thinking some random glitch may have caused things to seize up since I last checked on it some eight hours earlier. I held the line, waiting for Mark’s “All Clear”. But it wasn’t to be.
“Uh… I’m getting a message here on the server about not being able to load the OS…”
“Could you read that too me again?” I asked, not quite believing what I was hearing.
He read it again.
“That’s bad. ” I said, and the warm Florida day suddenly got a lot colder.
What followed was a long series of phone calls while Mark and the crew read cryptic message after cryptic message to me, while I suggested a number of increasingly more complicated troubleshooting steps, all of which failed. The only good news was that we had multiple, current backups of the server.
By the time I got back to the hotel room, we’d essentially ruled out any troubleshooting measure short of a full system restore. But even that ran aground. (Retrospect folks: we really gotta talk about your “disaster recovery” feature…). Shiaw-Ling acted as my hands and ears from afar, working long into the night, while I remoted in to try to get the machine on its feet.
By morning, it looked like we were nearly there, but I still knew I’d have to spend the rest of the day, at best, working on it. The kids (and I will be eternally grateful for this) decided that they didn’t want to do our second day of Disney without their Dad, so they wandered around Orlando while I—having slept all of 50 minutes or so that night—tried and tried to get the server working again from my remote connection.
Alas, I failed in my efforts and, exhausted, I decided around 8 pm that I *had* to get some sleep. I was just no good to anyone. Worse with the weekend ahead of me, there was nobody to reach on the West Coast to do hardware swapping and other such work.
In the morning, I tried a few ideas I’d had overnight, then did the only reasonable thing I could think of: I set the site to forward to a “System is Down” page, and went off to spend the day with my family at Universal Studios.
Universal Studios was also a delight, and the Simpsons Ride alone was almost worth the cost of admission. I’d never been to Universal Studios before, so I had no idea what to expect from any of the attractions, all of which added to my amazement when they all turned out to be so well executed (with the possible exception of the unintentionally hilarious “Twister”). Toward the end of the day, I called Shiaw-Ling at home and asked if she’d be willing to help out from the California end with another go-round at a system restore that night. To her eternal credit, she said yes.
That night, we labored well into the morning and actually succeeded in rolling back the server to precisely the point it was before all the trouble. The only thing missing which prevented us from resuming full operation was the need to get new SSL certificates from our Certification Authority. We put in the request for this, but the CA wasn’t working until Monday. Once again, there was nothing for me to do…so I went back to Disney World and had a great time with the family.
On Monday morning, there was still no word from our CA, but I managed to reach the CA and get the new certificates installed while riding in the passenger seat on the road up from Orlando. Other than a few mail hiccups, we were finally back on our game, after a nervewracking shutdown of almost four days.
So it’s hard to say how I feel about that leg of the trip. It was full of thrills and adventure in the theme park cities; and full of frayed nerves and dismal depression while I worked from an Orlando hotel room by modem to try to restore a machine on the other side of the country. We even contemplated aborting the whole vacation and having me jump a plane to get back to California to get the bloody server back up. I’m glad as heck it never came to that, but for a time, it felt like a near thing indeed.
And in the end, it all worked out. My terrific family stuck by their stressed-out Dad in his time of need (they even found me Dunkin Donuts and coffee for my late night work shifts), my daughter got to meet new BFF Minnie Mouse, and I finally, finally, got to ride Space Mountain with my son. And yes, the ride was everything I’d ever hope it would be. We even got to see the offices of the Daily Bugle and ride along with Spider-Man as he battled the Sinister Six at Universal Islands of Adventure. All’s well that ends well, I guess.
