Virgin on the Ridiculous

Yesterday I was supposed to fly from Sydney back to Melbourne but like around 100000 other jet setters on what is apparently the third busiest air corridor in the world I spend a leisurely six hours at the airport. Two to check into my flight and then 4 sampling the excellent gourmet fair of the airport food court and the lovely community atmosphere generated by several thousand grumpy people in a confined space. The $12 worth of vouchers that Virgin Blue (VB) kindly provided me with secured a pie and a coffee.

Anyway, this post isn't about venting (ok well maybe a little bit) but more an under informed investigation and discussion of the last 24 hours.

Navitaire: Virgin Blue's* Services Provider
I managed to discover last night that VBs service provider for its reservation system is Navitaire and after a little poking around it seems that they use the Open Skies system. A look around the Navitaire website reveals that all of the Australian airlines use the same system. As the outage affects all aspects of the VB operations that the Open Skies system would handle it seems of little doubt that the outage is related to the Navitaire system. It would also seem that each airline has its own servers etc running the Navitaire software other wise more of the airlines would be affected.

Redundancy?
As someone that used to work on a critical business infrastructure product, corporate telephone systems, I have experienced first hand at work the business impact that downtime on one of these systems can have. Our system was supposed to be five 9s reliable, which means that the system was supposed to be down for a maximum of 5.26 minutes per year or 6 seconds a week, including scheduled upgrades etc. If there are no further outages this year then VBs flight reservation system is currently running somewhere around 99.7% reliability. Through a combination of factors we once caused a customer outage that lasted around 7 minutes and where forced to compensate in the order of $50 million.

In light of the high costs associated with downtime our systems had redundancy and lots of it. We had high availability (HA) servers that ran in pairs, you could shoot one and the other one didn't skip a beat. Then if they both failed another set of servers could take over, if the network suffered a large scale outage, servers at each site could take over the calls for the local phones.

So one of the big questions that needs to be asked is "where is the redundancy?" Where are the backup servers located in another location? Why is there not replacement server on standby ready to jump in and run from a current back up of the data?

I hope that at some point in the future we get analysis of the failures so that we can work to avoid such incidences again in the future. While it does seem that Navitaire is the culprit in this instance I'm moving on to talk about VBs handling of the situation.

Information Flows and Transparency
Let me just start out this section by saying I don't blame this on the people at the front line, the customer service folks that managed to, for the most part, maintain the smiles. They are doing a brilliant job. What is not working is the information hierarchy, it was evident that the staff were quick to pass on any and all information that they received but they were evidently not getting enough information and what information they were getting could not be effectively communicated.

1. None of the display boards at the airport where updated. They were still showing the flights that where supposed to be going etc as if everything was running according to plan. We all know that even on a good day planes are delayed, cancelled etc and they should be able to reflect such changes on the boards quickly and easily.
2. The website updates where too slow. I left for the airport four hours after the system went down because there was nothing on the website to say that I shouldn't. As soon as I arrived at the airport people where told that all non-essential flights should be postponed and to get out of the airport.
3. Dissemination of credit and refund policies, I'm still not sure what the polices are!
4. Change the messages at the call centre, I didn't need to spend an hour on hold to be told the system was still down. How about having a message at the start telling me that nothing could be done?! Or having that play instead of the "Your call is important to us" malarkey!
5. You have email and phone details for a lot of the customers, use them! The only way I am finding semi-current information is Twitter. Check out the search for the latest. This morning I did find out that you would call or SMS people to let them know their new flight details. How about email as well? That could be easily automated!

I think I am done for now, I am sure that more things will spring to mind. But in the mean time Virgin Blue needs to be told that it is not what crises you face that defines you but rather how you respond to them and their response currently leaves a little to be desired. Studies have shown time and time again that customers will come back to a company after a failure such as this as long as the companies response is well handled.

On that note I am off to attempt to work from my mothers house on a tiny screen!

* I wonder as I write this whether I should be using Virgin's Blue as the plural of Virgin Blue.