Well now this is a surprise....

Virgin Blue's checkin/everything system is back up and almost 12 hours earlier than predicted. More info here.

To be honest I am a little surprised that they managed to pull it off, let alone so far ahead of schedule, I guess a pessimist set the time line, which is better than an optimist.

I suppose though, that there is a pretty good possibility of the whole thing falling over in a big heap again, but here is hoping that they have it all sorted out, after all I have to fly with them again at least once so that I can use my flight credits and to be honest I actually usually like their service etc at least for flights between Sydney and Melbourne.

What? They're cancelling flights again!

What a crock! Yup, in part of the continuing saga related to the complete failure of their Navitaire 'New Skies' booking system two weeks ago Virgin Blue is cancelling more flights.

I talked about the planned downtime in a post on Monday, This really is Virgin on the Ridiculous, which as you can read there seems to be a terrible display of system and software engineering that makes me a little ashamed to be in the industry. The need to cancel flights screams to me that Virgin Blue still hasn't got a good backup check-in and booking system organised. While apparently they've accommodated everyone

 "...on other flights and they will depart within an hour of their original departure time."

I am curious why they couldn't with a week or so notice fix the problems associated with speed of check-in in their manual backup process to the point where they didn't need to cancel any flights what so ever. Perhaps it is purely a cost saving exercise, there being no point to fly the extra flights, but given that they usually have these extra flights running and presumably full enough to justify, it seems that it is because they expect delays in the "manual" check-in system again.

So as this saga continues we continue to see flaws in both Virgin Blue and Navitaire, no system switch over should take 32 hours but really Virgin Blue should be able to handle the normal flight load on their backup system with advanced notice to bring in extra staff etc.

This really is Virgin on the Ridiculous

Virgin Blue Minor Incident Report

So, I'm sure by now if you're reading this blog that you have seen my thoughts on the Virgin Blue check-in/reservation/booking system fiasco (Here they are if you missed them, Virgin on the Ridiculous, More background on Virgins IT Systems and Naive Navitaire). It just got a little more ridiculous in my book, they are still using their backup system, which in itself is pretty poor form, and they are planning to switch back over to their main system tomorrow. Here is the statement.

Online and telephone reservations services will be unavailable from 9.00pm AEST on Tuesday 5 October;

• Check in for domestic flights will open two hours before scheduled departure time; three hours for international flights;
• Web check, Kiosk check in and Check-mate will close at 8pm AEST on Tuesday 5 October through to 5am on Thursday 7 October; 

That's right, they expect the switch over to mean that their system will be down for1 day and 8 hours or 32 hours altogether during which time they will be checking people in manually and the phone and online reservation systems will be down.

How crazy is that? So not only did it take Virgin Blue and Navitaire 21 hours to switch over to the backup system apparently it is going to take them 32 hours to switch back to the 'primary' system. What a crock! What an absolute shambles! Is the system that badly designed that they have to untangle a mass of inter related crap instead of changing a few key routers or something like that to switch between the two systems.

I can understand that it would be complicated but anything more than a few hours seems crazy, can't they get the primary up and running in parallel and then flick a switch?

Anyway, thought you might all appreciate an update on the tom foolery.

Naive Navitaire: Virgin on the Ridiculous part two

Following my 30 or so hour delay on a 90 minute trip, I've been investigating this failure a little bit. I guess I am still unsure how a single server failure could take out a major Australian airline for a day. Then there are the questions about why they don't have better backup and manual systems in place.

There is a bit more information around since my earlier post, Virgin Blue updated its press release on Monday afternoon says:

"We are advised by Navitaire that while they were able to isolate the point of failure to the device in question relatively quickly, an initial decision to seek to repair the device proved less than fruitful and also contributed to the delay in initiating a cutover to a contingency hardware platform."

and Virgin Blue and The Register reports that the failure was in a Solid State storage array.

So basically, the SAN array died, Navitaire guys thought that they could fix it that didn't work and it took them 21 hours to get the backup system working in its place (or to get the hardware replaced and the data restored I'm not sure which). I am flabbergasted that a service provider that at services every Australian Airline and another 70 or so airlines around the world could have such a terrible response to the failure.

From what I can gather around the net the New Skies System is based on .NET and I presume some sort of SQL back-end. This sort of setup lends itself very well to redundancy via data mirroring and load balancing across a group of servers. So why wasn't there a redundant data server sitting there ready? Early quotes (that I can't seem to put my hands on now) indicate that Virgin Blue have a "cheaper" back up solution than Qantas, that should have kicked in within three hours. Obviously 21 hours is a lot longer than 3.

A Virgin Blue spokesperson told iTWire that Navitaire was supposed to have a parallel system in place and in case of disaster this would go live within three hours. However, it did not actually come into play until almost a full day after the first incident.

But why? I guess we'll possibly never know. There is however one thing that I know, if Navitaire did have that backup system working within 3 hours then while I would have probably been delayed it certainly wouldn't have taken more than 30 hours for me to get home.

So for that I say screw you Navitaire! Where is my compensation?