Wednesday, September 29, 2010

Naive Navitaire: Virgin on the Ridiculous part two

Following my 30 or so hour delay on a 90 minute trip, I've been investigating this failure a little bit. I guess I am still unsure how a single server failure could take out a major Australian airline for a day. Then there are the questions about why they don't have better backup and manual systems in place.

There is a bit more information around since my earlier post, Virgin Blue updated its press release on Monday afternoon says:
"We are advised by Navitaire that while they were able to isolate the point of failure to the device in question relatively quickly, an initial decision to seek to repair the device proved less than fruitful and also contributed to the delay in initiating a cutover to a contingency hardware platform."
and Virgin Blue and The Register reports that the failure was in a Solid State storage array.

So basically, the SAN array died, Navitaire guys thought that they could fix it that didn't work and it took them 21 hours to get the backup system working in its place (or to get the hardware replaced and the data restored I'm not sure which). I am flabbergasted that a service provider that at services every Australian Airline and another 70 or so airlines around the world could have such a terrible response to the failure.

From what I can gather around the net the New Skies System is based on .NET and I presume some sort of SQL back-end. This sort of setup lends itself very well to redundancy via data mirroring and load balancing across a group of servers. So why wasn't there a redundant data server sitting there ready? Early quotes (that I can't seem to put my hands on now) indicate that Virgin Blue have a "cheaper" back up solution than Qantas, that should have kicked in within three hours. Obviously 21 hours is a lot longer than 3.
A Virgin Blue spokesperson told iTWire that Navitaire was supposed to have a parallel system in place and in case of disaster this would go live within three hours. However, it did not actually come into play until almost a full day after the first incident.
But why? I guess we'll possibly never know. There is however one thing that I know, if Navitaire did have that backup system working within 3 hours then while I would have probably been delayed it certainly wouldn't have taken more than 30 hours for me to get home.

So for that I say screw you Navitaire! Where is my compensation?

Tuesday, September 28, 2010

Reverting & Arguing with already merged revisions in SVN

So for the last few days I've been merging a crap shoot of a collection of features that have intermingled revisions and a whole bunch of interdependencies.

This lead me to the point yesterday that I had to revert to an earlier version of the branch I've been working (and in the process lose an afternoon of work) so that I could merge another feature that had lots of changes I needed.

After eventually remembering how to go back to a previous version (Merge 2 different trees and select the From Revision as the latest revision on the branch and then select the To Revision as the version you want to go back to) I discovered another problem, I couldn't get the revisions that had already been merged to cooperate and re-merge into the branch because subversion thought that the revisions where all in there already.

After some unsuccessful searching on the net and trawling Stack Overflow I couldn't find anything useful. So I went off and poked around a little more before it occurred to that I could probably edit the properties on the branch. So I opened up the properties via the Tortoise SVN menu as shown to the left. And discovered there was a field that holds the merge history, svn:merginfo. The mergeinfo tag is organised by source branch and revision.

Once I'd found this it was a simple (albeit tedious) process to remove the revisions that I wanted to re-merge from the list and it was all good, they once again merged properly. Hope this tip helps you guys out.

Oh and I know that we should move to Git or Mercurial (aka hg) but it is going to take a while to get the whole company on board.

Monday, September 27, 2010

Just hit 1000 page views on my blog

Pageviews today
39
Pageviews yesterday
29
Pageviews last month
356
Pageviews all time history
1,001



With Virgin Blue to thank for the last 50 or so views that pushed me over the edge.