Monday, August 30, 2010

FileSystemWatcher and Bluescreens

For a project at work we need to monitor a directory for any new folders added and to check for changes to a few of the files to update configuration and settings information in other parts of the program. the obvious solution was to use the MS FileSystemWatcher which sits in the System.IO namespace of .Net.

For the most par the FileSystemWatcher does a reasonably good job. That is until you start adding lots of files at once to the directories etc that are being 'watched'. That's when all hell breaks loose and you get lovely error messages like:
System.IO.InternalBufferOverflowException: Too many changes at once in directory
So, there are a few things that you can do to address these issues, the first is to make the Buffer bigger by setting  the FileSystemWatcher.InternalBufferSize property to something larger than its default 8192, with the caveat that it comes from non-paged memory, so it is never swapped when the process is running and it should be in multiples of the page size of where ever you're running it.

Setting the buffer did help us eliminate some of the problems, but as I tested more and more aggressively, trying to check memory usage of our new feature and its robustness, it was still quite easy to replicate the buffer overflow. The next plan is to catch only the file changed events and then trigger our program to go and check all the directories and do its thing. Supposedly only File changed events need to be used as when a file is created it is done so in steps, first created and then  the contents added, thus it triggers two events, Added and then modified. I'm not sure how well this'll work when deleting as that seems like it should only take a single operation.

While the buffer overflow is still a problem it is not insurmountable and once the program gets passed the test bench is unlikely to cause a problem. What is more of a problem is the BSOD that I started getting when I started testing the memory usage and processor impact of having many, many folders in our monitored directory. After four BSODs in controlled circumstances I was pretty sure it was related to the FileSystemWatcher when I was adding large numbers of folders to the monitored directory (with an awesomely hacky batch script, for loop + xcopy = win).

So, after spending the appropriate amount of time googling and wandering around Stackoverflow I determined that I had to dig out the minidumps (C:\Windows\Minidump) and have a look around. Right, so I naively double click on it only for Visual Studio 2010 to spit out this useless message.
Ahh, so how about I actually setup the windows debugging tools, which you can grab from here. So I installed it and then tried to run it, what I need the Symbols? Hmmm, so I pointed at the windows symbol server and run the "!analyze -v" command and get another useful error message.

Debugging Details:
------------------
***** Kernel symbols are WRONG. Please fix symbols to do analysis.
Right, so I need symbols for Windows Server 2008R2 specifically, so I find my way to the download and start downloading them, it just happens to be that this 400Mb download is the slowest that I've ever had from a MS server. Actually thats when I started writing this blog post, waiting for the symbols to download. So I install the symbols and run it again.

I get the same error! WTF? I decide I better read into the details a bit more.

Unable to load image \SystemRoot\system32\DRIVERS\eamonm.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for eamonm.sys
*** ERROR: Module load completed but symbols could not be loaded for eamonm.sys
 It was just telling me it couldn't find the symbols for 'eamonm.sys', when I actually look into the output a bit more it starts to make sense it is eamonm.sys that is causing me grief. This it turns out is one of the drivers for ESET's Nod32 Antivirus software, so apparently it is the antivirus software that is causing me problems. To confirm this I disabled the real time file system protection in Nod32.

Then I bludgeoned the directory with lots and lots of changes and then lots and lots more changes and it all looked good. Then I remembered that I needed to have my program running as well. Whoops! Better clear out the directory and do it again. Damn it looks like I've just been hit with the curse of the SSD, really slow delete times, been meaning to look into that as well.....


Ok, so program running, real time protection disabled and.... the problem is gone. Now to make sure excluding just that directory also fixes it. Well that and making sure no one runs the app I'm working on and adds 200 files to the folder....

For all of you that care (and if you're still reading when you get to here) here is the output from the analysis of the minidump.
MODULE_NAME: eamonm

FAULTING_MODULE: fffff80001649000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  4c18e225

READ_ADDRESS:  0000000000001f68 

CURRENT_IRQL:  0

FAULTING_IP: 
eamonm+833b
fffff880`0482e33b 493908          cmp     qword ptr [r8],rcx

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR:  0xD1

LAST_CONTROL_TRANSFER:  from fffff800016b8b69 to fffff800016b9600

STACK_TEXT:  
fffff880`06ff1118 fffff800`016b8b69 : 00000000`0000000a 00000000`00001f68 00000000`00000002 00000000`00000000 : nt+0x70600
fffff880`06ff1120 00000000`0000000a : 00000000`00001f68 00000000`00000002 00000000`00000000 fffff880`0482e33b : nt+0x6fb69
fffff880`06ff1128 00000000`00001f68 : 00000000`00000002 00000000`00000000 fffff880`0482e33b fffff880`048429ee : 0xa
fffff880`06ff1130 00000000`00000002 : 00000000`00000000 fffff880`0482e33b fffff880`048429ee 00000000`00000000 : 0x1f68
fffff880`06ff1138 00000000`00000000 : fffff880`0482e33b fffff880`048429ee 00000000`00000000 00000000`00000000 : 0x2


STACK_COMMAND:  .bugcheck ; kb

FOLLOWUP_IP: 
eamonm+833b
fffff880`0482e33b 493908          cmp     qword ptr [r8],rcx

SYMBOL_NAME:  eamonm+833b

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  eamonm.sys

Friday, August 27, 2010

Lets hold another election so that I can vote for these guys!

The missing party in Australian politics:
"My vote - and I suspect the votes of many others - would lie with a new party, headed by the likes of Turnbull or Rudd (although not necessarily them), that is truly of the 21st century. This would be a party that embraces the market, although believes in regulation to prevent exploitation by employers and big business. It would believe in sustainability and climate change, yet would encourage innovation and enterprise to solve these problems. It would be tolerant of cultural diversity, comfortable with same-sex marriage and strongly secular. It would believe that the government plays a crucial role in society, and that we need to pay sufficient tax for it to execute that role effectively, but government should be only as big as necessary and as small as possible. Moreover, the party wouldn't be tied to unions or the church."

This is an excerpt from an article in The Drum that someone just sent to me and I must say it appeals greatly to me. I find my self displeased with both Liberal and Labor and a new party that targets other like minded people could be a great success. I see the Greens attempting to fashion themselves as the funky party for the youth etc, but their extremism is their downfall, environmentalism needs to be balanced against the fiscal realities of the world. Perhaps if they tempered some of their views and policies they would become the political party for the under 40s.

Thursday, August 26, 2010

The problem with Green Homes

Some of you may have seen Scott Adams' latest article talking about his trials and tribulations as a builder of a green home. Basically he concludes that it can be done but there are countless obstacles in the way, from planning commissions to contractors that don't want to do a risky project.

While these are problems, I think they'll be over come with time. The largest problem that is stopping more green homes being built is home owner ignorance and this is being perpetuated by the contract building firms.

There is someone I know that is building a house at the moment and they didn't know what eaves where and how they could be used to ensure that summer sun stayed out but winter sun got in, thus ensuring better temperature control in the house. Now I don't think it is their job to go out there and completely inform themselves of every aspect of building a house (although I would totally be out there trying to find out everything that I could, but I am an engineer that thinks too much information is never enough) but it should be the job of the building firm that they are engaging to build their house.

It should be a requirement that the builders provide information on things like this to the future house owner. It should also be a requirement that they can't build a house without eaves or any other environmentally ridiculous ideas, but that is another matter, with the push for mandatory 6 star ratings failing in the back rooms of government once again. Even more so, for companies etc that have a heap of model homes etc surely they could provide modelling etc that shows the new home owner what the effects of each inclusion or exclusion has on the energy efficiency and heating and cooling costs for the new home. 

The owner needs to be able to make an informed decision and it is such modelling and cost benefit analysis that gives the home owner the ability to make the best possible decision.

Monday, August 23, 2010

Wednesday, August 18, 2010

Patents are Screwed! US Patent App: 0080270152

Ok, so there are plenty of articles floating around in the ether about the uselessness of the patent system in the US. I'd like to point out a couple that really bring it home for me.

The first is United States Patent Application: 0080270152 a patent on how to patent something that a competitor needs for a competing product and then use this to obtain a cross licensing agreement. Not only is it dodgy business practice at best but it hardly seems patentable.

Then there is the admission from James "Father of Java" Gosling that the engineers at Sun pretty had a competition to get the goofiest patent, here's a quote from his blog (via Zdnet).
There was even an unofficial competition to see who could get the goofiest patent through the system. My entry wasn't nearly the goofiest.
And I have to admit that we had similar competitions at my old place of employ. We tried to get a lot crazy patents idea through, and we also spent a lot of time discussing the silliness of the entire process.

And finally there are the continuous stream of stories of software and other patents coming out of Apple such as this one from Go Rumours. It's not that I think Apple is doing the wrong thing, in fact it would be stupid of them not to be out there patenting everything that they can think of.

The problem is that too many patents that should be chucked out get approved, too many patents that have prior art, are so obvious that everyone else with less patent lawyers sitting around have skipped past and too many patents that are on things that should be covered by copyright and not patents. Copyright is another legal construct that needs a serious overhaul.

So, what should we do? If you've been paying attention recently you would have seen a whole lot of stories about a P=NP proof, that was released. The merits of the proof seem to have been beaten down now but the process by how this occurred shows a lot of promise. It was peer reviewed by the most savage group of peers in existence, every mathematician with an internet connection. This led to the paper being quickly reviewed in a way much better than the old peer review via journal process.

How about for patents we get a peer review system? Obviously privacy needs to be considered so that ideas are stolen. Perhaps there is a pool of respected representatives in a given field. They each have the ability to review the patent application, subject to NDAs of course, with a system in place that allows them to be selected or opt in and out of reviewing a particular patent based on collection of categories, products etc that their companies work in. There are some obvious kinks to be worked out, how does one get the patent application reviewed by peers in the field without those peers getting influenced by ideas etc that they have seen in the patent. How about we start with retired people in the field, or people that have moved on to different areas of expertise.

Obviously this is all a work in progress, the ideas need to be mulled over by people smarter than me. In the mean time I am going to see if I can patent a patent vetting system so I can make some cash of it when the Patent Office starts using it!

Tuesday, August 17, 2010

Education, Terrorism and Floods

In the face of floods of an "unprecedented" scale in Pakistan the aid agencies are seeing a pitiful and underwhelming response.

Why is this?

Well it could have a lot to do with the less than spectacular spending that occurred after the 2005 earthquakes, luxury cars appearing in the driveways of people charged with distributing aid, money disappearing etc. Or it could be because the death toll is relatively low.

How about the perspective is changed slightly?

There are more people who have had their homes destroyed or have been displaced than there are people in Australia. We whine about our housing shortage, but can you even imagine our entire country being without housing overnight?

But all of this is not the main reason that we should be trying to help out our fellow human beings in Pakistan, the main reason is fundamentalists, fundamentalists that are aligned with extreme groups like the Taliban. They are already in Pakistan providing aid and establishing more of a foothold in otherwise non-fundamentalists areas. This is dangerous because the more presence they have in these areas the more people they have the ability to indoctrinate.

But you don't have to take my word for this. I urge you to read Three Cups of Tea: One Man's Mission to Promote Peace . . . One School at a Time it is an incredible story that begins some 20 years ago with a mountain climber that fails to make it to the top of K2 and finishes with a man who single mission in life is to educate Pakistan and the whole of Central Asia. In particular his vision is to educate the girls. It is his belief that through secular education we can address two of the most pressing issues in the world today, poverty and terrorism and he has achieved remarkable success.

So, while an immediate response is needed in Pakistan to address such things as water borne diseases, a long term view is needed to ensure that Pakistan recovers properly from this disaster and moves its population from below to above the poverty line.

So, go buy the book or read about what this guy has said and achieved. If you use the link above a percentage of the sale goes to the Central Asia Institute and to assist in building schools and providing educational supplies to the region. Before you go and read the book think about donating directly to the Central Asia Institute or to one of the various charities collecting for Flood Relief, such as Unicef or World Vision.

Friday, August 13, 2010

Memristors: The undergrad engineers are screwed!

So a while ago there was the announcement that Memristors had finally made the jump from theory into reality. Two years on and we see that there is a great deal of confusion over the implications of the technology and how to teach the technology. In a recent article from IEEE Spectrum: How Do You Teach the Memristor? they conducted an impromptu survey of Universities and besides the occasional mention of memristors in some early course materials the memristor is not studied at all.

The basic reasons for this are two fold (well three fold if you include the fact that courses just aren't updated that often), the main reason is that despite floating around in theory for since the early 70s no-one really knows how to teach them effectively and the second reason is they are non-linear and a pain in the arse to model.

Non-linear circuit modeling is, to say the least, extremely challenging. I can recall in my classes that there was enough trouble getting the whole class to understand Linear Time Invariant Circuits, the primary school version of circuit analysis compared non-linear modelling.

One thing that I'll watch with interest is the development and adoption of memristors in everyday electronics and their teaching in undergrad class rooms and lecture halls around the world. I looking forward to seeing them applied to new and novel applications, but sorry for the poor sods that end up getting stuck attempting to model them in a first year tute.

Wednesday, August 11, 2010

I want my toll back! Follow Up San Fran Parking Meeter

I saw this on Engaget shortly after posting my tollway story and it immediately appealed San Francisco rolls out new smart parking meters with 'demand-responsive pricing' -- Engadget.

This is exactly what I was talking about, albeit in parking meter form. The main component that I like, the pricing based on demand. This should be introduced on the tollways! Even if it is not automatically calculated like the Parking Meters appear to do, big charges during peak hours but then at other times, greatly reduced charges.

Taking the Cross City Tunnel as my continuing example encouraging people to use it outside of peak should be a priority of the Roads and Traffic Authority. When I lived in Sydney I avoided it as much as possible, only catching it when I was in a hurry and knew that the other routes would be congested and slow. However at night, I would always avoid it, $4.56 to save 5 minutes at 1 AM never really appealed to me.

Obviously, the Tollway owners would only ever be convinced of the benefits in purely economic terms, and to be honest if I was a share holder that is exactly how I would want them to think, what perhaps needs to be done is a subsidy from the RTA or in this particular case the City of Sydney Council, it seems to me that as Clover Moore is hell bent on reducing traffic in the city encouraging use of the CCT would assist with it. Subsidizing cross city traffic on Friday and Saturday nights might even reduce incidences in the city, I have had near misses numerous times where a drunken reveller has stepped out in front of my car, or cars around me. Reducing the cars would cut down on that.

Monday, August 9, 2010

I want my toll back! Motorways should be discounted when congested

I was up in Sydney on the weekend for my engagement party, as often happens when I'm up in Sydney there was a run to the airport and as I was running a little late I thought I'd better take the cross city tunnel because its supposed to be quicker.

Boy was that a mistake, no sooner had I entered the tunnel (after driving past no less than 5 electronic traffic advisory signs that said nothing) and past the point of no return than there was a message saying that there was congestion ahead. So, for the next 35 minutes (of a trip that should take less than ten) I crawled along in the tunnel. It was just after the beep of my eTag that the queue of cars required me to stop completely. If I hadn't managed to convince the firetruck to go ahead of me I think I could still be sitting in the tunnel!

Firstly, I was annoyed that none of the earlier signs had warned me (the cynic in me is convinced that they make it so the RTA doesn't change these signs so that people still head into the tunnel and pay the toll) and secondly I was annoyed because I just paid $4.56, or whatever the toll is now up to, to sit in an exhaust filled tunnel breathing stagnant air for half an hour.

So, obviously it would cost them some cash, but how about discounting the toll when things go pear shaped and screw you over in the tunnel. Would it be so hard to halve the toll or discount it even more when such an incident happens, just for the cars that got stuck? While, they are at it sorting that out they should introduce discounted tolls late at night. I once again won't be taking the Cross City Tunnel until I absolutely need to, mainly because I'm annoyed at paying to sit still. If I'd been comped or something then I wouldn't hesitate to use it again.

For Toll ways, such as the Cross City Tunnel that has notoriously low adoption rates, they should be looking at ways to encourage people to use it. Making it half price in the middle of the night would encourage people to use it, and if any of my friends are a good indication, more than twice the people would use it if it was half the price.

Anyway, surely the lost revenue of a couple of thousand cars would be more than offset by the positive publicity and the customers that return later, if they where to let people stuck after an accident travel for free. It's possible that they would be able to find an insurer to underwrite the loss of revenue as well.

Thursday, August 5, 2010

Google falls off the Crest of the Wave

When I awoke this morning (actually a couple of hours after I awoke this morning thanks to an early morning airport run) I was greeted with the news that Google’s Wave project has been canned. Here is the official word over at the Google Blog, basically not enough people where using it.

The general consensus from folks around the internet is that nobody understood how to use it, Tech Crunch seems to think that is was ahead of its time (and totally stole my blog title…)  while Gizmodo is just confused.
From my perspective (someone that spent several years working at a company developing next generation IP technologies, such as Video Telephony and collaboration) Wave was the future. Or rather the user experience that Wave was trying to be was the future. I spent years collaborating with developers, testers and managers spread across the globe, at one point I was working on a team that had people located in Denver (USA West Coast), New Jersey (USA East Coast), Pune (India), Dublin (Ireland) and Sydney (Australia), we needed a tool for our meetings that:
1.       Had audio conferencing to talk to everyone on the call
2.       Had video conferencing so that we could see the reactions of people on the call.
3.       Had the ability to have sidebar chats to discuss small side points while being active in the current call, e.g. insight between local members of a team.
4.       Allowed easy sharing, modification and discussion of documents.
5.       Enabled collaborative white boarding (to allow quick drawings and discussion points to be shared).
6.       Recorded the whole lot so that the members that where located in unfavourable time zones could see the results of the outcome and discussion. As well as allowing review and generation of minutes following the meeting.
Google Wave coupled with our existing technology enabled this, with a bit of work a seamless integration could be achieved. It seems to me that Google targeted the wrong group of users. It was always going to be hard to get a large enough following from the Tech crowd to push it into the mainstream, which is their usual method of developing and promoting new technology. They faced a barrage of other communication systems that people where happily using or adopting because of the simple use cases, Twitter, Email, Facebook Chat, Skype etc. Wave was kind of a combination of all of these crammed into one. Thus making it quite confusing at first (and second) glance, if the use cases weren’t immediately apparent to a user then they probably weren’t going to go to the trouble of signing up. Perhaps if it integrated better with gmail and google docs, these would have pulled people across. I feel the people that would have gotten the most benefit out of it never actually saw it or perhaps even heard about it.
Google have said that they intend to continue to work with some of the technology and to merge it into existing products so I hope that they work to get it into their corporate offering and integrated with Google Docs and Voice. If they do that then all the exciting parts of this technology can live on to see another day.

Tuesday, August 3, 2010

The problem with Cycling in Melbourne

Having cycled to work most days for the best part of a year I figured it was time to vent. There are plenty of articles and discussions floating around that discuss the problems with cycling in Melbourne and for the most part they raise legitimate concerns, the lack of interconnected cycle paths, cyclists slowing motorists down on major roads etc. As a cyclist once you've worked out a good route that minimizes the time spent on major roads and maximizes the use of cycle paths and quiet side streets these issues aren't so bad. You get used to the conditions on the route etc and it fades into oblivion.

Inner Melbourne, for the most part should be an awesome place to cycle, the streets are wide and have wide lanes and for the most part their are few hills, there are also some good paths along the rivers etc. The problem is that there are plenty of streets with tram tracks that should be perfect for riding on, the curb side lanes are wide enough to accommodate parked cars with plenty of room to spare, a few meters, for cyclists to ride in but there is a significant portion of drivers that find it difficult, if not impossible to stay in completely in the middle lane.

On at least half of my weekly bike trips I encounter a car that is so far into the curb side lane that there simply isn't enough room for me to fit past on my bike. I am not counting cars that may be double parked just cars that fail to stay in the middle lane to such an extent that I can't even ride past. What makes this doubly annoying and concerning is that often these cars have just driven past me to get to where they are, so either they are doing it deliberately or they are completely oblivious to the fact that they are sitting a good two meters into the other lane. Aren't they concerned about some one in a parked car opening a door?

When I have to stop completely I will often tap on the window and gesture for the driver to move over. This is greeted by a couple of different reactions, complete shock which tends to indicate a complete lack of awareness of surroundings or anger. Now, I always try to be polite when doing it, could you please move over etc, but I guess some people are just angry.

Just this morning I had an amusing encounter with a father driving a Land Cruiser that, drove past and almost side swiped me and then proceeded to stop with the car more in the curb side lane than in the actual driving lane. When I tapped on the window I got the finger and a big f*** off. When I pointed out that perhaps his wife wouldn't approve of him using such language in front of his children he went a particularly lovely shade of red and then sped off.