(a) The CAN bus (which reports wheel velocity and position of the steering wheel among other things) micro-controller hardware stopped sending interrupts which caused the main computer to think we were stopped. As far as I can tell this is just a straight up hardware bug with the Philips ARM chip we were using. This caused the accelerator controller to floor it because it was only seeing the last CAN message we ever got which happened to be zero velocity. Same thing with the steering (hence the big swerve).
or
(b) I failed to consider the contingency of not getting any CAN interrupts (either because the of the [very intermittent] hardware bug or because the connector got disconnected--turns out the symptoms are the same) and didn't have any code written to deal with it. Guess what I wrote that night. :-) Luckily I had all day to think about it while a new tire was fetched and by the time I figured out what was going on it took about 10 minutes to code the fix: a watchdog timer that shuts the world down if there are no inputs from the sensors for any reason.
It seems obvious in retrospect but when things are working you sometimes forget about weird failure cases.
Any chance of getting 'second opinion' style sensors in there that can provide you with sanity checks ? Such as 'GPS reports movement, but wheel sensors do not, we have a problem ?'
That way you can avoid a paralysis of the control software until the vehicle has really come to a halt.
I read somewhere about realtime applications that do something like this but redundant sensors holding a consensus polling algorithm. Have three sensors reporting the same thing and if they are not all in agreement within some kind of delta then go into some kind of limp mode or have the two sensors in agreement be the ones that the system uses for it's algorithms. I cannot recall where I read it though.
Me too. The back of my brain is telling me it was for some sort of plane control software? Maybe? An interesting tidbit that I recall was that they used different manufacturers to hedge their bets against bugs.
That sounds about right. I seem to recall that it was for aerospace too. Maybe NASA? Something about zero defect software. I cannot for the life of me find the article right now though. I also vaguely remember that it was a HN submission too.
