I worked on a taxi booking and dispatch system, written in c, and running on dos with custom networking via RS232. This system was installed at around 300 locations around the UK, and on one fateful day, every installation crashed.
It came down to me to find and fix the problem, and it was subtle. The clue lay in the fact that all of the sites that crashed did so within about a minute of one another.
Turns out that some of the old, old sections of the software had been written by the MD, who, despite referring to himself as 'the emperor of c', was in fact an atrocious programmer.
The actual trigger was the comms system looking at a byte that determined as to whether a message had been received. This byte was set to the character 'A' if a message was received. It just so happened that the first byte of the current value of the number of seconds since 1970 evaluated to 'A', and had been written into that memory location a negative index into an array that hadn't been initialised.
This negative index into an array that shouldn't have been empty caused a section of memory to be overwritten that made the comms system think that it had received a packet. This snowballed quickly, and took down the system within about five seconds of boot.
Took the best part of two days to track down, and, of course, it was everyone else's fault but the emperors.
It came down to me to find and fix the problem, and it was subtle. The clue lay in the fact that all of the sites that crashed did so within about a minute of one another.
Turns out that some of the old, old sections of the software had been written by the MD, who, despite referring to himself as 'the emperor of c', was in fact an atrocious programmer.
The actual trigger was the comms system looking at a byte that determined as to whether a message had been received. This byte was set to the character 'A' if a message was received. It just so happened that the first byte of the current value of the number of seconds since 1970 evaluated to 'A', and had been written into that memory location a negative index into an array that hadn't been initialised.
This negative index into an array that shouldn't have been empty caused a section of memory to be overwritten that made the comms system think that it had received a packet. This snowballed quickly, and took down the system within about five seconds of boot.
Took the best part of two days to track down, and, of course, it was everyone else's fault but the emperors.