A program frozen at the point of failure
detection may have already erased the evidence
that explains the failure.  

'''Therefore''',
trap each statement of the program and record
crucial information in a CircularBuffer.  Inspect
this buffer when the program halts to determine the 
circumstances leading to the failure.

The concept is similar to the flight voice and 
data recorders recovered from the wreckage of 
airplane crashes.  Information is easy to capture,
but, since failures are rare, all but the most recent
records are discarded.

I used a variation on this concept 
to hunt bugs in multi-microprocessor 
hardware and software.
After each statement in the program
I mechanically inserted ...

	* A call on a trap handler
	* the source line number
	* a pointer to a table of in-scope variables

As inserted, the calls were to a subroutine
that would freeze the program, record and display the 
line number, and wait for interactive requests
which included ...

	* display in-scope variables
	* disassemble code for the statement (to find compiler bugs)
	* examine system message queues
	* display the buffer of recorded line numbers
	* modify the trap to call a more passive handler

In essence, the program started with a breakpoint on 
every statement and this last interactive command 
removed them.  Except it didn't remove the breakpoints
completely.  It changed them to breakpoints that silently
recorded the progress of the program in the circular buffer.

I would step through a loop once watching
each statement go by.  The second time through the loop
I would remove the breakpoints as I went.  Then, zip, I 
would be out of the loop, at the next statement's breakpoint.
A quick check of the buffer would make sure the final
passes through the loop were as well behaved as the first.

This may all sound trivial compared to the heroics that
modern debuggers will perform on your behalf.  Where it 
saved me was when the systems crashed in bizarre ways.  I
was never left without a clue.  And, I could trust the
clues because of the absolute simplicity of the debugging
tool.
-- WardCunningham

''You're my hero, Ward. I have probably used plenty of stuff you touched over the years. -- MartySchrader, long time and extensive user of Tektronix instruments, development systems, and software''

-------

Hugh Blair-Smith Describes the same technique used to debug the ApolloGuidanceComputer in the 60's.

''Another function served by these auxiliary erasable memories was called the "Coroner," which recorded every instruction executed, with its inputs and results, writing over the oldest such record when it filled up. When a program "crashed," you could punch out a full record of what it was doing in most of its last second and analyze the problem at your ease. I have often wished that PCs offered such an advanced feature.''

http://hrst.mit.edu/hrs/apollo/public/blairsmith3.htm