Grand Unified Theory of logging
embedded, logging, testing
Solution
You could "instrument" your code in many different ways, everything from start-up/shut-down events to individual machine instruction execution (using a processor emulator). Of all the possibilities, what's worth doing? Don't just do it for the sake of completeness; have a specific goal in mind. A business case if you like, with a benefit you expect to receive. E.g.:
- Insight into CPU task execution times/patterns to enable optimisation (if you need to improve performance).
- Insight into other systems to resolve system integration issues (e.g. what messages is your VoIP box sending and receiving when it connects to a particular peer?)
- Insight into the nature of errors (for field diagnostics)
- Aid in development
- Aid in validation testing
I imagine that there's no grand unified theory of logging, because what you do would depend on many details:
- Quantity of data
- Type of data
- Events
- Streamed audio/video
- Available storage
- Storage speed
- Storage capacity
- Available channels to extract data
- Bandwidth
- Cost
- Availability
- Internet connected 24×7
- Site visit required
- Need to unlock a rusty gate, climb a ladder onto a roof, to plug in a cable, after filling out OHS documentation
- Need to wait until the Antarctic winter is over and the ice sheets thaw
- Random access vs linear access (e.g. if you compress it, do you need to read from the start to decompress and access some random point?)
- Need to survive error conditions
- Watchdog reboots
- Possible data corruption
- Due to failing power supply
- Due to unreliable storage media
- Need to survive a plane crash
As for ASCII vs binary, I usually prefer to keep the logging simple, and put any nice presentation in a PC application that decodes the data. It's usually easier to create a user-friendly presentation in PC software (written in e.g. Python) rather than in the embedded system itself.
Problem
Is their a Grand Unified Theory of logging? Shall we develop one? Question (just to show this is not a discussion :), how can I improve on the following? (note that I live mainly in the embedded world, but non-embedded suggestions are also welcome) How do you log, when do you log, what do you log, what do you do with log files? How do you log - I generally have macros, #ifdef TESTING, sort of thing. They write to RAM and a low priority process writes them out when the system is idle (using UDP, since I do embedded systems) When do you log - same as voting, early and often. At every (in)significant program event, I log at varying levels. Events received, transaction succeed/fail, data updated, etc What do you log - Fatal/Error/Warning/Info/Debug/Trace is covered in When to use the different log levels? What do you do with log files - 1) keep them (in CVS), both pass and fail 2) capture everything and filter later in case I can't repeat a problem. I have tools to filter the log by "level" (Fatal/Error/etc), process, file, etc. And to draw message sequence charts, dump data structures, draw histograms of memory usage - what am I missing? Hmmm, binary or ascii log file format? Ascii is bulkier, but binary requires more processing. I have done both, currently I use ascii Question - did I miss anything, and how can I improve on this?