How to Think About Application Logging

A colleague of mine talked about how Splunk has changed our way of thinking about application logging. The analogy he used was that we used to think about logging as storytelling (i.e. Job1 is starting, Doing task 1, Doing task 2, ..., Job1 finished). It is much like "In the beginning... something happened... then something else happened... The End." Splunk has gotten us into the habit of thinking about logging as a series of statements that can be grouped together with the Splunk search tools (i.e. Job=1, Message=Doing something; Job=1, Message=Doing something else; ...). This is a powerful shift in thinking that allows us to better troubleshoot and diagnose system issues.