Look for gaps in time stamps

bash, linux, shell

Solution

This uses GNU `date` to interpret the time. The code below reads from a file named `file` and, looping over each line, checks to see if a time gap of more than 300 seconds exists:

while read newline
do
    new=$(date -d "$(echo "$newline" | sed -E 's/-([0-9][0-9])\.([0-9][0-9])\./ \1:\2:/')" '+%s')
    if [ "$old" ] && (( $new - $old > 300))
    then
        printf "%4i seconds gap before %s" "$((new - old))" "$newline"
    fi
    old=$new
done <file

Example

Let's consider this test file:

$ cat file
2014-11-11-04.01.05.000000
2014-11-11-04.03.33.000000
2014-11-11-04.08.31.000000
2014-11-11-04.13.32.000000
2014-11-11-05.13.33.000000

The above script finds the two gaps that exceed 5 minutes:

 301 seconds gap before 2014-11-11-04.13.32.000000
3601 seconds gap before 2014-11-11-05.13.33.000000

Observe that this can detect gaps as small as 5 minutes and 1 second. It also detects the 1-hour gap even though the minutes didn't change.

How it works

To understand the time format in all its potential complexity, the GNU `date` utility is used to convert time to seconds-since-epoch. This is done simply as:

$ date -d '2014-11-11 04:01:05.000000' '+%s'
1415707265

My `date` (newer versions may differ) does not support the exact format of our input:

$ date -d '2014-11-11-04.01.05.000000' '+%s'
date: invalid date `2014-11-11-04.01.05.000000'

However, we can use `sed` to make the format look like the one above which worked:

$ date -d "$(echo "$newline" | sed -E 's/-([0-9][0-9])\.([0-9][0-9])\./ \1:\2:/')" '+%s'
1416384000

Next, it is a matter of getting those seconds into a shell variable. To do that, command substitution is used:

new=$(date -d "$(echo "$newline" | sed -E 's/-([0-9][0-9])\.([0-9][0-9])\./ \1:\2:/')" '+%s')

With the most recent time in the variable `new`, we can see if more than 5 minutes (300 seconds) has elapsed since the last time (stored in the variable `old`) and, if so, print out a message:

if [ "$old" ] && (( $new - $old > 300))
then
    printf "%4i seconds gap before %s\n" "$((new - old))" "$newline"
fi

The first test above, `[ "$old" ]` makes sure that the variable `old` has been defined. It will always be defined except for the first line that we read in. Hence, the effect of the test `[ "$old" ]` is to skip that first line.

The second test is `(( $new - $old > 300))`. This simply determines if more than 300 seconds has elapsed since the previous line.

What to do if your `sed` does not support `-E`

For GNU `sed`, `-E` means extended regex format. On Mac OSX, `-r` would be used in its place. If you are on an old linux system that doesn't support `-E`, we can try using basic regular expression syntax. Try:

$ echo 2014-11-11-04.01.05.000000 | sed  's/-\([0-9][0-9]\)\.\([0-9][0-9]\)\./ \1:\2:/'
2014-11-11 04:01:05.000000

And:

$ date -d "$(echo 2014-11-11-04.01.05.000000 | sed  's/-\([0-9][0-9]\)\.\([0-9][0-9]\)\./ \1:\2:/')" '+%s'
1415707265

As you can see, the difference between basic and extended regex is which characters have to be escaped.

If that works, then use:

while read newline
do
    new=$(date -d "$(echo "$newline" | sed  's/-\([0-9][0-9]\)\.\([0-9][0-9]\)\./ \1:\2:/')" '+%s')
    if [ "$old" ] && (( $new - $old > 300))
    then
        printf "%4i seconds gap before %s" "$((new - old))" "$newline"
    fi
    old=$new
done <file

Problem

I have the following time stamps in a file that goes on for maybe 24 hours or slightly more: ``` 2014-11-11-04.01.05.000000 2014-11-11-04.03.33.000000 2014-11-11-04.06.02.000000 2014-11-11-04.08.31.000000 ``` The gaps between each time stamp should be less than 5 minutes. How can I put a simple bash shell script together to parse through the file and tell me if there are gaps greater than 5 minutes? The simplest way I see would be to subtract the next line with the previous line. But I'm not really good with bash shell script. Can anyone help?

Original source