gawk / awk: piping date to getline *sometimes* won't work

Question

I'm attempting to convert dates from one format to another: From e.g. "October 29, 2005" to 2005-10-29. I have a list of 625 dates. I use Awk. The conversion works -- most of the time. Hovewer, sometimes the conversion won't happen at all, and the variable supposed to hold the (converted) date remains undefined. This always happens with the exact same rows. Running `date' explicitly (from the Bash shell) on the dates of those weird rows works fine (the dates are properly converted). -- It's not the textual contents of those rows that matters. Why this behavior, and how can I fix my script? Her it is: ``` awk 'BEGIN { FS = "unused" } { x = "undefined"; "date \"+%Y-%m-%d\" -d " $1 | getline x ; print $1 " = " x }' uBXr0r15.txt \ > bug-out-3.txt ``` If you want to reproduce this problem: - Download this file: uBXr0r15.txt. - Run the Awk skript. - Search for "undefined" in bug-out-3.txt. ("undefined" found 122 times, on my computer.) Then you could run the script again, and (on my compute…

Accepted Answer

Whenever you open a pipe or file for reading or writing in `awk`, the latter will first check (using an internal hash) whether it already has a pipe or file with the same name (still) open; if so, it will reuse the existing file descriptor instead of reopening the pipe or file.

In your case, all entries which end up as `undefined` are actually duplicates; the first time that they are encountered (i.e. when the corresponding command `date "..." -d "..."` is first issued) the proper result is read into `x`. On subsequent occurrences of the same date, `getline` attempts to read a second, third etc. lines from the original `date` pipe, even though the pipe has been closed by `date`, resulting in `x` no longer being assigned.

From the `gawk` man-page:

NOTE: If using a pipe, co-process, or socket to getline, or from print or printf within a loop, you must use close() to create new instances of the command or socket. AWK does not automatically close pipes, sockets, or co-processes when they return EOF.

You should explicitly `close` the pipe every time after you have read `x`:

```
close("date \"+%Y-%m-%d\" -d " $1)
```

Incidentally, would it be OK to `sort` and `uniq` `uBXr0r15.txt` before piping into `awk`, or do you need the original ordering/duplication?

gawk / awk: piping date to getline sometimes won't work

Solution

Problem