List of new files downloaded with wget

bash, wget

Solution

You can use wget's `-o` flag to pipe the standard output to a logfile. The logfile will have the same format as the regular output to a terminal, for example:

--2012-06-28 17:57:13--  http://cdn.sstatic.net/stackoverflow/img/sprites.png
Resolving cdn.sstatic.net (cdn.sstatic.net)... 67.201.31.70
Connecting to cdn.sstatic.net (cdn.sstatic.net)|67.201.31.70|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16425 (16K) [image/png]
Saving to: `sprites.png'

0K .......... ......                                     100%  131K=0.1s

2012-06-28 17:57:13 (131 KB/s) - `sprites.png' saved [16425/16425]

If you pipe this file through egrep as `egrep -e "--" logfile.txt` you will get only the lines specifying which files were downloaded, with timestamps.

--2012-06-28 17:57:13--  http://cdn.sstatic.net/stackoverflow/img/sprites.png

If you wish, you can then pipe it to cut `egrep -e "--" logfile.txt|cut -d ' ' -f 4` to get only to downloaded files.

http://cdn.sstatic.net/stackoverflow/img/sprites.png

You should add error-checks in here as well, but this is the basic outline.

Problem

I'm using wget to download new files from a FTP server. The new files that have been downloaded need to be processed by another script. ``` wget -N -r ftp://server/folder ``` So my question is: How do i get a list of all files that wget has been downloaded? Thanks in advance!

Original source