One-liner to split very large directory into smaller directories on Unix
filesystems, shell, unix, wget
Solution
Another option:
i=1;while read l;do mkdir $i;mv $l $((i++));done< <(ls|xargs -n100)
Or using `parallel`:
ls|parallel -n100 mkdir {#}\;mv {} {#}
`-n100` takes 100 arguments at a time and `{#}` is the sequence number of the job.
Problem
How do you to split a very large directory, containing potentially millions of files, into smaller directories of some custom defined maximum number of files, such as 100 per directory, on UNIX? Bonus points if you know of a way to have `wget` download files into these subdirectories automatically. So if there are 1 million `.html` pages at the top-level path at `www.example.com`, such as ``` /1.html /2.html ... /1000000.html ``` and we only want 100 files per directory, it will download them to folders something like ``` ./www.example.com/1-100/1.html ... ./www.example.com/999901-1000000/1000000.html ``` Only really need to be able to run the UNIX command on the folder after `wget` has downloaded the files, but if it's possible to do this with `wget` as it's downloading I'd love to know!