How do I pick random unique lines from a text file in shell?

awk, sed, shell

Solution

If `jot` is on your system, then I guess you're running FreeBSD or OSX rather than Linux, so you probably don't have tools like `rl` or `sort -R` available.

No worries. I had to do this a while ago. Try this instead:

$ printf 'one\ntwo\nthree\nfour\nfive\n' > input.txt
$ cat rndlines
#!/bin/sh

# default to 3 lines of output
lines="${1:-3}"

# default to "input.txt" as input file
input="${2:-input.txt}"

# First, put a random number at the beginning of each line.
while read line; do
  printf '%8d%s\n' $(jot -r 1 1 99999999) "$line"
done < "$input" |
sort -n |               # Next, sort by the random number.
sed 's/^.\{8\}//' |     # Last, remove the number from the start of each line.
head -n "$lines"        # Show our output

$ ./rndlines input.txt 
two
one
five
$ ./rndlines input.txt 
four
two
three
$

Here's a 1-line example that also inserts the random number a little more cleanly using awk:

$ printf 'one\ntwo\nthree\nfour\nfive\n' | awk 'BEGIN{srand()} {printf("%8d%s\n", rand()*10000000, $0)}' | sort -n | head -n 3 | cut -c9-

Note that different versions of `sed` (in FreeBSD and OSX) may require the `-E` option instead of `-r` to handle ERE instead or BRE dialect in the regular expression if you want to use that explictely, though everything I've tested works with escapted bounds in BRE. (Ancient versions of `sed` (HP/UX, etc) might not support this notation, but you'd only be using those if you already knew how to do this.)

Problem

I have a text file with an unknown number of lines. I need to grab some of those lines at random, but I don't want there to be any risk of repeats. I tried this: ``` jot -r 3 1 `wc -l<input.txt` | while read n; do awk -v n=$n 'NR==n' input.txt done ``` But this is ugly, and doesn't protect against repeats. I also tried this: ``` awk -vmax=3 'rand() > 0.5 {print;count++} count>max {exit}' input.txt ``` But that obviously isn't the right approach either, as I'm not guaranteed even to get `max` lines. I'm stuck. How do I do this?

Original source