Using select/poll/kqueue/kevent to watch a directory for new files
asynchronous, freebsd, polling, python
Solution
After running profiling I'm seeing a lot of time spent in the sleep, and I'm wondering if I should change this to use polling instead.
Looks like you already do synchronous polling, by checking the state at regular intervals. Don't worry about the time "spent" in `sleep`, it won't eat CPU time. It just passes control to the operating system which wakes the process up after a requested timeout.
You could consider asynchronous event loop using a library that listens to filesystem change notifications provided by the operating system, but consider first if it gives you any real benefits in this particular situation.
Problem
In my app I need to watch a directory for new files. The amount of traffic is very large and there are going to be a minimum of hundreds of new files per second appearing. Currently I'm using a busy loop with this kind of idea: ``` while True: time.sleep(0.2) if len(os.listdir('.')) > 0: # do stuff ``` After running profiling I'm seeing a lot of time spent in the sleep, and I'm wondering if I should change this to use polling instead. I'm trying to use one of the available classes in `select` to poll my directory, but I'm not sure if it actually works, or if I'm just doing it wrong. I get an fd for my directory with: ``` fd = os.open('.', os.O_DIRECT) ``` I've then tried several methods to see when the directory changes. As an example, one of the things I tried was: ``` poll = select.poll() poll.register(fd, select.POLLIN) poll.poll() # returns (fd, 1) meaning 'ready to read' os.read(fd, 4096) # prints largely gibberish but i can see that i'm pulling the files/folders contained in the directory at least poll.poll() # returns (fd, 1) again os.read(fd, 4096) # empty string - no more data ``` Why is poll() acting like there is more information to read? I assumed that it would only do that if something had changed in the directory. Is what I'm trying to do here even possible? If not, is there any other better alternative to `while True: look for changes` ?