Ruby file handle management (too many open files)

file-io, limits, ruby

Solution

Use dtrace or strace or whatever equivalent is on your system, and find out exactly what files are being opened.

Note that these could be sockets.

I agree that the code you have pasted does not seem to be capable of causing this problem, at least, not without a rather strange concurrency bug as well.

Problem

I am performing very rapid file access in ruby (2.0.0 p39474), and keep getting the exception `Too many open files` Having looked at this thread, here, and various other sources, I'm well aware of the OS limits (set to `1024` on my system). The part of my code that performs this file access is mutexed, and takes the form: ``` File.open( filename, 'w'){|f| Marshal.dump(value, f) } ``` where `filename` is subject to rapid change, depending on the thread calling the section. It's my understanding that this form relinquishes its file handle after the block. I can verify the number of `File` objects that are open using `ObjectSpace.each_object(File)`. This reports that there are up to 100 resident in memory, but only one is ever open, as expected. Further, the exception itself is thrown at a time when there are only 10-40 `File` objects reported by `ObjectSpace`. Further, manually garbage collecting fails to improve any of these counts, as does slowing down my script by inserting `sleep` calls. My question is, therefore: - Am I fundamentally misunderstanding the nature of the OS limit---does it cover the whole lifetime of a process? - If so, how do web servers avoid crashing out after accessing over `ulimit -n` files? - Is ruby retaining its file handles outside of its object system, or is the kernel simply very slow at counting 'concurrent' access? Edit 20130417: `strace` indicates that ruby doesn't write all of its data to the file, returning and releasing the mutex before doing so. As such, the file handles stack up until the OS limit. In an attempt to fix this, I have used `syswrite`/`sysread`, synchronous mode, and called `flush` before `close`. None of these methods worked. My question is thus revised to: Why is ruby failing to close its file handles, and how can I force it to do so?

Original source