How do I find out what inotify watches have been registered?

inotify, linux

Solution

Oct 31 2022 update

While my script below works fine as it is, Michael Sartain implemented a native executable that is much faster, along with additional functionality not present in my script (below). Worth checking out if you can spend a few seconds compiling it! I have also added contributed some PRs to align the functionality, so it should be pretty 1:1, just faster. Upvote his answer on the Unix Stackexchange.

Original answer with script

I already answered this in the same thread on Unix Stackexchange as was mentioned by @cincodenada, but thought I could repost my ready-made answer here, seeing that no one really has something that works:

I have a premade script, `inotify-consumers`, that lists the top offenders for you:

   INOTIFY   INSTANCES
   WATCHES      PER   
    COUNT     PROCESS   PID USER         COMMAND
------------------------------------------------------------
   21270         1       11076 my-user    /snap/intellij-idea-ultimate/357/bin/fsnotifier
     201         6           1 root       /sbin/init splash
     115         5        1510 my-user    /lib/systemd/systemd --user
      85         1        3600 my-user    /usr/libexec/xdg-desktop-portal-gtk
      77         1        2580 my-user    /usr/libexec/gsd-xsettings
      35         1        2475 my-user    /usr/libexec/gvfsd-trash --spawner :1.5 /org/gtk/gvfs/exec_spaw/0
      32         1         570 root       /lib/systemd/systemd-udevd
      26         1        2665 my-user    /snap/snap-store/558/usr/bin/snap-store --gapplication-service
      18         2        1176 root       /usr/libexec/polkitd --no-debug
      14         1        1858 my-user    /usr/bin/gnome-shell
      13         1        3641 root       /usr/libexec/fwupd/fwupd
...

   21983  WATCHES TOTAL COUNT

INotify instances per user (e.g. limits specified by fs.inotify.max_user_instances): 

INSTANCES    USER
-----------  ------------------
41           my-user
23           root
1            whoopsie
1            systemd-ti+
...

Here you quickly see why the default limit of 8K watchers is too little on a development machine, as just WebStorm instance quickly maxes this when encountering a `node_modules` folder with thousands of folders. Add a webpack watcher to guarantee problems ...

Even though it was much faster than the other alternatives when I made it initially, Simon Matter added some speed enhancements for heavily loaded Big Iron Linux (hundreds of cores) that sped it up immensely, taking it down from ten minutes (!) to 15 seconds on his monster rig.

Later on, Brian Dowling contributed instance count per process, at the expense of relatively higher runtime. This is insignificant on normal machines with a runtime of about one second, but if you have Big Iron, you might want the earlier version with about 1/10 the amount of system time :)

How to use

`inotify-consumers --help` To get it on your machine, just copy the contents of the script and put it somewhere in your `$PATH`, like `/usr/local/bin`. Alternatively, if you trust this stranger on the net, you can avoid copying it and pipe it into bash over http:

$ curl -s https://raw.githubusercontent.com/fatso83/dotfiles/master/utils/scripts/inotify-consumers | bash 

       INOTIFY
       WATCHER
        COUNT     PID USER     COMMAND
    --------------------------------------
        3044   3933 myuser node /usr/local/bin/tsserver
        2965   3941 myuser /usr/local/bin/node /home/myuser/.config/coc/extensions/node_modules/coc-tsserver/bin/tsserverForkStart /hom...

        6990  WATCHES TOTAL COUNT

How does it work?

For reference, the main content of the script is simply this (inspired by this answer)

find /proc/*/fd \
    -lname anon_inode:inotify \
    -printf '%hinfo/%f\n' 2>/dev/null \
    \
    | xargs grep -c '^inotify'  \
    | sort -n -t: -k2 -r

Changing the limits

In case you are wondering how to increase the limits

$ inotify-consumers --limits 

Current limits
-------------
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288


Changing settings permanently
-----------------------------
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p # re-read config

Problem

I have my inotify watch limit set to 1024 (I think the default is 128?). Despite that, yeoman, Guard and Dropbox constantly fail, and tell me to up my inotify limit. Before doing so, I'd like to know what's consuming all my watches (I have very few files in my Dropbox). Is there some area of /proc or /sys, or some tool I can run, to find out what watches are currently registered?

Original source