HHVM fastcgi + Nginx performance fluctuations
hhvm, mysql, nginx, performance, php
Solution
So we managed to figure it out, it was a combined problem of HHVM and the application we are using.
HHVM produces quite a bit of perf-****.map files and there where also heaps of sess_ files. In total there where over 8 million files in the /tmp directory. After removing those our problems went away and almost all our requests are significantly improved in performance. Also the session files shouldn't have been there in the first place but this was a problem which is already fixed.
Now we disabled the generation of the perf- files by adding -vEval.PerfPidMap to our startup arguments.
Problem
Currently we started using HHVM in a production envirionment and so far almost all results are pretty inpressive. Our overall transaction rate is greatly improved compared to PHP-FPM with APC. Almost all requests are under 500ms however every couple of requests (5 to 10 or so) results in a request time of 2 to even 5 seconds. The page requested does not seem to make any difference and also requesting the same page over and over again will trigger this behavior within a couple of requests. We are running HHVM in server mode with the following command line options: ``` /usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true ``` We are running nginx for the webserver with these relevant configurations (I am sorry if i forgot something importand here). ``` fastcgi_buffer_size 128k; fastcgi_buffers 256 16k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_read_timeout 240; fastcgi_intercept_errors on; ``` The server has 128GB of memory and 24 cores (hyperthreading so actually 12). We did a fair bit of searching on https://github.com/facebook/hhvm/wiki/Runtime-options however most options aren't explained very well so I have no idea what they do and testing them in a production environment is a bit scary. Had anybody here had a similar problem or could maybe point me in a direction with some of the HHVM options I would be very grateful. The HHVM version used is from http://www.hop5.in/yum/el6/ ``` HipHop VM 3.0.1 (rel) Compiler: Repo schema: e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 ``` And the the /etc/hhvm/config.hdf ``` Log { Level = Warning AlwaysLogUnhandledExceptions = true RuntimeErrorReportingLevel = 8191 } MySQL { TypedResults = false } ``` We are using supervisord to start HHVM so here the supervisor config as wel: ``` [program:hhvm] stopasgroup=true killasgroup=true command=/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true user=admin stdout_logfile=/var/log/hhvm/admin.log stderr_logfile=/var/log/hhvm/admin.error.log directory=/home/admin umask=000 ``` There is a php.ini in /etc/hhvm/php.ini but the content is empty. Also the pages tried all do some database connectivity but this is usually very minimal. For a complete picture also the my.cnf. The mysql version used is percona ``` [mysql] # CLIENT # port = 3306 socket = /var/lib/mysql/mysql.sock [mysqld] # GENERAL # user = mysql default-storage-engine = InnoDB socket = /var/lib/mysql/mysql.sock pid-file = /var/lib/mysql/mysql.pid [mysqld] # MyISAM # key-buffer-size = 32M myisam-recover = FORCE,BACKUP # SAFETY # max-allowed-packet = 128M max-connect-errors = 1000000 # DATA STORAGE # datadir = /var/lib/mysql/ # BINARY LOGGING # log-bin = /var/lib/mysql/mysql-bin expire-logs-days = 14 sync-binlog = 1 # CACHES AND LIMITS # tmp-table-size = 128M max-heap-table-size = 256M query-cache-size = 10G max-connections = 1000 thread-cache-size = 100 open-files-limit = 65535 table-definition-cache = 4096 table-open-cache = 4000 join-buffer-size = 1M # INNODB # innodb-flush-method = O_DIRECT innodb-log-files-in-group = 2 innodb-log-file-size = 512M innodb-flush-log-at-trx-commit = 1 innodb-file-per-table = 1 innodb-buffer-pool-size = 73G # LOGGING # log-error = /var/lib/mysql/mysql-error.log log-queries-not-using-indexes = 1 slow-query-log = 1 slow-query-log-file = /var/lib/mysql/mysql-slow.log ``` Mysql version: ``` innodb_version 5.6.17-65.0 protocol_version 10 slave_type_conversions version 5.6.17-65.0-56-log version_comment Percona Server (GPL), Release 65.0, Revision 587 version_compile_machine x86_64 version_compile_os Linux ```