.rtreport files (randomly?) disappear #450
Description
On our deployed OLS instances we run into issues with disappearing .rtreport files. All of a sudden the complete status folder has disappeared from the /dev/shm/ols path.
The error log shows the following after a systemctl restart lsws was triggered at 15:56:
2025-03-24 15:56:30.500768 [NOTICE] [960544] Server Restart Request via Signal...
2025-03-24 15:56:30.500976 [NOTICE] [960551] Start shutting down gracefully ...
2025-03-24 15:56:30.500983 [NOTICE] [960550] Start shutting down gracefully ...
2025-03-24 15:56:30.500984 [NOTICE] [960548] Start shutting down gracefully ...
2025-03-24 15:56:30.501001 [NOTICE] [960547] Start shutting down gracefully ...
2025-03-24 15:56:30.501016 [NOTICE] [960549] Start shutting down gracefully ...
2025-03-24 15:56:30.501026 [NOTICE] [960551] Event-loop lingering start.
2025-03-24 15:56:30.501032 [NOTICE] [960550] Event-loop lingering start.
2025-03-24 15:56:30.501038 [NOTICE] [960548] Event-loop lingering start.
2025-03-24 15:56:30.501046 [NOTICE] [960547] Event-loop lingering start.
2025-03-24 15:56:30.501071 [NOTICE] [960549] Event-loop lingering start.
2025-03-24 15:56:30.501075 [NOTICE] [960546] Start shutting down gracefully ...
2025-03-24 15:56:30.501112 [NOTICE] [960546] Event-loop lingering start.
2025-03-24 15:56:30.501717 [NOTICE] [960544] SIGHUP received, start count down to stop server...
2025-03-24 15:56:30.501935 [NOTICE] [960544] Forked [962395] for graceful restart.
2025-03-24 15:56:30.502019 [NOTICE] [960544] Start passing listener sockets through forked child [962395].
2025-03-24 15:56:30.502075 [INFO] [962395] [UDP *:7080] pass listener, copy fd 15 to 1007.
2025-03-24 15:56:30.502102 [INFO] [962395] [*:7080] Pass listener, copy fd 14 to 1006.
2025-03-24 15:56:30.502113 [INFO] [962395] [TCP *:8088], SO_REUSEPORT #1, pass listener copy fd 13 to 1005.
2025-03-24 15:56:30.502122 [INFO] [962395] [TCP *:8088], SO_REUSEPORT #2, pass listener copy fd 12 to 1004.
2025-03-24 15:56:30.502131 [INFO] [962395] [TCP *:8088], SO_REUSEPORT #3, pass listener copy fd 11 to 1003.
2025-03-24 15:56:30.502141 [INFO] [962395] [TCP *:8088], SO_REUSEPORT #4, pass listener copy fd 10 to 1002.
2025-03-24 15:56:30.502156 [INFO] [962395] [TCP *:8088], SO_REUSEPORT #5, pass listener copy fd 9 to 1001.
2025-03-24 15:56:30.502165 [INFO] [962395] [TCP *:8088], SO_REUSEPORT #6, pass listener copy fd 8 to 1000.
2025-03-24 15:56:30.502198 [NOTICE] [962395] Starting new instance: execute '/usr/local/lsws/bin/litespeed'.
2025-03-24 15:56:30.509231 [INFO] [config:server:basics] httpdWorkers: 6, Num of Processors: 6
2025-03-24 15:56:30.509275 [INFO] [config:server:basics] enableLVE: 0
2025-03-24 15:56:30.509280 [INFO] [config:server:basics] bubbleWrap: 0, cmd: '(null)'
2025-03-24 15:56:30.517862 [INFO] [PlainConf] [httpServerConfig:] start parsing file /usr/local/lsws/conf/httpd_config.conf
2025-03-24 15:56:30.517881 [INFO] [PlainConf] [httpServerConfig:] Finished parsing file /usr/local/lsws/conf/httpd_config.conf
2025-03-24 15:56:30.517885 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [checkprivatecache 1]
2025-03-24 15:56:30.517889 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [checkpubliccache 1]
2025-03-24 15:56:30.517892 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [maxcacheobjsize 10000000]
2025-03-24 15:56:30.517895 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [maxstaleage 200]
2025-03-24 15:56:30.517898 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [qscache 1]
2025-03-24 15:56:30.517901 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [reqcookiecache 1]
2025-03-24 15:56:30.517904 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [respcookiecache 1]
2025-03-24 15:56:30.517907 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [ignorereqcachectrl 1]
2025-03-24 15:56:30.517910 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [ignorerespcachectrl 0]
2025-03-24 15:56:30.517913 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [enablecache 0]
2025-03-24 15:56:30.517916 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [expireinseconds 3600]
2025-03-24 15:56:30.517920 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [enableprivatecache 0]
2025-03-24 15:56:30.517923 [INFO] [PlainConf] [httpServerConfig:] module [cache] add param [privateexpireinseconds 3600]
2025-03-24 15:56:30.517938 [NOTICE] Loading LiteSpeed/1.8.2 Open (lsquic 4.0.8, modgzip 1.1, cache 1.66, mod_security 1.4 (with libmodsecurity v3.0.13)) BUILD (built: Fri Dec 20 16:39:15 UTC 2024) ...
2025-03-24 15:56:30.517949 [NOTICE] Using [BoringSSL]
2025-03-24 15:56:30.518728 [NOTICE] [ADMIN] server socket: uds://usr/local/lsws/admin/tmp/admin.sock.7635, fd 6.
2025-03-24 15:56:30.518905 [NOTICE] [962401] [*:8088] Recovering server socket, fd: 8
2025-03-24 15:56:30.518921 [NOTICE] [962401] Recv listener *:8088, copy fd 1000 to 8.
2025-03-24 15:56:30.518936 [NOTICE] [962401] [*:8088] SO_REUSEPORT #2, recovering server socket, fd: 9.
2025-03-24 15:56:30.518946 [NOTICE] [962401] [*:8088] SO_REUSEPORT #3, recovering server socket, fd: 10.
2025-03-24 15:56:30.518955 [NOTICE] [962401] [*:8088] SO_REUSEPORT #4, recovering server socket, fd: 11.
2025-03-24 15:56:30.518965 [NOTICE] [962401] [*:8088] SO_REUSEPORT #5, recovering server socket, fd: 12.
2025-03-24 15:56:30.518974 [NOTICE] [962401] [*:8088] SO_REUSEPORT #6, recovering server socket, fd: 13.
2025-03-24 15:56:30.518983 [NOTICE] [962401] [*:7080] Recovering server socket, fd: 14
2025-03-24 15:56:30.518991 [NOTICE] [962401] Recv listener *:7080, copy fd 1006 to 14.
2025-03-24 15:56:30.519036 [NOTICE] [962401] chroot is disabled.
2025-03-24 15:56:30.519046 [INFO] [962401] old priority: 0, new priority: 0
2025-03-24 15:56:30.519198 [INFO] [962401] [config:server:basics2] For better obscurity, server version number is hidden in the response header.
2025-03-24 15:56:30.520208 [INFO] [962401] setuid is allowed in Ext Apps
2025-03-24 15:56:30.520477 [NOTICE] [962401] [PID: 962401]: forked cgid: 962402
2025-03-24 15:56:30.520650 [INFO] [962401] Add QUIC.cloud IPs from admin/conf/quic-cloud-ips to trusted ACL ...
2025-03-24 15:56:30.520756 [INFO] [962401] Add CloudFlare Subnets to trusted ACL ...
2025-03-24 15:56:30.520822 [INFO] [962401] [PlainConf] [adminConfig:] start parsing file /usr/local/lsws/admin/conf/admin_config.conf
2025-03-24 15:56:30.521135 [INFO] [962401] [PlainConf] [adminConfig:] Finished parsing file /usr/local/lsws/admin/conf/admin_config.conf
2025-03-24 15:56:30.521701 [INFO] [962401] QuicEngine::init(), pid: 962401, log level [warn].
2025-03-24 15:56:30.522275 [INFO] [962401] [Module: modcompress 1.1] has been initialized successfully
2025-03-24 15:56:30.522391 [INFO] [962401] [Module: moddecompress 1.1] has been initialized successfully
2025-03-24 15:56:30.522486 [INFO] [962401] [Module: cache 1.66] has been initialized successfully
2025-03-24 15:56:30.523052 [NOTICE] [962401] Reuse current listener [adminListener].
2025-03-24 15:56:30.523118 [NOTICE] [962401] Reuse current listener [Default].
2025-03-24 15:56:30.523124 [INFO] [962401] [*:8088] Enable SO_REUSEPORT .
2025-03-24 15:56:30.523133 [NOTICE] [962401] The maximum number of file descriptor limit is set to 50000.
2025-03-24 15:56:30.524554 [ERROR] [962401] [config:server:epsr:lsphp81] invalid path - /usr/local/lsws/lsphp81/bin/lsphp, it cannot be started by Web server!
2025-03-24 15:56:30.524571 [ERROR] [962401] [config:server:epsr:lsphp82] invalid path - /usr/local/lsws/lsphp82/bin/lsphp, it cannot be started by Web server!
2025-03-24 15:56:30.524587 [NOTICE] [962401] [config:server:epsr:lsphp83] 'Process Limit' probably is too low, adjust the limit to: 2100.
2025-03-24 15:56:30.524629 [NOTICE] [962401] [config:server:epsr:lsphp80] 'Process Limit' probably is too low, adjust the limit to: 2100.
2025-03-24 15:56:30.524753 [INFO] [962401] [PlainConf] [virtualHostConfig:] start parsing file /usr/local/lsws/conf/vhosts/Example/vhconf.conf
2025-03-24 15:56:30.525639 [INFO] [962401] [PlainConf] [virtualHostConfig:] Finished parsing file /usr/local/lsws/conf/vhosts/Example/vhconf.conf
2025-03-24 15:56:30.525653 [INFO] [962401] [PlainConf] [context:/] rewrite [] add rules [rewritefile .htaccess]
2025-03-24 15:56:30.525838 [WARN] [962401] [config:server:vhosts:vhost:Example] Uid of /usr/local/lsws/Example/html/ is 0, smaller than minimum requirement 11, use server uid!
2025-03-24 15:56:30.525846 [WARN] [962401] [config:server:vhosts:vhost:Example] Gid of /usr/local/lsws/Example/html/ is 1003, smaller than minimum requirement 10, use server gid!
2025-03-24 15:56:30.526057 [INFO] [962401] [config:server:vhosts:vhost:Example] config context /.
2025-03-24 15:56:30.526092 [INFO] [962401] RewriteFile [.htaccess] parsed, return 0.
2025-03-24 15:56:30.526097 [INFO] [962401] [config:server:vhosts:vhost:Example] config context /blocked/.
2025-03-24 15:56:30.526105 [INFO] [962401] [config:server:vhosts:vhost:Example] config context /cgi-bin/.
2025-03-24 15:56:30.526117 [INFO] [962401] [config:server:vhosts:vhost:Example] config context /docs/.
2025-03-24 15:56:30.526127 [INFO] [962401] [config:server:vhosts:vhost:Example] config context /protected/.
2025-03-24 15:56:30.526245 [INFO] [962401] [PlainConf] [virtualHostConfig:] start parsing file /usr/local/lsws/conf/vhosts/beta.example.com.conf
2025-03-24 15:56:30.526421 [INFO] [962401] [PlainConf] [virtualHostConfig:] start parsing file /home/user/ols/beta.example.com.conf
2025-03-24 15:56:30.526550 [INFO] [962401] [PlainConf] [virtualHostConfig:] Finished parsing file /home/user/ols/beta.example.com.conf
2025-03-24 15:56:30.526565 [INFO] [962401] [PlainConf] [virtualHostConfig:] Finished parsing file /usr/local/lsws/conf/vhosts/beta.example.com.conf
2025-03-24 15:56:30.526832 [INFO] [962401] [config:server:vhosts:vhost:beta.example.com] config context /.
2025-03-24 15:56:30.526884 [INFO] [962401] Rewrite directive: <IfModule mod_rewrite.c> bypassed.
2025-03-24 15:56:30.526895 [INFO] [962401] RewriteBase [/] parsed, set to HttpContext 0x188cc70.
2025-03-24 15:56:30.526904 [INFO] [962401] Rewrite directive: </IfModule> bypassed.
2025-03-24 15:56:30.526908 [INFO] [962401] RewriteFile [/var/www/beta.example.com/.htaccess] parsed, return 0.
2025-03-24 15:56:30.526985 [INFO] [962401] [PlainConf] [virtualHostTemplate:] start parsing file /usr/local/lsws/conf/templates/ccl.conf
2025-03-24 15:56:30.527583 [INFO] [962401] [PlainConf] [virtualHostTemplate:] Finished parsing file /usr/local/lsws/conf/templates/ccl.conf
2025-03-24 15:56:30.527597 [INFO] [962401] [PlainConf] [context:/] rewrite [] add rules [rewritefile .htaccess]
2025-03-24 15:56:30.527763 [INFO] [962401] [PlainConf] [virtualHostTemplate:] start parsing file /usr/local/lsws/conf/templates/rails.conf
2025-03-24 15:56:30.528295 [INFO] [962401] [PlainConf] [virtualHostTemplate:] Finished parsing file /usr/local/lsws/conf/templates/rails.conf
2025-03-24 15:56:30.528308 [INFO] [962401] [PlainConf] [context:/] rewrite [] add rules [rewritefile .htaccess]
2025-03-24 15:56:30.528460 [NOTICE] [962401] [ZConfManager] No VHosts added, do not send!
2025-03-24 15:56:30.528759 [NOTICE] [962401] Instance is ready for service. m_fdCmd 32, m_fdAdmin 6.
2025-03-24 15:56:30.529005 [NOTICE] [962401] [AutoRestarter] new child process with pid=962403 is forked!
2025-03-24 15:56:30.529236 [NOTICE] [962401] [AutoRestarter] new child process with pid=962404 is forked!
2025-03-24 15:56:30.529445 [INFO] [962403] Close SO_REUSEPORT #2 fd: 9.
2025-03-24 15:56:30.529461 [INFO] [962403] Close SO_REUSEPORT #3 fd: 10.
2025-03-24 15:56:30.529465 [INFO] [962403] Close SO_REUSEPORT #4 fd: 11.
2025-03-24 15:56:30.529468 [INFO] [962403] Close SO_REUSEPORT #5 fd: 12.
2025-03-24 15:56:30.529471 [INFO] [962403] Close SO_REUSEPORT #6 fd: 13.
2025-03-24 15:56:30.529477 [NOTICE] [962403] [*:8088] Worker #1 activates SO_REUSEPORT #1 socket, fd: 8
2025-03-24 15:56:30.529475 [NOTICE] [962401] [AutoRestarter] new child process with pid=962405 is forked!
2025-03-24 15:56:30.529503 [INFO] [962403] [UDP:0.0.0.0:7080] initPacketsIn: allocated 100 packets
2025-03-24 15:56:30.529732 [NOTICE] [962403] Successfully change current user to user
2025-03-24 15:56:30.529744 [NOTICE] [962403] Core dump is enabled.
2025-03-24 15:56:30.529738 [NOTICE] [962401] [AutoRestarter] new child process with pid=962406 is forked!
2025-03-24 15:56:30.529825 [INFO] [962403] [lsphp]: locked pid file [/tmp/lshttpd/lsphp.sock.pid].
2025-03-24 15:56:30.529837 [INFO] [962403] [lsphp] remove unix socket for detached process: /tmp/lshttpd/lsphp.sock
2025-03-24 15:56:30.529898 [NOTICE] [962403] [LocalWorker::workerExec] Config[lsphp]: suExec uid 1003 gid 1003 cmd /usr/local/lsws/lsphp83/bin/lsphp, final uid 1003 gid 1003, flags: 0.
2025-03-24 15:56:30.529991 [NOTICE] [962401] [AutoRestarter] new child process with pid=962407 is forked!
2025-03-24 15:56:30.530212 [NOTICE] [962403] [lsphp] add child process pid: 962408
2025-03-24 15:56:30.530239 [NOTICE] [962401] [AutoRestarter] new child process with pid=962409 is forked!
2025-03-24 15:56:30.530290 [INFO] [962403] [lsphp]: unlocked pid file [/tmp/lshttpd/lsphp.sock.pid].
2025-03-24 15:56:30.530399 [INFO] [962403] [lsphp83]: locked pid file [/tmp/lshttpd/lsphp83.sock.pid].
2025-03-24 15:56:30.530408 [INFO] [962403] [lsphp83] remove unix socket for detached process: /tmp/lshttpd/lsphp83.sock
2025-03-24 15:56:30.530492 [NOTICE] [962403] [LocalWorker::workerExec] Config[lsphp83]: suExec uid 1003 gid 1003 cmd /usr/local/lsws/lsphp83/bin/lsphp, final uid 1003 gid 1003, flags: 0.
2025-03-24 15:56:30.530880 [INFO] [962409] Close SO_REUSEPORT #1 fd: 8.
2025-03-24 15:56:30.530896 [INFO] [962409] Close SO_REUSEPORT #2 fd: 9.
2025-03-24 15:56:30.530900 [INFO] [962409] Close SO_REUSEPORT #3 fd: 10.
2025-03-24 15:56:30.530904 [INFO] [962409] Close SO_REUSEPORT #4 fd: 11.
2025-03-24 15:56:30.530908 [INFO] [962409] Close SO_REUSEPORT #5 fd: 12.
2025-03-24 15:56:30.530915 [NOTICE] [962409] [*:8088] Worker #6 activates SO_REUSEPORT #6 socket, fd: 13
2025-03-24 15:56:30.530930 [NOTICE] [962409] Stop listener *:7080, fd 14.
2025-03-24 15:56:30.531149 [NOTICE] [962403] [lsphp83] add child process pid: 962410
2025-03-24 15:56:30.531213 [NOTICE] [962409] Successfully change current user to user
2025-03-24 15:56:30.531227 [NOTICE] [962409] Core dump is enabled.
2025-03-24 15:56:30.531227 [INFO] [962403] [lsphp83]: unlocked pid file [/tmp/lshttpd/lsphp83.sock.pid].
2025-03-24 15:56:30.531248 [NOTICE] [962409] Setup swapping space...
2025-03-24 15:56:30.531292 [NOTICE] [962403] Setup swapping space...
2025-03-24 15:56:30.531325 [NOTICE] [962409] LiteSpeed/1.8.2 Open starts successfully!
2025-03-24 15:56:30.531456 [NOTICE] [962403] LiteSpeed/1.8.2 Open starts successfully!
2025-03-24 15:56:30.531497 [INFO] [962404] Close SO_REUSEPORT #1 fd: 8.
2025-03-24 15:56:30.531520 [INFO] [962404] Close SO_REUSEPORT #3 fd: 10.
2025-03-24 15:56:30.531526 [INFO] [962404] Close SO_REUSEPORT #4 fd: 11.
2025-03-24 15:56:30.531531 [INFO] [962404] Close SO_REUSEPORT #5 fd: 12.
2025-03-24 15:56:30.531544 [INFO] [962404] Close SO_REUSEPORT #6 fd: 13.
2025-03-24 15:56:30.531554 [NOTICE] [962404] [*:8088] Worker #2 activates SO_REUSEPORT #2 socket, fd: 9
2025-03-24 15:56:30.531572 [NOTICE] [962404] Stop listener *:7080, fd 14.
2025-03-24 15:56:30.531857 [NOTICE] [962404] Successfully change current user to user
2025-03-24 15:56:30.531883 [NOTICE] [962404] Core dump is enabled.
2025-03-24 15:56:30.531927 [NOTICE] [962404] Setup swapping space...
2025-03-24 15:56:30.531990 [NOTICE] [962404] LiteSpeed/1.8.2 Open starts successfully!
2025-03-24 15:56:30.532000 [INFO] [962407] Close SO_REUSEPORT #1 fd: 8.
2025-03-24 15:56:30.532016 [INFO] [962407] Close SO_REUSEPORT #2 fd: 9.
2025-03-24 15:56:30.532020 [INFO] [962407] Close SO_REUSEPORT #3 fd: 10.
2025-03-24 15:56:30.532023 [INFO] [962407] Close SO_REUSEPORT #4 fd: 11.
2025-03-24 15:56:30.532027 [INFO] [962407] Close SO_REUSEPORT #6 fd: 13.
2025-03-24 15:56:30.532034 [NOTICE] [962407] [*:8088] Worker #5 activates SO_REUSEPORT #5 socket, fd: 12
2025-03-24 15:56:30.532049 [NOTICE] [962407] Stop listener *:7080, fd 14.
2025-03-24 15:56:30.532304 [NOTICE] [962407] Successfully change current user to user
2025-03-24 15:56:30.532318 [NOTICE] [962407] Core dump is enabled.
2025-03-24 15:56:30.532338 [NOTICE] [962407] Setup swapping space...
2025-03-24 15:56:30.532401 [NOTICE] [962407] LiteSpeed/1.8.2 Open starts successfully!
2025-03-24 15:56:30.532977 [INFO] [962406] Close SO_REUSEPORT #1 fd: 8.
2025-03-24 15:56:30.532992 [INFO] [962406] Close SO_REUSEPORT #2 fd: 9.
2025-03-24 15:56:30.532996 [INFO] [962406] Close SO_REUSEPORT #3 fd: 10.
2025-03-24 15:56:30.533000 [INFO] [962406] Close SO_REUSEPORT #5 fd: 12.
2025-03-24 15:56:30.533004 [INFO] [962406] Close SO_REUSEPORT #6 fd: 13.
2025-03-24 15:56:30.533011 [NOTICE] [962406] [*:8088] Worker #4 activates SO_REUSEPORT #4 socket, fd: 11
2025-03-24 15:56:30.533027 [NOTICE] [962406] Stop listener *:7080, fd 14.
2025-03-24 15:56:30.533258 [NOTICE] [962406] Successfully change current user to user
2025-03-24 15:56:30.533272 [NOTICE] [962406] Core dump is enabled.
2025-03-24 15:56:30.533298 [NOTICE] [962406] Setup swapping space...
2025-03-24 15:56:30.533349 [NOTICE] [962406] LiteSpeed/1.8.2 Open starts successfully!
2025-03-24 15:56:30.533937 [INFO] [962405] Close SO_REUSEPORT #1 fd: 8.
2025-03-24 15:56:30.533952 [INFO] [962405] Close SO_REUSEPORT #2 fd: 9.
2025-03-24 15:56:30.533956 [INFO] [962405] Close SO_REUSEPORT #4 fd: 11.
2025-03-24 15:56:30.533960 [INFO] [962405] Close SO_REUSEPORT #5 fd: 12.
2025-03-24 15:56:30.533964 [INFO] [962405] Close SO_REUSEPORT #6 fd: 13.
2025-03-24 15:56:30.533971 [NOTICE] [962405] [*:8088] Worker #3 activates SO_REUSEPORT #3 socket, fd: 10
2025-03-24 15:56:30.533986 [NOTICE] [962405] Stop listener *:7080, fd 14.
2025-03-24 15:56:30.534225 [NOTICE] [962405] Successfully change current user to user
2025-03-24 15:56:30.534239 [NOTICE] [962405] Core dump is enabled.
2025-03-24 15:56:30.534258 [NOTICE] [962405] Setup swapping space...
2025-03-24 15:56:30.534309 [NOTICE] [962405] LiteSpeed/1.8.2 Open starts successfully!
2025-03-24 15:56:30.602411 [NOTICE] [960551] New litespeed process is ready, stops listeners
2025-03-24 15:56:30.602412 [NOTICE] [960547] New litespeed process is ready, stops listeners
2025-03-24 15:56:30.602422 [NOTICE] [960549] New litespeed process is ready, stops listeners
2025-03-24 15:56:30.602422 [NOTICE] [960550] New litespeed process is ready, stops listeners
2025-03-24 15:56:30.602431 [INFO] [960551] Stop all listeners.
2025-03-24 15:56:30.602456 [INFO] [960547] Stop all listeners.
2025-03-24 15:56:30.602462 [INFO] [960549] Stop all listeners.
2025-03-24 15:56:30.602458 [NOTICE] [960548] New litespeed process is ready, stops listeners
2025-03-24 15:56:30.602469 [INFO] [960550] Stop all listeners.
2025-03-24 15:56:30.602473 [NOTICE] [960551] Stop listener *:8088, fd 13.
2025-03-24 15:56:30.602477 [NOTICE] [960547] Stop listener *:8088, fd 9.
2025-03-24 15:56:30.602478 [NOTICE] [960549] Stop listener *:8088, fd 11.
2025-03-24 15:56:30.602496 [INFO] [960548] Stop all listeners.
2025-03-24 15:56:30.602505 [NOTICE] [960550] Stop listener *:8088, fd 12.
2025-03-24 15:56:30.602524 [NOTICE] [960551] Event-loop lingering done.
2025-03-24 15:56:30.602526 [NOTICE] [960548] Stop listener *:8088, fd 10.
2025-03-24 15:56:30.602533 [NOTICE] [960551] Shut down successfully!
2025-03-24 15:56:30.602533 [NOTICE] [960547] Event-loop lingering done.
2025-03-24 15:56:30.602539 [NOTICE] [960551] HttpServerImpl::releaseAll called.
2025-03-24 15:56:30.602540 [NOTICE] [960547] Shut down successfully!
2025-03-24 15:56:30.602540 [NOTICE] [960549] Event-loop lingering done.
2025-03-24 15:56:30.602560 [NOTICE] [960548] Event-loop lingering done.
2025-03-24 15:56:30.602563 [NOTICE] [960547] HttpServerImpl::releaseAll called.
2025-03-24 15:56:30.602565 [NOTICE] [960549] Shut down successfully!
2025-03-24 15:56:30.602569 [NOTICE] [960550] Event-loop lingering done.
2025-03-24 15:56:30.602567 [NOTICE] [960548] Shut down successfully!
2025-03-24 15:56:30.602577 [NOTICE] [960549] HttpServerImpl::releaseAll called.
2025-03-24 15:56:30.602577 [NOTICE] [960550] Shut down successfully!
2025-03-24 15:56:30.602581 [NOTICE] [960548] HttpServerImpl::releaseAll called.
2025-03-24 15:56:30.602591 [NOTICE] [960550] HttpServerImpl::releaseAll called.
2025-03-24 15:56:30.604885 [NOTICE] [960544] [AutoRestarter] cleanup children processes and unix sockets belong to process 960551 !
2025-03-24 15:56:30.604938 [NOTICE] [960544] [AutoRestarter] child process with pid=960551 exited with status=0!
2025-03-24 15:56:30.604981 [NOTICE] [960544] [AutoRestarter] cleanup children processes and unix sockets belong to process 960547 !
2025-03-24 15:56:30.605204 [NOTICE] [960546] New litespeed process is ready, stops listeners
2025-03-24 15:56:30.605213 [INFO] [960546] Stop all listeners.
2025-03-24 15:56:30.605217 [NOTICE] [960546] Stop listener *:8088, fd 8.
2025-03-24 15:56:30.605219 [NOTICE] [960544] [AutoRestarter] child process with pid=960547 exited with status=0!
2025-03-24 15:56:30.605224 [NOTICE] [960546] Stop listener *:7080, fd 14.
2025-03-24 15:56:30.605242 [NOTICE] [960546] Event-loop lingering done.
2025-03-24 15:56:30.605245 [NOTICE] [960546] Shut down successfully!
2025-03-24 15:56:30.605248 [NOTICE] [960544] [AutoRestarter] cleanup children processes and unix sockets belong to process 960549 !
2025-03-24 15:56:30.605249 [NOTICE] [960546] HttpServerImpl::releaseAll called.
2025-03-24 15:56:30.605480 [NOTICE] [960544] [AutoRestarter] child process with pid=960549 exited with status=0!
2025-03-24 15:56:30.605513 [NOTICE] [960544] [AutoRestarter] cleanup children processes and unix sockets belong to process 960550 !
2025-03-24 15:56:30.605779 [NOTICE] [960544] [AutoRestarter] child process with pid=960550 exited with status=0!
2025-03-24 15:56:30.605821 [NOTICE] [960544] [AutoRestarter] cleanup children processes and unix sockets belong to process 960548 !
2025-03-24 15:56:30.606846 [NOTICE] [960544] [AutoRestarter] child process with pid=960548 exited with status=0!
2025-03-24 15:56:30.606889 [NOTICE] [960544] [AutoRestarter] cleanup children processes and unix sockets belong to process 960546 !
2025-03-24 15:56:30.706968 [NOTICE] [960544] [AutoRestarter] child process with pid=960546 exited with status=0!
2025-03-24 15:56:30.707089 [NOTICE] [960544] Server Stopped!
2025-03-24 15:56:35.598638 [INFO] [962404] [lsphp80]: locked pid file [/tmp/lshttpd/lsphp80.sock.pid].
2025-03-24 15:56:35.598659 [INFO] [962404] [lsphp80] remove unix socket for detached process: /tmp/lshttpd/lsphp80.sock
2025-03-24 15:56:35.598824 [NOTICE] [962404] [LocalWorker::workerExec] Config[lsphp80]: suExec uid 1003 gid 1003 cmd /usr/local/lsws/lsphp80/bin/lsphp, final uid 1003 gid 1003, flags: 0.
2025-03-24 15:56:35.599561 [NOTICE] [962404] [lsphp80] add child process pid: 962420
2025-03-24 15:56:35.599630 [INFO] [962404] [lsphp80]: unlocked pid file [/tmp/lshttpd/lsphp80.sock.pid].
2025-03-24 16:19:37.005275 [ERROR] [962406] Failed to open the real time report: /dev/shm/ols/status/.rtreport.4!
2025-03-24 16:19:37.007480 [ERROR] [962407] Failed to open the real time report: /dev/shm/ols/status/.rtreport.5!
2025-03-24 16:19:37.009139 [ERROR] [962405] Failed to open the real time report: /dev/shm/ols/status/.rtreport.3!
2025-03-24 16:19:37.069373 [ERROR] [962409] Failed to open the real time report: /dev/shm/ols/status/.rtreport.6!
2025-03-24 16:19:37.082209 [ERROR] [962404] Failed to open the real time report: /dev/shm/ols/status/.rtreport.2!
2025-03-24 16:19:37.098026 [ERROR] [962403] Failed to open the real time report: /dev/shm/ols/status/.rtreport!
2025-03-24 16:19:38.006612 [ERROR] [962406] Failed to open the real time report: /dev/shm/ols/status/.rtreport.4!
2025-03-24 16:19:38.008856 [ERROR] [962407] Failed to open the real time report: /dev/shm/ols/status/.rtreport.5!
2025-03-24 16:19:38.011008 [ERROR] [962405] Failed to open the real time report: /dev/shm/ols/status/.rtreport.3!
2025-03-24 16:19:38.070855 [ERROR] [962409] Failed to open the real time report: /dev/shm/ols/status/.rtreport.6!
2025-03-24 16:19:38.083662 [ERROR] [962404] Failed to open the real time report: /dev/shm/ols/status/.rtreport.2!
2025-03-24 16:19:38.099425 [ERROR] [962403] Failed to open the real time report: /dev/shm/ols/status/.rtreport!
2025-03-24 16:19:39.000687 [ERROR] [962403] Failed to open the real time report: /dev/shm/ols/status/.rtreport!
2025-03-24 16:19:39.008025 [ERROR] [962406] Failed to open the real time report: /dev/shm/ols/status/.rtreport.4!
2025-03-24 16:19:39.010229 [ERROR] [962407] Failed to open the real time report: /dev/shm/ols/status/.rtreport.5!
2025-03-24 16:19:39.012880 [ERROR] [962405] Failed to open the real time report: /dev/shm/ols/status/.rtreport.3!
What confuses me is that after the restart completed at 15:56:30 the .rtreport status files were all there. But then, more than 20mins later, they suddenly disappear. A manual deletion by a human error can be ruled out.
This happens on all our OLS 1.8.x instances in irregular intervals (so far I would say once a week at least). Sometimes the .rtreport files stick around for several days, sometimes only a few minutes like reported in this case, after a lsws restart.
Any idea what would cause this? Let me know if I can help with additional logs or debugging.