I have a (formerly working) logrotate setup for OpenResty that I took from a previous Ubuntu 18.04 installation. Yet the logrotate.service fails with this error now ...
error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system ... and I have difficulties understanding why. The new machines are running Ubuntu 20.04, but I don't see why that should make a difference in this case.
First, here is the config:
$ cat /etc/logrotate.d/custom-openresty /usr/local/openresty/nginx/logs/access.log /usr/local/openresty/nginx/logs/error.log { daily rotate 60 maxsize 1G missingok notifempty compress compresscmd /usr/bin/zstd uncompresscmd /usr/bin/unzstd compressoptions -9 --long -T1 compressext .zst delaycompress sharedscripts postrotate test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid` endscript } /etc/logrotate.conf is unchanged and looks like this:
# see "man logrotate" for details # rotate log files weekly weekly # use the adm group by default, since this is the owning group # of /var/log/syslog. su root adm # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file #dateext # uncomment this if you want your log files compressed #compress # packages drop log rotation information into this directory include /etc/logrotate.d # system-specific logs may be also be configured here. Here is the state of my files (I would expect to get access.log.1 after logrotate ran):
$ ls -alhg /usr/local/openresty/nginx/logs/ total 10G drwxr-xr-x 2 root 4.0K Sep 16 20:04 . drwxr-xr-x 18 root 4.0K Sep 16 19:42 .. -rw-r--r-- 1 root 10G Sep 16 19:56 access.log -rw-r--r-- 1 root 5.5K Sep 16 19:56 error.log However, the logrotate.service fails with this error:
~$ systemctl status logrotate.service ● logrotate.service - Rotate log files Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago TriggeredBy: ● logrotate.timer Docs: man:logrotate(8) man:logrotate.conf(5) Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE) Main PID: 27403 (code=exited, status=1/FAILURE) $ sudo journalctl --unit logrotate.service Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files... Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result 'exit-code'. Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files. When I run it (without systemd) in debug mode as root, I get the following output:
# logrotate -v -d /etc/logrotate.d/custom-openresty reading config file /etc/logrotate.d/custom-openresty compress_prog is now /usr/bin/zstd uncompress_prog is now /usr/bin/unzstd compress_options is now -9 --long -T1 compress_ext is now .zst Reading state from file: /var/lib/logrotate/status Allocating hash table for state file, size 64 entries Creating new state Creating new state ... Creating new state Handling 1 logs rotating pattern: /usr/local/openresty/nginx/logs/access.log /usr/local/openresty/nginx/logs/error.log after 1 days (60 rotations) empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed considering log /usr/local/openresty/nginx/logs/access.log Now: 2020-09-16 20:24 Last rotated at 2020-09-16 20:11 log needs rotating considering log /usr/local/openresty/nginx/logs/error.log Now: 2020-09-16 20:24 Last rotated at 2020-09-16 19:00 log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet) rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60 dateext suffix '-20200916' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59), ... /logs/access.log.3.zst (rotatecount 60, logstart 1, i 2), renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1 running postrotate script running script with arg /usr/local/openresty/nginx/logs/access.log /usr/local/openresty/nginx/logs/error.log : " test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid` " To me, everything looks normal. If I run it, it also works:
# logrotate -v /etc/logrotate.d/custom-openresty reading config file /etc/logrotate.d/custom-openresty compress_prog is now /usr/bin/zstd uncompress_prog is now /usr/bin/unzstd compress_options is now -9 --long -T1 compress_ext is now .zst Reading state from file: /var/lib/logrotate/status Allocating hash table for state file, size 64 entries Creating new state ... Creating new state Handling 1 logs rotating pattern: /usr/local/openresty/nginx/logs/access.log /usr/local/openresty/nginx/logs/error.log after 1 days (60 rotations) empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed considering log /usr/local/openresty/nginx/logs/access.log Now: 2020-09-16 20:26 Last rotated at 2020-09-16 20:11 log needs rotating considering log /usr/local/openresty/nginx/logs/error.log Now: 2020-09-16 20:26 Last rotated at 2020-09-16 19:00 log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet) rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60 dateext suffix '-20200916' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist ... old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1 running postrotate script No error and in the end access.log.1 is created as expected:
# ls -algh /usr/local/openresty/nginx/logs/ total 10G drwxr-xr-x 2 root 4.0K Sep 16 20:26 . drwxr-xr-x 18 root 4.0K Sep 16 19:42 .. -rw-r--r-- 1 root 10G Sep 16 19:56 access.log.1 -rw-r--r-- 1 root 5.5K Sep 16 19:56 error.log Note that it said ...
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it ... instead of error renaming ... Read-only file system (as logrotate.service does)
Why does it work when running manually as root, but fails when it is executed by the logrotate.service?
I did not make any changes to logrotate.service. For completeness, here is the unit file:
$ systemctl cat logrotate.service # /lib/systemd/system/logrotate.service [Unit] Description=Rotate log files Documentation=man:logrotate(8) man:logrotate.conf(5) ConditionACPower=true [Service] Type=oneshot ExecStart=/usr/sbin/logrotate /etc/logrotate.conf # performance options Nice=19 IOSchedulingClass=best-effort IOSchedulingPriority=7 # hardening options # details: # no ProtectHome for userdir logs # no PrivateNetwork for mail deliviery # no ProtectKernelTunables for working SELinux with systemd older than 235 # no MemoryDenyWriteExecute for gzip on i686 PrivateDevices=true PrivateTmp=true ProtectControlGroups=true ProtectKernelModules=true ProtectSystem=full RestrictRealtime=true Now I'm running out of options. Any help in troubleshooting the problem is much appreciated.
02 Answers
I think, I found it. The logrotate file is not the problem.
Instead, it is caused by the hardening features in the systemd unit file. After I disabled the ProtectSystem=full hardening options, it worked. The reason is that logrotate has to operate on the /usr directory in my case, which is readonly if the option is enabled.
From the documentation on ProtectSystem=
Takes a boolean argument or the special values "full" or "strict". If true, mounts the /usr and the boot loader directories (/boot and /efi) read-only for processes invoked by this unit. If set to "full", the /etc directory is mounted read-only, too. If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys (protect these directories using PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=). This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration, and local mounts) is prohibited for the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. If this option is used, ReadWritePaths= may be used to exclude specific directories from being made read-only. This setting is implied if DynamicUser= is set. This setting cannot ensure protection in all cases. In general it has the same limitations as ReadOnlyPaths=, see below. Defaults to off.
To fix it properly, I added now the following line to /lib/systemd/system/logrotate.service:
ReadWritePaths=/usr/local/openresty/nginx/logs Then ProtectSystem=full will mount everything as readonly except the directory where the logs are. Now, the readonly error is gone:
# systemctl daemon-reload && systemctl start logrotate (completed successfully without output) I assume, I never hit the problem on Ubuntu 18.04 because the hardening options were not yet the defaults. Ubuntu 20.04 has it now, but I don't know in what Ubuntu release they were first introduced.
Nice find to the same problem i was seeing. Editing nginx.conf to log to different directory (eg. the common /var/log/nginx) is an alternative that also works.
