Units

      List units

      Display unit

      Editing units

      Unit properties

      Drop-in units

    Unit directives

      Restart

    Timers

      Pros and Cons

      Activation

      Schedule

      Certbot example

      Testing

      Resources

    Hostname

    Targets

    IP access control

      Resources

    Logging

    Templates

      OOM

    Troubleshooting

Units

List units

systemctl list-units

Filter by type:

systemctl list-units --type=socket

Filter by pattern:

systemctl list-units sympa*

Failed units:

systemctl --failed

It is recommended to examine why these units failed, but in some case these are old or irrelevant errors.

You can also filter the lists of failed units:

systemctl list-units --failed 'check_mk*'

You can remove them for the list individually:

systemctl reset-failed interchange

Or all of them:

systemctl reset-failed

Display unit

As there might be more than file involved, the following command is handy:

systemctl cat sympa

Editing units

Unit files can be edited through systemctl commands.

systemctl edit mybackup.service
systemctl edit --full mybackup.service
systemctl edit --full --force mybackup.service

Unit properties

This shows the properties of a currently running unit:

systemctl show ssh

You can query a specific property:

~# systemctl show --property=EnvironmentFiles ssh
EnvironmentFiles=/etc/default/ssh (ignore_errors=yes)

This relates to the following line in the unit file:

EnvironmentFile=-/etc/default/ssh

Drop-in units

List all drop-ins:

~# systemd-delta --type=extended
[EXTENDED]   /etc/systemd/system/check_mk.socket → /etc/systemd/system/check_mk.socket.d/10-ipacl.conf
[EXTENDED]   /usr/lib/systemd/system/rc-local.service → /usr/lib/systemd/system/rc-local.service.d/debian.conf
[EXTENDED]   /usr/lib/systemd/system/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service.d/resolvconf.conf
[EXTENDED]   /usr/lib/systemd/system/systemd-timesyncd.service → /usr/lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf

See also: Using systemd drop-in units

Unit directives

Restart

Restart policy for the unit.

For debugging a failed unit it might make sense to turn off restarting:

Restart: no

Timers

Timers are the equivalents of cron jobs. They consist of a service unit which defines the command to be executed and a timer unit which defines the schedule.

Pros and Cons

Advantages of using timers instead of cron jobs:

  • failed timers can be listed and monitored

  • output is automatically logged

  • timer is not executed if an instance is already running

  • easier to locate compared to the various files and directories with cron jobs

  • run job out of schedule without remembering the exact commandline

  • memory and CPU limits

  • precision is one second

Disadvantages:

  • takes more time to set them up

  • less control for users' jobs

Undecided:

Timers do not send automatically a notification on failed jobs.

With systemd timers ensure that your scripts provide a proper exit code on errors instead of relying on messages going to standard error output.

Activation

The timer needs to be enabled and started to be active:

~# systemctl enable certbot.timer
~# systemctl start certbot.timer

Display active timers:

~# systemctl list-timers

Display all timers:

~# systemctl list-timers --all

Schedule

You can always run the job manually independent with of the current schedule with:

~# systemctl start certbot.service

Every 20 minutes:

OnCalendar=*:0/20

Every even hour (0:00, 2:00, ...)

OnCalendar=0/2:00:00

Every odd hour (1:00, 3:00, ...)

OnCalendar=1/2:00:00

Once a day at midnight:

OnCalendar=daily

Once a day at 15:44:

OnCalendar=15:44

Monday noon:

OnCalendar=Mon 12:00

Start corresponding service immediately if last time was missed (e.g. system was down):

Persistent=true

Certbot example

Certbot service unit:

~# systemctl cat certbot.service
# /lib/systemd/system/certbot.service
[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://letsencrypt.readthedocs.io/en/latest/
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true

Certbot timer unit:

~# systemctl cat certbot.timer
# /lib/systemd/system/certbot.timer
[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target

Certbot timer status:

# systemctl status certbot.timer
● certbot.timer - Run certbot twice daily
   Loaded: loaded (/lib/systemd/system/certbot.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Wed 2019-04-03 08:00:27 CEST; 2 months 13 days ago
  Trigger: Mon 2019-06-17 10:36:39 CEST; 18h left

Testing

Testing OnCalendar settings:

systemd-analyze calendar $(systemctl cat myservice.timer | sed -n 's/^OnCalendar=//p')
  Original form: *:15:44
Normalized form: *-*-* *:15:44
    Next elapse: Sun 2020-01-05 18:15:44 CET
       (in UTC): Sun 2020-01-05 17:15:44 UTC
       From now: 45min left

Resources

archlinux Wiki

https://wiki.archlinux.org/index.php/Systemd/Timers

Hostname

Set hostname:

$ hostnamectl set-hostname buster-test-box

This is not persistent on Ubuntu. In order to change this, you need to edit /etc/cloud/cloud.cfg:

preserve_hostname: true

Targets

Show dependencies for a target:

~# systemctl list-dependencies network-online.target
network-online.target
● ├─networking.service
● └─NetworkManager-wait-online.service

IP access control

This feature was introduced with systemd 235, which is available in recent Linux distributions like Debian Buster, Ubuntu Bionic and CentOS 8.

The following example comes from a check_mk monitoring agent which is listening on port 6556. To restrict access to the monitoring server we add the following drop-in (e.g. etc/systemd/system/check_mk.socket.d/10-ipacl.conf):

[Socket]
IPAddressDeny=any
IPAddressAllow=203.0.113.114

Resources

IP Accounting and Access Lists with systemd

Logging

Messages since last reboot:

~# journalctl -b

Show kernel messages (equivalent to dmesg -T):

~# journalctl -k

Templates

Example: /lib/systemd/system/postgresql@.service on Debian for PostgreSQL cluster.

OOM

# prevent OOM killer from choosing the postmaster (individual backends will
# reset the score to 0)
OOMScoreAdjust=-900

Troubleshooting

● mariadb.service - MariaDB 10.3.22 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-03-25 11:32:09 CET; 1h 1min ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 638935 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=217/USER)

Mar 25 11:32:09 belukha systemd[1]: Starting MariaDB 10.3.22 database server...
Mar 25 11:32:09 belukha systemd[638935]: mariadb.service: Failed to determine user credentials: No such process
Mar 25 11:32:09 belukha systemd[638935]: mariadb.service: Failed at step USER spawning /usr/bin/install: No such process
Mar 25 11:32:09 belukha systemd[1]: mariadb.service: Control process exited, code=exited, status=217/USER
Mar 25 11:32:09 belukha systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 25 11:32:09 belukha systemd[1]: Failed to start MariaDB 10.3.22 database server.

Fixed by:

systemctl daemon-reexec
systemctl start mariadb