#title Systemd
#author Stefan Hornburg (Racke)
#LISTtitle Systemd
#SORTtopics systemd, cron, unit, service, timers, ntp
#lang en

** Units

*** Rules

=%= and =$= need to escape as =%%= and =$$=:

{{{
ExecStart=/bin/bash -c '/home/{{ shopify_api_user }}/shopify-api/getorders.pl --filter created_at_min=$$(date +%%Y-%%m-%%d --date="2 days ago")'
}}}

*** List units

{{{
$ systemctl list-units
}}}

Filter by type:

{{{
$ systemctl list-units --type=socket
}}}

Filter by pattern:

{{{
$ systemctl list-units sympa*
}}}

Failed units:

{{{
$ systemctl --failed
}}}

It is recommended to examine why these units failed, but in some case these are old or irrelevant errors.

You can also filter the lists of failed units:

{{{
$ systemctl list-units --failed 'check_mk*'
}}}

You can remove them for the list individually:

{{{
$ systemctl reset-failed interchange
}}}

Or all of them:

{{{
$ systemctl reset-failed
}}}

*** Display unit

As there might be more than file involved, the following command is handy:

{{{
$ systemctl cat sympa
}}}

*** Editing units

Unit files can be edited through *systemctl* commands.

{{{
$ systemctl edit mybackup.service
$systemctl edit --full mybackup.service
systemctl edit --full --force mybackup.service
}}}

*** Unit properties

This shows the properties of a currently running unit:

{{{
$ systemctl show ssh
}}}

You can query a specific property:

{{{
$ systemctl show --property=EnvironmentFiles ssh
EnvironmentFiles=/etc/default/ssh (ignore_errors=yes)
}}}

This relates to the following line in the unit file:

{{{
EnvironmentFile=-/etc/default/ssh
}}}

*** Drop-in units

List all drop-ins:

{{{
~# systemd-delta --type=extended
[EXTENDED]   /etc/systemd/system/check_mk.socket → /etc/systemd/system/check_mk.socket.d/10-ipacl.conf
[EXTENDED]   /usr/lib/systemd/system/rc-local.service → /usr/lib/systemd/system/rc-local.service.d/debian.conf
[EXTENDED]   /usr/lib/systemd/system/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service.d/resolvconf.conf
[EXTENDED]   /usr/lib/systemd/system/systemd-timesyncd.service → /usr/lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
}}}

See also: [[https://coreos.com/os/docs/latest/using-systemd-drop-in-units.html][Using systemd drop-in units]]

** Unit directives
#directives

*** ExecStartPre
#direxecstartpre

Commands to execute before the main process started.

This can be used to test the configuration:

{{{
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
}}}

Adding a plus sign in front of the command indicates that the root user is executing this command:

{{{
ExecStartPre=+/bin/mkdir -p /run/sympa
ExecStartPre=+/bin/chown sympa:sympa /run/sympa
}}}
 
*** Group
#dirgroup

User group used when running the service.

{{{
Group sympa
}}}

*** PermissionsStartOnly
#dirpermissionsstartonly

This directive is deprecated. It was used to run directives like
[[#direxecstartpre][*ExecStartPre*]] as privileged (root) user when your service is running as a non privileged
user.

Instead of using this directive you can prepend a plus sign to the command,
which indicates that the root user is executing this command.

{{{
ExecStartPre=+/bin/mkdir -p /run/sympa
ExecStartPre=+/bin/chown sympa:sympa /run/sympa
}}}
 
*** Restart
#dirrestart

Restart policy for the unit.

For debugging a failed unit it might make sense to turn off restarting:

{{{
Restart: no
}}}

*** RuntimeDirectory

This directive creates a directory in the default location (usually =/run=):

{{{
RuntimeDirectory sympa
}}}

If multiple related services are using the same directory it is important
that only one service declares the runtime directory.

The other service
units should declare *Requires* and *After* on the "base" service.

*** User
#diruser

User running the service.

{{{
User sympa
}}}
 
*** WorkingDirectory
#dirworkingdirectory

This configures the working directory for the service, equivalent to =cd= in
a shell.

{{{
WorkingDirectory=/home/sympa
}}}

** Timers
#timers

Timers are the equivalents of cron jobs. They consist of a *service* unit which defines the command to be executed and a *timer* unit which defines the schedule.

*** Pros and Cons
#timersproscons

Advantages of using timers instead of cron jobs:

  - failed timers can be listed and monitored
  - output is automatically logged
  - timer is not executed if an instance is already running
  - easier to locate compared to the various files and directories with cron
    jobs
  - run job out of schedule without remembering the exact commandline
  - memory and CPU limits
  - precision is one second

Disadvantages:

  - takes more time to set them up
  - less control for users' jobs

Undecided:

Timers do not send automatically a notification on failed jobs.

With systemd timers ensure that your scripts provide a proper exit code on
errors instead of relying on messages going to standard error output.

*** Activation

The timer needs to be enabled and started to be active:

{{{
$ systemctl enable certbot.timer
$ systemctl start certbot.timer
}}}

Display active timers:

{{{
$ systemctl list-timers
}}}

Display all timers:

{{{
$ systemctl list-timers --all
}}}

*** Schedule

You can always run the job manually independent with of the current schedule
with:

{{{
$ systemctl start certbot.service
}}}

Every 20 minutes:

{{{
OnCalendar=*:0/20
}}}

#oddandevenhours
Every even hour (0:00, 2:00, ...)

{{{
OnCalendar=0/2:00:00
}}}

Every odd hour (1:00, 3:00, ...)
{{{
OnCalendar=1/2:00:00
}}}

Once a day at midnight:

{{{
OnCalendar=daily
}}}

Once a day at 15:44:

{{{
OnCalendar=15:44
}}}

Monday noon:
{{{
OnCalendar=Mon 12:00
}}}

Start corresponding service immediately if last time was missed (e.g. system was down):

{{{
Persistent=true
}}}

*** Certbot example

Certbot service unit:

{{{
$ systemctl cat certbot.service
# /lib/systemd/system/certbot.service
[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://letsencrypt.readthedocs.io/en/latest/
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true
}}}

Certbot timer unit:

{{{
$ systemctl cat certbot.timer
# /lib/systemd/system/certbot.timer
[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target
}}}

Certbot timer status:

{{{
$ systemctl status certbot.timer
● certbot.timer - Run certbot twice daily
   Loaded: loaded (/lib/systemd/system/certbot.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Wed 2019-04-03 08:00:27 CEST; 2 months 13 days ago
  Trigger: Mon 2019-06-17 10:36:39 CEST; 18h left
}}}

*** Testing

Testing *OnCalendar* settings:

{{{
$ systemd-analyze calendar $(systemctl cat myservice.timer | sed -n 's/^OnCalendar=//p')
  Original form: *:15:44
Normalized form: *-*-* *:15:44
    Next elapse: Sun 2020-01-05 18:15:44 CET
       (in UTC): Sun 2020-01-05 17:15:44 UTC
       From now: 45min left
}}}

*** Resources

  archlinux Wiki :: [[https://wiki.archlinux.org/index.php/Systemd/Timers]]

** Hostname

Set hostname:

{{{
$ hostnamectl set-hostname buster-test-box
}}}

This is not persistent on Ubuntu. In order to change this, you need to edit
=/etc/cloud/cloud.cfg=:

{{{
preserve_hostname: true
}}}

** Targets

Show dependencies for a target:

{{{
$ systemctl list-dependencies network-online.target
network-online.target
● ├─networking.service
● └─NetworkManager-wait-online.service
}}}

** DNS resolver

Show status of DNS resolver:

{{{
$ sudo resolvectl status
}}}

** Time

*** Synchronization (NTP)

Configuration file: =/etc/systemd/timesyncd.conf=

Specific NTP servers (e.g. Active Domain Controllers) can be added in the NTP= line, separated by whitespace.
Fallback NTP servers are configured in the FallbackNTP= line, for example

{{{
[Time]
NTP=
FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org
}}}

Restart =systemd-timesyncd= service after changing the configuration:

{{{
$ sudo systemctl restart  systemd-timesyncd
}}}

*** Timezone

Show time and date details:

{{{
$ timedatectl
...
}}}

Change timezone:

{{{
$ timedatectl set-timezone Europe/Berlin
}}}

** IP access control
#ipacl

This feature was introduced with systemd 235, which is available in recent Linux distributions like Debian Buster, Ubuntu Bionic and CentOS 8.

The following example comes from a check_mk monitoring agent which is listening on port 6556. To restrict access to the monitoring server we add the following drop-in (e.g. =etc/systemd/system/check_mk.socket.d/10-ipacl.conf=):

{{{
[Socket]
IPAddressDeny=any
IPAddressAllow=203.0.113.114
}}}

*** Resources

[[http://0pointer.net/blog/ip-accounting-and-access-lists-with-systemd.html][IP Accounting and Access Lists with systemd]]

** Logging (journalctl)

*** Messages since last reboot

{{{
$ sudo journalctl -b
}}}

*** Show kernel messages

This equivalent to =dmesg -T=.

{{{
$ sudo journalctl -k
}}}

*** Line wrapping

Long log lines are not wrapped as with *less*, which makes it hard to see the complete message on the console.

You can change this behaviour with the *SYSTEMD_PAGER* environment variable:

{{{
$ export SYSTEMD_PAGER="less -r"
}}}

*** Vacuum logs

Retain only the past four days:

{{{
$ sudo journalctl --vacuum-time=4d
}}}

Retain only the past 750 MB:

{{{
$ sudo journalctl --vacuum-size=750M
}}}

** Templates

Example: =/lib/systemd/system/postgresql@.service= on Debian for PostgreSQL cluster.

*** OOM

{{{
# prevent OOM killer from choosing the postmaster (individual backends will
# reset the score to 0)
OOMScoreAdjust=-900
}}}

** Troubleshooting
#reexeccasemaria

{{{
● mariadb.service - MariaDB 10.3.22 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-03-25 11:32:09 CET; 1h 1min ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 638935 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=217/USER)

Mar 25 11:32:09 belukha systemd[1]: Starting MariaDB 10.3.22 database server...
Mar 25 11:32:09 belukha systemd[638935]: mariadb.service: Failed to determine user credentials: No such process
Mar 25 11:32:09 belukha systemd[638935]: mariadb.service: Failed at step USER spawning /usr/bin/install: No such process
Mar 25 11:32:09 belukha systemd[1]: mariadb.service: Control process exited, code=exited, status=217/USER
Mar 25 11:32:09 belukha systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 25 11:32:09 belukha systemd[1]: Failed to start MariaDB 10.3.22 database server.
}}}

Fixed by:

{{{
systemctl daemon-reexec
systemctl start mariadb
}}}

** Old init scripts

Run them without systemd stepping on its toes:

<example>
export _SYSTEMCTL_SKIP_REDIRECT=1
/etc/init.d/sympa start
</example>