SysV init vs Systemd

In today’s meeting with the great guys at Victoria Linux Users Group, discussion turned to SysV init scripts vs Systemd, because that topic never gets old.

The init script for postfix was raised as an example:

Looking at /etc/init.d/postfix (on Ubuntu Server 18.04, and skipping the INIT INFO preamble):

test -x $DAEMON && test -f /etc/postfix/main.cf || exit 0

Okay, we’re testing for executable (-x) file, and existence of /etc/postfix/main.cf – easy-ish if you already know about test, -x, -f, ||, etc. Scrolling further…

. /lib/lsb/init-functions
#DISTRO=$(lsb_release -is 2>/dev/null || echo Debian)

Brilliant, what does /lib/lsb/init-functions do? There’s no man page for it. There exists such a file, but it’s not executable. Is it safe to run it with –help to find out what it does? How about you try it and report back?

Anyway, it gets more cryptic from there – have a look for yourself: less /etc/init.d/postfix and see if it makes much sense to you? I say this as someone who enjoys bash scripting…

Now, let’s look at Systemd’s init script:

# systemctl cat postfix.service   
# /lib/systemd/system/postfix.service
[Unit]
Description=Postfix Mail Transport Agent
Conflicts=sendmail.service exim4.service
ConditionPathExists=/etc/postfix/main.cf

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/true
ExecReload=/bin/true

[Install]
WantedBy=multi-user.target

Legibility

systemd: +1 SysV init: 0

Okay, that’s at least legible and it’s easy to see the conflicts. Weird thing though: ExecStart = /bin/true. How does postfix actually start then?

But wait, there’s another postfix service: postfix@-.service (“@” indicating a specific instance):

# systemctl cat postfix@.service  
# /lib/systemd/system/postfix@.service
[Unit]
Description=Postfix Mail Transport Agent (instance %i)
Documentation=man:postfix(1)
PartOf=postfix.service
Before=postfix.service
ReloadPropagatedFrom=postfix.service
After=network-online.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
GuessMainPID=no
ExecStartPre=/usr/lib/postfix/configure-instance.sh %i
ExecStart=/usr/sbin/postmulti -i %i -p start
ExecStop=/usr/sbin/postmulti -i %i -p stop
ExecReload=/usr/sbin/postmulti -i %i -p reload

[Install]
WantedBy=multi-user.target

There’s more info there, still legible, but I don’t know all those options it’s showing. Of course, there’s lots of documentation if we run man systemd.unit (1598 lines of output).

Documentation

systemd: +2 SysV init: 0

A quick search in the man pages shows list-dependencies, when combined with –before, shows the services that:

--before
    With list-dependencies, show the units that are ordered after the
    specified unit. In other words, recursively list units following
    the Before= dependency.

# systemctl --before list-dependencies postfix@-.service
postfix@-.service
● ├─postfix.service
● └─shutdown.target

That’s lovely, actually. It’s shown above, in the unit file, that this service is PartOf=postfix.service and needs it to run first: Before=postfix.service.

Transparency

systemd: +3 SysV init: 0

And yet, what processes does running postfix actually start? There is no postfix process running, it just launches some other stuff, from man postfix there are a lot of other processes:

  Daemon processes:
   anvil(8), Postfix connection/rate limiting
   bounce(8), defer(8), trace(8), Delivery status reports
   cleanup(8), canonicalize and enqueue message
   discard(8), Postfix discard delivery agent
   dnsblog(8), DNS black/whitelist logger
   error(8), Postfix error delivery agent
   flush(8), Postfix fast ETRN service
   local(8), Postfix local delivery agent
   master(8), Postfix master daemon
   oqmgr(8), old Postfix queue manager
   pickup(8), Postfix local mail pickup
   pipe(8), deliver mail to non-Postfix command
   postlogd(8), Postfix internal logging service
   postscreen(8), Postfix zombie blocker
   proxymap(8), Postfix lookup table proxy server
   qmgr(8), Postfix queue manager
   qmqpd(8), Postfix QMQP server
   scache(8), Postfix connection cache manager
   showq(8), list Postfix mail queue
   smtp(8), lmtp(8), Postfix SMTP+LMTP client
   smtpd(8), Postfix SMTP server
   spawn(8), run non-Postfix server
   tlsmgr(8), Postfix TLS cache and randomness manager
   tlsproxy(8), Postfix TLS proxy server
   trivial-rewrite(8), Postfix address rewriting
   verify(8), Postfix address verification
   virtual(8), Postfix virtual delivery agent

# ps -e | grep -Ei "postfix|anvil|postsuper|postqueue|qmgr"
3353956 ?        00:00:01 anvil
3705336 ?        00:00:17 qmgr

So far, the where and how of postfix’s sub-process daemons initialization happens when /usr/sbin/postmulti is run. It’s called in /etc/init.d/postfix, and seen above in the unit file under ExecStart=/usr/sbin/postmulti -i %i -p start. Personally, I find it easier to look for ExecStart.

Shifting topic a bit, it cannot be overstated how handy it is for systemd to fully implement tab completion. So systemd [tab][tab] shows the sub-command systemd-analyse, etc. And auto-completing that, then entering another [tab][tab] shows an option called blame. Running systemd-analyse blame | grep postfix shows us the time it took to initialize the services:

# systemd-analyze blame | grep postfix
1.878s postfix@-.service                                      
  10ms postfix.service

There are many, many more options (plot sounds interesting – it makes an SVG of the init processes!).

Usability

systemd: +4 SysV init: 0

Finally, starting processes in parallel (systemd) vs sequential (SysV init) is a win, especially when starting & stopping many VMs and / or containers daily.

Speed of boot process

systemd: +5 SysV init: 0

So, be it resolved, systemd > SysV init. Case closed.