Debug systemd and define a loglevel

In the past days I had to debug quite old (up2date “enterprise”) software. It ships with multiple bash and sh scripts that together start a java service. All nicely wrapped into a sysv init script. Wrapped in a systemd unit. sucks…

A systemctl start $unit / systemctl stop $unit works most of the time, a systemctl restart $unit never worked. The process forks multiple bash/sh scripts which fork the java process at the end. systemd isn’t able to keep track of the correct pid for the main process, it thinks the main process is one of the sh scripts. The whole startup only works because of RemainAfterExit=yes. A nasty workaround would be to set PIDFile=. The software even creates one, but that isn’t owned by root. systemd declines reading pidfiles that aren’t owned by root.

All in all this sucks hard. The scripts come from the vendor so I cannot easily change them. One way to improve debugging and the lost pid is by increasing the systemd-internal log level:

systemd-analyze set-log-level debug

Afterwards the journal of the restarted unit contains more information. systemd in my case complained that it received a SIGCHLD from a forked process which sometimes(?) triggers systemd to immediately stop the unit again. Now it’s time for futher debugging!

Posted in General, Linux, Short Tips | Leave a comment

systemd unit hardening followup followup

I did some more research on systemd hardening and found another blogpost series that I can highly recommend: https://www.ctrl.blog/entry/systemd-service-hardening.html . The first article is quite similar to mine, but the followup articles go a bit more into details. Check them out!

Posted in General, IT-Security, Linux, Short Tips | Leave a comment

Puppet PQL Queries

PQL syntax can be a bit tricky/ugly. It took me some time to figure this out so I thought sharing it isn’t a bad idea.

Get all nodes with a specific class in their last catalog

puppet query 'nodes[certname] {resources {type = "class" and title = "CapitalizedClassname"}}'

This gives us a list of all nodes that have have a class, for example Apache, in their catalog. Maybe want to have a list of all nodes that use a specific module (that could be Apache::Vhosts but not Apache):

puppet query 'nodes[certname] {resources {type = "class" and title ~ "CapitalizedClassname"}}'

The query can be combined with more matchers, for example on the certificate name:

puppet query 'nodes[certname] {certname ~ "^bla-" and resources {type = "class" and title = "CapitalizedClassname"}}'
Posted in General, Linux, Puppet, Short Tips | Leave a comment

PostgreSQL: Do a VACUUM FULL without exclusive locks!

So, a strange title today. What’s an exclusive lock, what’s a vacuum, why can it be full and what has all this to do with PostgreSQL you might ask yourself.

How PostgreSQL deletes data

In very short: If you delete a row or dataset (also called tuple in the PostgreSQL world) in a table, this is marked as ‘please delete me from disk later whenever you have time but please before the disk is actually full’. PostgreSQL runs a vacuum in the background from time to time. This deletes old tuples from disk and frees up diskspace. This is called autovacuum.

Performance problems with autovacuum

Scanning the data on disk for tuples that are marked to-be-deleted costs IO. People using the database and doing lots of updates/inserts also require IO. If autovacuum is too agressive, the other database operations will be slow. If autovacuum is disabled/ or too slow your disk will fill up. That’s the moment you have to run a VACUUM FULL. This is a manual command that locks the tables with an exclusive lock (nobody else can write) and cleans up the tuples. Also the manual vacuum can often delete a few tuples that the autovacuum cannot.

Using pg_repack

Now comes pg_repack into play! This is a PostgreSQL extension that can cleanup dead tuples like VACUUM FULL, but without an exclusive lock! pg_repack is currently not distributed as a package from the PostgreSQL people, so you’ve to compile it yourself. Afterwards load it into the database and it’s ready to be used! Their documentation is quite good so I won’t copy and paste their installation docs: reorg.github.io/pg_repack/

Posted in General, Linux | Leave a comment

systemd unit hardening followup

at https://blog.bastelfreak.de/2022/01/systemd-unit-hardening/ I blogged about systemd hardening. While doing some research for a followup post I discovered https://docs.arbitrary.ch/security/systemd.html. This covers *a lot* about systemd hardening and general linux optimization. I can highly recommend reading the whole documentation (and it kinda makes my planned blogpost obsolete, so I will postpone it).

Posted in General, IT-Security, Linux, Short Tips | 1 Comment

Migrate CentOS 8 to AlmaLinux

CentOS 8 is dead since the end of 2021 (while CentOS 7 still has support but is really really old). There are a few alternatives. You can upgrade to CentOS Stream, to AlmaLinux or Rocky Linux. CentOS Stream is an odd rolling release distribution that I’m not interested in (I already maintain Gentoo and Arch Linux boxes). I wanted to do a huge article about migrating CentOS to AlmaLinux, sadly that’s really simple:

dnf --assumeyes update
sync
reboot
curl --silent --remote-name https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
bash almalinux-deploy.sh
reboot

It’s a really good idea to review the script before you execute it. it basically updates the repositories and reinstalls all packages.

Posted in Linux, Short Tips | Leave a comment

DNS Setup for own domains

There are many different options to operate your own domain. From a registrar you buy a domain name. The registrar publishes NS records to the registry. Those NS records point to nameservers (or DNS servers or authoritative DNS servers). Registrars usually offer you to host the DNS zone on their nameservers. Think about a DNS Zone as a txt file. Here is a short zone file for bastelfreak.de:

$ORIGIN .
_dmarc.bastelfreak.de   86400   IN      TXT     "v=DMARC1; p=none; rua=mailto:dmarc-reports@bastelfreak.de; ruf=mailto:dmarc-reports@bastelfreak.de"
bastelfreak.de  86400   IN      A       144.76.249.219
bastelfreak.de  86400   IN      AAAA    2a01:4f8:171:1152::8
bastelfreak.de  86400   IN      CAA     0 issue "letsencrypt.org"
bastelfreak.de  86400   IN      CAA     0 iodef "mailto:tim@bastelfreak.de"
bastelfreak.de  86400   IN      MX      1 mail.bastelfreak.de.
bastelfreak.de  86400   IN      NS      robotns3.second-ns.com.
bastelfreak.de  86400   IN      NS      robotns2.second-ns.de.
bastelfreak.de  86400   IN      NS      ns3.inwx.eu.
bastelfreak.de  86400   IN      NS      ns1.first-ns.de.
bastelfreak.de  86400   IN      NS      ns2.inwx.de.
bastelfreak.de  86400   IN      NS      ns.inwx.de.
bastelfreak.de  86400   IN      SOA     ns.inwx.de. postmaster.bastelfreak.de. 2021122302 1200 1800 1209600 3600
bastelfreak.de  86400   IN      SSHFP   3 2 2006a5beab176e9061e0f8c4ab49097ebc2c4566093822a5e98b09a66a7627e3
bastelfreak.de  86400   IN      SSHFP   2 2 eec6ad6e8b3b46ee2d27e24c5c299732bba2ca04f93435573f8391ad1193e116
bastelfreak.de  86400   IN      SSHFP   1 2 d36e80e044c0ed9db13c623b4b480ef3b8c5c3d3a96dbb13a5434dc6ff152079
bastelfreak.de  86400   IN      SSHFP   4 2 479269b5636c85b0b071cf084e6235168bd14309471da1cbd620a7cef9cab05e
bastelfreak.de  86400   IN      TXT     "v=spf1 ip4:144.76.249.221 ip6:2a01:4f8:171:1152::7 mx -all"
blog.bastelfreak.de     86400   IN      A       144.76.249.219
blog.bastelfreak.de     86400   IN      AAAA    2a01:4f8:171:1152::a
default._domainkey.bastelfreak.de       86400   IN      TXT     "v=DKIM1; k=rsa; s=email; " "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCkElHw8QtjcK39chikZjBgbN2pc6kI4z4Xa3TsfbtizWEbjnjPuO7WX0mvo+ARJBNeOuBN+Ez6fPo/UOBCjx/mIuHJFY68Vea81qeM5NSYvo16fUxEONojYTPAK7tn+Zf80n+e17MJGADNFTF7YcbhRJtxtK9jeRK0kNOm5qGMxwIDAQAB"
mail.bastelfreak.de     86400   IN      A       144.76.249.221
mail.bastelfreak.de     86400   IN      AAAA    2a01:4f8:171:1152::7
www.bastelfreak.de      86400   IN      CNAME   bastelfreak.de.

Often you can edit your zonefile via a webinterface from the registrar. Which alternatives exist? You can host your own nameserver(s)! Most Registries (for example the DENIC, that operates .de tld) require you to provide multiple nameservers, and if you want to have good connectivity around the globe and low latency, you need many many nameservers that are anycasted. So Instead of operating all nameservers for your domains, you can also just run one, called hidden primary, which sends updates via AXFR to the public nameservers, that will be queried by the rest of the internet / are written down via NS resource records at the registry. That’s the approach that I’m doing since some time.

I’ve all my domains registered via INWX. They are cheap and they support a hidden primary server. For each change on a domain, my DNS server needs to send a notify to their special nameserver. That will notify their public servers and they will do an IXFR from my DNS server. To make this more redundant, I also do AXFR to the Hetzner nameservers. This means all domains are served by three INWX DNS servers and three Hetzner DNS servers. I do all this with PowerDNS on Arch Linux.

How does that work you might ask yourself. Let me explain! First, I configured the desired nameserver in the registrar, INWX, to publish them into the .de zone:

# whois bastelfreak.de | grep Nserver
Nserver: ns1.first-ns.de
Nserver: ns2.inwx.de
Nserver: ns3.inwx.eu
Nserver: ns.inwx.de
Nserver: robotns2.second-ns.de
Nserver: robotns3.second-ns.com

Afterwards I configured the zone within PowerDNS, as shown in the zone above. In addition, we need to allow AXFR from the Hetzner nameserver and notify the special INWX nameserver when we update the zone:

 pdnsutil get-meta bastelfreak.de
Metadata for 'bastelfreak.de'
ALLOW-AXFR-FROM = 185.181.104.96, 2a01:4f8:0:a101::a:1, 213.133.105.6, 2a01:4f8:d0a:2004::2, 193.47.99.3, 213.239.242.238, 2001:67c:192c::add:a3
ALSO-NOTIFY = 185.181.104.96
Posted in Linux | Leave a comment

Setup Gentoo on a Hetzner server

I really like Gentoo for their awesome package manager, Portage. Gentoo is a really flexible distribution that you can customize (and break) in many ways. It’s a good opportunity to learn a lot about linux. I documented the installation process. Given is an EX server from Hetzner, booted into the Debian rescue system.

As a first step, we setup the partitioning + mdadm Raid + LVM + filesystems:

parted /dev/nvme0n1 --script mklabel gpt
parted /dev/nvme1n1 --script mklabel gpt
parted /dev/nvme0n1 --script mkpart primary ext3 2048s 4095s
parted /dev/nvme1n1 --script mkpart primary ext3 2048s 4095s
parted /dev/nvme0n1 --script mkpart primary ext3 4096s 1953791s
parted /dev/nvme1n1 --script mkpart primary ext3 4096s 1953791s
parted /dev/nvme0n1 --script mkpart primary ext3 1953792s 100%
parted /dev/nvme1n1 --script mkpart primary ext3 1953792s 100%
parted /dev/nvme0n1 --script set 1 bios_grub on
parted /dev/nvme1n1 --script set 1 bios_grub on
mdadm --verbose --create /dev/md/0 --level=1 --raid-devices=2 --metadata=1.2 /dev/nvme0n1p2 /dev/nvme1n1p2
mdadm --verbose --create /dev/md/1 --level=1 --raid-devices=2 --metadata=1.2 /dev/nvme0n1p3 /dev/nvme1n1p3
echo 999999999 > /proc/sys/dev/raid/speed_limit_min
echo 999999999 > /proc/sys/dev/raid/speed_limit_max
until ! grep -q resync /proc/mdstat; do echo "sleeping for 2s"; sleep 2; done
mkfs.ext4 -v /dev/md/1
pvcreate --verbose /dev/md/2
vgcreate --verbose vg0 /dev/md/2
lvcreate --verbose --name root --size 50G vg0
mkfs.ext4 -v /dev/mapper/vg0-root
mount /dev/mapper/vg0-root /mnt/

Next, we download a stage 3 tarball (minimal precompiled Gentoo basically) and verify it:

url='https://mirror.netcologne.de/gentoo/releases/amd64/autobuilds/current-stage3-amd64-systemd'
latest='stage3-amd64-systemd-20220102T170545Z.tar.xz'
for file in '' {.CONTENTS.gz,.DIGESTS.asc}; do wget "${url}/${latest}${file}"; done
gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0xBB572E0E2D182910
gpg --verify "${latest}.DIGESTS.asc"
grep " ${latest}.CONTENTS.gz" *.DIGESTS.asc
openssl dgst -r -sha512 "${latest}.CONTENTS.gz"

This will verify the gpg signature in the .asc File. Afterwards we grep the SHA512 checksum for CONTENTS.gz from the .asc file. Then we run openssl to compute the SHA512 checksum on the actual stage 3. Compare the two SHA512 checksums, they have to be identical. If they are, continue with extracting the tarball:

mv stage3-amd64-hardened-selinux-*.tar.xz /mnt/
cd /mnt/
tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner
mount /dev/md/1 /mnt/boot/

Now we can prepare the chroot:

cp --dereference /etc/resolv.conf /mnt/etc/
mount --types proc /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys
mount --rbind /dev /mnt/dev
mount --make-rslave /mnt/dev
mkdir /mnt/hostlvm
mount --bind /run/lvm /mnt/hostlvm
chroot /mnt/gentoo /bin/bash
ln -s /hostlvm /run/lvm
source /etc/profile
export PS1="(chroot) ${PS1}"

Now we can configure portage:

mkdir /etc/portage/repos.conf
cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf
echo 'MAKEOPTS="-j6"' >> /etc/portage/make.conf
echo 'USE="systemd ipv6"' >> /etc/portage/make.conf
sed -i 's/^COMMON_FLAGS.*/COMMON_FLAGS="-march=native -O2 -pipe"/g' /etc/portage/make.conf
emerge --sync
emerge --ask --verbose --update --deep --newuse @world
emerge --depclean
echo '=sec-policy/selinux-base-policy-9999 **' >> /etc/portage/package.accept_keywords
# set profile to hardened..., see eselect profile list
emerge --ask --verbose --unmerge sysvinit eudev
echo 'sys-fs/mdadm static' >> /etc/portage/package.use/mdadm
echo 'sys-fs/lvm2 lvm2create_initr' >> /mnt/etc/portage/package.use/lvm2
echo 'app-admin/puppet augeas diff doc rrdtool' > /etc/portage/package.use/puppet
emerge --ask --autounmask-write --verbose sys-kernel/gentoo-sources sys-apps/systemd vim htop nload iftop iptraf-ng strace lsof gentoolkit intel-microcode pciutils genkernel dstat grub mdadm lvm2 smartmontools dropbear ccze fail2ban tcpdump dev-vcs/git puppet dfc gptfdisk ethtool net-misc/ipcalc ndisc6
echo 'LANG="en_US.utf8"' >> /etc/locale.conf
# setup /etc/systemd/network/50-dhcp.network
mdadm --detail --scan >> /etc/mdadm.conf
sed -i 's/.*LVM=.*/LVM="yes"/' /etc/genkernel.conf
sed -i 's/.*MICROCODE=.*/MICROCODE="yes"/' /etc/genkernel.conf
sed -i 's/.*SSH=.*/SSH="yes"/' /etc/genkernel.conf
sed -i 's/.*BUSYBOX=.*/BUSYBOX="yes"/' /etc/genkernel.conf
sed -i 's/.*MDADM=.*/MDADM="yes"/' /etc/genkernel.conf
sed -i 's/.*E2FSPROGS=.*/E2FSPROGS="yes"/' /etc/genkernel.conf
genkernel all
sed -i 's/.*GRUB_DISABLE_SUBMENU.*/GRUB_DISABLE_SUBMENU=y/g' /etc/default/grub
sed -i 's/.*GRUB_TIMEOUT=.*/GRUB_TIMEOUT=15/g' /etc/default/grub
sed -i 's|.*#GRUB_CMDLINE_LINUX=.*|GRUB_CMDLINE_LINUX="init=/usr/lib/systemd/systemd dolvm domdadm dossh rootfstype=ext4"|' /etc/default/grub
for dev in /dev/nvme?n1; do grub-install "${dev}"; done
grub-mkconfig -o /boot/grub/grub.cfg
# get UUID with `blkid /dev/mapper/vg0-root /dev/md1` and update /etc/fstab
passwd
mkdir ~/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKC4uaKuYzMGK4jlTvPlbnMP9n+gdac65480/eDTMWRw bastelfreak" > ~/.ssh/authorized_keys
sed -i 's/.*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl enable systemd-networkd sshd

Setup language/locales

echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen
# verify that our locale is present:
locale -a | grep en_US.utf8
eselect locale set en_US.utf8
echo 'dns_domain_lo="bastelfreak.org"' >> /etc/conf.d/net
echo 'hostname="hypervisor01"' >> /etc/conf.d/hostname
. /etc/profile
env-update && source /etc/profile

Setup timesyncd and DNS

sed -i 's/#NTP=/NTP=ntp1.hetzner.de ntp2.hetzner.com ntp3.hetzner.net/' /etc/systemd/timesyncd.conf
rm /etc/localtime
ln -s /usr/share/zoneinfo/Europe/Berlin /etc//localtime
sed -i 's/#DNS=/DNS=2a01:4f8:0:1::add:1010 2a01:4f8:0:1::add:9999 2a01:4f8:0:1::add:9898/' /etc/systemd/resolved.conf
sed -i 's/#Domains=/Domains=bastelfreak.org/' /etc/systemd/resolved.conf
systemctl enable systemd-timesyncd systemd-resolved

Now we can reboot! I suggest to configure a password first and/or create a user. After the first boot we can continue with some fancy pancy stuff:

Setup LLDP:

echo 'net-misc/lldpd jansson' > /etc/portage/package.use/lldpd
emerge --ask lldpd
systemctl enable lldpd
systemctl start lldpd

Setup all the fancy dotfiles:

cd ~
git clone https://github.com/bastelfreak/scripts.git
ln -s ~/scripts/vimrc ~/.vimrc
ln -s ~/scripts/bashrc ~/.bashrc
ln -s ~/scripts/bash_profile ~/.bash_profile
mkdir -p ~/.vim/backupdir/
mkdir ~/.vim/ftplugin
echo "set colorcolumn=80" >> ~/.vim/ftplugin/tex.vim
git clone https://github.com/gmarik/Vundle.vim.git ~/.vim/bundle/Vundle.vim
vim +PluginInstall +qall

And last but not least, you might want to run some virtual machines, so install Qemu and Libvirt:

echo 'app-emulation/libvirt zeroconf virt-network pcap parted lvm' > /etc/portage/package.use/libvirt
echo 'app-emulation/qemu spice virtfs usb usbredir' > /etc/portage/package.use/qemu
emerge --ask qemu libvirt ebtables dmidecode openvswitch bridge-utils
dispatch-conf
emerge --ask qemu libvirt ebtables dmidecode openvswitch bridge-utils
systemctl enable ovsdb-server
systemctl start ovsdb-server
systemctl start ovs-vswitchd
systemctl enable ovs-vswitchd
ovs-vsctl add-br br0

You now have a proper Gentoo box. I suggest to configure at least backups and a firewall. Please keep in mind that Gentoo is a rolling release distribution, so some of the commands might be obsolete after some time or you need to configure something differently. Still, this walkthrough should give you a good first impression.

Posted in General, Linux, Virtualization | Leave a comment

systemd-networkd + wireguard configuration

As mentioned in the previous post, networkd is quite nice for network configurations. It can also configure network devices, such as wireguard tunnels. The following config can go into a .netdev file (like /etc/systemd/network/as3668-1.netdev):

[NetDev]
Name=as3668-1
Kind=wireguard
MTUBytes=1412

[WireGuard]
PrivateKeyFile=/etc/wireguard/as3668-1
ListenPort=1337

[WireGuardPeer]
PublicKey=WiN46vGCfAGuH7p6mc+9zLvtmuACdyMtXULETbGP2SM=
Endpoint=router01.tld:1337
PersistentKeepalive=5
AllowedIPs=fe80::/64
AllowedIPs=fd00::/8
AllowedIPs=0.0.0.0/0

The configuration reads the private key from /etc/wireguard/as3668-1, it connects to router01.tld on port 1337, sends a keepalive packet every 5 seconds and allows all traffic to flow through the tunnel. The is configured to 1412 because this goes through a VDSL line. Of course you can configure all of this via Puppet! I published a module at https://forge.puppet.com/modules/puppet/wireguard. The following snippet creates the above configuration:

wireguard::interface {'as3668-1':
  source_addresses      => ['144.76.249.216', '2a01:4f8:171:1152::11'],
  public_key            => 'WiN46vGCfAGuH7p6mc+9zLvtmuACdyMtXULETbGP2SM=',
  endpoint              => 'router01.tld1337',
  dport                 => 1337,
  input_interface       => $facts['networking']['primary'],
  addresses             => [{'Address' => '169.254.0.8/32', 'Peer' =>'169.254.0.7/32'},{'Address' => 'fe80::beef:e/64'},],
  destination_addresses => [], 
  persistent_keepalive  => 5,
}

It will also create a .network file:

[Match]
Name=as3668-1

[Network]
DHCP=no
IPv6AcceptRA=false
IPForward=yes

# for networkd >= 244 KeepConfiguration stops networkd from
# removing routes on this interface when restarting
KeepConfiguration=yes

[Address]
Address=169.254.0.8/32
Peer=169.254.0.7/32

[Address]
Address=fe80::beef:e/64

Posted in General, Linux, Puppet | Leave a comment

systemd-networkd configuration

Systemd is used in all major Linux distributions. One of the components, systemd-networkd, provides a unified way to manage network interfaces and related settings (like routes, MTU) in a inifile-like way. This is quite awesome because it enables system administrators to use the same configuration style on all their operating systems.

Simple layer 2 IP assignment

Here is a very simple configuration to assign an IPv4 and IPv6 address to one interface

[Match]
Name=uplink

[Address]
Address=192.168.1.2/32
Peer=192.168.1.1/32

[Address]
Address=2a01:4f8:181:1157::3/128

[Route]
Gateway=192.168.1.1
GatewayOnLink=yes
PreferredSource=192.168.1.2
InitialCongestionWindow=10
InitialAdvertisedReceiveWindow=10

[Route]
Gateway=fe80::506e:45ff:fef1:ba4d
GatewayOnLink=yes
PreferredSource=2a01:4f8:181:1157::3
InitialCongestionWindow=10
InitialAdvertisedReceiveWindow=10

This can be saved in /etc/systemd/network/ with a .network suffix. I recommend to use the interface name in the file: /etc/systemd/network/uplink.network. This example assumes that the system is in 192.168.1.2/24. But configured is the address as /32. That means it has no layer 2 peers. It has an onlink connection to the router, 192.168.1.1. This is an important detail, especially for larger environments. The configuration does no ARP traffic. And there won’t be much ARP on the switches as well. The same happens with IPv6 and NDP.

Rename an interface

You might ask yourself, why is the interface named uplink? You can rename interfaces with systemd-network too! And I like to name them by their purpose. Throw the following into a .link file (the MAC address is used as an identifier, many other options are available as well):

# /etc/systemd/network/10-uplink.link
[Match]
Driver=virtio_net
MACAddress=52:54:00:78:63:9c

[Link]
Description=Uplink interface
Name=uplink
Posted in General, Linux, Virtualization | Leave a comment