OSDC 2016 Talk Recommendations

OSDC – The Open Source Datacenter Conference – just happend a few days back. Here is a short list of videos that I can recommend:

A Job and an Adventure – Dawn Foster

I met Dawn once at Puppetcamp Düsseldorf, she organized the event. Dawn is a powerful woman, speaking about open source and getting a job in that area.

An Introduction to Software Defined Networking SDN – Martin Loschwitz

Martin is well known for his work at Hastexo and now at Sys11. He is responsible for the Sys11-Stack. I had the opportunity to see his work at the last Sys11 Conference.

Introduction to Testing Puppet Modules – David Schmitt

David is a Senior Software Engineer at Puppet(labs), he puts a huge effort in making rspec and Puppet Modules better, rspec basics are the central point of his talk. I’m lucky to have David as an advisor for my bachelor thesis this summer.

What´s wrong with my Puppet – Felix Frank

Felix is a featured Puppet Community member since a very long time. You can always recognize him by his hair. I was able to meet him at the last config mangement camp and we had a great time. In this talk is goes into detailed puppet debugging for different use cases with different tools.

Posted in 30in30, General, Internet found pieces | Leave a comment

Let’s Encrypt automation – The awesome way

Free SSL for the mass \o/
Cryptography is important. I like to encrypt as much traffic and data as possible, not only the important stuff. Let’s Encrypt is a new project sponsored by multiple big companies and the Linux Foundation to provide free and automated SSL certificates for everyone. There are a few – not so awesome – solutions to get a certificate. The project ships a little daemon which can communicate with their API, but I don’t like that. Running a daemon is always a security challenge. It it possible to use the daemon as a client only, start it once, renew cert/get a new one, exit.

My fellow aibo blogged about this in January and created a nice systemd service + timer for that. You had to run the command from the service once via terminal because it asks you to accept their Terms of Service and to provide an email address.

I recently made a little adjustment together with aibo to also provide these to information, now you can completely automate the SSL setup. Here is out modified service file:

Setup:

[Unit]
Description=renew certificates for %I

[Service]
Type=oneshot
ExecStartPre=/usr/bin/mkdir -p /tmp/letsencrypt-auto
ExecStart=/usr/bin/letsencrypt certonly \
  --webroot \
  --webroot-path=/tmp/letsencrypt-auto \
  --renew-by-default \
  --keep \
  --agree-tos \
  --email tim@bastelfreak.de \
  -d %I
ExecStartPost=/usr/bin/nginx -s reload

[Install]
WantedBy=multi-user.target

Save that as /etc/systemd/system/letsencrypt-renew@.service, also get the following timer for /etc/systemd/system/letsencrypt-renew@.timer:

[Unit]
Description=run cert renew for %I every two month

[Timer]
OnCalendar=*-*/2-4 1:0:0
Persistent=true

[Install]
WantedBy=multi-user.target

You now want a SSL cert for myawesomestuff.example.com? Just do systemctl enable letsencrypt-renew@myawesomestuff.example.com.timer and wait until the timer starts. Or if you want a new cert now, just run systemctl start letsencrypt-renew@myawesomestuff.example.com.serice. You need more certificates? Just enable the timer again with a different domain name \o/

Webserver integration:
Here is a snippet from my nginx vhost:

upstream jenkins {
  server 127.0.0.1:8090 fail_timeout=0;
}

server {
  listen 80;
  listen [::];
  server_name ci.virtapi.org;

  location /.well-known {
    root /tmp/letsencrypt-auto;
  }

  location / {
    return 301 https://$host$request_uri;
  }
}

server {
  listen 443 ssl;
  listen [::]:443 ssl;

  server_name ci.virtapi.org;

  ssl_certificate /etc/letsencrypt/live/ci.virtapi.org/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/ci.virtapi.org/privkey.pem;

  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
  }
}

Conclusion:
ssl all the things
Let’s encrypt is really cool, systemd is also cool, the combination is even cooler. This brings us a lightweight solution to get as many certificates as we want.

Posted in 30in30, General, Internet found pieces, IT-Security, Linux | Leave a comment

Short Tip: Fiddling around with login shells

I’m currently playing around with LARS, this script collection creates a arch ISO with some post-install magic, for example setting a login shell for root. The used code is:

usermod -s /usr/bin/bash root

I could boot up the image and ssh into it with my key. But three things weren’t working:

  • Login as normal user and do su
  • Login at a TTY
  • Login via ssh with a password

I got a “Access Denied” message in all three cases. I digged around for two days. I know that Arch Linux moved binaries around a few years back, /bin is now a symlink to /usr/bin. Just for fun a colleague changed the shell in the /etc/passwd to /bin/bash, just like in the good old days, and WTF everything was working?!
We found the /etc/shells file:

#
# /etc/shells
#

/bin/sh
/bin/bash

# End of file

This file lists all shells that are allowed to be used as a login shell. By default it lists the symlink for bash, not the absolute path itself… I still don’t know why the keybased login was working, but this is another mystery.

Posted in 30in30, General, Linux, Short Tips | Leave a comment

VirtAPI-Stack Review Part 1 – What happend in the last six months?

I announced the VirtAPI-Stack during the last “30 Posts in 30 Days” Challenge. Let us do a recap, what did we achive in the past 6 months?

VirtAPI-Stack description:

Let us start with a short description, what was that VirtAPI stack again? This is a combination of four different projects to solve one common problem: How do you do unattended linux installations in a large scale environment like a datacenter?

  • installimage – bash scripts for deploying every linux system
  • LARS – Live Arch Rescue System
  • VirtAPI – API for KVM Cloud Nodes and Instances
  • arch-package-builder – Toolbox to create a continuous Delivery Platform with Jenkins-CI for Arch Packages
  • Marmoset – Python based API to deal with LARS, installimage and PXE/DHCP servers

The installimage:
Today we will talk about the installimage. What exactly is this? A collection of bash scripts that creates partitions on your system, with mdadm raid and LVM2 if you like. Then it unzips a packed image of $yourmostloveddistro and throws it on the disk, does a bit of chroot stuff aaaand your system is ready. This works with Fedora, CentOS, Debian, Ubuntu; Suse and CoreOS. There is also support for Arch Linux, in which case an actual installation is performed. The cool thing is that installimage allowes you to use a guided installation with a lot of help and comments, but also to automate every step.

What happend?
We validated the complete installimage against shellcheck! I started this in the middle of October 2015, we finished it in February 2016 I think. We did a huge refactoring, added several hundred quotes and created a styleguide based on our decisions which I would like to share with you:

What else happened? We finally got a README.md which describes the project itself and every file in it in detail. We also created a flowchart to visualize the data flow, this was needed because it isn’t obvious to new people joining the team. Speaking of the team…, we’re already 18 people in the VirtAPI-Stack Github organisation, most of us are always present in our IRC channel #virtapi. We got code for the installimage from 6 different people, everybody else probably reviewed or helped by design decisions. Besides that we also did some general improvements. We removed legacy code, for example wrapping and parsing ifconfig and switched to ifdata and iputils2. A lot of iteration and ls parsing got removed, we try to use local instead of global variables as often as possible and we added Arch support. You can now start your installimage from an Arch system! (before that, only Debian was tested).Hetzner Online GmbH is the original author of the script. We contributed our changed back to them and vice versa to ensure compatibility.

Prospect for the future:
What will come in the next six months? Two companies will implement VirtAPI in their stack, one of them is Host Europe Group. We’re working on a report hook to create notifications for each installation step which will allow you a better forecast of the estimated installation time and better debugging solutions. Support to manage hardware raid and a few others are also on the agenda.

Contribution:
You’re interested in this project and would like to contribute or only use it? Let us know and join our IRC channel! You’re annoyed by foreman because it is too slow? You don’t want to manage Kickstart XML stuff + debootstrap but have to support both worlds? Let us know and we help you implementing the installimage!

Posted in 30in30, General, Linux | Leave a comment

Creating a reliable Arch Linux Package Build Environment

The Arch Linux developers created some fancy scripts to setup a build environment. It consists of a btrfs subvolume with a minimal Arch Linux installation (only packages from base-devel). A snapshot will be created for each build, the builds will be executed in separated systemd-nspawn containers.

Basic Requirements:
We need a mounted btrfs somewhere in our system, you can easily create a loopbackimage and mount that as a btrfs volume if you do not own any free partitions or logical volumes:

# create 75GB image, throw btrfs onto it and mount it
dd if=/dev/zero of=arch-test.btrfs bs=1G count=75
pacman -Syyu --noconfirm btrfs-progs
mkfs.btrfs arch-test.btrfs
mkdir /mnt/aur
mount arch-test.btrfs /mnt/aur/

First Installation:
We start with the installation of the Arch Linux devtools, this is the packaged collection of scripts for dealing with packages:

pacman -Syyu --noconfirm devtools

Now we define an environment variable with the path to our snapshots. This can be any writable sub-directory in the btrfs mount (our example is mounted at /mnt/aur). We will also setup the masterimage (the following block is intended to be run as a normal user and not root):

mkdir /mnt/aur/build_test
CHROOT=/mnt/aur/build_test
echo "export CHROOT=$CHROOT" >> ~/.bashrc
sudo mkarchroot $CHROOT/root base-devel

Now we can fetch a package source and build it (again as a normal user):

git clone https://aur.archlinux.org/puppetdb.git
cd puppetdb
makechrootpkg -c -r $CHROOT

This will spin up an nspawn container in a snapshot of our btrfs image, cleans the build env, build the packages and places it in our current directory. If you do `ls` now you should see `puppetdb-2.3.8-2-any.pkg.tar.xz`. Awesome, package is build.

Next step: automate this for a list of packages
I guess I don’t need to write much about that script as it is very simple:

#!/bin/bash

BASEDIR=/tmp/
CHROOT=/mnt/aur/build_test2
function clone_repo() {
  [ -n "$1" ] || exit
  cd "$BASEDIR" || exit
  rm -rf "$1"
  git clone "https://aur.archlinux.org/$1.git"
}

function build_package() {
  [ -n "$1" ] || exit
  cd "$BASEDIR/$1" || exit
  echo "$BASEDIR/$1"
  ls
  makechrootpkg -c -r $CHROOT
}

cd "$BASEDIR" || exit
while read -r package; do
  clone_repo "$package"
  if build_package "$package"; then
    echo -e "\033[01;32mwe just build $package for you\033[00m"
  else
    echo -e "\033[01;31mwe failed on $package /o\ \033[00m"
  fi
done < "$BASEDIR/packages"

This will work really good, unless one package has a dependency on another AUR package or needs an unknown GPG key to validate a signature. What comes next?

  • Moving to Jenkins-CI as build System
  • Move the package list to Github to allow collaboration
  • Automatic fetching for GPG keys
  • Resolving dependencies
  • Support for concurrent builds
  • Seperate jobs + history for each package
  • Pushing new packages to a mirror

Setup Jenkins-CI:
I plan to build packages, many many packages. Probably so many that I need to spread the load across multiple servers. Also, to make administration easier, a working Web UI + API + proper build history would be cool. GitHub repository, this holds a aur-packages file with all packages that we would like to build, also we’ve got a groovy script for Jenkins:

def PackagesFile = new File('/var/lib/jenkins/jobs/Arch Package Builder/workspace/aur-packages')
PackagesFile.eachLine { line ->
  packageName = line.trim()
  job("Arch_Package_${packageName}") {
    description("This Job builds the ${packageName} package for archlinux")
    concurrentBuild()
    label('master')
    scm {
      git{
        remote{
          name('origin')
          url("https://aur.archlinux.org/${packageName}.git")
        }
        branch('master')
        extensions {
          cleanBeforeCheckout()
        }
      }
    }
    triggers {
      scm('H/20 * * * *')
    }
    steps {
      shell("sudo /usr/bin/makechrootpkg -u -c -r /mnt/aur/build_test -l ${packageName}")
    }
    publishers {
      artifactDeployer {
        artifactsToDeploy {
          includes('*.pkg.tar.xz')
          remoteFileLocation("/var/www/archlinux/aur/os/x86_64/")
          failIfNoFiles()
          deleteRemoteArtifacts()
        }
      }
      postBuildScripts {
        steps {
          // add a sync() call, which may prevent broken repo DB
          shell('sync;')
          // remove old release from repodb, add new one
          shell("/usr/bin/repo-add --remove --quiet /var/www/archlinux/aur/os/x86_64/aur.db.tar.gz /var/www/archlinux/aur/os/x86_64/${packageName}*.pkg.tar.xz")
          // delete the unneded btrfs subvol to free up diskspace
          shell("sudo /usr/bin/btrfs subvolume delete /mnt/aur/build_test/${packageName}")
        }
        onlyIfBuildSucceeds(true)
      }
      // display fancy jokes and a picture of chuck
      chucknorris()
    }
  }
  //queue("Arch_Package_${packageName}")
}

This is a standard jenkins installation (via `pacman -Syu jenkins`) on a plain Arch Linux server. I only added a few plugins: GitHub Plugin, Artifact Deployer Plug-in, Job DSL.

Jenkins is configured with one job “Arch Package Builder”, this one gets notified about changes from GitHubs Jenkins hook for every commit. Than it will run the groovy script. This iterates at the text file from the repo. Jenkins configures one new job for every package, this consists of:

  • A description
  • A git repo at https://aur.archlinux.org
  • A trigger which checks every 20minutes for changes on the repo
  • a build step to actually build the package (with our well known makechrootpkg command from earlier)

There are a few adjustments needed to the base nspawn image and the Jenkins host to work properly (you can run arch-nspawn /mnt/aur/build_test/root bash to get a shell in the master image):

1) Enable 32bit Repos
It is possible that we will build something with 32bit dependencies so we need to enable the multirepo in pacman (we need that on the host and in the image):

arch-nspawn /mnt/aur/build_test/root bash awk 'BEGIN { r=0 } /\[community\]/ { r=1; print "[community]"; next } r == 0 { print $0 } r == 1 { gsub(/#/, "", $0); print $0; r=0 }' /etc/pacman.conf
awk 'BEGIN { r=0 } /\[community\]/ { r=1; print "[community]"; next } r == 0 { print $0 } r == 1 { gsub(/#/, "", $0); print $0; r=0 }' /etc/pacman.conf

2) Update sudoers to allow the jenkins for our tooling:

echo "jenkins ALL = NOPASSWD: /usr/bin/makechrootpkg" > /etc/sudoers.d/jenkins
echo "jenkins ALL = NOPASSWD: /usr/bin/arch-nspawn" >> /etc/sudoers.d/jenkins
echo "jenkins ALL = NOPASSWD: /usr/bin/btrfs" >> /etc/sudoers.d/jenkins

3) Tell gpg to automatically search for missing keys:

echo "keyserver-options auto-key-retrieve" >> /var/lib/jenkins/.gnupg/gpg.conf

4) Use more than one core for compilation (set it to the amount of cores you have or armount+1):

sed -i 's/#MAKEFLAGS="-j2"/MAKEFLAGS="-j9"' /etc/makepkg.conf

Analyze the mkchrootpkg command:
what jenkins does is

sudo /usr/bin/makechrootpkg -u -c -r /mnt/aur/build_test -l ${packageName}

This tells the command to use our master image (-r /path) as a base and create a snapshot from it, the snaptshot is named after the package (-l ${packageName}). This means that every package will get his own snapshot, and this means that we can run as many concurrent builds as we want. The -c option will delete any existing snapshot with the package name before the actual build, this guarantees that the build env is fresh an clean. The -u flag tells makechrootpkg to update the system in the container before the actual packaging begins. This is a requirement of the Arch Linux packaging guidelines, every package has to be build against the latest dependencies.

Final missing step: Create a Repository to serve the files (one mirror to rule them all!)
So, everything is awesome right now, we just need to push an updated package list to github, this will automatically add a jenkins job, build the package and saves it in a safe directory. Now I want to make this accessible to other Arch machines so that the central build system is useful. The default Arch repositories only hold one version of each package, it is not supported to have a history. The Arch Linux Archive is a project that saves all versions of packages and offers snapshots for each day (more about that in an upcoming post). This allows users to do safely down/upgrade (You’ve to know that PKGBUILDs have dependencies on other packages, but without specifying a version number, a package always depends on the newest version of the dependency that was available at the release date of the actual package [was this understandable?]).
I also want to serve the packages from my Jenkins as an ALA repository to have a history. The goal is to have one ALA server serving all repositories, the default ones and our AUR custom repository. Arch Linux Archive tools can only sync from one central place so we need to install a normal Tier2 Arch Linux mirror, add our AUR repository to it and then ise that as a source for the ALA.

We start with the installation of a sync script (save at /usr/local/bin/syncrepo):

#!/bin/bash

##
# created by bluewind
# modified by bastelfreak
##

home="/var/www"
target="${home}/archlinux"
tmp="${home}/tmp/"
lock='/tmp/mirrorsync.lck'
#bwlimit=4096
# NOTE: you very likely need to change this since rsync.archlinux.org requires you to be a tier 1 mirror
source='rsync://mirror.selfnet.de/archlinux'
lastupdate_url="http://mirror.selfnet.de/archlinux/lastupdate"
[ ! -d "${target}" ] && mkdir -p "${target}"
[ ! -d "${tmp}" ] && mkdir -p "${tmp}"
exec 9>"${lock}"
flock -n 9 || exit
# if we are called without a tty (cronjob) only run when there are changes
if ! tty -s && diff -b <(curl -s "$lastupdate_url") "$target/lastupdate" >/dev/null; then
  exit 0
fi
if ! stty &>/dev/null; then
    QUIET="-q"
fi
rsync -rtlvH --safe-links --delete-after --progress -h ${QUIET} --timeout=600 --contimeout=60 -p \
  --delay-updates --no-motd \
  --temp-dir="${tmp}" \
  --exclude='*.links.tar.gz*' \
  --exclude='/other' \
  --exclude='/sources' \
  --exclude='/aur' \
  ${source} \
  "${target}"
#echo "Last sync was $(date -d @$(cat ${target}/lastsync))"

Then we add a systemd service and timer:

root@ci ~ # systemctl cat syncrepo.{service,timer}
# /etc/systemd/system/syncrepo.service
[Unit]
Description=Sync Archlinux Repo

[Service]
Type=oneshot
ExecStart=/usr/local/bin/syncrepo

# /etc/systemd/system/syncrepo.timer
# /usr/lib/systemd/system/archive.timer
[Unit]
Description=Hourly Archlinux Mirror Update

[Timer]
OnCalendar=hourly
AccuracySec=1m
Persistent=true

[Install]
WantedBy=timers.target
root@ci ~ #

This will sync every official repository every hour to /var/www/archlinux. You can now setup your most loved webserver (nginx) to point to /var/www and serve the content. Now we install the archivetools (as a nonroot user):

git clone https://github.com/seblu/archivetools.git
cd archivetools/
makepkg -i
sed -i "s/#ARCHIVE_RSYNC=*/ARCHIVE_RSYNC='/var/www/archlinux/'" /etc/archive.conf
sudo systemctl enable archive.timer
sudo systemctl start archive.service

The last command will start the initial sync, this will take some time depending on the speed of your internet connection. The timer will run once a day and sync your mirror to the ALA. You can now setup another vhost to serve /srv/archive. There is the Arch Linux Archive including our AUR repository \o/

Dealing with AUR build dependencies:
It is possible that an AUR package needs another AUR package during the build. This brings us into two issues.

  • Detect the dependency and add it to the aur-packages file
  • Bring the package in the nspawn container to install it

Job one is a fancy ruby script, you can use that as a precommit hook for the repo that stores the aur-packages file:

#!/usr/bin/ruby

##
# Written by bastelfreak
##
# enable your local multilib package
##

require 'json'
require 'net/http'

@path = 'aur-packages'
@aur_packages = []
@aur_url = 'https://aur.archlinux.org/rpc/?v=5&type=info&arg[]='
@http = ''
@matches = []
# read files
def get_all_packages path
  @aur_packages = []
  f = IO.readlines path
  f.each do |line|
    line.delete!("\n")
    @aur_packages << line
  end
end

def aur_api_connect
  uri = URI.parse(@aur_url)
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  @http = http
end

def get_deps_for_package package
  uri = URI.parse("#{@aur_url}#{package}")
  res = @http.request(Net::HTTP::Get.new(uri.request_uri))
  ary = JSON.load(res.body)['results']
  #ary[0].key?("Depends") ? ary[0]["Depends"] : ''
  ary.length > 0 ? ary[0]["Depends"] : ''
end

def is_no_official_package? package
  !system("pacman -Ssq #{package}", :out => File::NULL)
end

def add_deps deps
#  unless deps.nil?
    deps.each do |dep|
      add_dep dep
    end
#  end
end

def add_dep dep
  dep = dep.slice(/^[a-zA-Z0-9@.+_-]+/)
  puts "processing dep #{dep}"
  if (is_no_official_package?(dep)) && (!@aur_packages.include? dep)
    puts "found dep #{dep}"
    #@aur_packages << dep
    @matches << dep
  end
end

def get_all_deps_for_every_package
  @aur_packages.each do |package|
    puts "processing package #{package}"
    deps = get_deps_for_package package
    add_deps deps if deps.is_a? Array
  end
end

def cycle_until_all_deps_are_found
  get_all_deps_for_every_package
  if @matches.length > 0
    puts "we found one or more deps, adding them to the file and rescan"
    @matches = @matches.uniq
    @aur_packages = @matches
    File.open(@path, "a") do |f|
      f.puts(@matches)
    end
    @matches = []
    cycle_until_all_deps_are_found
  end
end

# let the magic happen
aur_api_connect
get_all_packages @path
cycle_until_all_deps_are_found

Fixing the second issue: We now have a AUR repo \o/ we just add that to our /etc/pacman.conf in the btrfs base image. The content is:

[aur]
SigLevel = Optional TrustAll
Server = http://mirror.virtapi.org/archlinux/$repo/os/$arch/

Conclusion:
And we are finally done \o/

We now have a sweet Jenkins-CI that builds all packages we want to. They are automatically pushed to a mirror and adding new packages is done via github PRs. The Arch Linux Archive project creates a nice history of all packages which allows us to easily downgrade and upgrade to other package versions.

Sources:

  • Check out the amazing Jenkins-CI API documentation
  • Big thanks to Bluewind who helped me out with the Arch build details
Posted in 30in30, General, Internet found pieces, Linux, Nerd Stuff, Virtualization | Leave a comment

#vDM30in30 – Summer Edition

Hello everybody!
Yes, this Blog is still alive. I was very busy in the past months, but this will change in May. Rob Nelson and I talked about our #vDM30in30 challange las November, and we will do a summer edition next month, so stay tuned! Upcoming topics are:

  • Getting your feet into the open source community
  • Current state of the VirtAPI-Stack
  • Current state of the Puppet Community (now known as Voxupupli)
  • Automation and monitoring with Zabbix
  • Archlinux stuff

You want to have a post about a certain topic? let me know!

Posted in 30in30, General | Leave a comment

Short Tip: git magic (copy changed files from branch A to B)

Let us consider the following case: I’ve got a huge project which you want to refactor. You checkout master, create a branch called cleanup and commit every change. Most of the refactoring is only style guide related so no change involves two or more files. After a time your branch has 30 commits. Reviewing this might cause a lot of pain because there are many changed files and all in one Branch. A solution would be: extract the changed files, create new Branches + Pull Requests for each file, get them reviewed separately. How do we do this nicely?

Go to master, create a new branch:

git checkout master
git checkout -b myrefactoringpart1

Now we tell git to give us a list of changed files between our new branch and the cleanup branch in our most loved editor. We can remove all files from the list except for the one that we want to place in our new branch, than save + quit the editor. Git will checkout the file from the cleanup branch:

git checkout cleanup -- $(git diff --name-status cleanup | cut -f 2- | vipe)

(thanks to i7c from #virtapi on freenode for this awesome snippet)

Now we can commit + push the changes and create a Pullrequest. You can redo these steps for all files you want to put in new branches. You can also automate this to create a new branch for each file, but this method here allows you to edit the files before the commit, in case you missed important stuff.

git commit -am 'refactoring $cooltexthere'
git push --set-upstream origin myrefactoringpart1
Posted in General, Linux, Short Tips | Leave a comment

Short Tip: tuning Zabbix Database Part 2

[part 1]

Besides disabling autovacuum you can also try to increase the performance if you have got enough free resources. There are three important options for this:

autovacuum_vacuum_cost_delay
I lowered the sleep between two autovacuums to 0.2ms

autovacuum_vacuum_cost_limit/vacuum_cost_limit
(setting autovacuum_vacuum_cost_limit to -1 will instruct zabbix to use the vacuum_cost_limit value for autovacuum). The accumulated cost limit that will cause a vacuum to sleep, increasing it will result in a way higher IO usage. Be careful with this if you do not have a very fast IO storage which can deliver high random IO power (SSD Raid in my case). I increased the default 200 to 6000.

autovacuum_max_workers
The default is pretty low and my machine had some free cores so I increased it to 12.

Puppet code to apply these changes to a Postgres 9.4 instance with Puppetlabs/PostgreSQL module:

  postgresql::server::config_entry{'autovacuum_max_workers':
    value => 12,
  }
  postgresql::server::config_entry{'autovacuum_vacuum_cost_delay':
    value => '2ms',
  }
  postgresql::server::config_entry{'autovacuum_vacuum_cost_limit':
    value => 6000,
  }
  postgresql::server::config_entry{'vacuum_cost_limit':
    value => 6000,
  }
Posted in General, Linux, Short Tips | Leave a comment

Short Tip: tuning Zabbix Database

I am currently running a zabbix setup which is a bit bigger than the average ones, right now it receives ~2100 new values per second, generated by 169052 items. The data is stored in a postgresql database, and I am using auto partitioning to manage the ~780gb database. This makes it easy to delete old data, so there is no need for doing an autovacuum on huge tables. The vacuum process can lock the tables which prevents the zabbix server from inserting new values for several hours. However autovacuum is useful in general, and I am glad that postgresql allows us to disable it on a per table basis. The history_uint is the table where it failes, so we disable it here:

zabbixdb=# ALTER TABLE history_uint SET (autovacuum_enabled = false, toast.autovacuum_enabled = false);
ALTER TABLE
zabbixdb=#

This will not disable the autovacuum completely on this table, sometimes it still needs to be done to prevent transaction ID wraparound failures.

Posted in General, Linux, Short Tips | 1 Comment

Short Tip: Moving commits between branches

Sometimes you are developing on a huge new feature on a software project. If you are smart you do it in a separate (feature-) branch. During the development cycle you may modify existing code and after a few days you notice that this was a security bugfix. Merging the complete branch to get the fix is a really bad idea because your feature isn’t ready yet. Thankfully git offers a solution for moving commits from one branch to another, so we can easily create a bugfix branch, move the commits into it, review and than merge it. Here is a little example:


bastelfreak@tmu ~/virtapi $ git checkout master
Switched to branch 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
bastelfreak@tmu ~/virtapi $ git pull
Updating 061bb98..1bae1ed
Fast-forward
database/images/virtapi.svg | 5306 ++++++++++++++++++++++++++++++++++++++-----------------------------------------
database/virtapi.sql | 295 +++--
database/workbench/virtapi.mwb | Bin 23848 -> 25329 bytes
3 files changed, 2727 insertions(+), 2874 deletions(-)
bastelfreak@tmu ~/virtapi $ git checkout -b fix-markdown
Switched to a new branch 'fix-foobar'
bastelfreak@tmu ~/virtapi $ git cherry-pick 9d1792019fe903a986e18f4f1e95701f4a0a0154 faae8809d5f802fa1d9c258820812f4d5ddb7719 94cbb0702e67a0b76142bdc7a7482364e0ae6ee9
[fix-markdown 57af32e] fix intendation
Date: Fri Jan 8 16:17:57 2016 +0100
1 file changed, 9 insertions(+), 9 deletions(-)
[fix-markdown 85782a3] fix intendation
Date: Fri Jan 8 16:09:13 2016 +0100
1 file changed, 23 insertions(+), 23 deletions(-)
[fix-markdown fa71ab5] fix intendation
Date: Fri Jan 8 15:25:51 2016 +0100
1 file changed, 13 insertions(+), 13 deletions(-)
bastelfreak@tmu ~/virtapi $ git push
Username for 'https://github.com': bastelfreak
Password for 'https://bastelfreak@github.com':
Counting objects: 8, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 820 bytes | 0 bytes/s, done.
Total 8 (delta 5), reused 0 (delta 0)
To https://github.com/virtapi/virtapi.git
+ fa71ab5...c6f62ac fix-markdown -> fix-markdown
bastelfreak@tmu ~/virtapi $

Tip: Always take a look which branch is the base for your new one, if you don’t specify a base branch during the checkout -b, git will use the currently checked out branch. During my first try I created the fix-markdown branch based on my dev-branch, this will result in very strange issues.

Posted in General, Linux, Short Tips | Leave a comment