the world needs more puppet!

Sometimes you check your fileserver and your raid looks like this:
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active (auto-read-only) raid6 sdl[1] sdh[7] sdf[5] sdk[2] sdg[6] sdm[8] sdi[0] sdn[10] sde[4]
23441080320 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/9] [UUU_UUUUUU]
bitmap: 17/22 pages [68KB], 65536KB chunk

Noticed the _ in [UUU_UUUUUU]? This means your raid is degraded and one drive is missing. You had one job harddrive! You can use the output of find /dev/sd? and lsblk to check which device is the missing one. Hopefully the drive is still present to the system, than it should appear. In that case it also should be possible to query it for the serial number:

smartctl -i /dev/sdj | awk '/Serial/ {print $3}'

You should now be able to determine the failed drive in the system and replace it. After that you can add the new one to the raid:

mdadm /dev/md125 --add /dev/sdj

Tip1:
Hard drive failure rates work like the bathtub curve. They die in the first hours of their usage or add the end, mostly around 25.000-30.000 (depends on many many many factors, this is just a rough estimate). But just keep that in mind for the next hours while your raid rebuild, the new drive may fail and you have to replace it again.

Tip2
You can speed up the rebuild with:

echo 2000000000 > /proc/sys/dev/raid/speed_limit_max
echo 200000000 > /proc/sys/dev/raid/speed_limit_min

This forces the kernel to use all available IO power for the rebuild. You can watch it with watch cat /proc/mdstat.

I mentioned ferm in my last post about gluster (an iptables/ip6tables abstraction layer in perl with a nice firewall config). The default rule-set looks like this:

table filter {
    chain INPUT {
        policy DROP;

        # connection tracking
        mod state state INVALID DROP;
        mod state state (ESTABLISHED RELATED) ACCEPT;

        # allow local connections
        interface lo ACCEPT;

        # respond to ping
        proto icmp icmp-type echo-request ACCEPT;

        # allow SSH connections
        proto tcp dport ssh ACCEPT;

        # ident connections are also allowed
        proto tcp dport auth ACCEPT;

        # the rest is dropped by the above policy
    }

    # outgoing connections are not limited
    chain OUTPUT policy ACCEPT;

    # this is not a router
    chain FORWARD policy DROP;
}

Most people attend to C&P that blog and wrap it in domain ip6 {} if they also want to filter IPv6. This works fine accept for the ICMP part. It only allows echo requests (a normal ping/mtr). But IPv6 does Neighbor Discovery Protocol (the successor of Address Resolution Protocol) via ICMP. Your device won’t be able to reach your gateway without allowing ICMP completely, so a simple C&P will break existing IPv6 setups. My fixed version is:

## respond to ping
#proto icmp icmp-type echo-request ACCEPT;
# allow all icmp (needed for ipv6 NDP)
proto icmp ACCEPT;

I made a detailed tutorial for a Arch Linux installation a few days back. This is a quick follow up post to create a Distributed-Replicated gluster share.

The goal is to create a mirror for several linux distributions. A mirror needs a lot of disk space, but big machines are expensive. Also I would like to have a bit of redundancy. The idea is to use 4 machines, each of them has around 4,2TB storage. Gluster allows us a good combination of replication and distribution, so we can use 8,4TB of storage across two machines, the rest of the free storage is used for replication.

The installation is quite easy. We need to install the rpcbind and gluster packages. I’ve got a seperate partition on all machines, mountet at /glusterfs. After the installation of all packages we can directly create the share (do the following on any of the nodes). Important note: do not do the probe command for the server you are currently working on (we’re on server4 in this example):

gluster peer probe server1
gluster peer probe server2
gluster peer probe server3
gluster volume create mirror replica 2 transport tcp \
server1:/glusterfs/mirror \
server2:/glusterfs/mirror \
server3:/glusterfs/mirror \
server4:/glusterfs/mirror
gluster volume start mirror

Now we should have a working share, hooray. We can verify the peer and share state:

gluster peer status
gluster volume info mirror

We’re now able to mount the share via the glusterfs fuse module on any node:

mount -t glusterfs server1:/mirror /srv/mirror

There isn’t any encryption on the gluster traffic so we should do some firewalling. In my setup nobody except for the gluster nodes itself will mount the share. Here is an example ferm config:

@def $node1_ipv4 = ( ipv4 );
@def $node1_ipv6 = ( ipv6 );
@def $node2_ipv4 = ( ipv4 );
@def $node2_ipv6 = ( ipv6 );
@def $node3_ipv4 = ( ipv4 );
@def $node3_ipv6 = ( ipv6 );
@def $node4_ipv4 = ( ipv4 );
@def $node4_ipv6 = ( ipv6 );

table filter {
  chain INPUT {
    policy DROP;

    # connection tracking
    mod state state INVALID DROP;
    mod state state (ESTABLISHED RELATED) ACCEPT;

    # allow local connections
    interface lo ACCEPT;

    # respond to ping
    proto icmp icmp-type echo-request ACCEPT;

    # allow SSH connections
    proto tcp dport ssh ACCEPT;

    # allow gluster
    proto tcp dport (111 24007 24008 49152 49153 49154 49155 49156 49157) saddr ( $node1_ipv4 $node2_ipv4 $node3_ipv4 $node4_ipv4 ) ACCEPT;
    # the rest is dropped by the above policy
  }

  # outgoing connections are not limited
  chain OUTPUT policy ACCEPT;

  # this is not a router
  chain FORWARD policy DROP;
}

domain ip6 {
 table filter {
    chain INPUT {
      policy DROP;

      # connection tracking
      mod state state INVALID DROP;
      mod state state (ESTABLISHED RELATED) ACCEPT;

      # allow local connections
      interface lo ACCEPT;

      ## respond to ping
      #proto icmp icmp-type echo-request ACCEPT;
      # allow all icmp (needed for ipv6 ND and so on)
      proto icmp ACCEPT;

      # allow SSH connections
      proto tcp dport ssh ACCEPT;

      # allow gluster
      proto tcp dport (111 24007 24008 49152 49153 49154 49155 49156 49157) saddr ( $node1_ipv6 $node2_ipv6 $node3_ipv6 $node4_ipv6 ) ACCEPT;
      # the rest is dropped by the above policy
    }

    # outgoing connections are not limited
    chain OUTPUT policy ACCEPT;

    # this is not a router
    chain FORWARD policy DROP;
  }
}

I’m currently playing around with LARS, this script collection creates a arch ISO with some post-install magic, for example setting a login shell for root. The used code is:

usermod -s /usr/bin/bash root

I could boot up the image and ssh into it with my key. But three things weren’t working:

• Login as normal user and do su
• Login at a TTY
• Login via ssh with a password

I got a “Access Denied” message in all three cases. I digged around for two days. I know that Arch Linux moved binaries around a few years back, /bin is now a symlink to /usr/bin. Just for fun a colleague changed the shell in the /etc/passwd to /bin/bash, just like in the good old days, and WTF everything was working?!
We found the /etc/shells file:

#
# /etc/shells
#

# End of file


This file lists all shells that are allowed to be used as a login shell. By default it lists the symlink for bash, not the absolute path itself… I still don’t know why the keybased login was working, but this is another mystery.