Short Tip: Setup glusterfs share on Arch Linux

I made a detailed tutorial for a Arch Linux installation a few days back. This is a quick follow up post to create a Distributed-Replicated gluster share.

The goal is to create a mirror for several linux distributions. A mirror needs a lot of disk space, but big machines are expensive. Also I would like to have a bit of redundancy. The idea is to use 4 machines, each of them has around 4,2TB storage. Gluster allows us a good combination of replication and distribution, so we can use 8,4TB of storage across two machines, the rest of the free storage is used for replication.

The installation is quite easy. We need to install the rpcbind and gluster packages. I’ve got a seperate partition on all machines, mountet at /glusterfs. After the installation of all packages we can directly create the share (do the following on any of the nodes). Important note: do not do the probe command for the server you are currently working on (we’re on server4 in this example):

1gluster peer probe server1
2gluster peer probe server2
3gluster peer probe server3
4gluster volume create mirror replica 2 transport tcp \
5server1:/glusterfs/mirror \
6server2:/glusterfs/mirror \
7server3:/glusterfs/mirror \
8server4:/glusterfs/mirror
9gluster volume start mirror

Now we should have a working share, hooray. We can verify the peer and share state:

1gluster peer status
2gluster volume info mirror

We’re now able to mount the share via the glusterfs fuse module on any node:

1mount -t glusterfs server1:/mirror /srv/mirror

There isn’t any encryption on the gluster traffic so we should do some firewalling. In my setup nobody except for the gluster nodes itself will mount the share. Here is an example ferm config:

1@def $node1_ipv4 = ( ipv4 );
2@def $node1_ipv6 = ( ipv6 );
3@def $node2_ipv4 = ( ipv4 );
4@def $node2_ipv6 = ( ipv6 );
5@def $node3_ipv4 = ( ipv4 );
6@def $node3_ipv6 = ( ipv6 );
7@def $node4_ipv4 = ( ipv4 );
8@def $node4_ipv6 = ( ipv6 );
9 
10table filter {
11  chain INPUT {
12    policy DROP;
13 
14    # connection tracking
15    mod state state INVALID DROP;
16    mod state state (ESTABLISHED RELATED) ACCEPT;
17 
18    # allow local connections
19    interface lo ACCEPT;
20 
21    # respond to ping
22    proto icmp icmp-type echo-request ACCEPT;
23 
24    # allow SSH connections
25    proto tcp dport ssh ACCEPT;
26 
27    # allow gluster
28    proto tcp dport (111 24007 24008 49152 49153 49154 49155 49156 49157) saddr ( $node1_ipv4 $node2_ipv4 $node3_ipv4 $node4_ipv4 ) ACCEPT;
29    # the rest is dropped by the above policy
30  }
31 
32  # outgoing connections are not limited
33  chain OUTPUT policy ACCEPT;
34 
35  # this is not a router
36  chain FORWARD policy DROP;
37}
38 
39domain ip6 {
40 table filter {
41    chain INPUT {
42      policy DROP;
43 
44      # connection tracking
45      mod state state INVALID DROP;
46      mod state state (ESTABLISHED RELATED) ACCEPT;
47 
48      # allow local connections
49      interface lo ACCEPT;
50 
51      ## respond to ping
52      #proto icmp icmp-type echo-request ACCEPT;
53      # allow all icmp (needed for ipv6 ND and so on)
54      proto icmp ACCEPT;
55 
56      # allow SSH connections
57      proto tcp dport ssh ACCEPT;
58 
59      # allow gluster
60      proto tcp dport (111 24007 24008 49152 49153 49154 49155 49156 49157) saddr ( $node1_ipv6 $node2_ipv6 $node3_ipv6 $node4_ipv6 ) ACCEPT;
61      # the rest is dropped by the above policy
62    }
63 
64    # outgoing connections are not limited
65    chain OUTPUT policy ACCEPT;
66 
67    # this is not a router
68    chain FORWARD policy DROP;
69  }
70}
This entry was posted in 30in30, General, Linux, Short Tips. Bookmark the permalink.

One Response to Short Tip: Setup glusterfs share on Arch Linux

  1. Pingback: Linux Short Tip: Correct IPv6 with ferm firewalling | the world needs more puppet!

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.