DIY Big Data

UPDATE – I have rebuilt this cluster to use Ubuntu 16.04. You can find updated instruction for Ubuntu 16.04 here.

As was discussed in the network design post, we will set up the master node as a router to manage network traffic in and out of the cluster.  Before starting, ensure that all of the slave nodes have been powered down, that your home network is still connected directly to the open port on the cluster’s ethernet switch, that you have collected each node’s MAC address, and that the master node is powered up and you are logged into it via SSH.

The first step is to explicitly set up the networking interfaces for both the eth0 and eth1 device on the master node. Note that by default the Ubuntu system that was installed on the master node treats eth0 a as requesting a DHCP lease on the network it is attached to. This is why it got an IP address when it was first powered up. However, we are going to change this. The eth0 interface will be connected to the cluster’s internal network, and the eth1 interface will connect the cluster to the external network. Since the master node is also the DHCP server and a node cannot request an IP address from itself, we will have to configure the eth0 interface to have a static IP address. This is done by creating interface descriptions in the /etc/network/interfaces.d directory.

Create and open the eth0 interface file for editing:

sudo vi /etc/network/interfaces.d/eth0

Then edit the file to have the following contents:

auto eth0
iface eth0 inet static
        address 10.10.10.1
        netmask 255.255.255.0
        network 10.10.10.0
        broadcast 10.10.10.255

This assigns the eth0 interface to having a static IP address of 10.10.10.1.

To set up the eth1 interface to request an IP address from a DHCP server, create and open the eth1 interface file for editing:

sudo vi /etc/network/interfaces.d/eth1

Then edit the file to have the following contents:

auto eth1
iface eth1 inet dhcp

Now we will to set up the DHCP and NAT service on the master node. First, we need to tell the system to forward IP traffic between its network interfaces. Do this by first installing iptables if it isn’t already, then editing the following file:

sudo apt-get install iptables
sudo vi /etc/sysctl.conf

Find and uncomment the following line in the file (simply remove the leading # character):

net.ipv4.ip_forward=1

And we also need to set up the IP tables rules to enable NAT. Open the /etc/rc.local file for editing:

sudo vi /etc/rc.local

Then add the following two lines just before the exit 0 line at the end of the file:

/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables --table nat -A POSTROUTING -o eth1 -j MASQUERADE

Next we are going to install and configure the DHCP server.

sudo apt-get install isc-dhcp-server -y
sudo vi /etc/default/isc-dhcp-server

Find the line in the file such that looks like:

INTERFACES=""

To tell the DHCP server to only hand out IP addresses on the cluster’s internal network, change the line to look like:

INTERFACES="eth0"

Next, set up the DHCP server’s policies. Open it’s configuration file for editing:

sudo vi /etc/dhcp/dhcpd.conf

Comment out these lines:

#option domain-name "example.org";
#option domain-name-servers ns1.example.org, ns2.example.org;

Uncomment this line:

authoritative;

And then add the following text at the end of the file. Note that this is where the slave nodes’ MAC addresses get used. Be sure to replace the example MAC address – what follows hardware ethernet in each host block – with the appropriate MAC address of each specific slave node in your cluster.

subnet 10.10.10.0 netmask 255.255.255.0 { 
        range 10.10.10.100 10.10.10.200;
        option domain-name-servers 8.8.8.8;              
        option routers 10.10.10.1;
        option broadcast-address 10.10.10.255;
        default-lease-time 600;
        max-lease-time 7200;

        host slave1 {
        	option host-name "slave1";
                hardware ethernet 00:1e:06:32:2a:70;
                fixed-address 10.10.10.2;
        }
        host slave2 {
        	option host-name "slave2";
                hardware ethernet 00:1e:06:32:2a:7e;
                fixed-address 10.10.10.3;
        }
        host slave3 {
        	option host-name "slave3";
                hardware ethernet 00:1e:06:32:2a:7c;
                fixed-address 10.10.10.4;
        }

}

The DHCP and NAT services have been set up on your master node, but there are a few more things we need to do. But before we do them, let’s start using the master node as a router for the cluster. Shutdown the master node:

sudo shutdown -h now

All nodes in the cluster should now be off. Remove your home network’s cable from the cluster’s ethernet switch, and reattach it to the ethernet dongle attached to the master node. Once this is done, power up all nodes in the cluster. Wait a few minutes, then SSH into the master node using the IP address that you set up your home network router to assign to the ethernet dongle’s MAC address.

ssh odroid@192.168.1.50

Test the network set up by pinging each node in the cluster by name (press control-C to cancel each ping):

ping slave1
ping slave2
ping slave3

Now let’s install a distributed SSH program that will allow you to issue commands on all nodes in the cluster at the same time:

sudo apt-get install pssh

Create a file in the odroid home directory that lists all the slaves:

mkdir ~/cluster
vi ~/cluster/slaves.txt

Set the file contents to:

slave1
slave2
slave3

Also create a list for all nodes in the cluster:

vi ~/cluster/all.txt

Set the file contents to:

master
slave1
slave2
slave3

And now create an SSH key on the master node and share it with each node’s odroid and root account so that you can issue command via SSH without a password. Note that you should create no password when generating the key.

ssh-keygen -t rsa -P ""
ssh-copy-id odroid@slave1
ssh-copy-id odroid@slave2
ssh-copy-id odroid@slave3
ssh-copy-id root@slave1
ssh-copy-id root@slave2
ssh-copy-id root@slave3
ssh-copy-id root@master

Test your newfound cluster powers by setting up the locale on each node then shutting down all of the slave nodes at once, and then shutting down the master node:

parallel-ssh -i -h ~/cluster/all.txt -l root "locale-gen en_US.UTF-8"
parallel-ssh -i -h ~/cluster/slaves.txt -l root "sudo /sbin/shutdown -h now"
sudo /sbin/shutdown -h now

Congratulate yourself, as you have fully configured the networking and can now truly say you have a compute cluster with its own network backbone.

Optional – Inbound Routing

As the cluster’s network is set up, you cannot directly connection to nodes on the internal network as-is. However, if you tell a computer external to the cluster’s internal network the route to the nodes, it can directly connect. Since I only connect to the cluster from my Mac laptop, the simple solution is to tell my laptop about the route between it and the gateway into the cluster’s private network, which is the master node. To add the route on a Mac, issue the following comment in Terminal:

sudo route add 10.10.10.0/24 192.168.1.50

Replace the final IP address with the address that your cluster’s IP address that it gets on your home network out the USB ethernet dongle. Ideally, you have created a static DHCP address assignment for the dongle in your home network’s router so that your cluster will consistently get the same IP address. Note that this route assignment will only persist until you reboot your computer. This blog post describes how to make static routes like this persist across reboots. After adding the route, update you computers /etc/hosts files to additionally have the following lines:

10.10.10.1      master
10.10.10.2      slave1
10.10.10.3      slave2
10.10.10.4      slave3

Your computer now will be able to connect to any node as if your computer were on the cluster’s private network.