I decided to rebuild my ODROID XU4 cluster’s OS with a fresh build using Ubuntu 16.04. The previous build used Ubuntu 14.04. But seeing as 14.04 is several years old at this point, I wanted to upgrade my cluster to the most recent major build of Ubuntu Linux. For the most part, the process of building ou the OS was pretty much the same, but there are some key differences on the master node. This post is just a wholesale copy of my previous two posts (here and here) covering the same topic, but updated for Ubuntu 16.04.
Configuring the ODROID XU4 Operating System
Temporary Networking Setup
When setting up the nodes initially, you will need to SSH into them to configure their settings. However, if we go straight to our network design, we will not be able to connect to any node between the master node is not yet set up as a router. So we will need to connect each node directly to the external network (e.g., you home network).
Since the bill of materials called for a 5-port ethernet switch and there are only 4 nodes, we will start by having the external network connect directly to the switch.
Flashing the eMMC Drives
The eMMC drives I ordered from Hard Kernel came with Ubuntu 15.10 preinstalled. Due to wanting to use Ubuntu 16.04, I found it best just to re-flash the drives with a fresh OS install.
If the eMMC drives are attached to your nodes, remove them. Each drive came with a MicroSD interface that allows you to connect the drive to anything that accepts a MicroSD card. Attach the drive to this interface, then attach the interface to something that will allow you to connect the drive to your computer. My computer is a MacBook Pro, which comes with a builtin SD card reader. So I attached the eMMC drive MicroSD interface to a MicroSD to SD adapter.
After doing this, I downloaded the Ubuntu 16.04 LTS server (“minimal”) image with the Linux 4.9 kernel for the XU4 from HardKernel, which is here. From there, I followed the instructions in the XU4 user manual for flashing the drive. I describe the steps to use on OS X, since that is what I use. Please refer to the XU4 manual for other platforms.
Once downloading the Ubuntu image, uncompress it:
xz -d /path/to/compressed-ubuntu-image-file
Then, for each eMMC drive to be flashed, do the following steps. Attach the eMMC drive to your computer using the SD card adapter. Then, you will need to find the the device name for the eMMC drive. It will have the form of /dev/diskX, and can be found by using the OS X command line utility diskutil. Once the device name is found, unmount it, and then us dd to flash the Ubuntu image eMMC drive. Please be careful here. If you mistype the device name, you could seriously mess up the drive on your computer.
diskutil list diskutil unmountdisk /dev/diskX sudo dd of=/dev/diskX bs=1m if=/path/to/ubuntu-16.04.3-4.9-minimal-odroid-xu4-20170824.img
Alternatively, you can use the open source tool is called Etcher, which is built for Mac, Windows, and Linux. Grab a copy, and then use it to flash all the eMMC drives withe the Ubuntu 16.04 image file.
Configuring the Operating System
After all the eMMC drives have been flashed, attached them to the XU4s in the cluster. Then power up the ethernet switch, but do not yet add power to any node. Then, repeat the following steps for each of the nodes in the cluster. For the sake of ordering, start with the master node. Before starting, ensure that the MicroSD card is not attached to a node, and unsure that the XU4s’ boot selector switch is set to the eMMC drive.
- Attach the power to the node. You will hear the node’s fan whirl and see the various LEDs on the device flash. The default configuration of the OS we flashed has the device grab an IP address from a DHCP server. Since the cluster switch is connected to your home network right now, the device should grab an IP address from your house router. Log into your router or use a network scanning utility to find the IP address assigned to the node that just booted. I use iNet Network Scanner from my iPad. there is also an iPhone version.
- Once you have the device’s IP address, remotely SSH into it using the account odroid and password odroid. From the terminal on your personal computer:
ssh odroid@192.168.1.50
If you get the login prompt but cannot login with the ordroid account, that means the odroid account wasn’t created as part of the Ubuntu image. Login instead as user root with password odroid. Then, add the odroid user with:
adduser odroid adduser odroid sudo apt-get update apt-get upgrade exit
You will be prompted for a password and other information. Once completed, exit out and log into the machine with user odroid. Sometimes I had to remove the
/var/lib/dpkg/lockto allow theapt-get upgradecommand to work. If you get an error indicating that apt-get can’t proceed because of a lock being in place, perform the following:rm /var/lib/dpkg/lock rm /var/cache/apt/archives/lock
- We need to get and record the hardware MAC address for each node. This will be needed in the next post when we set up the router service. Issue the command
ifconfiginto the shell that you just opened on the newly booted XU4. You will get a bunch of text that looks something like this:eth0 Link encap:Ethernet HWaddr 00:1e:06:32:2a:7f inet addr:192.168.1.116 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21e:6ff:fe32:2a7f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:589 errors:0 dropped:0 overruns:0 frame:0 TX packets:146 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:110461 (110.4 KB) TX bytes:20988 (20.9 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:16 errors:0 dropped:0 overruns:0 frame:0 TX packets:16 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1216 (1.2 KB) TX bytes:1216 (1.2 KB)<span id="mce_marker" data-mce-type="bookmark"></span>Look for the block that has the
inet addrwith the same value as the IP address that you logged into, and in that section look for theHWaddritem. What follows is the MAC address for the onboard ethernet device. Record it in association with what node it is for. On the master node with the USB ethernet dongle attached, you can runifconfig -ato get all available network interfaced. Theeth1block will be the USB ethernet dongle’s network interface. You will want to record the dongle’s MAC address so that you can set up a static DHCP IP address assignment in your home network’s router (see your router’s manual for instructions). You might device IDs other thaneth0andeth1. That’s OK, we will take care of that later on the master node. - Install and run a very useful tool called Odroid Utility made by Hard Kernel. To do this, issue the following commands:
sudo -s wget -O /usr/local/bin/odroid-utility.sh https://raw.githubusercontent.com/mdrjr/odroid-utility/master/odroid-utility.sh chmod +x /usr/local/bin/odroid-utility.sh odroid-utility.sh
- You are going to do three things in this utility:
- Name the Node – The “Change Hostname” menu option will give you the opportunity to name the node. Name the node with the ethernet dongle
master, and the other threeslave1,slave2, andslave3. - Turn off Xorg – We are running these nodes as headless servers. The Ubuntu image you just flashed runs a GUI by default. We don’t need our nodes loaded with that overhead, so wee need to turn off the X server that gets launched by default. Use the menu item “Xorg On/Off” to do this.
- Resize the Boot Drive – When you flashed the the eMMC drive, it created the root file system with a 4 GB partition. The drives we ordered were 16 GB, and the smallest you can buy are 8 GB. Either way, there is unclaimed space on your drive. Use the “Resize your root partition” menu option to resize the root file system to make available all remaining space on the eMMC drive.
- Name the Node – The “Change Hostname” menu option will give you the opportunity to name the node. Name the node with the ethernet dongle
- Update the hosts file on the machine to define short host names for each node in the cluster. Do that by editing the hosts file:
vi /etc/hosts
Update the file’s contents to the following:
127.0.0.1 localhost 10.10.10.1 master 10.10.10.2 slave1 10.10.10.3 slave2 10.10.10.4 slave3 ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters
Next, we need to turn off IPv6 on the internal cluster network. Open the following file for editing:
sudo vi /etc/sysctl.conf
Add the following lines at the end of the file:
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1
- Shutdown the node.
shutdown -h now
Configuring DHCP and NAT in ODROID XU4 Cluster
As was discussed in the network design post, we will set up the master node as a router to manage network traffic in and out of the cluster. Before starting, ensure that all of the slave nodes have been powered down, that your home network is still connected directly to the open port on the cluster’s ethernet switch, that you have collected each node’s MAC address, and that the master node is powered up and you are logged into it via SSH.
The first step is to explicitly set up the networking interfaces for both the eth0 and eth1 device on the master node. However, before we do that, we need to compensate for some Ubuntu 16.04 default configuration. Specifically, I would like to use systemd rather than NetworkManager to initialize the network interfaces on the master node at boot, and I want the interfaces to be called eth0 and eth1 always. I am less concerned about the slave nodes because the default NetworkManager setup causes the node to claim a dynamic address from the DHCP server, which is exactly what we want. So, all the configuration for the clusters network backbone only needs to happen on the master node.
Power up the master node only and log in as odroid. Issue the following command to disable the NetworkManager:
sudo systemctl disable NetworkManager.service
Now let’s configure the master node to always call the built-in ethernet eth0 and the USB dongle eth1. Do this be creating and editing the /etc/udev/rules.d/70-persistent-net.rules file:
sudo vi /etc/udev/rules.d/70-persistent-net.rules
Set the contents of the file to:
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1e:06:32:2a:7f", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0e:c6:c2:16:cc", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth1"
Note that ATTR{address} setting on each line should be set the the MAC address for the relevant ethernet port identified by the NAME parameter on that line.
By default the Ubuntu system that was installed on the master node treats eth0 a as requesting a DHCP lease on the network it is attached to. This is why it got an IP address when it was first powered up. However, we are going to change this. The eth0 interface will be connected to the cluster’s internal network, and the eth1 interface will connect the cluster to the external network. Since the master node is also the DHCP server and a node cannot request an IP address from itself, we will have to configure the eth0 interface to have a static IP address. This is done by creating interface descriptions in the /etc/network/interfaces.d directory.
Create and open the eth0 interface file for editing:
sudo vi /etc/network/interfaces.d/eth0
Then edit the file to have the following contents:
auto eth0
iface eth0 inet static
address 10.10.10.1
netmask 255.255.255.0
network 10.10.10.0
broadcast 10.10.10.255
This assigns the eth0 interface to having a static IP address of 10.10.10.1.
To set up the eth1 interface to request an IP address from a DHCP server, create and open the eth1 interface file for editing:
sudo vi /etc/network/interfaces.d/eth1
Then edit the file to have the following contents:
auto eth1 iface eth1 inet dhcp
Now we will to set up the DHCP and NAT service on the master node. First, we need to tell the system to forward IP traffic between its network interfaces. Do this by first installing iptables if it isn’t already, then editing the following file:
sudo apt-get install iptables sudo vi /etc/sysctl.conf
Find and uncomment the following line in the file (simply remove the leading # character):
net.ipv4.ip_forward=1
And we also need to set up the IP tables rules to enable NAT. Open the /etc/rc.local file for editing:
sudo vi /etc/rc.local
Then add the following two lines just before the exit 0 line at the end of the file:
/sbin/iptables -P FORWARD ACCEPT /sbin/iptables --table nat -A POSTROUTING -o eth1 -j MASQUERADE
Next we are going to install and configure the DHCP server.
sudo apt-get install isc-dhcp-server -y sudo vi /etc/default/isc-dhcp-server
Find the line in the file such that looks like:
INTERFACES=""
To tell the DHCP server to only hand out IP addresses on the cluster’s internal network, change the line to look like:
INTERFACES="eth0"
Next, set up the DHCP server’s policies. Open it’s configuration file for editing:
sudo vi /etc/dhcp/dhcpd.conf
Comment out these lines:
#option domain-name "example.org"; #option domain-name-servers ns1.example.org, ns2.example.org;
Uncomment this line:
authoritative;
And then add the following text at the end of the file. Note that this is where the slave nodes’ MAC addresses get used. Be sure to replace the example MAC address – what follows hardware ethernet in each host block – with the appropriate MAC address of each specific slave node in your cluster.
subnet 10.10.10.0 netmask 255.255.255.0 {
range 10.10.10.100 10.10.10.200;
option domain-name-servers 8.8.8.8;
option routers 10.10.10.1;
option broadcast-address 10.10.10.255;
default-lease-time 600;
max-lease-time 7200;
host slave1 {
option host-name "slave1";
hardware ethernet 00:1e:06:32:2a:70;
fixed-address 10.10.10.2;
}
host slave2 {
option host-name "slave2";
hardware ethernet 00:1e:06:32:2a:7e;
fixed-address 10.10.10.3;
}
host slave3 {
option host-name "slave3";
hardware ethernet 00:1e:06:32:2a:7c;
fixed-address 10.10.10.4;
}
}
The DHCP and NAT services have been set up on your master node, but there are a few more things we need to do. But before we do them, let’s start using the master node as a router for the cluster. Shutdown the master node:
sudo shutdown -h now
All nodes in the cluster should now be off. Remove your home network’s cable from the cluster’s ethernet switch, and reattach it to the ethernet dongle attached to the master node. Once this is done, power up all nodes in the cluster. Wait a few minutes, then SSH into the master node using the IP address that you set up your home network router to assign to the ethernet dongle’s MAC address.
ssh odroid@192.168.1.50
Test the network set up by pinging each node in the cluster by name (press control-C to cancel each ping):
ping slave1 ping slave2 ping slave3
Now let’s install a distributed SSH program that will allow you to issue commands on all nodes in the cluster at the same time:
sudo apt-get install pssh
Create a file in the odroid home directory that lists all the slaves:
mkdir ~/cluster vi ~/cluster/slaves.txt
Set the file contents to:
slave1 slave2 slave3
Also create a list for all nodes in the cluster:
vi ~/cluster/all.txt
Set the file contents to:
master slave1 slave2 slave3
And now create an SSH key on the master node and share it with each node’s odroid and root account so that you can issue command via SSH without a password. Note that you should create no password when generating the key.
ssh-keygen -t rsa -P "" ssh-copy-id odroid@slave1 ssh-copy-id odroid@slave2 ssh-copy-id odroid@slave3 ssh-copy-id root@slave1 ssh-copy-id root@slave2 ssh-copy-id root@slave3 ssh-copy-id root@master
Test your newfound cluster powers by setting up the locale on each node then shutting down all of the slave nodes at once, and then shutting down the master node:
parallel-ssh -i -h ~/cluster/all.txt -l root "locale-gen en_US.UTF-8" parallel-ssh -i -h ~/cluster/slaves.txt -l root "sudo /sbin/shutdown -h now" sudo /sbin/shutdown -h now
Congratulate yourself, as you have fully configured the networking and can now truly say you have a compute cluster with its own network backbone.
Optional – Inbound Routing
As the cluster’s network is set up, you cannot directly connection to nodes on the internal network as-is. However, if you tell a computer external to the cluster’s internal network the route to the nodes, it can directly connect. Since I only connect to the cluster from my Mac laptop, the simple solution is to tell my laptop about the route between it and the gateway into the cluster’s private network, which is the master node. To add the route on a Mac, issue the following comment in Terminal:
sudo route add 10.10.10.0/24 192.168.1.50
Replace the final IP address with the address that your cluster’s IP address that it gets on your home network out the USB ethernet dongle. Ideally, you have created a static DHCP address assignment for the dongle in your home network’s router so that your cluster will consistently get the same IP address. Note that this route assignment will only persist until you reboot your computer. This blog post describes how to make static routes like this persist across reboots. After adding the route, update you computers /etc/hosts files to additionally have the following lines:
10.10.10.1 master 10.10.10.2 slave1 10.10.10.3 slave2 10.10.10.4 slave3
Your computer now will be able to connect to any node as if your computer were on the cluster’s private network.
Adding the MicroSD Data Drives
Adding the MicroSD cards as data drives to each node in the cluster is done the same was as previously described for Ubuntu 14.04. Follow those instructions to finish the operating system set up of your ODROID XU4 cluster with Ubuntu 16.04.
At this point, the cluster hardware and networking has been completely set up. It is for us to start installing some big data application.
2 thoughts on “Upgrading ODROID Cluster to Ubuntu 16.04”