Adding a New Node to the ODROID XU4 Cluster

michael

I recently acquired another ODROID XU4 device (and a MicroSD card for bulk storage) to add it to my XU4 cluster. This new node brings my node count to five. Adding a new node to the cluster is relatively straight forward, but there are a lot of details. In the modern datacenter, this operation would be accomplished through a package manager, which would build the new node according to an image. However, I haven’t set up package management on my cluster so I will need to sit up the new node manually.

In the original cluster configuration, I used a 5-port ethernet switch for the internal network. Given that there was an open port, no additional hardware beyond the new node is needed. However, if I were to add a sixth node (or beyond), I would need to update my ethernet switch to something such as an 8-port switch. I will also note that using the 40 mm PCB spacers I originally ordered makes the top node in a 5 node stack too far away to reach the ethernet switch with the original 6 inch ethernet cables I ordered, so I ordered an 8 inch cable to connect the top node to the switch.

Setup Up a New Node

The first step in adding a new XU4 node is to set it up as a slave node as described in my original post on setting up the node’s operating system. After doing this, you will need to set up the cluster’s internal network such that the new node get’s a predictable IP address. Edit the /etc/dhcp/dhcpd.conf file on the master node and add the following subblock such as this to the subnet 10.10.10.0 block:

        host slave4 {
                option host-name "slave4";
                hardware ethernet 00:1e:06:30:55:63;
                fixed-address 10.10.10.5;
        }

Of course, update the host name, assign IP address and ethernet MAC address as appropriate. After doing this, log into each node and update the /etc/hosts file to include an entry for the new node. One the new node itself, be sure to fully populate the /etc/hosts file for all the nodes in the cluster. Now restart the entire cluster to ensure the new networking configuration takes effect.

On the master node, add the new node to the cluster configuration files we originally created at ~odroid/cluster/all.txt and ~odroid/cluster/slaves.txt. then, as the odroid user on the master node, copy the master node SSH keys to the new node.

ssh-copy-id odroid@slave4
ssh-copy-id root@slave4

After do this, set up the MicroSD card for the new node as described in my post for formatting and adding a MicroSD card as bulk storage for a node.

Installing Hadoop on a New Node

To set up the new node for Hadoop, first log into it as user odroid to create the hduser user and install Java

sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser
sudo adduser hduser sudo
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt-get install rsync

On the master node as user hduser, update the Hadoop configuration to see the new node, sync out the updated configuration to the existing slaves, then sync out Hadoop to the new node.

# copy hduser SSH keys to new node
ssh-copy-id hduser@slave4

# update Hadoop configuration to see new node by adding the slave to teh slaves file
cd /usr/local/hadoop/etc/hadoop/
vi slaves

# sync out new configuration to existing nodes
rsync -avxP /usr/local/hadoop/etc/hadoop/ hduser@slave1:/usr/local/hadoop/etc/hadoop/
rsync -avxP /usr/local/hadoop/etc/hadoop/ hduser@slave2:/usr/local/hadoop/etc/hadoop/
rsync -avxP /usr/local/hadoop/etc/hadoop/ hduser@slave3:/usr/local/hadoop/etc/hadoop/

# sync out hadoop to new node
sudo rsync -avxP /opt/hadoop-2.7.2/ root@slave4:/opt/hadoop-2.7.2/

Back on the new node as hduser, perform the final configuration of the node:

sudo mkdir -p /data/hdfs/tmp
sudo chown -R hduser:hadoop /data/hdfs
sudo chown -R hduser:hadoop /opt/hadoop-2.7.2
sudo ln -s /opt/hadoop-2.7.2 /usr/local/hadoop

Now you can start HDFS on the master node as hduser:

start-dfs.sh

You will see the new node get an HDFS daemon launched. If you check out the HDFS monitor page at http://<cluster IP address>:50070 , you can see that the new node does not have any data installed. It would be a good idea to balance your cluster. This will take a while:

hdfs balancer

Installing Spark on the New Node

The first step here is to update the configuration of Spark on the master node to be able to see the new node.

cd /usr/local/spark/conf/
vi slaves

Add the new node name to the slaves file. Sync out the new configuration to the existing slaves, and Spark to the new node.

rsync -avxP /usr/local/spark/conf/ hduser@slave1:/usr/local/spark/conf/
rsync -avxP /usr/local/spark/conf/ hduser@slave2:/usr/local/spark/conf/
rsync -avxP /usr/local/spark/conf/ hduser@slave3:/usr/local/spark/conf/
sudo rsync -avxP /opt/spark-1.6.2-bin-hadoop2.6 root@slave4:/opt/

Then on the new node, finalize the Spark set up:

sudo chown -R hduser:hadoop /opt/spark-1.6.2-bin-hadoop2.6
sudo ln -s /opt/spark-1.6.2-bin-hadoop2.6 /usr/local/spark
sudo mkdir -p /data/spark
sudo chown hduser:hadoop /data/spark
sudo apt-get install python3

To start Spark with the new node, on the master node as user hduser:

/usr/local/spark/sbin/start-all.sh

With that, you have fully added the new node to the ODROID XU4 cluster.

Leave a Reply