DIY Big Data

We are nearly done with the basic set up of the ODROID XU4 four node cluster. The final step is to configure the MicroSD cards that were in the bill of materials as data drives for each node. Before we start, ensure that the cluster is completely powered down and remove the power cords from each node. We will be removing and adding components to the ODROID XU4’s, and it is just best practice to have no power to the device when removing or adding components.

The first step is to format each MicroSD card with the ext4 filesystem. Each platform has it’s own instructions for how to do this. I will provide the Mac OS X instructions since that is what I use. In the terminal, do the following:

1. If you don’t have it already, install the Homebrew package management app for OS X. Visit the brew.sh website for instructions.
2. Install the e2fsprogs utility using brew install e2fsprogs
3. With the MicroSD card attached to your computer, determine the name of the mounted MicroSD partition using diskutil list. For me, the MicroSD partition was named /dev/disk4s1.
4. Run the command sudo $(brew --prefix e2fsprogs)/sbin/mkfs.ext4 /dev/disk4s1 to format the MicroSD card. Ensure that you use the correct partition name if yours is different than /dev/disk4s1. Note that this will take a few minutes. If you get the message that the resource is busy, unmount the MicroSD card using the diskutil unmountdisk /dev/disk4s1  command.

Repeat steps #3 and #4 for each MicroSD card. Once each MicroSD card has been formatted, install them onto each of the unpowered ODROID XU4 devices in your cluster. Power up all the nodes and SSH into the master node.

Now we need to configure each node to mount the MicroSD data drive while simultaneously booting from the eMMC drive. Use the following steps for each node in the cluster.

1. From the master node, SSH into the node. Skip this step if you are setting up the master node itself. For example, if you are going to work on the slave3 node:

   ssh odroid@slave3
   

2. Determine the device name that the MicroSD card attached as by listing the dev directory:

   ls /dev

   For me, the MicroSD card was named /dev/mmcblk1p1. Assuming you set up the node as I described in all the prior posts, I would expect that your MicroSD card got the same name, but do check to be sure. If your MicroSD card got a different device name, update all the instructions below appropriately.

3. Create a /data mount point and mount the MicroSD card to it:

   sudo mkdir /data
   sudo mount /dev/mmcblk1p1 /data

4. Confirm that you successfully mounted the MicroSD card:

   df -H

5. Now find the UUID of the MicroSD card. Issue the command:

   sudo blkid

   You will see a block of text that looks like this:

   /dev/mmcblk0: PTUUID="0008aae4" PTTYPE="dos"
   /dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="5F78-D927" TYPE="vfat" PARTUUID="0008aae4-01"
   /dev/mmcblk0p2: LABEL="rootfs" UUID="e139ce78-9841-40fe-8823-96a304a09859" TYPE="ext4" PARTUUID="0008aae4-02"
   /dev/mmcblk1: PTTYPE="dos"
   /dev/mmcblk1p1: UUID="c1f7210a-293a-423e-9bde-1eba3bcc9c34" TYPE="ext4"

   Record the UUID that follows the /dev/mmcblk1p1 device.

6. Edit the file system table so that the MicroSD card will be automatically mounted at each boot:

   sudo cp /etc/fstab /etc/fstab.old 
   sudo vi /etc/fstab

   Then add the following line at the end of the file, replacing the UUID with the one you recorded in the previous step:

   UUID=c1f7210a-293a-423e-9bde-1eba3bcc9c34 /data ext4 defaults 0 0

7. Restart the node with sudo shutdown -r now, and then log back in when the node is fully up. Use the df -H command to verify that the MicroSD card automatically mounted to /data after restarting.

At this point, the cluster hardware and networking has been completely set up. It is for us to start installing some big data application.