Low Cost ODROID XU4 Compute Cluster

The ODROID XU4 cluster fully assembled
ODROID XU4 Cluster

This project walks you through all the steps necessary to create a distributed compute cluster with four ODROID XU4 single board computers and then install everything needed to run  Apache Spark to perform data analysis. Total project cost is less than US$600.

Building the Cluster

  1. Cluster and Network Design
  2. Hardware Selection
  3. Construction
  4. Operating System Set Up
  5. Configuring the DHCP and NAT Services
  6. Adding the MicroSD Data Drives

Installing Data Analysis Software

Latest Software Builds

These are the latest builds for the arm71 platform for each of the software packages listed below.

 

Apache Hadoop

Here we install Hadoop onto the cluster and demonstrate some of its usage.Apache Hadoop

  1. Installing Java
  2. Getting Hadoop for ARM processor
  3. Preparing Cluster
  4. Installing and Configuring Hadoop
  5. Starting HDFS
  6. Mounting HDFS vis NFS
  7. Running the Word Count job with Hadoop
  8. More to come …

Apache SparkApache Spark

  1. Installing Spark
  2. Installing Jupyter Notebooks
  3. Running the Word Count job with Spark
  4. Mote to come …

Quantcast File System

The Quantcast File System (QFS) is a more efficient alternative to using HDFS to store data on the cluster.

  1. Installing QFS
  2. Installing Spark to use QFS
  3. Running the Word Count job with Spark on QFS

Data Analysis Projects