Skip to main content

Big Data Cluster

ACCRE currently has a development environment set up for Vanderbilt and VUMC researchers to access, and plan to build out a production environment over the next 2-3 years. The development environment, (aka bigdata) is running the Cloudera implementation of the Hadoop ecosystem.

Getting Access to the bigdata Cluster

The bigdata cluster is available for use by the Vanderbilt community. Users should contact ACCRE to get access to the cluster.

Verifying the Connection

Once approved for access, use your terminal client to log in to Your username is your VUnetID and your password is the one you use to log in to the traditional ACCRE cluster. (In the past your password would have been the ePassword, the one used to log in to Vanderbilt email and other services. It has since been changed to the ACCRE password.)

Overview of Cloudera Services

YARNYet Another Resource Negotiator
OozieWeb app for scheduling Hadoop jobs
MapReduce 2MapReduce jobs running on top of YARN
HueUser interface for constructing Jobs
SparkMapReduce-like + cacheing
HiveETL transformations expressed as SQL
ImpalaInteractive SQL
HBaseRandom, realtime read/write access to distributed big data store
PigHigh-level language for expressing data analysis programs
SolrText search engine supporting free form queries

Hadoop command line

To interact with the HDFS filesystem, use the hadoop fs command:

[fido@abd740 ~]$ hadoop fs -ls /
Found 5 items
drwxr-xr-x   - hdfs  supergroup          0 2017-04-19 13:47 /data
drwxr-xr-x   - hbase hbase               0 2017-04-02 21:09 /hbase
drwxrwxr-x   - solr  solr                0 2017-02-24 17:20 /solr
drwxrwxrwx   - hdfs  supergroup          0 2017-05-06 00:26 /tmp
drwxr-xr-x   - hdfs  supergroup          0 2017-02-17 12:14 /user
[fido@abd740 ~]$ hadoop fs -ls /data
Found 9 items
-rw-r--r--   3 hdfs    supergroup       3359 2017-02-14 09:57 /data/
drwxr-xr-x   - hdfs    supergroup          0 2017-03-06 16:25 /data/babs
drwxr-xr-x   - hdfs    supergroup          0 2017-03-06 11:52 /data/capitalbikeshare-data
drwxr-xr-x   - hdfs    supergroup          0 2017-03-06 12:10 /data/citibike-tripdata
drwxr-xr-x   - hdfs    supergroup          0 2017-02-14 21:10 /data/google-ngrams
-rw-r--r--   3 hdfs    supergroup  274188932 2017-04-19 13:47 /data/hadoop-2.5.0-cdh5.2.0.tar.gz
drwxr-xr-x   - hdfs    supergroup          0 2017-01-18 19:06 /data/nyc-tlc
drwxr-xr-x   - hdfs    supergroup          0 2016-12-21 15:14 /data/stack-archives

Note: The Hadoop file system is separate from the Linux file system. hadoop fs -ls /data will return the example files above; ls /data will return nothing.

Logging on to the Cluster via Hue

Once approved, users will be able to connect to via ssh, but Cloudera Manager provides a WebUI to interact with the cluster called Hue. To access Hue, simply to in your web browser and enter your credentials.

Using the HDFS file browser

If you’ve used the web UIs for Dropbox, Google Drive, etc., then this step is a piece of cake. The File Browser is accessed from the dog-eared-piece-of-paper icon near the top right of the screen. In the file broswer, you’re able to navigate the directory structure of HDFS and even view the contents of text files.

  • When a new user logs into Hue, Hue creates an HDFS directory for that user at /user/<vunetid> which becomes that user’s home directory.
  • Note that, by default, logging in to Hue creates a new user’s home directory with read and execute permissions enabled for the world!
  • Files can be uploaded to your directories using the drag-and-drop mechanism; however, the file size for transferring through the WebUI is capped at around 50GB, so other tools like scp or rsync are necessary for moving large files onto the cluster.

Hosted datasets

In addition to your own data, ACCRE hosts some publicly available datasets at /data/:

/data/babsBay Area bikeshare data
/data/capitalbikeshare-dataDC area bikeshare data
/data/citibike-tripdataNYC bikeshare data
/data/google-ngramsn-grams collected from Google Books
/data/nyc-tlcNYC taxi trip data
/data/stack-archiveshistoric posts from StackOverflow et al.

As stated above, the example datasets are on the Hadoop file system, not the Linux file system.

If you know of other datasets that may appeal to the Vanderbilt community at large, just let us know!

Building an application

Hue uses Oozie to compose workflows on the cluster, and to access it, you’ll need to follow the tabs Workflows -> Editors -> Workflows.

From here, click the + Create button, and you’ll arrive at the workflow composer screen. You can drag and drop an application into your workflow, for instance a Spark job. Here you can specify the jar file (which, conveniently, you can generate from our GitHub repo) and specify options and inputs.

If you want to interactively select your input and output files each time you execute the job, you can use the special keywords ${input} and ${output}, which is a nice feature for generalizing your workflows.

Wordcount in Spark

Here, we present an example Spark application. This content is adapted slightly from the Spark getting started guide. Users can execute the same commands in the Spark REPL, which is launched by running in bash


where the environment variable $SPARK_HOME points to an installation of Spark. This can be a local Spark installation, or on bigdata, users can type spark-shell for Spark 1.6.2 or spark2-shell for Spark 2.0.0.

Read in a text file using the SparkContext

  • The SparkContext sc is the entry point for Spark’s data structures (provided automatically in the Spark REPL).
  • The value textFile becomes an RDD (Resilient Distributed Dataset)
val textFile = sc.textFile("spark_read_me.txt")
  • The RDD cannot be viewed directly in the REPL (in practice it will be distributed across many nodes!!). Thus, in order to view all the data, we have to gather the data at a single node using collect. A summary of RDD functions can be found here.
(textFile collect) foreach println
# Apache Spark

Spark is a fast and general cluster computing system for Big Data. It provides
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
MLlib for machine learning, GraphX for graph processing,
and Spark Streaming for stream processing.


Filter lines containing Spark

  • The RDD (and Scala collections) support filtering
val linesWithSpark = textFile.filter(line => line.contains("Spark"))
(linesWithSpark collect) foreach println
# Apache Spark
Spark is a fast and general cluster computing system for Big Data. It provides
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
and Spark Streaming for stream processing.
You can find the latest Spark documentation, including a programming
## Building Spark
Spark is built using [Apache Maven](
To build Spark and its example programs, run:
["Building Spark"](
The easiest way to start using Spark is through the Scala shell:
Spark also comes with several sample programs in the `examples` directory.
    ./bin/run-example SparkPi
    MASTER=spark://host:7077 ./bin/run-example SparkPi
Testing first requires [building Spark](#building-spark). Once Spark is built, tests
Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported
Hadoop, you must build Spark against the same version that your cluster runs.
in the online documentation for an overview on how to configure Spark.

Map lines from String to Array[String]

The RDD (and Scala collections) support mapping. For example:

val foo = "a line with Spark"
foo.split(" ") 
Array(a, line, with, Spark)

Scala Bonus! The dot operator can be omitted in Scala, so that splitting operation can be written as:

foo split (" ")
Array(a, line, with, Spark)

Let’s split each line

val arraysWithSpark = => line split(" "))
val wordsPerLine = arraysWithSpark map (a => a.size)
(wordsPerLine collect) foreach println

Counting words per line

  • Spark and Scala allow for chaining operations together. Thus, we can just write:
val wordsPerLine = => line split(" ")).map(a => a.size)
(wordsPerLine collect) foreach println
  • Equivalently:
val wordsPerLine = linesWithSpark map(_ split " ") map(_ size)
(wordsPerLine collect) foreach println

Here, Scala is smart enough to understand that the underscore implies that the map function is taking a single argument, thus avoiding the need for the foo => foo split(" ") pattern.

Scala Bonus! this pattern defines an anonymous function.

Counting total “Spark” occurrences with flatMap

If we wanted to count total occurrence of Spark we could filter our Arrays from the previous step to keep only those words that match Spark. Since we don’t actually care about which line contains occurrences of Spark but rather how many Sparks are in our entire document, we can consider all the individual words at once. Spark provides a mechanism to do this called flatMap.

val allSparkWords = linesWithSpark flatMap (line => line split " ")
(allSparkWords collect) foreach println

To get the Spark occurrences, simply filter:

val allSparkWords = linesWithSpark flatMap (_ split "\\s+") filter (_ == "Spark")
(allSparkWords collect) foreach println

Scala Bonus! "\\s+" is a regex that matches one or more consecutive whitespace characters.

Counting occurrences of each word

  • The classic MapReduce example (as popularized by Hadoop)
  • Split each line into words
  • Map each word into a word value pair, or Tuple, e.g. (the, 100)
  • First element is the key which serves as identifier
  • Second element is the value which signifies that each word has occurred one time.

Since each word is not unique, we need to group them together and count the occurrences per group. When we perform an action on an RDD, all the pairs with identical keys are sent to the same node and then we can aggregate these together. This is precisely what the reduceByKey function does:

val wordFrequencies = textFile flatMap (_ split ("\\s+") map (word => (word, 1))) reduceByKey (_ + _)
(wordFrequencies take 20) foreach println


The argument to reduceByKey is a function describing how to combine values (which must have the same type, otherwise see aggregateByKey). To print the output, we’ve usen the take function to take the first 20 results. We can also use the aptly named takeOrdered function:

(wordFrequencies.takeOrdered(50)(Ordering[Int].reverse.on(_._2))) foreach println