Spark Architecture Model

Coordination is done by the SparkContext object in your main program(called the driver program)

Cluster manager

An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)

Worker Node

Any node that can run application code in the cluster

Spark MLlib — Machine Learning Basics

Gather and prepare data

Select features

• Current age for predicting the probability of ending up in hospital in the coming five years.
• $\frac{\text{weight}}{\text{length}^2}$ for predicting percentage of body fat.
• You fall in category $A$, so you are very likely to do $Z$.

Train a model

Select the best model

• Train model with different parameters.
• Model might be too complex, or too simple.
• Evaluate model on some validation set and use cross-validation to find an appropriate model.

Predict

Spark MLlib — Programming Abstractions

• (Dense/Sparse) Vector
• LabeledPoint
• Matrix
• Rating
• Model classes
• Pipeline API

(Dense/Sparse) Vector

• A mathematical vector containing numbers.
• Both dense and sparse vectors.
• Constructed via the mllib.linalg.Vectors class.
• Do not provide arithmetic operations.

        val denseVec1 = Vectors.dense(1.0, 2.0, 3.0)
        val denseVec2 = Vectors.dense(Array(1.0,2.0,3.0))
        val sparseVec = Vectors.sparse(4, Array(0,2), Array(1.0, 2.0))

LabeledPoint

             val lp = LabeledPoint(1,Vectors.dense(3.14,1.68,1.41))

Matrix

• Integer typed row and column indices
• Double values
• Different implementations for distribution purposes (RowMatrix, BlockMatrix, CoordinateMatrix,...).
• Dense and sparse variants

Rating

         case class Rating(user: Long, item: Long, rating: Double)

Model classes

• Work on RDD[Vector], RDD[LabeledPoint], etc.
• Often follow naming pattern: <problem>With<Algorithm>, e.g. LinearRegresionWithSGD.
• Either the model follows a builder pattern and has a run() method, or it has static train() and predict() methods:

val points: RDD[LabeledPoint] = // ...
val lr = new LinearRegressionWithSGD()
                    .setNumIterations(200)
                    .setIntercept(true)
val model = lr.run(points)

val model = DecisionTree.trainClassifier(
                    input = data,
                    numClasses = 10,
                    categoricalFeaturesInfo = Map[Int, Int](),
                    impurity = "gini",
                    maxDepth = 15,
                    maxBins = 5
                    )

Pipeline API

• Advanced API for chaining machine learning operations in one workflow
• Uses the more advanced DataFrame features compared to RDD's of simple MLlib abstractions.
• Possible pipeline: Automated feature selection -> Model training -> Validation -> Model selection -> Prediction

Spark MLlib — Example: Predicting Digits

• 42,000 drawings of digits.
• Given a drawing, predict the written digit.
• Classification problem.
• Use Decision Tree approach.

CODE TIME

Spark GraphX — Graph Fundamentals

• Graphs are all about relationships between objects.
• Vertices are the objects we are interested in.
• Edges describe the relationship between two vertices.
• The alternate way of looking at data, allows us to ask different questions, or give answers to questions much easier. These questions are typically about relationships.

Spark GraphX — Programming Abstractions in Spark

Spark GraphX uses these terms as well, and has programming abstractions for them.

Graph

class Graph[VD, ED] {
        val vertices: VertexRDD[VD]
        val edges: EdgeRDD[ED]
}

A very rich wrapper around a collection of vertices and edges.

Vertices

• A VertexRDD[VD] is a rich wrapper around an RDD[(VertexID, VD)].
• A VertexID is a unique 64-bits Long. It allows for easy lookups and other operations on Vertices.

Edges

case class Edge[ED] (srcId: VertexId, dstId: VertexId = 0,attr: ED)

• An EdgeRDD[ED] is an RDD[Edge[ED].
• An Edge[ED] is basically a 3-tuple of (VertexId, VertexId, ED), where the edge is directed and ED is the type of relationship between the two Vertices.

EdgeTriplet

Constructing a Graph

Several options for creating a Graph in GraphX:

• Graph.apply(vertices,edges), or Graph(vertices,edges) results in a Graph instance.
• The Graph object also has fromEdges() and fromEdgeTuples() methods for Graph instantiation.
• GraphLoader.edgeListFile() provides a way to load a graph from a list of edges on disk.

Spark GraphX — Example: Medline Topics

source: ftp://ftp.nlm.nih.gov/nlmdata/sample/medline

Data

Code Time

Spark GraphX — Example: Maven Dependencies

source: https://github.com/ogirardot/meta-deps

Data

• Directed graph of maven dependencies.
• Data structure:

Code Time