Telco simulator with Spark + Akka + Kafka – Real Impact Analytics – Who am I?

View Github Repository Open presentation in a new window

RealImpactAnalytics

See all presentation from RealImpactAnalytics

Telco simulator with Spark + Akka + Kafka – Real Impact Analytics – Who am I?

1 0

big-data-meetup

On Github RealImpactAnalytics / big-data-meetup

Telco simulator with Spark + Akka + Kafka

Real Impact Analytics

Thoralf Gutierrez

Who am I?

  • Computer EngineeringArtificial Intelligence

  • Data ScientistSoftware Engineer

Data

Call Detail Record (CDR) Date — Time — Duration Billing — Recharge — Money transfer Social Network Antenna's geolocalisation Network information Data usage — URL — App

Missions

Operational optimization Campaign managment Drive field actions Improve network coverage Performance tracking Stock management Managed services Data commercialization Improve marketing Strengthen subscriber acquisitions Acquire competitor market Increase cross-sales Reduce churn Target high value subscribers A/B test campaign messages Compute campaign's ROI

Treating telco/banking data of 300,000,000 subscribers daily

Active in 35+ countries

Is this a lot?

4.1% of world population

10% of africa's population

Still Room for growth

What do we struggle with?

  • Benchmarking
  • Stress testing for data quantity increase
  • Prepare for worst case scenario
  • Convince prospect

Solution?

Simulate a real world telecommunication provider

Telco Simulator

Genesis

Requirements

  • Be as close as possible to realitysocial behaviours, mobility paterns, activity distibutions, ...
  • Be able to simulate the bigest telco providersChina Mobile: 800M subscribers

What did we build?

Nature

God

What's what ?

Egnine for large-scale data processingGraphX is Spark's API for graph-parallel computation

Framework to build actor-based concurrent systems

Distributed log-based messaging system

God

God creates the world

  • the antennas based on the geography
  • the subscribers based on
    • operators + number of subs
    • profiles for each operator (social + mobility)
    • geographical distribution of the population
  • the social graph based on social profiles

Nature

Nature brings it to life

Future challenges

  • mobilitywho is where ?
  • synchronisationis eveybody at the same time ?

Synchronisation : Async

  • The clock sends the time to everyone
  • Every actor advances at his own pace (his message queue)
  • Simple
  • But we can't make any supposition about time ...

Synchronisation : Sync

  • The clock waits for eveybody to finishbefore sending the next tick
  • Every actor advances at the same pace
  • As slow as the slowest actor
  • We can make suppositions about time

Synchronisation : Sync

Relying on synchronized clocks in a distributed system is an anti-pattern @jboner

— Nitro Engineering (@NitroEng) August 19, 2015

The cost of maintaining the illusion of time in distributed systems increases with consistency and coherence - #reactive by @jboner

— Nitro Engineering (@NitroEng) August 19, 2015
black Star Glassesimage/svg+xmlLayer 1

We are hiring (nerds)!

Product Owners, MEAN Stack, QA, Devops Lead Architect, Head of Engineering

tg@ria.bi

Telco simulator with Spark + Akka + Kafka Real Impact Analytics Thoralf Gutierrez