EVE Probe – Why, How, and Where We Are Now

View Github Repository Open presentation in a new window

breynisson

See all presentation from breynisson

EVE Probe – Why, How, and Where We Are Now

0 0

eveprobe_presentation

On Github breynisson / eveprobe_presentation

EVE Probe

Why, How, and Where We Are Now

Björgvin Reynisson bjorgvin@ccpgames.com @breynisson

What's this all about?

How we made a crowd-testing app that is also used for development

I will tell you about why we made a crowd-testing app, which is used heavily for development. Why did we do it? And what is the current status?

Who is your presenter?

Björgvin Reynisson

QA Engineer at CCP Games

Background in mobile phones

Electrical Engineer. At CCP since 2013. Previously at Nokia doing manufacturing tunings and tests. Now work with the graphics engine platform team.
A few words about CCP.
  • Founded 1997
  • EVE launches May 2003
  • Dust launches 2013
  • Valkyrie in development
  • Massively multiplayer online game (MMOG)
  • Launched May 2003
  • Single-shard

What is EVE Probe?

An app for:

Crowd-testing the EVE Online Graphics Engine Development Performance Benchmarks/Monitoring To have the ability to run a controlled test on YOUR machine. A way to stop reacting to defects. To make test-scenes while developing. To continuously run the same scenes to monitor the code performance.

Why EVE Probe?

Has anyone experienced this?

Anyone seen this?

How about this?

We do get reports about these sort of errors from time to time.

"We cannot reproduce this..."

...is neither satisfying for us nor our players

Why is reproduction hard?

Some issues may not be deterministic, i.e., may not be 100% reproducible even with same setup. Those are not the scope of this talk. It can be hard to recreate the conditions present when an issue happens: What is the hardware used? What is the software (OS, drivers)?

EVE Players use a LOT of different configurations

  • Windows
  • Mac
  • Linux (although we don't support it)
  • 855 different Graphics cards (GPU's)

Let's make an app!

EVE Probe

Works on your machine!

A shift from the tried and tired Dev quote of: "It works on my machine"

EVE Probe

Repeatable test-scenes that you play back on your computer

Crowd-testing!

Increased coverage

Works on your machine.

Demo:

This is one of the scenes currently in the EVE Probe.

What metrics are collected?

  • System Info
  • Settings
  • EVE Probe version
  • Logs
  • Performance Metrics
  • Crash info
  • Data is not personally identifiable
  • OS, GPU, Processor, RAM
  • Monitor resolution
  • Self-explanatory
  • If any
  • 149 different metrics for each scene.
  • We primarily look at Frametime and Memory metrics.
  • If the EVE Probe crashes during run we collect crash info.

Data Pipeline

  • EVE Probe sends data to a server on an AWS EC2 instance
  • This server zips up the incoming information.
  • We have an internal server which queries the AWS servers for data.
  • If there is new data, we send it to our internal server, where it is unpacked and sent to our Mongodb,
  • We query our Mongodb to process the data.

Data Pipeline

Initially used R and IPython notebooks for cleaning/processing/visualizing data

IPython Notebook Example

IPython Notebooks and R are EXCELLENT tools for exploratory data analysis. When your data becomes too big to handle in-memory, then you might want to explore other ways of digging into your data.

Where's my Dashboard!?

EveProbe Dashboard

Voila!

  • Python Flask application
  • Javascript visualization
  • Map/Reduce queries Mongodb

Reports

A collection of reports.

Example:

An example of one the reports.

Performance Metrics

From the "performance stat" field, you can choose any one of the 149 different metrics.

Performance Metrics

Deployment Process

Builds from each branch

Automatic builds. The page shows EVE Probe builds from the different branches. The publish button can be used to publish the selected build.

Getting the app to the players

This is a part of the process we've done the least amount of work on. We did a soft-launch, only putting up a forum thread with some information, and a download link. Since then, we have not really pushed the app, except for a one-time news-item in the EVE-Online launcher. Clearly, we suspect the adoption numbers reflect this.

Link:

http://bit.ly/1HkUMoY

Delivery

Resources are delivered On-Demand.

Binaries are delivered On-Demand.

Resources have been delivered on-demand from the start. This means that the needed resources for each scenes are downloaded at runtime. Binaries on-demand introduced recently. Means the binary version is downloaded at runtime.
Due to the Binaries being downloaded on-demand, you can choose the available versions. I.e., the versions that have been published by CCP show up in "Available versions:"
Due to the Binaries being downloaded on-demand, you can choose the available versions. I.e., the versions that have been published by CCP show up in "Available versions:"

Developer Tool

  • A test-framework
  • Repeatable scenes

Heavily used by the core graphics team.

Helps when doing any kind of refactoring or optimization. We know that the scene looks the same every time. So when when changing code, and comparing results, we know that the code is the variable, and not the test-setup.

Demo:

Success?

  • Great development tool
  • Has proven its worth for hard-to-reproduce defects
  • Big potential for benchmarking use

Also used the tool to prove the DoD concept.

Adoption

Total test-runs below 10K.

Improvements?

  • Low uptake
  • Soft launch with minimal push
  • Integrate into release process
  • We have not really been pushing the app.

Questions?

EVE Probe Why, How, and Where We Are Now Björgvin Reynisson bjorgvin@ccpgames.com @breynisson