Generative Integration Tests – Ashton Kemerling – So, Let's Talk TDD

View Github Repository Open presentation in a new window

AshtonKem

See all presentation from AshtonKem

Generative Integration Tests – Ashton Kemerling – So, Let's Talk TDD

0 0

Conj-Talk

On Github AshtonKem / Conj-Talk

Generative Integration Tests

Ashton Kemerling

@ashton

Thank You

My Wife Leah

The Conj Organizers

Countless open source contributors

Don't forget:
  • Married exactly 2 weeks ago
  • No working on honeymoon

Who I Am

Pivotal Labs

Pivotal Tracker

An agile project management tool. More reliable software: easier to reason & tests.

So, Let's Talk TDD

TDD at Tracker

  • 52k lines of Ruby
  • 12k+ RSpec examples
  • 39k lines of Javascript
  • 7k+ Jasmine Examples
  • 3 dedicated QA personnel + 1 on support
Also, external code quality tooling

What's the Issue

  • Time
  • Creativity
  • Edge/ordering cases
  • Emergent behavior
You usually don't have enough time, and you can't tell when you're short on creativity.

Generative Testing

Check to see who has used Test.Check or QuickCheck.

The Benefits of Generative Testing

  • Let the machine come up with data
  • Run more cases than you want to write
  • Shrinking
Explain shrinking

The Downsides of Generative Testing

  • Test duration
  • Reduced assertion power

Integration Generative Tests

Represent actions as data

How We Use Generative Tests

Reproduction Step Finding

Shrinking finds the minimal user actions to provoke a bug.

Trade blind hunting for code that searches for you

Hitting Known Trouble Spots

  • Concurrency
  • Bits of code post refactor

Blind Luck

Canvasing the application as a whole can find missed errors.

Creates a very large regression net

Tools

  • Clojure
  • Test.Check (or DoubleCheck)
  • clj-webdriver
  • JDBC (or similar)
  • clj-http (optional)

We use JDBC for setup and assertions, replace with whatever driver you need

We use simple queries, if you need more power be very mindful of performance

Test.Check

Port of QuickCheck

Written by Reid Draper

How Test.Check Works

Macros create test case Create bookkeeping code Run cases Record failures Shrink failing cases Report Pause and make sure everyone is caught up

Example Test

(defspec first-element-is-min-after-sorting
         100
         (prop/for-all [v (gen/not-empty (gen/vector gen/int))]
           (= (apply min v)
              (first (sort v)))))
This is from the Test.Check docs. More complicated examples left as an exercise to reader.

The Process

Before All Tests

Copy/Setup database state Setup browser(s)

Before each run

Restore database state Refresh browser(s) and clear caches Generate actions Execute actions Assert

The Parts

  • Generators
  • Action runners
  • Assertions

Generators

Functions to produce data

Simple or higher-ordered

Sample Generators

  • int
  • string
  • vector
  • not-empty
  • hash-map
  • return
  • elements
  • one-of
We'll see some code samples in a bit.

An Action

{:story 2198
 :type ::drag-drop
 :via ::selenium
 :args 2192}

A Simple Test

(defspec drop-drop-spec 10
  (prop/for-all [actions (gen/not-empty
                           (gen/vector
                             (gen/hash-map
                               :type (gen/return ::drag-drop)
                               :via (gen/return ::selenium)
                               :story (gen/elements @story-ids)
                               :args (gen/elements @story-ids))))]
    (perform-actions db project-id actions)))

Unexpected Techniques

  • Each Action is Atomic
  • Impossible sequences are inconsequential
  • Actions must completely describe all test behavior

Actions

High Level Basics

Multimethods are your friends

(defmulti perform-action
  (fn [type via project-id story args] [type via]))

Strategies

Separate context from action

Allow for different execution order (parallel, sequential)

Strategies must setup the environment for themselves

Assertions

  • Belong in strategies only
  • May change depending on strategy
  • You can assert on alert text

Tolerance

  • Actions must be tolerant to application changes & state
  • Invalid actions are a noop
  • Don't be afraid to call to JS if it makes actions more repeatable

Assertions

The Basics

  • Assertions cannot depend on input data
  • Each assertion must be true every time
  • Aim to catch serious bugs

Trackers Assertions

  • All changes should synchronize (localhost)
  • Client/Server state parity
  • Presence of alerts and error dialogs
  • No crashes

A Word on Performance

Assertions can be slower than actions

Generative tests are very sensitive to performance issues

Slower assertions reduce the number of runs you can do

Demo Time

Glenn Vanderburg

Questions?

Thank You!