Goals

• Improve Testing Coverage
• Test Concurrency in a Sane Way
• Help Find Better Unit Test Cases
• Provide Confidence in Application Correctness

Non-Goals

• Replace Unit Tests
• Replace JS, or Ruby
• Test Visual Behavior (CSS & Rendering)

The Differences Between Unit Tests and Generative

The difference is how the example data is made.

Unit tests require hand entered behavior

Generative tests require description of possible behaviors, the machine makes the combinations

Example Based Tests

A series of imperative actions

Simple but tedious to write, easy to debug, easy to miss edges

drag_story = 'Learn to love as humans do'

within expand_story(drag_story) do
  select_in_dropdown '1 point', from: '.estimate'
  wait_until_empty_command_queue
  click_save_button
end

story_preview(drag_story).drag_to(story_preview('Glen identified as hostile'))

click_on 'Split Current/Backlog'

expect_story_order_to_be_in_one_browser :backlog_102,
drag_story, 'Glen identified as hostile'

Shape Based Testing

Depend on describing the "shape" of expected data

Harder to write, much better at catching edge cases

(check-that "Factoring 2 * a prime returns 2 and the prime" 1000
  (prop/for-all [prime (gen/elements primes)]
    (= [2 prime] (prime-factors (* 2 prime)))))

(check-that "The product of primes should factor to the original primes" 1000
   (prop/for-all [prime-seq (gen/vector (gen/elements primes))]
     (= prime-seq
        (prime-factors (apply * prime-seq)))))

How Generators Work

There are 2 kinds of generators, basic and composite

Basic generators produce primitive types like ints and strings

Composite generators used to produce collection types

They can be combined to produce arbitrary complexity

Basic Generators

• gen/int
• gen/string
• gen/pos-int
• gen/byte
• gen/elements

Composite Generators

These accept other generators as input, and modify them

• gen/vector
• gen/one-of
• gen/such-that
• gen/hash-map
• gen/map
• gen/frequency

Example Generators

List of integers

(gen/vector gen/int)

non-empty list of strings

(gen/not-empty (gen/vector gen/string))

Hashmap with keys :first, :last, and :age

(gen/hash-map
  :first gen/string-ascii
  :last gen/string-ascii
  :age gen/nat)

List of mixed integers and strings

(gen/vector (gen/one-of [gen/int gen/string]))

Testing Techniques

Generators are nice, but how do we make them work for us?

Reliable Server State

• Database state copied before any tests run
• Reset original story state and expire cache after each run
• Ensures that each test starts at a clean state quickly

Actions as Data

Each test generates a list of hashmaps, each representing an action a user could take.

A single function then runs those actions & checks that nothing horrible happened.

[{:type ::change-state
  :story 2186
  :args [:finished]}]

Easy copy & paste to a unit test for repeatability

Makes generating series of actions very easy

Perform Actions

Each type of action has a single function dedicated to execution

Use a mixture of dom state and internal state to validate actions

Isolated and very easy to test for correctness

Server Client Parity

Tests can touch the database and the browser for verification

The following checks are performed after all actions are run

• Check project version parity between database and browsers
• Compare story order correct between database and browsers
• Ensure that no force reload scrim exists

Examples!

Change Story Type

(defspec ^:selenium change-story-type 20
  (prop/for-all [actions (gen/not-empty (gen/vector (gen/hash-map
                       :type (gen/return ::change-type)
                       :story (gen/elements (map :id @stories))
                       :args (gen/elements
                               [[:feature] [:release] [:bug] [:chore]]))))]
                (perform-actions actions)))

Or factoring out the reusable action generator

(def type-generator #(gen/hash-map
                       :type (gen/return ::change-type)
                       :story (gen/elements (map :id @stories))
                       :args (gen/elements
                               [[:feature] [:release] [:bug] [:chore]])))

(defspec ^:selenium change-story-type 20
  (prop/for-all [actions (gen/not-empty (gen/vector (type-generator)))]
                (perform-actions actions)))

Adding Comments

(def comment-generator  #(gen/hash-map
                          :type (gen/return ::add-comment)
                          :story (gen/elements (map :id @stories))
                          :args (gen/such-that (fn [v] (= (count v) 1))
                                               (gen/vector gen/string))))

(defspec ^:selenium add-comment 10
  (prop/for-all [actions (gen/not-empty (gen/vector (comment-generator)))]
                (perform-actions actions)))

Change Story Type

Generator

(def state-generator #(gen/one-of
                       [(gen/hash-map
                         :type (gen/return ::change-state)
                         :story (gen/elements (map :id
                                                   (filter (comp
                                                            (partial = "feature")
                                                            :story_type)
                                                           @stories)))
                         :args (gen/elements
                                [[:started] [:finished] [:delivered] [:accepted]]))
                        (gen/hash-map
                         :type (gen/return ::change-state)
                         :story (gen/elements (map :id
                                                   (filter (fn [story]
                                                             (#{"chore" "bug"}
                                                              (:story_type story)))
                                                           @stories)))
                         :args (gen/elements [[:started] [:accepted]]))]))

The Tests

(defspec ^:selenium change-story-state 10
    (prop/for-all [actions (gen/not-empty (gen/vector (state-generator)))]
                  (perform-actions actions)))

Mixed Actions

(defspec ^:selenium mixed-actions 10
  (prop/for-all [actions (gen/not-empty (gen/vector (gen/one-of
                                                      [(comment-generator)
                                                       (state-generator)
                                                       (type-generator)])))]
                (perform-actions actions)))