explorative-tdd-talk
Explorative Test-Driven Development
Welcome everyone, thank you for having me here.Oleksii Fedorov
Software Craftsperson
Software Engineer@ Pivotal Labs
My name is Oleksii Fedorov. I am a Software Craftsperson and this is my twitter handle. I work as a Software Engineer @ Pivotal Labs.Eliminate Fear of Changing Legacy Code
- Increase test coverage
- Increase understanding of the code
- Test-drive your tests
Legacy Code
- Hard to understand
- No tests
- Brings value
Agenda
Knowledge in Code Mutation Code <-> Test Relationship Most Useful Coverage Metric Mutational Testing Explorative TDD Step-by-step Example Outside of Legacy Code We will define what Knowledge in the ProdCode and Mutation means. We will take a look at the relationship of the ProdCode and TestSuite. Then we will see what is the most useful coverage metric is. Then we will explore Mutational Testing and Explorative TDD techniques. And we will go through the example. Shall we get started?Knowledge in Code
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
some_var = some.value(from, other, source)
if some_var.predicate?
something.do_action(some, arguments, here)
elsif some_var == SPECIAL_VALUE
return UNKNOWN
else
some_var.split(",").each do |item|
collaborator.send_item(item)
end
end
And so on.
I think you get the idea. And if we change small piece of knowledge, we are introducing...Mutation
Mutation - granular change of knowledge, that changes behavior of the system.
Example
if cell_is_alive
do_this
else
do_some_other_thing
end
..Explain what this code does..
So let's pick a first bit of knowledge here:
if cell_is_alive
do_this
else
do_some_other_thing
end
if true cell_is_alive
do_this
else
do_some_other_thing
end
if false condition
Other available mutations here: Changing if condition to always be false.
if !condition condition
Inverting if condition.
if condition
# if_body
if_body
else ...
Commenting out if body
if condition
if_body
else
# else_body
else_body
end
Commenting out else bodyCode and Test Suite Relationship
How does Test Suite affect Production Code?
- Makes sure Code is correct
- Enables refactoring
- Gives courage to introduce change
- Coupled to Code
How Does Production Code affect Test Suite?
- Knowledge should be verified by Test Suite
- Mutation should lead to a Test failure
Knowledge Change
==
Test for the Test
Agenda
Knowledge in Code Mutation Code <-> Test Relationship Most Useful Coverage Metric Mutational Testing Explorative TDD Step-by-step Example Outside of Legacy Code ..Great point to stop, give audience chance to ask questions and drink some water..Knowledge Coverage
How to check if knowledge is covered?
Break it.
Introduce a mutation.
Introduce a very small change to the knowledge. The test suite should fail. If it doesn't - knowledge is not covered well enough. And that leads us to the term called...Semantic Test Stability
Semantic Test Stability. Test Suite can be considered semantically stable if for any mutation to any bit of knowledge it tests there is a failing test. There are techniques that allow us to keep this metric up high. One of them is...Mutational testing
Example
if cell_is_alive
do_this
else
do_some_other_thing
end
First, we need to narrow our scope to a single bit of knowledge.
if cell_is_alive
do_this
else
do_some_other_thing
end
Second, we need to introduce a mutation:
if true cell_is_alive
do_this
else
do_some_other_thing
end
Third, we need to make sure there is a test failure:
$ rake test
....
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
4 examples, 0 failures
Oh no, it didn't fail, so we have a "failing test" for our test suite. In this case we need to add the test for the negative case:
cell_is_alive = false
expect(did_some_other_thing).to eq(true)
cell_is_alive = false
expect(did_some_other_thing).to eq(true)
cell_is_alive = false
expect(did_some_other_thing).to eq(true)
$ rake test
....F
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
5 examples, 1 failure
if true
do_this
else
do_some_other_thing
end
if cell_is_alive true
do_this
else
do_some_other_thing
end
if cell_is_alive
do_this
else
do_some_other_thing
end
$ rake test
.....
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
5 examples, 0 failures
Usually, to accomplish any useful behavior we would like to combine multiple bits of knowledge. So if we want to understand how system works better, we need to focus on groups of pieces of knowledge. This is what Explorative TDD is about:Explorative TDD
The technique used to increase code coverage and understanding of the knowledge in the code.
1. Narrow scope to some manageable knowledge and isolate it
(manageable knowledge = method/function/class/module)2. Read and try to understand one piece of knowledge
3. Write a test to verify this assumption
4. Make sure it passes
by altering the assumption, or fixing production code (bugs)5. Apply Mutational Testing repeatedly
Apply Mutational Testing to each related granular piece of knowledge to verify that the understanding (and the test) is correct (this may introduce more tests)6. Go back to 2
Recap
Narrow scope and isolate it Read, try to understand, pick granular piece of knowledge Write a test Make sure test passes Apply Mutational Testing repeatedly Go back to 2 I think this is a good time to have some questions... I think we should go through a small example...Agenda
Knowledge in Code Mutation Code <-> Test Relationship Most Useful Coverage Metric Mutational Testing Explorative TDD Step-by-step Example Outside of Legacy CodeI want to see it in action!
(step-by-step example)
Narrow & Isolate
Means of isolation:
- Extract completely to module/class/package of its own.
- Duplicate the code under the test and put it into function/method(s) with different distinguishable name.
Our scope
class User
def notifications
notifications = Database
.where("notifications") do |x|
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
(x[1][0] == "favorited_notification" &&
StatusUpdate.find(x[1][2].to_i)
.owner_id == id) ||
(x[1][0] == "replied_notification" &&
StatusUpdate.find(x[1][2].to_i)
.owner_id == id) ||
(x[1][0] == "reposted_notification" &&
StatusUpdate.find(x[1][2].to_i)
.owner_id == id)
end.map do |row|
id, values = row
kind = values[0]
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif kind == "favorited_notification"
{
kind: kind,
favoriter: User.find(values[1].to_i),
status_update: StatusUpdate
.find(values[2].to_i),
}
elsif kind == "replied_notification"
{
kind: kind,
sender: User.find(values[1].to_i),
status_update: StatusUpdate
.find(values[2].to_i),
reply: StatusUpdate
.find(values[3].to_i),
}
elsif kind == "reposted_notification"
{
kind: kind,
reposter: User.find(values[1].to_i),
status_update: StatusUpdate.find(values[2].to_i),
}
end
end
Analytics.tag({name: "fetch_notifications",
count: notifications.count})
notifications
end
As you might guess, this code is really overwhelming. So, let's start by duplicating the whole method in an isolated method to test it:
class User
def notifications
notifications = Database
.where("notifications") do |x|
...
end
class User
def notifications
notifications = Database
.where("notifications") do |x|
...
end
class User
def notifications ... end
def notifications_isolated
notifications = Database
.where("notifications") do |x|
...
end
Next we need to read the code and try to understand a concrete part of the behavior it has. We need to identify related knowledge for this behavior as a whole:
...
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
...
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
...
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
These bits of knowledge indeed look related, so let's try to guess what behavior they are responsible for:
it "can load followed notifications" do
# TODO
end
Wait. I think we are making to big assumption. There is a smaller assumption that we need to validate here:
it "can load some followed notifications" do
# TODO
end
it "can load some notifications" do
# TODO
end
it "can load some notifications" do
user = User.new(email: "john@example.org",
password: "welcome")
# TODO
end
it "can load some notifications" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
end
it "can load some notifications" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
notifications = user.notifications_isolated
end
it "can load some notifications" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
notifications = user.notifications_isolated
expect(notifications.count).to eq(1)
end
it "can load some notifications" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
notifications = user.notifications_isolated
expect(notifications.count).to eq(1)
end
Next step is to make sure that this test passes:
$ rake test
.
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
1 example, 0 failures
Now we need to apply mutational testing repeatedly to these bits of knowledge until we are confident that it is well-tested. For example:
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
Let's take a closer look at this boolean condition:
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
For example, We could remove it:
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
$ rake test
F
Finished in 0.03511 seconds (files took 0.11877 seconds to load)
1 example, 1 failure
So it fails, great! Let's take a detailed look at the failure itself:
1) User#notifications looks like it loads some notifications from the database
Failure/Error: expect(notifications.count).to eq(1)
expected: 1
got: 0
(compared using ==)
# ./lemon_spec.rb:63
1) User#notifications looks like it loads some notifications from the database
Failure/Error: expect(notifications.count).to eq(1)
expected: 1
got: 0
(compared using ==)
# ./lemon_spec.rb:63
1) User#notifications looks like it loads some notifications from the database
Failure/Error: expect(notifications.count).to eq(1)
expected: 1
got: 0
(compared using ==)
# ./lemon_spec.rb:63
This mutant is not surviving, which means that our test is good. Or is it? Let's see what other mutations we can introduce for this bit of knowledge:
false (x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
true (x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
Replacing condition with true results in:
$ rake test
.
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
1 example, 0 failures
Now we have to either:
- change our understanding if it is not what we expect, or
- change our test to cover that, or
- add more tests.
it "ignores records of an invalid kind" do
end
it "ignores records of an invalid kind" do
user = User.new(email: "john@example.org",
password: "welcome")
end
it "ignores records of an invalid kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["invalid", "986", user.id.to_s])
end
it "ignores records of an invalid kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["invalid", "986", user.id.to_s])
notifications = user.notifications_isolated
end
it "ignores records of an invalid kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["invalid", "986", user.id.to_s])
notifications = user.notifications_isolated
expect(notifications.count).to eq(0)
end
it "ignores records of an invalid kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["invalid", "986", user.id.to_s])
notifications = user.notifications_isolated
expect(notifications.count).to eq(0)
end
$ rake test
.F
Finished in 0.21531 seconds (files took 0.11582 seconds to load)
2 examples, 1 failure
1) User#notifications ignores records of invalid kind
Failure/Error: expect(notifications.count).to eq(0)
expected: 0
got: 1
(compared using ==)
# ./lemon_spec.rb:131
1) User#notifications ignores records of invalid kind
Failure/Error: expect(notifications.count).to eq(0)
expected: 0
got: 1
(compared using ==)
# ./lemon_spec.rb:131
1) User#notifications ignores records of invalid kind
Failure/Error: expect(notifications.count).to eq(0)
expected: 0
got: 1
(compared using ==)
# ./lemon_spec.rb:131
Great, it means that this mutation is covered by our tests too. It is important to undo the mutation and see all tests pass:
(x[1][0] == "followed_notification" &&
x[1][2] == id.to_s) true ||
...
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
..
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
2 examples, 0 failures
Apply Mutational Testing repeatedly
...
kind = values[0]
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
...
kind = values[1] values[0]
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
...
kind = values[1] values[0]
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
$ rake test
..
Finished in 0.02343 seconds (files took 0.11584 seconds to load)
2 examples, 0 failures
It seems we need another test:
it "loads notifications with a correct kind" do
end
it "loads notifications with a correct kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
end
it "loads notifications with a correct kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
notifications = user.notifications_isolated
expect(notifications[0][:kind])
.to eq("followed_notification")
end
it "loads notifications with a correct kind" do
user = User.new(email: "john@example.org",
password: "welcome")
Database.insert("notifications",
["followed_notification", "986",
user.id.to_s])
notifications = user.notifications_isolated
expect(notifications[0][:kind])
.to eq("followed_notification")
end
..F
Finished in 0.14636 seconds (files took 0.12127 seconds to load)
3 examples, 1 failure
1) User#notifications loads followed notifications with correct kind
Failure/Error: expect(notifications[0][:kind]).to eq("followed_notification")
NoMethodError:
undefined method `[]' for nil:NilClass
# ./lemon_spec.rb:72
1) User#notifications loads followed notifications with correct kind
Failure/Error: expect(notifications[0][:kind]).to eq("followed_notification")
NoMethodError:
undefined method `[]' for nil:NilClass
# ./lemon_spec.rb:72
1) User#notifications loads followed notifications with correct kind
Failure/Error: expect(notifications[0][:kind]).to eq("followed_notification")
NoMethodError:
undefined method `[]' for nil:NilClass
# ./lemon_spec.rb:72
And this failure is pointing to the code in the test:
# ./lemon_spec.rb:72
notifications[0][:kind]
# ./lemon_spec.rb:72
notifications[0][:kind]
^ nil ^[:kind]
# ./lemon_spec.rb:72
notifications[0][:kind]
^ nil ^[:kind]
# ./lemon_spec.rb:72
notifications[0][:kind]
^ nil ^[:kind] => NoMethodError
We made slightly wrong assumption about what will happen after the mutation and the failing test has proven us wrong. We had to investigate what really has happened and therefore we have deepened our understanding of this knowledge in the code.
...
kind = values[1]
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
...
kind = values[0] values[1]
if kind == "followed_notification"
{
kind: kind,
follower: User.find(values[1].to_i),
user: User.find(values[2].to_i),
}
elsif ...
...
Finished in 0.14636 seconds (files took 0.12127 seconds to load)
3 examples, 0 failures
Continue applying mutational testing
(until there is enough confidence)
Go back to step 2 and repeat
Choose new group of bits of knowledge responsible for other behaviors of the code.
And repeat.
(until there is enough confidence)
This step-by-step example can be viewed as commit history here:
This concludes our example and I think it is time for questions... With that done, we should see, if that technique can be used outside of the context of Legacy Code...Outside of the context of Legacy Code
Useful during big refactorings
(extract class/module/package)
Useful when refactoring tests
- verify that test suite is still correct after refactoring
- verify that test suite is not rigid (and identify parts requiring refactoring)
(rigid = one change -> 70% tests fail)
Let's recap the technique itself and draw a bottom line.Recap & Q & A
Narrow scope and isolate it Read, try to understand, pick granular piece of knowledge Write a test Make sure test passes Apply Mutational Testing repeatedly Go back to 2 It is time for questions now.Thank you
Twitter: twitter.com/waterlink000
Github: github.com/waterlink
Blog: tddfellow.com
0
Explorative Test-Driven Development
Welcome everyone, thank you for having me here.