The Perl test ecosystem

How to write tests in Perl

Eric Johnson / @kablamo_

Topics

Prove

prove is a tool for running Perl tests

prove  # runs tests in 't' and 'xt' directories
          

prove t  # runs tests in 't' directory
          

prove t/cowboy t/alien
          

prove -r t/cowboy t/alien # recursive
          

prove -lvr # with the 'lib' dir, verbose, recursive
          

prove          \
  --lib        \ # add 'lib' dir to your $PERL5LIB
  --verbose    \ # more output
  --recurse    \ # run tests in the t dir recursively
  -Pretty      \ # use the prove plugin App::Prove::Plugin::retty
  --jobs 10    \ # run tests in parallel
  --state=slow   # run slowest tests first
          

Prove states

• slow: tests that are slow run first
• fast: tests that are fast run first
• failed: only run tests that failed the last run
• passed: only run tests that passed the last run
• hot: tests that fail frequently run first
• new: newest tests based on modification time run first
• old: oldest tests based on modification time run first
• fresh: only run tests modified since the last run
• save: save test info to .prove in the cwd

.proverc

save your options in your ~/.proverc

--lib         # add 'lib' dir to your $PERL5LIB
--verbose     #
--recurse     # run tests in the t dir recursively
--state=save  # save test info to .prove in the cwd
-Pretty       # use the prove plugin App::Prove::Plugin::retty
          

Topics

Test directory layout

t
├── 00_compile.t        # this test runs first
├── data                # dir for test data
│   ├── alien.json
│   └── cowboy.json
├── lib                 # dir for test libraries
│   └── t
│       └── Helper.pm   # t::Helper
├── Transmogrifier
│   ├── In
│   │   └── Tiger.t     # unit tests for Transmogrifier::In::Tiger
│   └── Out
│       ├── Alien.t     # unit tests for Transmogrifier::Out::Alien
│       └── Cowboy.t    # unit tests for Transmogrifier::Out::Cowboy
└── Transmogrifier.t    # integration tests for Transmogrifier.pm
xt                      # extra tests are run only during a release
├── reallyslow.t
└── slow.t
          

• prove runs tests in alphabetical order by default
• prove only runs .t files
• layout of .t files mirrors organization of .pm files

Topics

Test::More

ok( $got, $test_name);
ok(!$got, $test_name);

is  ($got, $expected, $test_name);
isnt($got, $expected, $test_name);

like  ($got, qr/expected/, $test_name);
unlike($got, qr/expected/, $test_name);
          

is_deeply($got_complex_thing, $expected_complex_thing, $test_name);
          

isa_ok($object, $class);
          

# rare, but used for complex logic: 
# eg, if I get to this place, pass()
pass "this test will always pass";
fail "this test will always fail";
          

# Used to make sure you meant to stop testing here and
# didn't exit/return/die and skip the rest of your tests.
done_testing();
          

Test::More

SKIP: {
    skip $why, $how_many unless $have_some_feature;

    ok   $got1,                $test_name1;
    is   $got2, $expected,     $test_name2;
    like $got3, qr/$expected/, $test_name3;
};
          

$ prove t/tiger.t
t/tiger.t .. 
ok 1 - created a tiger
ok 2 - tiger has sharp teeth
ok 3 - tiger has stripes
ok 4 # skip third party tests
ok 5 # skip third party tests
ok 6 # skip third party tests
1..6
ok
All tests successful.
          

Test::More

use Test::More skip_all => 'third party tests';

# ...test code here...

done_testing;
          

$ prove t/tiger.t
t/tiger.t .. 
1..0 # SKIP third party tests
skipped: third party tests
Files=1, Tests=0,  0 wallclock secs ( 0.01 usr  0.01 sys +  0.04 cusr  0.00 csys =  0.06 CPU)
Result: NOTESTS
          

Test::More

TODO: {
    local $TODO = $why;

    is $got, $expected, $test_name;
};
          

$ prove t/tiger.t
t/tiger.t .. 
ok 1 - created a tiger
ok 2 - tiger has sharp teeth
ok 3 - tiger has stripes
not ok 4 - the tiger ate # TODO third party tests

#   Failed (TODO) test 'the tiger ate'
#   at t/Transmogrifier/In/Tiger.t line 18.
#          got: 'not implemented yet at /home/eric/code/slides-testing-in-perl/src/lib/Transmogrifier/In/Tiger.pm line 9.
# '
#     expected: undef
1..4
ok
All tests successful.
          

Subtests

Subtests are really useful and are highly recommended

subtest $name => sub {
    ok $got1,            $test_name1;
    is $got2, $expected, $test_name2;
    is $got3, $expected, $test_name3;
};
          

• Subtests are an organizational tool
• Subtests help prevent global variables
• We don't want 1000 line .t files for the same reasonwe don't want 1000 line functions

Topics

TAP

Test Anything Protocol

TAP is the stuff your tests output

TAP Syntax

# comic book noises test sequence commencing
# (this line and line before are ignored by the tap parser)
ok 1 - zap()
not ok 2 - bam()
#   Failed test 'bam()'
#   at xt/slow.t line 6.
not ok 3 - pow() # TODO not implemented yet
#   Failed (TODO) test 'pow()'
#   at xt/slow.t line 10.
ok 4 # skip because ComicBook::Noises::Batman is not installed
yargle blargle (this line is ignore by the tap parser)
ok 5 - kaboom()
1..5
          

• Any output not recognizable as TAP is ignored
• Any commented output is ignored
• Parser counts test failures and successes

TAP workflow

• Your test produces TAP output
• prove is a small wrapper around TAP::Harness
• TAP::Harness parses TAP output and calculates a summary
• So when you run prove, you are both producing TAP and consuming TAP
• Continuous integration systems might also consume TAP output

TAP recommendations

• Don't print messages because this can cause TAP parsing to fail
• If you do print messages, start them with #
• Or use note(), diag(), explain() which add # in front for you

note "here is a note";         # only displayed in verbose output
diag "here's what went wrong"; # always displayed
explain \%foo;                 # dump a data structure
          

Topics

Test::Differences

Uses Text::Diff to display a diff when the test fails

eq_or_diff $got, "$line1\n$line2\n$line3\n", "testing strings";
          

$ prove t/boop.t
t/boop.t....1..1
not ok 1 - testing strings
#     Failed test ((eval 2) at line 14)
#     +---+----------------+----------------+
#     | Ln|Got             |Expected        |
#     +---+----------------+----------------+
#     |  1|this is line 1  |this is line 1  |
#     *  2|this is line b  |this is line 2  *
#     |  3|this is line 3  |this is line 3  |
#     +---+----------------+----------------+
# Looks like you failed 1 test of 1.
          

Topics

Testing data structures

my $person = make_person();
like $person->{name},  '/^(Mr|Mrs|Miss) \w+ \w+$/', "name ok";
like $person->{phone}, '/^0d{6}$/',                 "phone ok";
is scalar keys %$person, 2, "number of keys";
          

Testing data structures

my $person = make_person();
if (ref($person) eq "HASH")
{
    like $person->{name},  '/^(Mr|Mrs|Miss) \w+ \w+$/', "name ok";
    like $person->{phone}, '/^0d{6}$/',                 "phone ok";
    is scalar keys %$person, 2, "number of keys";
}
else {
    fail "person is not a hash";
}
          

Testing nested data structures

my $person = make_person();
if (ref($person) eq "HASH") {
    like $person->{name},  '/^(Mr|Mrs|Miss) \w+ \w+$/', "name ok";
    like $person->{phone}, '/^0d{6}$/',                 "phone ok";
    my $cn = $person->{child_names};
    if (ref($cn) eq "ARRAY") {
        foreach my $child (@$cn) {
            like($child, $name_pat);
        }
    }
    else {
        fail "child names not an array";
    }
}
else {
    fail "person not a hash";
}
          

Conclusion?

Testing data structureswith Test::Moreis doable

Testing horrible complex nested data structureswith Test::Moreis painful

Test::Deep

my $person  = make_person();
my $name_re = re('^(Mr|Mrs|Miss) \w+ \w+$');
cmp_deeply
    $person,
    {
        name        => $name_re,
        phone       => re('^0d{6}$'),
        child_names => array_each($name_re),
    },
    "person ok";
          

Test::Deep

cmp_deeply
    \@array,
    [$hash1, $hash2, ignore()],
    "first 2 elements are as expected, ignoring third element";
          

Test::Deep

my $expected = {
  name        => ignore(),
  dna         => subsetof('human', 'mutant', 'alien'),
  is_evil     => bool(1),
  weaknesses  => set(@weaknesses), # ignores order, duplicates
  powers      => bag('flight', 'invisibility', 'fire'), # ignores order
  weapons     => any( undef, array_each(isa("Weapon")) );
  home_planet => all(
    isa("Planet"),
    methods(
       number_of_moons   => any(0..100),
       distance_to_earth => code(sub { return 1 if shift > 0 }),
    ),
  ),
};
cmp_deeply $superhero, $expected, "superhero hashref looks ok";
          

Topics

Exceptions as strings

Using standard Perl exception syntax

eval { die "Something bad happened" };
warn $@ if $@;
          

Using Try::Tiny

try   { die "Something bad happened" }
catch { warn $_ };
          

Exceptions as objects

Using standard Perl exception syntax

eval {
    die My::Exception->new(
        error    => 'Something bad happened',
        request  => $request,
        response => $response,
    );
};
warn $@->error if $@;
          

eval {
    My::Exception->throw(
        error    => 'Something bad happened',
        request  => $request,
        response => $response,
    );
};
warn $@->error if $a;
          

Exceptions as objects

Using Try::Tiny

try {
    My::Exception->throw(
        error    => 'Something bad happened',
        request  => $request,
        response => $response,
    );
}
catch {
    warn $_->error;
};
          

Test::Fatal

The standard module for testing exceptions

# Check the exception matches a given regex
like
    exception { $obj->method() },
    qr/Something bad happened/,
    "caught the exception I expected";
          

# Check method() did not die
is
    exception { $obj->method() },
    undef,
    "method() did not die";
          

# Check method() did not die
lives_ok { $obj->method } 'expecting to live';
          

Note: Test::Fatal is recommended over Test::Exception

Topics

Test::Warnings

use Test::More;
use Test::Warnings;
like 
    warning { $obj->method() },
    qr/BEEP/,
    'got a warning from method()';
          

use Test::More;
use Test::Warnings;
is_deeply
    [ warnings { $obj->method() } ],
    [ 
      'INFO: BEEP!',
      'INFO: BOOP!',
    ],
    'got 2 warnings from method()';
          

use Test::More;
use Test::Warnings;
use Test::Deep;
cmp_deeply
    [ warnings { $obj->method() } ],
    bag(
        re(qr/BEEP/),
        re(qr/BOOP/),
    ),
    'got 2 warnings from method()';
          

Note: Test::Warnings is recommended over Test::Warn

Test::Warnings

use Test::More;
use Test::Warnings;
pass 'yay!;
done_testing;
          

$ prove t/test.t
ok 1 - yay!
ok 2 - no (unexpected) warnings (via done_testing)
1..2
ok
All tests successful.
          

Note: Test::Warnings is recommended over Test::NoWarnings

Topics

Test::Lib

Searches upward from the calling file for a directory 't' with a 'lib' directory inside it and adds that to the module search path (@INC).

use Test::Lib;
use t::Private::Test::Library;
          

instead of

use FindBin qw($Bin);
use lib "$Bin/lib";
use t::Private::Test::Library;
          

Topics

Test::Cmd

use Test::More;
use Test::Cmd;
 
my $test = Test::Cmd->new(prog => '/bin/ls', workdir => '');

# ls -l /var/log
$test->run(args => '-l /var/log');
is $test->stdout, $expected_stdout, 'standard out';
is $test->stderr, $expected_stderr, 'standard error';
is $? >> 8,       1,                'exit status';

# ls -a
$test->run(args => '-a');
is $test->stdout, $expected_stdout, 'standard out';
is $test->stderr, $expected_stderr, 'standard error';
is $? >> 8,       1,                'exit status';

done_testing;
          

TIP: You can strip ansi color codes from your program's output using Term::ANSIColor::colorstrip().

Topics

Test::DescribeMe

Tell test runners what kind of test you are

use Test::More;
use Test::DescribeMe qw/extended/;

pass "foo";

done_testing;
          

$ prove t/test.t

1..0 # SKIP Not running extended tests
skipped: Not running extended tests

          

$ EXTENDED_TESTING=1 prove xt/reallyslow.t 
xt/reallyslow.t .. 
ok 1 - very very slow test
1..1
ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.03 cusr  0.00 csys =  0.04 CPU)
Result: PASS
          

See also Test::Settings and the Lancaster Consensus

Topics

Plack::Test

Web servers are event driven.

Event driven code looks weird.

The traditional Plack::Test UI is weird.

When a request happens, a code reference (aka callback) gets run.

Plack::Test

The traditional Plack::Test UI (widely used)

use Plack::Test;

test_psgi
    app    => My::WebApp->app,
    client => sub {
        # $cb is a callback (aka code reference)
        my $cb  = shift;  
        my $req = HTTP::Request->new(GET => "http://localhost/hello");
        my $res = $cb->($req);
        like $res->content, qr/Hello World/;
    };
          

$cb is a wrapper around the PSGI app which does the following:

Plack::Test

Alternate UI for Plack::Test (less common)

use Plack::Test;
use HTTP::Request::Common;

my $app  = My::WebApp->app;
my $test = Plack::Test->create($app);

my $res = $test->request(GET "/");
like $res->content, qr/Hello World/;
          

Plack::Test

No forking.

No network calls.

Makes things very fast and light.

Plack::Test

But if you want forking and network calls there are 2 options for you:

• It can fork and run a server in another process
• It can send your request to an external web server

Topics

Devel::Cover

Generate reports about test coverage

Devel::Cover

$ HARNESS_PERL_SWITCHES=-MDevel::Cover prove -lvr
                

$ cover
                

Devel::Cover

The report tells you the coverage of each of the following categories as a percentage.

• stmt: % of statements (lines of code) which were hit by tests
• bran: % of branches, eg if/else blocks
• cond: % of conditionals, eg $x && $y || $z(useful due to short circuiting)
• sub: % of subroutines which were hit by tests

Topics

Test::Most

you can type this

use strict;
use warnings;
use Test::More;
use Test::Exception;
use Test::Differences;
use Test::Deep;
use Test::Warn;
          

or you can type this

use Test::Most;
          

but

• Test::Fatal is recommended over Test::Exception
• Test::Warnings is recommended over Test::Warn

Topics

Test::Modern

you can type this

use strict;
use warnings;
use Test::More;
use Test::Fatal;
use Test::Warnings;
use Test::API;
use Test::LongString;
use Test::Deep;
use Test::Pod;
use Test::Pod::Coverage;
use Test::Version;
use Test::Moose;
use Test::CleanNamespace;
use Test::Benchmark;
use Test::Requires;
use Test::RequiresInternet;
use Test::Without::Module;
use Test::DescribeMe;
use Test::Lib;
          

or you can type this (well its roughly equivalent)

use Test::Modern;
          

but

Test::Modern doesn't work with Test::Pretty

Topics

Without Test::Pretty

~/code/text-vcard ⚡ prove t/vcard.t                                                                                                                                                                                                            
t/vcard.t .. 
    # Subtest: output methods
    ok 1 - as_string()
    ok 2 - as_file()
    ok 3 - file contents ok
    1..3
ok 1 - output methods
    # Subtest: simple getters
    ok 1 - full_name
    ok 2 - title
    ok 3 - photo
    ok 4 - birthday
    ok 5 - timezone
    ok 6 - version
    1..6
ok 2 - simple getters
    # Subtest: photo
    ok 1 - returns a URI::http object
    ok 2 - returns a URI::http object
    ok 3 - photo
    1..3
ok 3 - photo
    # Subtest: complex getters
    ok 1 - family_names()
    ok 2 - given_names()
    ok 3 - prefixes
    ok 4 - suffixes
    ok 5 - work phone
    ok 6 - cell phone
    ok 7 - work address
    ok 8 - home address
    ok 9 - work email address
    ok 10 - home email address
    1..10
ok 4 - complex getters
1..4
ok
All tests successful.
Files=1, Tests=4,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.07 cusr  0.02 csys =  0.12 CPU)
Result: PASS
          

With Test::Pretty

~/code/text-vcard ⚡ prove t/vcard.t                                                                                                                                                                                                            

  output methods
    ✓  as_string()
    ✓  as_file()
    ✓  file contents ok
  simple getters
    ✓  full_name
    ✓  title
    ✓  photo
    ✓  birthday
    ✓  timezone
    ✓  version
  photo
    ✓  returns a URI::http object
    ✓  returns a URI::http object
    ✓  photo
  complex getters
    ✓  family_names()
    ✓  given_names()
    ✓  prefixes
    ✓  suffixes
    ✓  work phone
    ✓  cell phone
    ✓  work address
    ✓  home address
    ✓  work email address
    ✓  home email address

ok
          

How to use Test::Pretty

Type this on the command line

prove -Pretty t
          

Or save some typing by adding this to your ~/.proverc

-Pretty
          

Topics

Other test modules

• Test::Fixme
• Test::NoTabs
• Test::Version
• Test::Pod
• Test::Pod::Coverage
• Test::Synopsis
• Test::Vars
• Test::LoadAllModules

Topics

THE END

These slides are online at

• kablamo.org/slides-perl-testing
• github.com/kablamo/slides-perl-testing

Test::Exception

The standard module for testing exceptions for years

Use Test::Fatal instead!

# Check that the stringified exception matches given regex
throws_ok { $obj->method } qr/division by zero/, 'caught exception';
          

# Check an exception of the given class (or subclass) is thrown
throws_ok { $obj->method } 'Error::Simple', 'simple error thrown';
like $@->error, qr/division by zero/;
          

# Check method() died -- we do not care why
# dies_ok() is for lazy people; throws_ok() is best practice
dies_ok { $obj->method } 'died as expected';
          

# Check method() did not die
lives_ok { $obj->method } 'lived as expected';