SCALACHECK

Property-based testing in SCALA

ScalaMAD (Scala Programming Meetup @ Madrid)

17 Nov 2015

By: Luis Rodero-Merino (Habla Computing)

luis.rodero@hablapps.com

Property-based?

Usually we set the expected output for some given input

// Specs2 example
class HelloWorldSpec extends Specification {
  (...)
  def e1 = "Hello world" must haveSize(11)
}

A property is set for any input

val prSqrt: Prop = forAll{(n: Int) => scala.math.sqrt(n*n) == n}
val notPrSqrt: Prop = exists{(n: Int) => scala.math.sqrt(n*n) != n}

"(...) ScalaCheck can be used to state props about isolated parts - units - of your code (usually methods)"

Mostly, this refers to testing pure functions, i.e. with no side-effects, especially changes on state.

Combining Properties

Logically

val pOr:Prop  = p1 || p2  
val pAnd:Prop = p1 && p2
val pEq:Prop  = p1 == p2  // p1 p2 both pass, proved or fail
val pIm:Prop  = p1 ==> p2 // if p1 holds, then p2 must hold

By grouping

object SomeSpec extends Properties("SomethingImportant") {
  property("p1") = forAll{...}
  property("p2") = forAll{...}
  ...
}
object FullSpec extends Properties("TheWholeBunch") {
  include(SomeSpec)
  include(AnotherSpec)
  ...
}

Checking Properties

val concatStrPr: Prop = forAll {(s1:String, s2:String) =>
  val concat = (s1+s2);
  concat.endsWith(s2) &&
  concat.startsWith(s1) &&
  concat.size == s1.size + s2.size
}
concatStrPr.check // <-- cheching!

val sumListPr: Prop = forAll {(l:List[Int]) =>
  (l.sum > 0) ==> (l.indexWhere(_ > 0) > 0)
}
sumListPr.check

Results Discussion

Not really what we expected, huh?

Properties are subtle. Or, rather, our domain is

Ok, but... how is the 'magic' done?

Generators

(and shrinking)

"Magic"

Generators

Create random values (of any kind)

Used by ScalaCheck to generate test cases

Can be used in any other application or testing environment

class Gen[+T]{
  (...)
  def sample: Option[T] 
  (...)
}

Provided Gens - Basic

val g1: Gen[Int] = Gen.choose(0,10)
val g2: Gen[Double] = Gen.choose(0,0.5)
val g3: Gen[Int] = Gen.chooseNum(-10,10)
val g4: Gen[Double] = Gen.posNum[Double]

Gen.alphaStr: Gen[String]; Gen.numStr: Gen[String]
Gen.identifier: Gen[String]; Gen.uuid: Gen[java.util.UUID]

val g5: Gen[String] = Gen.oneOf("Meat","Vegs")
val g6: Gen[String] = Gen.frequency((4,"Meat"),(1,"Vegs"))
val g7: Gen[Seq[String]] = Gen.someOf("Apple", "Orange", "Banana")

// Applicable also with other gens
val g8: Gen[Int] = Gen.frequency((4,Gen.choose(0,10)),
                                 (1,Gen.choose(10,100000)))

Provided Gens - Containers

val unifRand: Gen[Int] = Gen.choose(Int.MinValue,Int.MaxValue)
val gL1: Gen[List[Int]] = Gen.listOf(unifRand)
val gL2: Gen[List[String]] = Gen.listOfN(5,Gen.alphaStr)

val gA: Gen[Array[Int]] = Gen.containerOf[Array,String](unifRand)
val gS: Gen[Set[Int]] = Gen.containerOfN[Set,Int](10,unifRand)

val gT: Gen[(Int,Int)] = Gen.zip(unifRand,unifRand)

val gM: Gen[Map[Int,String]] =
        Gen.mapOfN[Int,String](5,unifRand,alphaStr))

Careful with final size of Sets and Maps!

Generators - Creation

class Gen[+T] {
  def suchThat(f: T => Boolean): Gen[T]
  def map[U](f: (T) => U): Gen[U]
  def flatMap[U](f: (T) => Gen[U]): Gen[U]
}

val gL1: Gen[List[Int]] = Gen.listOfN(5,Gen.choose(0,100))
val gL2: Gen[List[Int]] = gL1 suchThat (_.sum % 2 == 0)
// RECALL! gL2.sample returns Option[List[Int]]

val gI: Gen[Int] = Gen.choose(0,100) map (_ * 2)
val gB: Gen[Boolean] = Gen.choose(0,1) map (_ == 0)

case class User(name: String, age: Int)
val userGen: Gen[User] = 
  for {
    name <- Gen.alphaStr
    age  <- Gen.choose(0,200)
  } yield User(name, age)

val gLU1: Gen[List[User]] = Gen.listOfN[User](100, userGen)
val gLU2: Gen[List[User]] = Gen.choose(0,10) flatMap
                            (i => Gen.listOfN[User](i,userGen)

Passing Gens to Props

(Use of Arbitrary)

case class User(name:String, age:Int)
// Let's assume we have a generator for our class 'User', userGen
// How can ScalaCheck find it?  
Prop.forAll{(u:User) => u.name != ""} // Error!

// First solution, pass the gen explicitly:
Prop.forAll(userGen){(u:User) => u.name != ""} // Will work

// Second solution: use Arbitrary
// Scala uses instances of Arbitrary (which encapsulate a Gen)
// to get the generators required by its properties.
implicit val arbUserGen: Arbitrary[User] = Arbitrary(userGen)
Prop.forAll{(u:User) => u.name != ""} // Will work

// Now the gen will be available everywhere by Arbitrary
val uG2: Gen[User] = Arbitrary.arbitrary[User] //uG2 eq userGen

// Several Arbitrary types are predefined:
val gI: Gen[Int] = Arbitrary.arbitrary[Int]
val gS: Gen[String] = Arbitrary.arbitrary[String] // :)
// Also for BitSet, Date, BigInt, Future...

Generators - Special Test Cases

// Arbitrary.scala:88
implicit lazy val arbInt: Arbitrary[Int] = Arbitrary(
  Gen.chooseNum(Int.MinValue, Int.MaxValue)
)

// Gen.scala:570
def chooseNum[T](minT: T, maxT: T, specials: T*)(
  implicit num: Numeric[T], c: Choose[T]
): Gen[T] = {
  import num._
  val basics = List(minT, maxT, zero, one, -one)
  val basicsAndSpecials = for {
    t <- specials ++ basics if t >= minT && t <= maxT
  } yield (1, const(t))
  val allGens = basicsAndSpecials ++ List(
    (basicsAndSpecials.length, c.choose(minT, maxT)) )
  // allGens is now (if specials is empty)
  //  ( (1,minT),(1,maxT),(1,zero),(1,one),(1,-one),
  //    (5,choose(minT,maxT)) ) 
  frequency(allGens: _*)
}

Shrinking

Aka Testing Case Minimization, done if error in property

It needs an implicit Shrink[T] instance

sealed abstract class Shrink[T] {
  def shrink(x: T): Stream[T] // Shrinking T instances
}

Implementations for common types are available

implicit lazy val shrinkInt: Shrink[Int] = Shrink { n =>

  def halfs(n: Int): Stream[Int] =
    if(n == 0) empty else cons(n, halfs(n/2))

  if(n == 0) empty else {
    val ns = halfs(n/2).map(n - _)
    cons(0, interleave(ns, ns.map(-1 * _)))
  }
} // Shrink.shrinkInt.shrink(100) take 20 foreach println
//   Shrink.shrink[Int](100)

Stateful systems

What about stateful systems?

Welcome the org.scalacheck.Commands package

Functional Prog & ScalaCheck

• ScalaCheck follows ideas of FP:
  • First we code tests gens and props, then we run them!
  • Gens and props form a language that we interpret
• Also, ScalaCheck fits FP testing
  • In FP we define (algebraic) operators with certain laws (e.g. flatMap associativity in monads)
  • Scalaz uses ScalaCheck to test such laws

$ scala -cp [scalacheck & scalaz (core and scalacheck bind) jars]
scala> :paste
import scalaz._
import Scalaz._
import scalacheck.ScalazProperties._
import scalacheck.ScalazArbitrary._
import scalacheck.ScalaCheckBinding._
scala> functor.laws[List].check
scala> monad.laws[List].check

Questions?

Thank you!

(Where is my beer?)