Property-based testing

ITT8060 Advanced Programming · Autumn 2026

Tallinn University of Technology

Full prose version with runnable examples: notes.html

ITT8060

Introduction

Designing, implementing and reasoning about large systems requires abstraction.

We treat a component as a black box (hiding the implementation) and are only interested in its behaviour (external interface): what does it require, what does it guarantee.

Assumption: components adhere to their specification.

Property-based testing: we specify (propose, hypothesise) a property of a component, the tool tries to falsify it (using randomness).

A "side effect" of testing in this style is that the specification of a component is made explicit (a set of properties).

ITT8060

1 · Properties

ITT8060

What is a property?

In property-based testing, a property is a falsifiable statement about the behaviour of a component.

We express this statement in the host language (via an embedded domain-specific language).

Non-example: every accepted job eventually completes. (Cannot falsify with a finite test.)

Non-example: the system is reliable when operated correctly. (Failure criterion undefined/fluid.)

Say that we are testing a sorting function.

How can we conclude that it is implemented incorrectly? What observation would falsify the claim that it is implemented correctly?

Property: for all xs : 'a list, List.sort xs is a permutation of xs. Falsification: a list ys (a counterexample) such that List.sort ys is not a permutation of ys.

Property: for all xs : 'a list, List.sort xs is non-decreasing. Falsification: a list ys such that List.sort ys is not non-decreasing.

ITT8060

Simple properties

Given a function f : 'a -> 'b, a simple property about f is a predicate on 'a (a function of type 'a -> bool).

For example, if we take List.rev : 'a list -> 'a list as our f, what property does the following predicate (of type 'a list -> bool) express?

let p1 xs = List.length (List.rev xs) = List.length xs

The body of the function checks that the length of the reversal of xs is the same as the length of xs.

> p1 [1; 2; 3];;
val it: bool = true

Reversing a list preserves the length.

ITT8060

Concrete example

List.rev is its own inverse.

#r "nuget: FsCheck, 3.6.0"
open FsCheck
open FsCheck.FSharp

let revRevIsId (xs: int list) = List.rev (List.rev xs) = xs

FsCheck tries to falsify this property (find a counterexample).

Generate ys : int list. If revRevIsId ys = false, then we have found a counterexample and the process stops. Otherwise, we continue. By default, up to 100 iterations.

> Check.Quick revRevIsId;;
Ok, passed 100 tests.
val it: unit = ()
ITT8060

What inputs are generated?

Check.Verbose also prints the generated input. We truncate the output.

> Check.Verbose revRevIsId;;
0:
[0; 2]
1:
[1; 0; 1]
2:
[-3]
3:
[-1]
4:
[-3; 0]
5:
[0; -1; 0; -2; 0]
...
Ok, passed 100 tests.
val it: unit = ()
ITT8060

2 · Shrinking and replay

ITT8060

Counterexamples

By default, Check.Quick starts from "small" inputs. If the generated input is not a counterexample, then the next input may be slightly "larger".

Since the search for counterexamples (test data generation) is not exhaustive, the first counterexample we find might not be the best explanation of the root cause.

Once FsCheck has found a counterexample, it tries to find a "smaller" counterexample by "shrinking" the current counterexample.

A "smaller" counterexample hopefully is a better explanation.

ITT8060

Remove from BST

remove contains a bug: at the value, it keeps only the left subtree.

let rec remove x t =
  match t with
  | Empty -> Empty
  | Node (l, y, r) ->
      if   x < y then Node (remove x l, y, r)
      elif x > y then Node (l, y, remove x r)
      else l

Removing x must keep every other value:

let removeKeepsOthers (x: int) (xs: int list) =
  let t = build xs
  inOrder (remove x t) = List.filter (fun y -> y <> x) (inOrder t)

build xs is a search tree for any list xs.

ITT8060

3 · More on properties

ITT8060

Conditional properties

The properties we have covered so far are of the form \(\forall (x : X), P(x)\). That is, we require every possible value \(x : X\) to have the property \(P\).

Sometimes we want to talk about a certain subset of values \(x : X\). For example, we may want to say that those values \(x : X\) that satisfy the property \(P\) must satisfy the property \(Q\).

\(\forall (x : X), P(x) \implies Q(x)\)

In FsCheck, for a particular x, with properties p and q, we could say

if p x then q x else true

The FsCheck syntax for such properties is precondition ==> property.

ITT8060

Conditional properties (ctd.)

The following property should check that, for non-empty lists, the head element of the reversed list is the last element of the original list.

fun (xs: int list) -> xs <> [] ==> (List.head (List.rev xs) = List.last xs)
System.ArgumentException: The input list was empty. (Parameter 'list')

What went wrong?

Arguments are evaluated before the function body. This applies to ==> as well. (In particular, also when xs = [].)

Solution: wrap the property after ==> in lazy so that it is evaluated only when its result is actually needed.

> let headRevIsLast (xs: int list) = xs <> [] ==> lazy (List.head (List.rev xs) = List.last xs);;
val headRevIsLast: xs: int list -> Property

headRevIsLast passes 100 tests. (Note its type.)

ITT8060

How many tests were useful?

let palindrome xs = List.rev xs = xs

let headLastPalindrome (xs : int list) =
  (xs <> [] && palindrome xs) ==> lazy (List.head xs = List.last xs)
> Check.Quick headLastPalindrome;;
Arguments exhausted after 8 tests.
val it: unit = ()

FsCheck only managed to (non-trivially) test the property 8 times.

Trivial test of p ==> q: if p is false, then the whole property is true no matter what q is. We care about those cases where we test q.

FsCheck tried to test the property 100 times but most of the generated lists were not palindromes and did not count. There is a limit on how many generated inputs can be rejected.

> Config.Quick.MaxRejected;;
val it: int = 1000
ITT8060

Ordered lists

let rec ordered (f : 'a -> 'a -> bool) (xs : 'a list) : bool =
  match xs with
  | [] | [ _ ]   -> true
  | x :: y :: xs -> f x y && ordered f (y :: xs)
let revFlipsOrder (xs : int list) =
  ordered (<=) xs ==> lazy (ordered (>=) (List.rev xs))

> Check.Quick revFlipsOrder;;
Arguments exhausted after 18 tests.
val it: unit = ()

Randomly generated lists are not likely to be ordered.

let sortIsOrdered (xs : int list) = ordered (<=) (List.sort xs)
Check.Quick sortIsOrdered // passes

let revFlipsOrder2 (xs : int list) = ordered (>=) (List.rev (List.sort xs))
Check.Quick revFlipsOrder2 // passes
ITT8060

Insert into ordered list

let rec insert ord x ys =
  match ys with
  | []      -> [ x ]
  | y :: ys -> if ord x y then x :: y :: ys else y :: insert ord x ys
let insertKeepsOrder (x: int) xs =
  ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))

Check.Quick insertKeepsOrder  // Arguments exhausted after 14 tests.
let insertTrivial (x: int) xs =
  ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
  |> Prop.trivial (List.length xs = 0)
> Check.Quick insertTrivial;;
Arguments exhausted after 21 tests (23% trivial).

Out of the 21 successful tests (generated list xs was ordered; precondition satisfied) 23% were cases where the length of xs was 0.

ITT8060

Classify generated data

let insertClassify (x: int) xs =
  ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
  |> Prop.classify (ordered (<=) (x :: xs)) "at-head"
  |> Prop.classify (ordered (<=) (xs @ [ x ])) "at-last"
> Check.Quick insertClassify;;
Arguments exhausted after 21 tests.
38% at-head.
33% at-last, at-head.
23% at-last.

38% out of the 21 generated lists that were ordered were such that adding the generated element at the beginning resulted in an ordered list.

23% out of the 21 generated lists that were ordered were such that adding the generated element at the end resulted in an ordered list.

ITT8060

Categorise generated data

let insertCollect (x: int) xs =
  ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
  |> Prop.collect (List.length xs)
> Check.Quick insertCollect;;
Arguments exhausted after 24 tests.
29% 1.
25% 0.
12% 4.
12% 2.
8% 5.
8% 3.
4% 8.

29% (out of 24 generated lists that were ordered) were lists with one element, 25% were empty, 12% had four elements, ...

ITT8060

Conjunctions, disjunctions, labels

We can form compound properties (conjunction p .&. q, disjunction p .|. q) and label the parts so that we know which of them failed.

let insertBothHeadTail (x: int) (xs : int list) =
  ordered (<=) xs ==>
  lazy ((Prop.label "head" <| ordered (<=) (x :: xs))
        .&.
        (Prop.label "tail" <| ordered (<=) (xs @ [x])))
> Check.Quick insertBothHeadTail;;
Falsifiable, after 2 tests (2 shrinks).
Label of failing property: head
Original:
-2
[0]
Shrunk:
1
[0]
with exception:
System.Exception: Expected true, got false.
ITT8060

Expect an exception

Sometimes we may need to say that an exception is the expected result.

let expectDivideByZero x =
  Prop.throws<System.DivideByZeroException, _> (lazy (1 / (x - x)))
> Check.Quick expectDivideByZero;;
Ok, passed 100 tests.
let expectDivideByZero' x =
  Prop.throws<System.DivideByZeroException, _> (lazy (1 / (x - 2)))
> Check.Quick expectDivideByZero';;
Falsifiable, after 3 tests (2 shrinks). 
Original:
-3
Shrunk:
0
with exception:
System.Exception: Expected exception, none was thrown
val it: unit = ()
ITT8060

4 · Generators

ITT8060

What are generators?

A generator of values of type X is a description of how to generate values of type X.

We can think of Gen<'a>, that is, a generator of values of type 'a as a function of type

int -> Randomness -> 'a

where the first argument bounds the size of the value to be generated and the second argument is a source of "randomness".

Why would we want to define our own generators and not use the default ones?

Recall properties about ordered lists. We can define a generator that is much more likely to generate an ordered list. This allows us to test such properties much more thoroughly.

FsCheck provides us with combinators for building complex generators from simpler ones.

ITT8060

Basic generators

A generator that ignores the size and randomness parameters and always generates the same value.

let genFive : Gen<int> = Gen.constant 5

As a function of type int -> Randomness -> 'a, think of it as fun s r -> 5.

A key building block is the function Gen.choose : int * int -> Gen<int> that randomly chooses a value from the given interval (with uniform distribution).

let smallPositive : Gen<int> = Gen.choose (1, 10)

Gen.choose ignores the size but uses randomness to pick an element in the given range.

ITT8060

Sample

As part of the process of developing and testing a generator we often want to sample a generator to see what kind of data it generates.

Gen.sample : int -> Gen<'a> -> 'a array

The first argument is the number of samples to produce and the second is the generator to sample.

> Gen.sample 5 smallPositive;;
val it: int array = [|9; 5; 8; 1; 10|]

NB! Gen.sample is not something that should be used to define a generator. Only use this to see what your generator would produce.

Why? Because the result is not a generator (just an array).

We want to build generators from other generators so that when we run the generator, the size and randomness arguments are propagated appropriately.

ITT8060

Gen.map

If we know how to generate values of type 'a and how to turn an 'a into a 'b, then we should be able to generate values of type 'b.

Gen.map : ('a -> 'b) -> Gen<'a> -> Gen<'b>
let chooseFromList (xs: 'a list) : Gen<'a> =
  let idxGen = Gen.choose (0, List.length xs - 1)
  Gen.map (fun i -> List.item i xs) idxGen

Compare with Gen.elements : 'a seq -> Gen<'a>.

> Gen.sample 10 (chooseFromList [50 .. 100]);;
val it: int array = [|98; 79; 100; 66; 55; 66; 77; 68; 79; 60|]

> Gen.sample 10 (Gen.elements [50 .. 100]);;
val it: int array = [|57; 73; 60; 58; 55; 72; 75; 63; 64; 64|]
ITT8060

Gen.bind

Gen.bind : ('a -> Gen<'b>) -> Gen<'a> -> Gen<'b>

Generate an 'a, then take the generator of 'b-s for that particular 'a and generate a 'b. Which generator of 'b-s is used depends on the 'a that was generated. (Sequencing.)

How to generate a pair of integers where the first one is smaller than the second?

Generate the first one and then use the generator that can only generate larger numbers.

let larger i : Gen<int * int> =
  Gen.choose (i + 1, 100)
  |> Gen.map (fun j -> (i, j))

let ordPair : Gen<int * int> =
  Gen.choose (0, 99)
  |> Gen.bind larger
> Gen.sample 5 ordPair;;
val it: (int * int) array = [|(63, 100); (85, 96); (54, 96); (64, 68); (39, 87)|]
ITT8060

Choosing a generator

To generate a value, first pick a generator. (NB! This does not pick a generated value.)

Gen.oneof : Gen<'a> seq -> Gen<'a>

Therefore, Gen.oneof [ Gen.constant true; Gen.constant false ] is a Gen<bool> that, every time it needs to generate a value, first picks a generator.

We can control the distribution: pick a generator from a list of generators but weight the probability of choosing a generator by the given factor.

Gen.frequency : (int * Gen<'a>) seq -> Gen<'a>

let boolGenOptimist = Gen.frequency [ (2, Gen.constant true); (1, Gen.constant false) ]
> Gen.sample 100 boolGenOptimist |> Array.filter id |> Array.length;;
val it: int = 73
ITT8060

Summary

  1. FsCheck tries to falsify a property.
  2. A failing input is shrunk to a small counterexample.
  3. Conditional properties (and filtering) require care. Measure.
  4. Control test data distribution with custom generators.
ITT8060

Homework, reading and next week

Homework 6: defence this week.

Reading:

  • the FsCheck documentation (we will assume that you are familiar with the API)
  • Claessen and Hughes, "QuickCheck: a lightweight tool for random testing of Haskell programs" (2000)
  • Scott Wlaschin's series on property-based testing
ITT8060