ITT8060 Advanced Programming · Autumn 2026
Tallinn University of Technology
Full prose version with runnable examples: notes.html
ITT8060Designing, implementing and reasoning about large systems requires abstraction.
We treat a component as a black box (hiding the implementation) and are only interested in its behaviour (external interface): what does it require, what does it guarantee.
Assumption: components adhere to their specification.
Property-based testing: we specify (propose, hypothesise) a property of a component, the tool tries to falsify it (using randomness).
A "side effect" of testing in this style is that the specification of a component is made explicit (a set of properties).
ITT8060
ITT8060In property-based testing, a property is a falsifiable statement about the behaviour of a component.
We express this statement in the host language (via an embedded domain-specific language).
Non-example: every accepted job eventually completes. (Cannot falsify with a finite test.)
Non-example: the system is reliable when operated correctly. (Failure criterion undefined/fluid.)
Say that we are testing a sorting function.
How can we conclude that it is implemented incorrectly? What observation would falsify the claim that it is implemented correctly?
Property: for all xs : 'a list, List.sort xs is a permutation of
xs. Falsification: a list ys (a counterexample) such that List.sort
ys is not a permutation of ys.
Property: for all xs : 'a list, List.sort xs is
non-decreasing. Falsification: a list ys such that List.sort ys is
not non-decreasing.
ITT8060Given a function f : 'a -> 'b, a simple property about f is a
predicate on 'a (a function of type 'a -> bool).
For example, if we take List.rev : 'a list -> 'a list as our f,
what property does the following predicate (of type 'a list -> bool)
express?
let p1 xs = List.length (List.rev xs) = List.length xs
The body of the function checks that the length of the reversal of
xs is the same as the length of xs.
> p1 [1; 2; 3];;
val it: bool = true
Reversing a list preserves the length.
ITT8060List.rev is its own inverse.
#r "nuget: FsCheck, 3.6.0"
open FsCheck
open FsCheck.FSharp
let revRevIsId (xs: int list) = List.rev (List.rev xs) = xs
FsCheck tries to falsify this property (find a counterexample).
Generate ys : int list. If revRevIsId ys = false, then we have
found a counterexample and the process stops. Otherwise, we
continue. By default, up to 100 iterations.
> Check.Quick revRevIsId;;
Ok, passed 100 tests.
val it: unit = ()
ITT8060Check.Verbose also prints the generated input. We truncate the output.
> Check.Verbose revRevIsId;;
0:
[0; 2]
1:
[1; 0; 1]
2:
[-3]
3:
[-1]
4:
[-3; 0]
5:
[0; -1; 0; -2; 0]
...
Ok, passed 100 tests.
val it: unit = ()
ITT8060Is the reversal of xs @ ys related to the reversal of xs and ys?
let revAppendWrong (xs: int list) (ys: int list) =
List.rev (xs @ ys) = List.rev xs @ List.rev ys
> Check.Quick revAppendWrong;;
Falsifiable, after 2 tests (5 shrinks).
Original:
[0; 0; 2]
[3]
Shrunk:
[0]
[1]
with exception:
System.Exception: Expected true, got false.
val it: unit = ()
let revAppend (xs: int list) (ys: int list) =
List.rev (xs @ ys) = List.rev ys @ List.rev xs
ITT8060NonNegativeInt is a single-constructor DU provided by FsCheck.
let rec fact n =
match n with
| 0 -> 1
| 7 -> fact 6
| n -> n * fact (n - 1)
let factStep (NonNegativeInt n) = fact (n + 1) = (n + 1) * fact n
> Check.Quick factStep;;
Falsifiable, after 13 tests (0 shrinks).
Original:
NonNegativeInt 6
with exception:
System.Exception: Expected true, got false.
val it: unit = ()
Note that while fact n computes an incorrect result when n >= 7,
factStep only fails on NonNegativeInt 6.
ITT8060We saw that revRevIsId passed 100 tests (FsCheck generated 100
values ys : int list and for all of those List.rev (List.rev ys) =
ys evaluates to true).
Lists are polymorphic in the type of values. Reversing a list of, say, strings should behave the same as reversing a list of integers.
Can we conclude that List.rev (List.rev xs) = xs for all lists xs?
let revRevIsIdFloat (xs: float list) = List.rev (List.rev xs) = xs
> Check.Quick revRevIsIdFloat;;
Falsifiable, after 8 tests (8 shrinks).
Original:
[1.028093409; -0.0; -0.0; -3.731234951; -0.5817794469; nan; 0.0; -0.0; infinity]
Shrunk:
[nan]
with exception:
System.Exception: Expected true, got false.
val it: unit = ()
> nan = nan;;
val it: bool = false
ITT8060
ITT8060By default, Check.Quick starts from "small" inputs. If the
generated input is not a counterexample, then the next input may be
slightly "larger".
Since the search for counterexamples (test data generation) is not exhaustive, the first counterexample we find might not be the best explanation of the root cause.
Once FsCheck has found a counterexample, it tries to find a "smaller" counterexample by "shrinking" the current counterexample.
A "smaller" counterexample hopefully is a better explanation.
ITT8060To shrink a value x : 'a just means to propose other candidate
values of type 'a that we consider to be "smaller" than x.
This mechanism is represented as a function of type 'a -> 'a seq.
If the first proposed value is also a counterexample, then recursively shrink that. Otherwise consider the next proposed value.
> let intArb = ArbMap.arbitrary<int> ArbMap.defaults;;
val intArb: Arbitrary<int>
What are some values "smaller" than 10 that we could try?
> intArb.Shrinker 10 |> Seq.toList;;
val it: int list = [0; 5; 8; 9]
Suppose we see that 0 is not a counterexample but 5 is. Then we
try to shrink 5.
> intArb.Shrinker 5 |> Seq.toList;;
val it: int list = [0; 3; 4]
ITT8060Shrink the data structure itself or shrink the values in the data structure.
> let intListArb : Arbitrary<int list> = ArbMap.arbitrary ArbMap.defaults;;
val intListArb: Arbitrary<int list>
> intListArb.Shrinker [3; 2; 1] |> Seq.toList;;
val it: int list list =
[[2; 1]; [3; 1]; [3; 2];
[0; 2; 1]; [2; 2; 1]; [3; 0; 1]; [3; 1; 1]; [3; 2; 0]]
Note how the first three elements are subsequences of the original list.
The rest are lists that are otherwise exactly the same as the original except that one element has been replaced with a "smaller" element.
> intArb.Shrinker 3;;
val it: int seq = seq [0; 2]
> intArb.Shrinker 2;;
val it: int seq = seq [0; 1]
ITT8060We can direct FsCheck to show the successful shrinking steps. (Trimmed and aligned output.)
> Check.One (Config.Quick.WithEveryShrink (sprintf "%A\n"), revRevIsIdFloat);;
[[0.4163758021; -0.0; -0.0; 0.0; -0.8829789142; nan; -0.0; 0.0; -0.0]]
[[ -0.0; -0.0; 0.0; -0.8829789142; nan; -0.0; 0.0; -0.0]]
[[ -0.0; 0.0; -0.8829789142; nan; -0.0; 0.0; -0.0]]
[[ 0.0; -0.8829789142; nan; -0.0; 0.0; -0.0]]
[[ -0.8829789142; nan; -0.0; 0.0; -0.0]]
[[ nan; -0.0; 0.0; -0.0]]
[[ nan; 0.0; -0.0]]
[[ nan; -0.0]]
[[ nan ]]
Falsifiable, after 17 tests (9 shrinks).
Original:
[-infinity; 0.4163758021; -0.0; -0.0; 0.0; -0.8829789142; nan; -0.0; 0.0; -0.0]
Shrunk:
[nan]
with exception:
System.Exception: Expected true, got false.
val it: unit = ()
ITT8060remove contains a bug: at the value, it keeps only the left subtree.
let rec remove x t =
match t with
| Empty -> Empty
| Node (l, y, r) ->
if x < y then Node (remove x l, y, r)
elif x > y then Node (l, y, remove x r)
else l
Removing x must keep every other value:
let removeKeepsOthers (x: int) (xs: int list) =
let t = build xs
inOrder (remove x t) = List.filter (fun y -> y <> x) (inOrder t)
build xs is a search tree for any list xs.
ITT8060Check.Quick removeKeepsOthers:
Original:
1
[0; 0; -1; 1; 0; 1; 0; 29; 0]
Shrunk:
0
[0; 1]
Removing 0 from the shrunk tree (right) also drops 1 (the result
is the left subtree, which is Empty).
Note how the shrunk counterexample directly points us to the issue.
ITT8060> Check.Quick revRevIsIdFloat;;
Falsifiable, after 9 tests (8 shrinks) (15443889507916823751,6318131138842723703).
Last step was invoked with size of 10 and seed of (14959874787232813416,10819090044508898283).
Replay directly at failing step with (14959874787232813416,10819090044508898283,10).
Original:
[-1.296238501; 0.9008574721; -0.08371394787; -0.0; 0.0; 1.821240888;
0.8605732909; nan; -0.0]
Shrunk:
[nan]
with exception:
System.Exception: Expected true, got false.
val it: unit = ()
Falsifiable, after 9 tests: the ninth input that was generated
refuted the property; (15443889507916823751,6318131138842723703) is
the seed of the whole run. 8 shrinks: found 8 smaller inputs, each
still failing.
Last step: the seed and size of the failing test. Replay directly: the seed and size that regenerate exactly that input.
Original: the first input that failed. Shrunk: the smallest failing input found.
ITT8060The report printed the seed and size of the failing step. A configuration with that seed and size regenerates the same input.
let failingSeed = Rnd (14959874787232813416UL,10819090044508898283UL)
let replayFailure =
Config.Quick.WithReplay (Some { Rnd = failingSeed; Size = Some 10 })
> Check.One (replayFailure, revRevIsIdFloat);;
Falsifiable, after 1 test (8 shrinks) (14959874787232813416,10819090044508898283).
Last step was invoked with size of 10 and seed of (14959874787232813416,10819090044508898283):
Original:
[-1.296238501; 0.9008574721; -0.08371394787; -0.0; 0.0; 1.821240888;
0.8605732909; nan; -0.0]
Shrunk:
[nan]
with exception:
System.Exception: Expected true, got false.
val it: unit = ()
We can reproduce the exact same search that found the counterexample.
ITT8060
ITT8060The properties we have covered so far are of the form \(\forall (x : X), P(x)\). That is, we require every possible value \(x : X\) to have the property \(P\).
Sometimes we want to talk about a certain subset of values \(x : X\). For example, we may want to say that those values \(x : X\) that satisfy the property \(P\) must satisfy the property \(Q\).
\(\forall (x : X), P(x) \implies Q(x)\)
In FsCheck, for a particular x, with properties p and q, we
could say
if p x then q x else true
The FsCheck syntax for such properties is precondition ==> property.
ITT8060The following property should check that, for non-empty lists, the head element of the reversed list is the last element of the original list.
fun (xs: int list) -> xs <> [] ==> (List.head (List.rev xs) = List.last xs)
System.ArgumentException: The input list was empty. (Parameter 'list')
What went wrong?
Arguments are evaluated before the function body. This applies to
==> as well. (In particular, also when xs = [].)
Solution: wrap the property after ==> in lazy so that it is
evaluated only when its result is actually needed.
> let headRevIsLast (xs: int list) = xs <> [] ==> lazy (List.head (List.rev xs) = List.last xs);;
val headRevIsLast: xs: int list -> Property
headRevIsLast passes 100 tests. (Note its type.)
ITT8060let palindrome xs = List.rev xs = xs
let headLastPalindrome (xs : int list) =
(xs <> [] && palindrome xs) ==> lazy (List.head xs = List.last xs)
> Check.Quick headLastPalindrome;;
Arguments exhausted after 8 tests.
val it: unit = ()
FsCheck only managed to (non-trivially) test the property 8 times.
Trivial test of p ==> q: if p is false, then the whole property is
true no matter what q is. We care about those cases where we test
q.
FsCheck tried to test the property 100 times but most of the generated lists were not palindromes and did not count. There is a limit on how many generated inputs can be rejected.
> Config.Quick.MaxRejected;;
val it: int = 1000
ITT8060let rec ordered (f : 'a -> 'a -> bool) (xs : 'a list) : bool =
match xs with
| [] | [ _ ] -> true
| x :: y :: xs -> f x y && ordered f (y :: xs)
let revFlipsOrder (xs : int list) =
ordered (<=) xs ==> lazy (ordered (>=) (List.rev xs))
> Check.Quick revFlipsOrder;;
Arguments exhausted after 18 tests.
val it: unit = ()
Randomly generated lists are not likely to be ordered.
let sortIsOrdered (xs : int list) = ordered (<=) (List.sort xs)
Check.Quick sortIsOrdered // passes
let revFlipsOrder2 (xs : int list) = ordered (>=) (List.rev (List.sort xs))
Check.Quick revFlipsOrder2 // passes
ITT8060let rec insert ord x ys =
match ys with
| [] -> [ x ]
| y :: ys -> if ord x y then x :: y :: ys else y :: insert ord x ys
let insertKeepsOrder (x: int) xs =
ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
Check.Quick insertKeepsOrder // Arguments exhausted after 14 tests.
let insertTrivial (x: int) xs =
ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
|> Prop.trivial (List.length xs = 0)
> Check.Quick insertTrivial;;
Arguments exhausted after 21 tests (23% trivial).
Out of the 21 successful tests (generated list xs was ordered;
precondition satisfied) 23% were cases where the length of xs was 0.
ITT8060let insertClassify (x: int) xs =
ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
|> Prop.classify (ordered (<=) (x :: xs)) "at-head"
|> Prop.classify (ordered (<=) (xs @ [ x ])) "at-last"
> Check.Quick insertClassify;;
Arguments exhausted after 21 tests.
38% at-head.
33% at-last, at-head.
23% at-last.
38% out of the 21 generated lists that were ordered were such that adding the generated element at the beginning resulted in an ordered list.
23% out of the 21 generated lists that were ordered were such that adding the generated element at the end resulted in an ordered list.
ITT8060let insertCollect (x: int) xs =
ordered (<=) xs ==> lazy (ordered (<=) (insert (<=) x xs))
|> Prop.collect (List.length xs)
> Check.Quick insertCollect;;
Arguments exhausted after 24 tests.
29% 1.
25% 0.
12% 4.
12% 2.
8% 5.
8% 3.
4% 8.
29% (out of 24 generated lists that were ordered) were lists with one element, 25% were empty, 12% had four elements, ...
ITT8060We can form compound properties (conjunction p .&. q, disjunction p
.|. q) and label the parts so that we know which of them failed.
let insertBothHeadTail (x: int) (xs : int list) =
ordered (<=) xs ==>
lazy ((Prop.label "head" <| ordered (<=) (x :: xs))
.&.
(Prop.label "tail" <| ordered (<=) (xs @ [x])))
> Check.Quick insertBothHeadTail;;
Falsifiable, after 2 tests (2 shrinks).
Label of failing property: head
Original:
-2
[0]
Shrunk:
1
[0]
with exception:
System.Exception: Expected true, got false.
ITT8060Sometimes we may need to say that an exception is the expected result.
let expectDivideByZero x =
Prop.throws<System.DivideByZeroException, _> (lazy (1 / (x - x)))
> Check.Quick expectDivideByZero;;
Ok, passed 100 tests.
let expectDivideByZero' x =
Prop.throws<System.DivideByZeroException, _> (lazy (1 / (x - 2)))
> Check.Quick expectDivideByZero';;
Falsifiable, after 3 tests (2 shrinks).
Original:
-3
Shrunk:
0
with exception:
System.Exception: Expected exception, none was thrown
val it: unit = ()
ITT8060
ITT8060A generator of values of type X is a description of how to generate
values of type X.
We can think of Gen<'a>, that is, a generator of values of type 'a
as a function of type
int -> Randomness -> 'a
where the first argument bounds the size of the value to be generated and the second argument is a source of "randomness".
Why would we want to define our own generators and not use the default ones?
Recall properties about ordered lists. We can define a generator that is much more likely to generate an ordered list. This allows us to test such properties much more thoroughly.
FsCheck provides us with combinators for building complex generators from simpler ones.
ITT8060A generator that ignores the size and randomness parameters and always generates the same value.
let genFive : Gen<int> = Gen.constant 5
As a function of type int -> Randomness -> 'a, think of it as fun s
r -> 5.
A key building block is the function Gen.choose : int * int ->
Gen<int> that randomly chooses a value from the given interval (with
uniform distribution).
let smallPositive : Gen<int> = Gen.choose (1, 10)
Gen.choose ignores the size but uses randomness to pick an element
in the given range.
ITT8060As part of the process of developing and testing a generator we often want to sample a generator to see what kind of data it generates.
Gen.sample : int -> Gen<'a> -> 'a array
The first argument is the number of samples to produce and the second is the generator to sample.
> Gen.sample 5 smallPositive;;
val it: int array = [|9; 5; 8; 1; 10|]
NB! Gen.sample is not something that should be used to define a
generator. Only use this to see what your generator would produce.
Why? Because the result is not a generator (just an array).
We want to build generators from other generators so that when we run the generator, the size and randomness arguments are propagated appropriately.
ITT8060If we know how to generate values of type 'a and how to turn an 'a
into a 'b, then we should be able to generate values of type 'b.
Gen.map : ('a -> 'b) -> Gen<'a> -> Gen<'b>
let chooseFromList (xs: 'a list) : Gen<'a> =
let idxGen = Gen.choose (0, List.length xs - 1)
Gen.map (fun i -> List.item i xs) idxGen
Compare with Gen.elements : 'a seq -> Gen<'a>.
> Gen.sample 10 (chooseFromList [50 .. 100]);;
val it: int array = [|98; 79; 100; 66; 55; 66; 77; 68; 79; 60|]
> Gen.sample 10 (Gen.elements [50 .. 100]);;
val it: int array = [|57; 73; 60; 58; 55; 72; 75; 63; 64; 64|]
ITT8060Gen.bind : ('a -> Gen<'b>) -> Gen<'a> -> Gen<'b>
Generate an 'a, then take the generator of 'b-s for that
particular 'a and generate a 'b. Which generator of 'b-s is used
depends on the 'a that was generated. (Sequencing.)
How to generate a pair of integers where the first one is smaller than the second?
Generate the first one and then use the generator that can only generate larger numbers.
let larger i : Gen<int * int> =
Gen.choose (i + 1, 100)
|> Gen.map (fun j -> (i, j))
let ordPair : Gen<int * int> =
Gen.choose (0, 99)
|> Gen.bind larger
> Gen.sample 5 ordPair;;
val it: (int * int) array = [|(63, 100); (85, 96); (54, 96); (64, 68); (39, 87)|]
ITT8060To generate a value, first pick a generator. (NB! This does not pick a generated value.)
Gen.oneof : Gen<'a> seq -> Gen<'a>
Therefore, Gen.oneof [ Gen.constant true; Gen.constant false ] is a
Gen<bool> that, every time it needs to generate a value, first picks a
generator.
We can control the distribution: pick a generator from a list of generators but weight the probability of choosing a generator by the given factor.
Gen.frequency : (int * Gen<'a>) seq -> Gen<'a>
let boolGenOptimist = Gen.frequency [ (2, Gen.constant true); (1, Gen.constant false) ]
> Gen.sample 100 boolGenOptimist |> Array.filter id |> Array.length;;
val it: int = 73
ITT8060Gen.sized : (int -> Gen<'a>) -> Gen<'a>
The argument int -> Gen<'a> specifies a generator for every possible
size s : int.
Example: generate natural numbers that are not larger than the given size.
let sizedNats s = Gen.choose (0, s)
We have a generator for every s : int.
Generator of natural numbers that respect the size parameter.
let naturalGen = Gen.sized sizedNats
> Gen.sampleWithSize 10 10 naturalGen;;
val it: int array = [|9; 2; 0; 0; 2; 1; 3; 4; 0; 10|]
> Gen.sampleWithSize 20 10 naturalGen;;
val it: int array = [|15; 6; 9; 3; 4; 6; 5; 19; 3; 20|]
> Gen.sampleWithSize 100 10 naturalGen;;
val it: int array = [|2; 78; 21; 99; 20; 1; 51; 75; 90; 85|]
ITT8060An Arbitrary<'a> is just the combination of a generator of 'a-s
and a shrinker of 'a-s.
Arb.fromGenShrink : Gen<'a> * ('a -> 'a seq) -> Arbitrary<'a>
This is what FsCheck needs to try to falsify a property 'a ->
bool. Generate an arbitrary value of type 'a. If it is a
counterexample, try to shrink it.
It is preferable that custom generators also respect the size parameter.
let intArb = ArbMap.arbitrary<int> ArbMap.defaults
intArb.Generator // Gen<int>
intArb.Shrinker // int -> int seq
Arb.fromGen // new Arbitrary from a given generator, empty shrinker
Arb.fromGenShrink // from given generator and shrinker
Arb.filter // filter generator and shrinker
Arb.mapFilter // map and filter the generator, filter the shrinker
ITT8060Prop.forAll : Arbitrary<'a> -> ('a -> 'b) -> Property
Thus Prop.forAll arb p requires that every value a from arb
satisfies the property p.
By supplying a custom arbitrary we can control the distribution of test data.
let primeIsOdd =
Prop.forAll (Arb.fromGen (chooseFromList [2; 3; 5; 7; 11; 13; 17; 19]))
(fun p -> p % 2 = 1)
> Check.Quick primeIsOdd;;
Falsifiable, after 6 tests (0 shrinks).
Original:
2
with exception:
System.Exception: Expected true, got false.
ITT8060One way to generate ordered lists (and not rely on filtering).
let intListArb : Arbitrary<int list> = ArbMap.arbitrary ArbMap.defaults
let orderedLists : Arbitrary<int list> = Arb.mapFilter List.sort (ordered (<=)) intListArb
Now we can say that for all x and for all ordered lists xs,
insert x xs is an ordered list. Note that we do not need conditional
properties to exclude some of the generated data.
let insertKeepsOrder' x =
Prop.forAll orderedLists (fun xs -> ordered (<=) (insert (<=) x xs))
> Check.Quick insertKeepsOrder';;
Ok, passed 100 tests.
ITT8060type Tree = Leaf of int
| Branch of Tree * Tree
The FsCheck documentation provides a "naive" generator. It does not work. Try it to see why.
let rec unsafeTree () =
printfn "UNSAFE!"
Gen.oneof
[ Gen.map Leaf intArb.Generator
Gen.map2
(fun x y ->
printfn "BRANCH"
Branch(x, y))
(unsafeTree ())
(unsafeTree ()) ]
unsafeTree () // Careful!
ITT8060We define a function int -> Gen<Tree>. In other words, a generator
of trees for every s : int.
let rec treeWithSize (s: int) : Gen<Tree> =
match s with
| 0 -> Gen.map Leaf intArb.Generator
| _ ->
let subTreeGen = treeWithSize (s / 2)
Gen.frequency
[ (1, Gen.map Leaf intArb.Generator)
(2, Gen.map2
(fun left right -> Branch (left, right))
subTreeGen
subTreeGen) ]
let treeGen = Gen.sized treeWithSize
> Gen.sampleWithSize 10 1 treeGen;;
val it: Tree array =
[|Branch
(Branch (Leaf 4, Branch (Leaf -8, Branch (Leaf -8, Leaf 5))), Leaf 8)|]
ITT8060gen { ... } gives us a more imperative notation for generators.
let! x = g binds the value generated by g to x (in the following
computation expression).
return v is a constant generator of values v.
let ordPairGen =
gen {
let! j = Gen.choose (1, 100)
let! i = Gen.choose (0, j - 1)
return (i, j)
}
> Gen.sample 10 ordPairGen;;
val it: (int * int) array =
[|(11, 15); (8, 30); (38, 70); (71, 91); (30, 40); (12, 13); (26, 83);
(17, 21); (3, 12); (38, 97)|]
How this notation works: Lecture 11.
ITT8060
ITT8060Homework 6: defence this week.
Reading:
ITT8060