Customize FsCheck Output
I am testing FsCheck and NUnit in VisualStudio.
Currently the problem is this: I was able to generate random plots (for testing some of the plot functions), but when the test fails, FsCheck spits out the entire graph and doesn't use ToString, so it literally dumps the raw list of records and you can't see what -not there.
Also I need not only the input graph to validate, but some other data that I create when I run the property.
So how can I change the behavior of the FsCheck output so that
- actually call my ToString method on the input graph
- output of additional information
when did the test fail?
EDIT: Here is my current test setup.
module GraphProperties
open NUnit.Framework
open FsCheck
open FsCheck.NUnit
let generateRandomGraph =
gen {
let graph: Graph<int,int> = Graph<_,_>.Empty()
// fill in random nodes and transitions...
return graph
}
type MyGenerators =
static member Graph() =
{new Arbitrary<Graph<int,int>>() with
override this.Generator = generateRandomGraph
override this.Shrinker _ = Seq.empty }
[<TestFixture>]
type NUnitTest() =
[<Property(Arbitrary=[|typeof<MyGenerators>|], QuietOnSuccess = true)>]
member __.cloningDoesNotChangeTheGraph (originalGraph: Graph<int,int>) =
let newGraph = clone originalGraph
newGraph = originalGraph
source to share
FsCheck is used sprintf "%A"
to convert test parameters to test output strings, so you need to control how your types are formatted %A
. According to How do I customize custom type output using printf? , the way to do it with StructuredFormatDisplay
. The value of this attribute should be a string in the format PreText {PropertyName} PostText
where it PropertyName
should be a property ( not a function!) For your type. For example, let's say you have a tree structure with some complex information in the leaves, but for your testing, you only need to know the number of leaves, not what is in them. So, you start with a datatype like:
// Example 1
type ComplicatedRecord = { ... }
type Tree =
| Leaf of ComplicatedRecord
| Node of Tree list
with
member x.LeafCount =
match x with
| Leaf _ -> 1
| Node leaves -> leaves |> List.sumBy (fun x -> x.LeafCount)
override x.ToString() =
// For test output, we don't care about leaf data, just count
match x with
| Leaf -> "Tree with a total of 1 leaf"
| Node -> sprintf "Tree with a total of %d leaves" x.LeafCount
Now, not yet what you want. This type has no declared format %A
, so FsCheck (and whatever it uses sprintf "%A"
to format it) will eventually create the entire complex tree structure and all of its non-test sheet data. To make FsCheck output what you want to see, you will need to set up a property, not a function ( ToString
will not work for that purpose) that will output what you want to see. For example:.
// Example 2
type ComplicatedRecord = { ... }
[<StructuredFormatDisplay("{LeafCountAsString}")>]
type Tree =
| Leaf of ComplicatedRecord
| Node of Tree list
with
member x.LeafCount =
match x with
| Leaf _ -> 1
| Node leaves -> leaves |> List.sumBy (fun x -> x.LeafCount)
member x.LeafCountAsString = x.ToString()
override x.ToString() =
// For test output, we don't care about leaf data, just count
match x with
| Leaf -> "Tree with a total of 1 leaf"
| Node -> sprintf "Tree with a total of %d leaves" x.LeafCount
NOTE. I haven't tested this in F #, just typed it into the comments field with a stack overflow - so maybe I messed up the part ToString()
. (I don't remember, and can't find with a quick google if the overrides should be after or before the keyword with
). But I know the attribute StructuredFormatDisplay
is what you want because I myself have used this to get custom output from FsCheck.
By the way, you could also set an attribute StructuredFormatDisplay
on a complex post type in my example. For example, if you have a test where you care about the tree structure but not the content of the leaves, you should write it like this:
// Example 3
[<StructuredFormatDisplay("LeafRecord")>] // Note no {} and no property
type ComplicatedRecord = { ... }
type Tree =
| Leaf of ComplicatedRecord
| Node of Tree list
with
member x.LeafCount =
match x with
| Leaf _ -> 1
| Node leaves -> leaves |> List.sumBy (fun x -> x.LeafCount)
override x.ToString() =
// For test output, we don't care about leaf data, just count
match x with
| Leaf -> "Tree with a total of 1 leaf"
| Node -> sprintf "Tree with a total of %d leaves" x.LeafCount
Now all your instances ComplicatedRecord
, regardless of their content, will appear as text LeafRecord
in your output, and you can better focus on the tree structure instead, and there was no need to set an attribute StructuredFormatDisplay
on the type Tree
.
This is not exactly an ideal solution, as you may need to tweak the attribute from time to time StructuredFormatDisplay
, as needed, with the various tests you are using. (For some tests, you might focus on one part of the sheet data, for others you want to ignore the sheet data entirely, etc.) And you probably want to grab an attribute before moving on to production. But until FsCheck acquires "Give me a function to format failed test data with a" config parameter ", this is the best way to get your test data in the format you want.
source to share
You can also use labels to display whatever you want when the test fails: https://fscheck.github.io/FsCheck/Properties.html#And-Or-and-Labels
source to share