How much should one control the `Depth` parameter in smallcheck?
haskell, smallcheck
Solution
While the "exhaustiveness" of smallcheck (for the small cases anyway) is an intriguing property, I would rather suggest quickcheck in this context. While JSON has a lightweight structure, from the point of view of actual bits of data, it is pretty heavy.
The testing time also quite crucially depend on how you define "size" (depth) for smallcheck in the Series instances for your types. If your types have a lot of branching (many constructors), the number of tests will grow quickly. It is exponential in "depth", while the base of the exponential is the amount of branching in your Series instances relevant to a particular testcase.
In other words, if you have 2 constructors on average, you are looking at something like 32 testcases to run, but if you have 20, it's more like 3200000.
However, your coverage is affected as well -- if you reduce the branching (making depth increase faster) in your testcases, you will get less coverage with given depth. With quickcheck, you trade losing some of the "small" testcases in favour of sampling a number of bigger examples which you can't reach with smallcheck.
Problem
I'm doing my first bit of real work with `smallcheck`, and I'm a little confused about how to use the `Depth` parameter. Before I go into that, let me state what I'm using `smallcheck` for. At work, we're building a simple web service in front of our own in-house database. The web service does some queries, and responds with the query results serialized to JSON. What I'm working on at the moment is the assurance that: given an object that represents the results of a query, this object produces the expected JSON. For example: ``` data Action = Action { actionType :: !ActionType , actionDescription :: !Text , actionPerformedAt :: !UTCTime , actionAgentName :: !Text } ``` Must produce JSON such as: ``` { "type": "Booking", "description": "Whatever", "performedAt": "2012-01-04", "agent": "Tom" } ``` This looked like an ideal task for `smallcheck`, and I have this formulated as the following: ``` testAction :: Tasty.TestTree testAction = Tasty.testGroup "Action" [ SmallCheck.testProperty "type" $ SmallCheck.over actions $ match $ Aeson.key "type" --> Aeson.toJSON . actionType , SmallCheck.testProperty "dateActioned" $ SmallCheck.over actions $ match $ Aeson.key "dateActioned" --> expectedUTCTimeEncoding . actionPerformedAt -- and so on ] -- (-->) :: Eq a => lens-aeson traversal a -> (b -> a) -> b -> Bool -- actions :: Monad m => SmallCheck.Series m Action ``` The default `smallcheck` depth in the `tasty` framework is 5, which results in test runs that I've yet to see finish. `smallcheck` has the `changeDepth` and `changeDepth1` functions, so I could use these as `changeDepth (const 3)` to ensure that my tests always run in a sensible amount of time. However, by doing this I can't help but feel I'm missing the point somewhere? For example, it is now not possible to run a longer test, perhaps overnight, by just changing the command line options to run the test. On the otherhand, if I used `changeDepth (- 2)`, it still feels as though I am making an assumption of how the tests are ran! Perhaps it's best to assume that a global test depth of 5 runs in n seconds, and it's up to each property to adjust the depth as it sees fit? Would love to hear some feedback on this more practical side of `smallcheck`.