Methods of property-based testing like QuickCheck?

Tools like QuickCheck (Haskell, Erlang) and Hypothesis (Python) are wonderful for generating thousands of tests against your code. They use a method called property-based testing, demoed here to great effect.

There are no libraries for Zig as of yet, but I know some solutions are cross language, and a few are known to target C. Has anyone used one of these libraries bound to Zig code, or have any ideas how to best integrate these sorts of tests at this time?

Funnily enough just this weekend I was writing some hypothesis tests for some maths functions, comparing Zig output to GCC output (and found a bug in exp()!): https://github.com/LewisGaul/zig-f128math/blob/main/hypothesis/test.py

I’m not sure this really answers your question, but thought I’d share as an example of just using Python to orchestrate the tests, interacting with Zig code via subprocess/stdout.

2 Likes

Awesome! This is exactly what I was looking for!

It’s one of those things where I had an idea, but no clue how to glue the things together. Studying this code will be a huge help. :slight_smile:

PS – You seem to have a better grasp at Hypothesis than I do, but in case you missed it: you can cement in test cases with @hypothesis.example(), which is very nice for repeating failures.

Oh cool, what a nice coincidence that I’d done that so recently then! I think my use of hypothesis is fairly basic in this case (constrained to integers in a fixed range), but just the randomness involved ended up finding a real bug (as well as other minor differences between Zig and GCC).

you can cement in test cases with @hypothesis.example() , which is very nice for repeating failures.

Thanks, I was aware of this, but at least until the failures are fixed I actually want the known ones not to be hit repeatedly! I have a hypothesis ‘strategy’ that doesn’t use the cache of previous failures for this reason.

If you have any questions about the code feel free to get in touch (e.g. create a discussion on the GitHub repo).

1 Like