[Introduction]
About six months ago shoenig/test was
created as a modern, generics based alternative to the popular stretchr/testify
testing assertions library for Go. As a developer of Hashicorp’s Nomad project,
I am lucky to work with a team willing to endure my experiments. This blog-post
is a look back on how well Go generics have (and haven’t) served us in using the
library on a real, very large Go project.
There were three guiding principles in designing the library.
-
Use of generics to eliminate the need for type-casting parameters.
-
Use of google/go-cmp for differentials.
-
Should be easy to accidentally use correctly.
That last point warrants some explanation. One of my personal gripes with testify
is that folks often get mixed up by the variadic arguments at the end of each
assertion function. It’s too easy to make incorrect assumptions about the parameters of
a function when every parameter type is interface{}, and there is not
a definite number of them. Another gripe is with method vs. function duality in
testify. For example, we have plenty of legacy code that does something like,
require := require.New(t)
require.Equal(...)
This pattern shadows the package name, which any reasonable code linter will complain
about. We are slowly removing this pattern from our codebase in favor of the Equal(t, ...)
equivalent, and the new test library supports only that form.
[Good]
One of the primary motivations for test was to eliminate the need to type-cast
parameters that were ambiguous to testify. For example, we can now simply do
must.Eq(t, 42, idx)
in places where we previous needed to do something like,
require.Equal(t, uint64(42), idx)
That may seem like a small improvement, but it’s those repetitive paper-cuts
that make all the difference in the world on a large project. This is of course
made possible by the use of generic parameters, where the expectation and actual
value are constrained by the same type [T any].
Another motivation was to make use of the go-cmp library for creating differential
output on failed assertions. This library does a very nice job of formatting human
readable diffs of arbitrary Go objects. Just a quick example,
test_test.go:1004: expected maps of same values via .Equal method
↪ Assertion | differential ↷
map[int]*test.Person{
0: &{ID: 100, Name: "Alice"},
- 1: &{ID: 101, Name: "Bob"},
+ 1: &{ID: 200, Name: "Bob"},
}
The diff-style output makes understanding the assertion failure quick and easy.
Another key feature of test is that we can make use of implementations of common
interfaces for assertions. For example, many structs satisfy interfaces such as
type Equal interface {
Equal(other T) bool
}
which can be used to assert the equality of structs in a custom way, including non-exported struct fields. Other common interfaces include,
type Contains interface {
Contains(item T) bool
}
for asserting something contains an element, or
type Size interface {
Size() int
}
for asserting something contains a certain number of elements, etc.
One interesting novelty of test is how it handles adding additional context to
output in failing tests. As described above the variadic interface{} arguments
of testify are a lingering source of buggy tests in our legacy code, so we
wanted to fix that.
In doing so, the assertion functions in test end in a variadic PostScript
parameter. Each PostScript prints a block of custom output on failure. Anything
that implements the PostScript interface can be used.
type PostScript interface {
Label() string
Content() string
}
We then include Sprint and Sprintf implementations of the PostScript interface
for the most common use cases of adding print statements. So you might see assertions
in complex scenarios that look something like
must.Eq(t, 42, value, must.Sprintf("a: %s, b: %d, c: %x", a, b, c))
or if you prefer, the key-value Sprint variant
must.Eq(t, 42, value, must.Sprint("a", a, "b", b, "c", c))
At one point, my co-worker discovered a bug in our code that a testify test was
incorrectly reporting as working correctly. It boiled down to a pointer comparison
vs. dereference value comparison, e.g.
type F struct {
B *bool
}
func (f *F) Equal(o *F) bool {
return f.B == o.B // bug!
}
Full example in Go Playground.
In this snippet, the equality check is comparing pointers which was not the intended
behavior. But if you use testify::Equal() without a custom comparator, it will magically
do the dereference for you, which is not indicative of the actual implementation. test
on the other hand does not do the magic dereference, and correctly reported the objects as
not equal.
[Bad]
There are some rough spots in test. Although the use of common interfaces described
above is convenient for the types that implement them, not all types implement them,
even though perhaps they would be good candidates. There is no official standard for
these interfaces. Even if a type implements the idea of a particular interface, the
spelling needs to be consistent, too! This was a problem in Nomad, where we were
using Equals() instead of Equal() until a major refactoring. Some other languages
come with certain built-in interfaces for describing things like Collections, Maps,
Sets to be implemented by built-in or custom types. I think Go could provide something
similar, expanding on the x/constraints package.
An earlier version of test tried to emphasize the use of the io/fs package for
filesystem testing. In practice, the multitude of interfaces around filesystem
operations provided by io/fs proved to be ineffective. In practice, the fact
that all the real filesystem operations are provided by package-level functions in
the os package means io/fs is not usable. Package functions cannot satisfy
an interface.
Using these io/fs interfaces would look something like this,
func DirExists(fs fs.FS, dir string)
which seems fine, but then the caller needs to do something like
DirExists(os.DirFS("usr/local"), "bin")
to check if "/usr/local/bin" exists. It’s not ideal. There needs to be some
kind of official os.Filesystem implementation of all those interfaces in io/fs
before they are actually useful. For now, test has kept the io/fs functions,
but also provides os variants, which is what folks actually use.
[Ugly]
Coming from a Java background, the lack of auto-boxing of built-in types becomes
painfully apparent when working with Go generics. In particular, to avoid the use
of reflection the test library has functions that are specific to map, slice,
and string values. This has been a painful pill to swallow and is something I
hope can be addressed in future versions of Go. For example consider a simple Empty
assertion. In fact the operation would be the same for each of those built-in types.
Just check if len(x) == 0. But there is no way to declare a type constraint that
describes a type that works with the built-in len operation. So you end up with
two options: either accept an interface{} and use a type-switch, or implement
variations of Empty for each built-in type. You might be thinking, “the type switch
doesn’t sound so bad, what’s wrong with that?”. The problem is, what about the
non-built-in types? You probably want to support “container”-like types that implement
an Empty interface, defering to that type’s Empty method. Since we were forced
to accept interface{} because of the len problem, now we need reflection to check
if the type implements Empty, add that to our type switch, and then cry a little
because we violated one of our key design goals - avoid reflection. So the test
library chooses the alternative pattern - creating specific functions for each of the
built-in types. It’s not elegant, but it works.
[Conclusion]
As described there are trade-offs between shoenig/test and stretchr/testify.
Which one is better is I think a matter of personal opinion - at the end of the
day they are both functioning assertions libraries, and they both have warts that
will make you stop and think, “hmm how do I do X with this?”. If Go generics continue
to improve - particularly around the problems with using built-ins in a generic context
as described above, I think there could be a bright future for test - and/or maybe a
generics based v2 of testify.
gl;hf!