/ website / blog

retro on a generic Go test assertions library

October 16, 2022


About six months ago shoenig/test was created as a modern, generics based alternative to the popular stretchr/testify testing assertions library for Go. As a developer of Hashicorp’s Nomad project, I am lucky to work with a team willing to endure my experiments. This blog-post is a look back on how well Go generics have (and haven’t) served us in using the library on a real, very large Go project.

There were three guiding principles in designing the library.

  1. Use of generics to eliminate the need for type-casting parameters.

  2. Use of google/go-cmp for differentials.

  3. Should be easy to accidentally use correctly.

That last point warrants some explanation. One of my personal gripes with testify is that folks often get mixed up by the variadic arguments at the end of each assertion function. It’s too easy to make incorrect assumptions about the parameters of a function when every parameter type is interface{}, and there is not a definite number of them. Another gripe is with method vs. function duality in testify. For example, we have plenty of legacy code that does something like,

require := require.New(t)

This pattern shadows the package name, which any reasonable code linter will complain about. We are slowly removing this pattern from our codebase in favor of the Equal(t, ...) equivalent, and the new test library supports only that form.


One of the primary motivations for test was to eliminate the need to type-cast parameters that were ambiguous to testify. For example, we can now simply do

must.Eq(t, 42, idx)

in places where we previous needed to do something like,

require.Equal(t, uint64(42), idx)

That may seem like a small improvement, but it’s those repetitive paper-cuts that make all the difference in the world on a large project. This is of course made possible by the use of generic parameters, where the expectation and actual value are constrained by the same type [T any].

Another motivation was to make use of the go-cmp library for creating differential output on failed assertions. This library does a very nice job of formatting human readable diffs of arbitrary Go objects. Just a quick example,

test_test.go:1004: expected maps of same values via .Equal method
↪ Assertion | differential ↷
  	0: &{ID: 100, Name: "Alice"},
- 	1: &{ID: 101, Name: "Bob"},
+ 	1: &{ID: 200, Name: "Bob"},

The diff-style output makes understanding the assertion failure quick and easy.

Another key feature of test is that we can make use of implementations of common interfaces for assertions. For example, many structs satisfy interfaces such as

type Equal interface {
    Equal(other T) bool

which can be used to assert the equality of structs in a custom way, including non-exported struct fields. Other common interfaces include,

type Contains interface {
    Contains(item T) bool

for asserting something contains an element, or

type Size interface {
    Size() int

for asserting something contains a certain number of elements, etc.

One interesting novelty of test is how it handles adding additional context to output in failing tests. As described above the variadic interface{} arguments of testify are a lingering source of buggy tests in our legacy code, so we wanted to fix that.

In doing so, the assertion functions in test end in a variadic PostScript parameter. Each PostScript prints a block of custom output on failure. Anything that implements the PostScript interface can be used.

type PostScript interface {
	Label() string
	Content() string

We then include Sprint and Sprintf implementations of the PostScript interface for the most common use cases of adding print statements. So you might see assertions in complex scenarios that look something like

must.Eq(t, 42, value, must.Sprintf("a: %s, b: %d, c: %x", a, b, c))

or if you prefer, the key-value Sprint variant

must.Eq(t, 42, value, must.Sprint("a", a, "b", b, "c", c))

At one point, my co-worker discovered a bug in our code that a testify test was incorrectly reporting as working correctly. It boiled down to a pointer comparison vs. dereference value comparison, e.g.

type F struct {
	B *bool

func (f *F) Equal(o *F) bool {
	return f.B == o.B // bug!

Full example in Go Playground.

In this snippet, the equality check is comparing pointers which was not the intended behavior. But if you use testify::Equal() without a custom comparator, it will magically do the dereference for you, which is not indicative of the actual implementation. test on the other hand does not do the magic dereference, and correctly reported the objects as not equal.


There are some rough spots in test. Although the use of common interfaces described above is convenient for the types that implement them, not all types implement them, even though perhaps they would be good candidates. There is no official standard for these interfaces. Even if a type implements the idea of a particular interface, the spelling needs to be consistent, too! This was a problem in Nomad, where we were using Equals() instead of Equal() until a major refactoring. Some other languages come with certain built-in interfaces for describing things like Collections, Maps, Sets to be implemented by built-in or custom types. I think Go could provide something similar, expanding on the x/constraints package.

An earlier version of test tried to emphasize the use of the io/fs package for filesystem testing. In practice, the multitude of interfaces around filesystem operations provided by io/fs proved to be ineffective. In practice, the fact that all the real filesystem operations are provided by package-level functions in the os package means io/fs is not usable. Package functions cannot satisfy an interface.

Using these io/fs interfaces would look something like this,

func DirExists(fs fs.FS, dir string)

which seems fine, but then the caller needs to do something like

DirExists(os.DirFS("usr/local"), "bin")

to check if "/usr/local/bin" exists. It’s not ideal. There needs to be some kind of official os.Filesystem implementation of all those interfaces in io/fs before they are actually useful. For now, test has kept the io/fs functions, but also provides os variants, which is what folks actually use.


Coming from a Java background, the lack of auto-boxing of built-in types becomes painfully apparent when working with Go generics. In particular, to avoid the use of reflection the test library has functions that are specific to map, slice, and string values. This has been a painful pill to swallow and is something I hope can be addressed in future versions of Go. For example consider a simple Empty assertion. In fact the operation would be the same for each of those built-in types. Just check if len(x) == 0. But there is no way to declare a type constraint that describes a type that works with the built-in len operation. So you end up with two options: either accept an interface{} and use a type-switch, or implement variations of Empty for each built-in type. You might be thinking, “the type switch doesn’t sound so bad, what’s wrong with that?”. The problem is, what about the non-built-in types? You probably want to support “container”-like types that implement an Empty interface, defering to that type’s Empty method. Since we were forced to accept interface{} because of the len problem, now we need reflection to check if the type implements Empty, add that to our type switch, and then cry a little because we violated one of our key design goals - avoid reflection. So the test library chooses the alternative pattern - creating specific functions for each of the built-in types. It’s not elegant, but it works.


As described there are trade-offs between shoenig/test and stretchr/testify. Which one is better is I think a matter of personal opinion - at the end of the day they are both functioning assertions libraries, and they both have warts that will make you stop and think, “hmm how do I do X with this?”. If Go generics continue to improve - particularly around the problems with using built-ins in a generic context as described above, I think there could be a bright future for test - and/or maybe a generics based v2 of testify.

Happy hacking!

➡️ related posts in the go-code series ...