SamWhited|blog

Supporting Go Modules

Over the past few weeks I’ve taken to submitting similar patches to any Go project that I depend on, directly or indirectly (and some that I don’t). These patches have all been relatively simple: basic support for Go Modules. They mostly comprise a go.mod file, a go.sum file if required, and minor tweaks to the Makefile or CI configuration. I also submitted a few fixes to packages that had existing, but broken module support.

Here is an incomplete (probably) list of patches that I could find by skimming my browser history:

And here are a handful of issues and patches that report broken module support (but don’t add support for modules themselves), either directly or when importing the project as a dependency:

From this I learned a few things (although some of it I, admittedly, already knew):

Tooling and the import compatibility rule

The tooling (or documentation if this tooling exists and I just couldn’t find it?) is lacking. For large projects that are above version one the import compatibility rule makes supporting modules very time consuming. Your life becomes a mix of running sed(1) and then finding and fixing changes that weren’t actually part of an import line.

Once those large projects are supporting modules, upgrading your project to use new major versions is also very time consuming. Contrary to the Go team’s claim that no one will use modules that change their version often, so the import compatibility rule is not a problem, sometimes you’re stuck with a service (I’m looking at you, Stripe) that can’t stop making breaking changes and doesn’t backport bug fixes. When this happens there’s nothing you can do about it, and it’s infuriatingly difficult to perform upgrades. Even if the breaking change itself was minor and easy to fix, you still have to update the import line every single place you used it and wait for any other dependencies you use to do the same.

The upgrade path from legacy dependency management tools like dep for modules that require use of the import compatibility rule is also very difficult or impossible. Modules did not attempt to be backwards compatible with the existing ecosystem, so supporting it in your post-v1 library means breaking support for consumers of your library who are using old tooling. Because of this, Stripe has decided to revert Go Modules support in their package. I was told several times by someone on the Go team (I’m paraphrasing) that: “people won’t use projects that change major versions frequently, so we don’t have to worry about the import compatibility rule”, but sometimes you’re using a service like Stripe whether you want to be or not, so you’re also stuck with their SDK and the design decisions they’ve made in the past.

Documentation

Users with no prior experience with modules are likely to merge broken support. The various combinations of semver tags on the repo, the import compatibility rule, having or not having a mod file at all, etc. are numerous, confusing, and not well documented (or hard to find if they are documented). I saw far more broken module support than I saw correct module support on the first try. Common mistakes included not adding the major version to the module name where required, not adding the major version to the import path and accidentally importing older versions of the module from the newer version, leaving out dependencies from build tags, leaving out dependencies entirely, and not comitting the go.sum file. I’m putting this down to a documentation problem because I don’t think the Go project leaders are likely to back down and change anything at this point, but realistically it’s a complexity problem.

Dependencies

A few unmaintained packages are widely used across the small set of packages I looked at (eg. github.com/pmezard/go-difflib and possibly github.com/BurntSushi/toml), but they’re not always clearly marked as unmaintained. The dependency on difflib was most commonly pulled in as an indirect dependency by way of github.com/stretchr/testify; if you use Testify not only are you pulling in unnecessary dependencies, but you’re pulling in unmaintained ones. Please take this risk into account before pulling in dependencies for minor convenience functions.

Luckily, authors appear to be more likely to recognize dependency bloat and move to fix it when they have a list in front of them. Several authors removed unnecessary testing dependencies, or started removing small dependencies that didn’t need to be external, after I submitted a patch with a large go.mod file that made the problem obvious. However, lots of small libraries I looked at had no dependencies at all; yay!

Conclusion

People accuse me of complaining about modules a lot, and while I don’t think they should have been released in Go 1.11 without a few changes, and shouldn’t be made official in Go 1.12, I actually really like them overall. As more projects are updated to support modules and the tooling improves, many of these problems are likely to go away or become easier to deal with.

For now, I recommend supporting modules in your project as soon as possible and filing an experience report about your upgrade. The benefits to modules far outweigh the cost, even for large projects. Being able to see a dependency list and known good versions of dependencies has already saved me from build and packaging issues several times; let’s get the Go ecosystem on a single standard packaging technology and then, if we can’t delay the full release, work out the kinks from there.

Corrections

2019–01–14

I previously left out patches for github.com/ortuman/jackal, and github.com/go-chi/chi/v3. Previously I also did not list any of the issues filed against projects using modules incorrectly, or which would be broken by using modules.

2019–01–16

Added a few more patches to the list and mentioned that at least one project has decided to revert module support.

2019–01–24

Previously the date on these corrections was wrong. I have also added a few more missing links.