Go 1.11 is out now and it includes preliminary support for Go modules. This is an experimental opt-in feature and the goal is to finalize it as a first class feature for the upcoming Go 1.12 release.

There is a lengthy documentation, with plenty of videos and tutorials if you want to learn Go modules. If you didn't use Go modules yet, I recommend to do it first and then continue reading this blog post.

I've also read the excellent blog post "Using Go modules with Travis CI" blog post from Dave Cheney. Definitely take a look at it and read it as well.

This blog post is more in-depth and tries to explain how Go modules work with vendoring in mind. After reading it, you should have a clear understanding on how .travis.yml works within a Go project and how you can make sure your Go project supports both old & new Go versions with vendoring enabled.

Let's jump in.

The "vendor/" folder

I have many personal Go projects that vendors the dependencies to their respective Git repositories. I've decided to move one of my projects to make use of Go modules, so I can learn more about how to use Go modules properly and also adjust the repository layout if needed. (This is mostly needed so we can add new features & improvements to vim-go).

All my open source projects use Travis CI for pull requests. Whenever someones create a new pull request with new contributions, Travis CI kicks in and tests the packages (by running go test -v ./...).

With Go 1.5, the famous vendor/ was added as an experiment. This would mean for many that we could now vendor (bundle) our dependencies together with our source code, enabling us various kind of advantages (such as locally reproducible packages, not depending on source outages, etc..).

As I said, I vendor all my dependencies for every single of my projects. This allows me to make sure to provide a stable version of my package. After defining a new module (go mod init) for my project, I also made sure to vendor my projects.  For this, I've learned that we have to call the following code:

go mod vendor

This makes sure to vendor the dependencies to a vendor/ folder. If you have already an existing vendor/ folder, just remove it completely and re-run the above command.

Set up .travis.yml for a Go Project

If you setup Travis and add the following .travis.yml file to the root of your Git repository, Travis CI automatically setups your Go project:

language: go
go: 
 - "1.10.x"

It'll do the followings:

• Setup a GOPATH
• Put it into the appropriate place
• Call some go commands to test your package (go get, go test, etc...)

What important here is, if you have a vendor/ folder, this also means that the go command makes sure to include this folder while executing any go subcommand (such as go build, go test, etc...).

This is all good and works perfectly. Now let us try to add the latest version
(Go 1.11 w/ modules support) to this file:

language: go
go: 
 - "1.10.x"
 - "1.11.x"

If you add this, you'll see that everything still works, but if you check the logs carefully, you'll see that something is different:

If you expand the install step, you will see the following warning:

go get: warning: modules disabled by GO111MODULE=auto in GOPATH/src;
	ignoring go.mod;
	see 'go help modules'

So this still uses the old way of building a package, it'll not respect the
go.mod that we added. Why? Let us read the documentation (from go help modules):

If GO111MODULE=auto or is unset, then the go command enables or
disables module support based on the current directory.
Module support is enabled only when the current directory is outside
GOPATH/src and itself contains a go.mod file or is below a directory
containing a go.mod file.

Travis CI puts the directory (the git repo itself) inside a GOPATH, therefore it's not enabled.

Enabling modules support

Let's enable it explicitly by setting the GO111MODULE  environment variable to on:

language: go
go: 
 - "1.10.x"
 - "1.11.x"

env:
  - GO111MODULE=on

This time, it all looks fine. However, Travis CI still downloads the dependencies every time with go get -t -v ./... command:

This is because the install statement in the .travis.yml is automatically set to go get -t -v ./... if you use language:go. That's the magic they do for you.

We don't want to download anything, as we already have vendor/ folder and that should be the sole truth of source for our Go dependencies.

To skip the install process, we have to set the field explicitly to true:

language: go
go: 
 - "1.10.x"
 - "1.11.x"

env:
  - GO111MODULE=on

install: true

Now it should be all good. If we check the build logs, you'll be surprised though! :

What is wrong here? You'll see that it still downloads the dependencies. There is no install step anymore, so the only command that is called is go test -v ./....

Modules and vendoring

With the new Go modules support, if the dependencies are not available in the cache (see go env | grep GOCACHE), it'll try to download it.

However, we already have a vendor/ folder. So why does it not respect the folder and still tries to download the dependencies? This is written in the go help modules page:

Once the go.mod file exists, no additional steps are required:
go commands like 'go build', 'go test', or even 'go list' will automatically
add new dependencies as needed to satisfy imports.

That's the reason go test adds (downloads) the dependencies. But it still doesn't answer why it doesn't respect the vendor/ folder. Let us continue to read:

Modules and vendoring

When using modules, the go command completely ignores vendor directories.

By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.

All right, seems like vendor/ is ignored when you use Go modules.  Bummer.

At least there is a solution to it, we just have to add the -mod=vendor flag. Now, we need to replace the default go test command in our Travis CI environment.

Remember the install field, there is also a script field which is running the actual command of the CI build. This is set to default to go test -v ./... in Travis CI. We can override this by adding an explicit script field with a customized command in our .travis.yml file:

language: go
go: 
 - "1.10.x"
 - "1.11.x"

env:
  - GO111MODULE=on

install: true

script: go test -v -mod=vendor ./...

Now we should be set. Let's check the CI logs:

Great! Everything works as expected. Travis uses Go version 1.11 and uses our vendor/ folder.

But... something is still wrong. Now, you'll see that Go 1.10 version started to fail:

If we check the logs, we'll see why:

That's a pity. The -mod flag was added with Go 1.11 and therefore is not available in the older version of Go.

So how do we fix this?

Supporting multiple Go versions with Build Matrix

We need to call two different commands for each Go version:

• 1.10: go test -v ./...
• 1.11: go test -v -mod=vendor ./...

Fortunately, this is possible with something called "Build Matrix" in Travis CI. Since the beginning, there was a build matrix for our travis setup already, because of the following lines:

go: 
 - "1.10.x"
 - "1.11.x"

Travis always triggers two builds for each listed Go version. Now what we have to do is, to change the script directive for each of these versions. This was implicit. Just like we did with install and script, we're going to replace the go directive and change it explicitly:

language: go

matrix:
  include:
  - go: "1.10.x"
    script:  go test -v ./...
  - go: "1.11.x"
    script: go test -v -mod=vendor ./...

env:
  - GO111MODULE=on

install: true

If we check our builds we'll see that both versions successful:

Both logs for 1.10 and 1.11 show us that a different command was executed:

Yay, we did it \o/

Verdict

We have now a CI system that tests our Go package against two different Go versions. This will help us to make sure our code still works for older versions of Go and newer versions that use Go modules.

This whole exercise was very useful for me as it helped me to understand the concepts better. Using go modules still requires learning these new concepts. The docs are well written, but it needs to be read carefully to see the differences between older and newer Go versions.