There is a lengthy documentation, with plenty of videos and tutorials if you want to learn Go modules. If you didn't use Go modules yet, I recommend to do it first and then continue reading this blog post.
This blog post is more in-depth and tries to explain how Go modules work with vendoring in mind. After reading it, you should have a clear understanding on how
.travis.yml works within a Go project and how you can make sure your Go project supports both old & new Go versions with vendoring enabled.
Let's jump in.
The "vendor/" folder
I have many personal Go projects that vendors the dependencies to their respective Git repositories. I've decided to move one of my projects to make use of Go modules, so I can learn more about how to use Go modules properly and also adjust the repository layout if needed. (This is mostly needed so we can add new features & improvements to vim-go).
All my open source projects use Travis CI for pull requests. Whenever someones create a new pull request with new contributions, Travis CI kicks in and tests the packages (by running
go test -v ./...).
With Go 1.5, the famous
vendor/ was added as an experiment. This would mean for many that we could now vendor (bundle) our dependencies together with our source code, enabling us various kind of advantages (such as locally reproducible packages, not depending on source outages, etc..).
As I said, I vendor all my dependencies for every single of my projects. This allows me to make sure to provide a stable version of my package. After defining a new module (
go mod init) for my project, I also made sure to vendor my projects. For this, I've learned that we have to call the following code:
go mod vendor
This makes sure to vendor the dependencies to a
vendor/ folder. If you have already an existing
vendor/ folder, just remove it completely and re-run the above command.
Set up .travis.yml for a Go Project
If you setup Travis and add the following
.travis.yml file to the root of your Git repository, Travis CI automatically setups your Go project:
language: go go: - "1.10.x"
It'll do the followings:
- Setup a GOPATH
- Put it into the appropriate place
- Call some
gocommands to test your package (
go test, etc...)
What important here is, if you have a
vendor/ folder, this also means that the
go command makes sure to include this folder while executing any
go subcommand (such as
go test, etc...).
This is all good and works perfectly. Now let us try to add the latest version
(Go 1.11 w/ modules support) to this file:
language: go go: - "1.10.x" - "1.11.x"
If you add this, you'll see that everything still works, but if you check the logs carefully, you'll see that something is different:
If you expand the
install step, you will see the following warning:
go get: warning: modules disabled by GO111MODULE=auto in GOPATH/src; ignoring go.mod; see 'go help modules'
So this still uses the old way of building a package, it'll not respect the
go.mod that we added. Why? Let us read the documentation (from
go help modules):
If GO111MODULE=auto or is unset, then the go command enables or disables module support based on the current directory. Module support is enabled only when the current directory is outside GOPATH/src and itself contains a go.mod file or is below a directory containing a go.mod file.
Travis CI puts the directory (the git repo itself) inside a GOPATH, therefore it's not enabled.
Enabling modules support
Let's enable it explicitly by setting the
GO111MODULE environment variable to
language: go go: - "1.10.x" - "1.11.x" env: - GO111MODULE=on
This time, it all looks fine. However, Travis CI still downloads the dependencies every time with
go get -t -v ./... command:
This is because the
install statement in the
.travis.yml is automatically set to
go get -t -v ./... if you use
language:go. That's the magic they do for you.
We don't want to download anything, as we already have
vendor/ folder and that should be the sole truth of source for our Go dependencies.
To skip the
install process, we have to set the field explicitly to
language: go go: - "1.10.x" - "1.11.x" env: - GO111MODULE=on install: true
Now it should be all good. If we check the build logs, you'll be surprised though! :
What is wrong here? You'll see that it still downloads the dependencies. There is no
install step anymore, so the only command that is called is
go test -v ./....
Modules and vendoring
With the new Go modules support, if the dependencies are not available in the cache (see
go env | grep GOCACHE), it'll try to download it.
However, we already have a
vendor/ folder. So why does it not respect the folder and still tries to download the dependencies? This is written in the
go help modules page:
Once the go.mod file exists, no additional steps are required: go commands like 'go build', 'go test', or even 'go list' will automatically add new dependencies as needed to satisfy imports.
That's the reason
go test adds (downloads) the dependencies. But it still doesn't answer why it doesn't respect the
vendor/ folder. Let us continue to read:
Modules and vendoring When using modules, the go command completely ignores vendor directories. By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification, as described in the previous section). To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module. To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -mod=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.
All right, seems like
vendor/ is ignored when you use Go modules. Bummer.
At least there is a solution to it, we just have to add the
-mod=vendor flag. Now, we need to replace the default
go test command in our Travis CI environment.
install field, there is also a
script field which is running the actual command of the CI build. This is set to default to
go test -v ./... in Travis CI. We can override this by adding an explicit
script field with a customized command in our
language: go go: - "1.10.x" - "1.11.x" env: - GO111MODULE=on install: true script: go test -v -mod=vendor ./...
Now we should be set. Let's check the CI logs:
Great! Everything works as expected. Travis uses Go version 1.11 and uses our
But... something is still wrong. Now, you'll see that Go 1.10 version started to fail:
If we check the logs, we'll see why:
That's a pity. The
-mod flag was added with Go 1.11 and therefore is not available in the older version of Go.
So how do we fix this?
Supporting multiple Go versions with Build Matrix
We need to call two different commands for each Go version:
go test -v ./...
go test -v -mod=vendor ./...
Fortunately, this is possible with something called "Build Matrix" in Travis CI. Since the beginning, there was a build matrix for our travis setup already, because of the following lines:
go: - "1.10.x" - "1.11.x"
Travis always triggers two builds for each listed Go version. Now what we have to do is, to change the
script directive for each of these versions. This was implicit. Just like we did with
script, we're going to replace the
go directive and change it explicitly:
language: go matrix: include: - go: "1.10.x" script: go test -v ./... - go: "1.11.x" script: go test -v -mod=vendor ./... env: - GO111MODULE=on install: true
If we check our builds we'll see that both versions successful:
Both logs for
1.11 show us that a different command was executed:
Yay, we did it \o/
We have now a CI system that tests our Go package against two different Go versions. This will help us to make sure our code still works for older versions of Go and newer versions that use Go modules.
This whole exercise was very useful for me as it helped me to understand the concepts better. Using go modules still requires learning these new concepts. The docs are well written, but it needs to be read carefully to see the differences between older and newer Go versions.