Ten Useful Techniques in Go

Here are my own best practices that I've gathered from my personal experiences with dealing lots of Go code for the past years. I believe they all scale well. With scaling I mean:
  1. Your applications requirements are changing in an agile environment. You don't want to refactor every piece of it after 3-4 months just because you need to. New features should be added easily.
  2. Your application is developed by many people, it should readable and easy to maintain.
  3. Your application is used by a lot of people, there will be bugs which should be find easily and fixed quickly

With time I've learned these things are important in long-term. Some of them are minor, but they affect a lot of things. These are all advices, try to adapt them and let me know if it works out for you. Feel free to comment :)

1. Use a single GOPATH

Multiple GOPATH's doesn't scale well. GOPATH itself is highly self-contained by nature (via import paths). Having multiple GOPATH's can have side effects such as using a different version for a given package. You might have updated it in one place, but not in another. Having said that, I haven't encountered a single case where multiple GOPATH's are needed. Just use a single GOPATH and it will boost your Go development process.

I need to clarify one thing that come up and lots of people disagree with this statement. Big projects like etcd or camlistore are using vendoring trough freezing the dependencies to a folder with a tool like godep. That means those projects have a single GOPATH in their whole universe. They only see the versions that are available inside that vendor folder. Using differents GOPATH for every single project is just an overkill unless you think your project is big and important one. If you think that your project needs its own GOPATH folder go and create one, however until that time don't try to use multiple GOPATH's. It will just slow down you.

2. Wrap for-select idiom to a function

If there is a situation where you need to break out of from a for-select idiom, you need to use labels. An example would be:

func main() {

L:
    for {
        select {
        case <-time.After(time.Second):
            fmt.Println("hello")
        default:
            break L
        }
    }

    fmt.Println("ending")
}

As you see you need use break in conjunction with a label. This has his place, but I don't like it. The for loop seems to be small in our example, but usually its much more larger and tracking the state of break is tedious.

I'm wrapping for-select idioms into a function if I need to break out:

func main() {
    foo()
    fmt.Println("ending")
}

func foo() {
    for {
        select {
        case <-time.After(time.Second):
            fmt.Println("hello")
        default:
            return
        }
    }
}

This has the beauty that you can also return an error (or any other value), and then it's just:

// blocking
if err := foo(); err != nil {
    // do something with the err
}

3. Use tagged literals for struct initializations

This is a untagged literal example :

type T struct {
    Foo string
    Bar int
}

func main() {
    t := T{"example", 123} // untagged literal
    fmt.Printf("t %+v\n", t)
}

Now if you go add a new field to your T struct your code will fail to compile:

type T struct {
    Foo string
    Bar int
    Qux string
}

func main() {
    t := T{"example", 123} // doesn't compile
    fmt.Printf("t %+v\n", t)
}

Go's compatibility rules (http://golang.org/doc/go1compat) covers your code if you use tagged literals. This was especially true when they introduced a new field called Zone to some net package types, see: http://golang.org/doc/go1.1#library. Now back to our example, always use tagged literals:

type T struct {
    Foo string
    Bar int
    Qux string
}

func main() {
    t := T{Foo: "example", Bar: 123}
    fmt.Printf("t %+v\n", t)
}

This compiles fine and is scalable. It doesn't matter if you add another field to the T struct. Your code will always compile and is guaranteed to be compiled by further Go versions. go vet will catch untagged struct literals, just run it on your codebase.

4. Split struct initializations into multiple lines

If you have more than 2 fields just use multiple lines. It makes your code much more easier to read, that means instead of:

T{Foo: "example", Bar:someLongVariable, Qux:anotherLongVariable, B: forgetToAddThisToo}

Use:

T{
    Foo: "example",
    Bar: someLongVariable,
    Qux: anotherLongVariable,
    B: forgetToAddThisToo,
}

This has several advantages, first it's easier to read, second it makes disabling/enabling field initializations easy (just comment them out or removing), third adding another field is much more easier (adding a newline).

5. Add String() method for integers const values

If you are using custom integer types with iota for custom enums, always add a String() method. Let's say you have this:

type State int

const (
    Running State = iota 
    Stopped
    Rebooting
    Terminated
)

If you create a new variable from this type and print it you'll just get an integer (http://play.golang.org/p/V5VVFB05HB):

func main() {
    state := Running

    // print: "state 0"
    fmt.Println("state ", state)
}

Well here 0 doesn't mean much until you lookup your consts variables again. Just adding the String() method to your State type fixes it (http://play.golang.org/p/ewMKl6K302):

func (s State) String() string {
    switch s {
    case Running:
        return "Running"
    case Stopped:
        return "Stopped"
    case Rebooting:
        return "Rebooting"
    case Terminated:
        return "Terminated"
    default:
        return "Unknown"
    }
}

The new output is: state: Running. As you see it's now much more readable. It will make your life a lot of easier when you need to debug your app. You can do the same thing with by implementing the MarshalJSON(), UnmarshalJSON() methods etc..

6. Start iota with a +1 increment

In our previous example we had something that is also open the bugs and I've encountered several times. Suppose you have a new struct type which also stores a State field:

type T struct {
    Name  string
    Port  int
    State State
}

Now if we create a new variable based on T and print it you'll be surprised (http://play.golang.org/p/LPG2RF3y39) :

func main() {
    t := T{Name: "example", Port: 6666}

    // prints: "t {Name:example Port:6666 State:Running}"
    fmt.Printf("t %+v\n", t)
}

Did you see the bug? Our State field is uninitialized and by default Go uses zero values of the respective type. Because State is an integer it's going to be 0 and zero means basically Running in our case.

Now how do you know if the State is really initialized? Is it really in Running mode? There is no way to distinguish this one and is a way to cause unknown and unpredictable bugs. However fixing it easy, just start iota with a +1 offset (http://play.golang.org/p/VyAq-3OItv):

const (
    Running State = iota + 1
    Stopped
    Rebooting
    Terminated
)

Now your t variable will just print Unknown by default, neat right? :) :

func main() {
    t := T{Name: "example", Port: 6666}

    // prints: "t {Name:example Port:6666 State:Unknown}"
    fmt.Printf("t %+v\n", t)
}

But starting your iota with a reasonable zero value is another way to solve this. For example you could just introduce a new state called Unknown and change it to:

const (
    Unknown State = iota 
    Running
    Stopped
    Rebooting
    Terminated
)

7. Return function calls

I've seen a lot of code like (http://play.golang.org/p/8Rz1EJwFTZ):

func bar() (string, error) {
    v, err := foo()
    if err != nil {
        return "", err
    }

    return v, nil
}

However you can just do:

func bar() (string, error) {
    return foo()
}

Simpler and easier to read (unless of course you want to log the intermediates values).

8. Convert slices,maps,etc.. into custom types

Converting slices or maps into custom types again and makes your code much more easier to maintain. Suppose you have a Server type and a function that returns a list of servers:

type Server struct {
    Name string
}

func ListServers() []Server {
    return []Server{
        {Name: "Server1"},
        {Name: "Server2"},
        {Name: "Foo1"},
        {Name: "Foo2"},
    }
}

Now suppose you want to retrieve only servers that with a specific name. Let's change our ListServers() function a little bit and add a simple filter support:

// ListServers returns a list of servers. If name is given only servers that
// contains the name is returned. An empty name returns all servers.
func ListServers(name string) []Server {
    servers := []Server{
        {Name: "Server1"},
        {Name: "Server2"},
        {Name: "Foo1"},
        {Name: "Foo2"},
    }

    // return all servers
    if name == "" {
        return servers
    }

    // return only filtered servers
    filtered := make([]Server, 0)

    for _, server := range servers {
        if strings.Contains(server.Name, name) {
            filtered = append(filtered, server)
        }
    }

    return filtered
}

Now you can use it for filter servers that has the Foo string:

func main() {
    servers := ListServers("Foo")

    // prints: "servers [{Name:Foo1} {Name:Foo2}]"
    fmt.Printf("servers %+v\n", servers)
}

As you see our servers are now filtered. However this doesn't scale well. What if you want to introduce another logic for your server set? Like checking health of all servers, creating a DB record for each server, filtering by another a new field, etc...

Let's introduce another new type called Servers and change our initial ListServers() to return this new type:

type Servers []Server

// ListServers returns a list of servers.
func ListServers() Servers {
    return []Server{
        {Name: "Server1"},
        {Name: "Server2"},
        {Name: "Foo1"},
        {Name: "Foo2"},
    }
}

What we do now is, we just add a new Filter() method to our Servers type:

// Filter returns a list of servers that contains the given name. An
// empty name returns all servers.
func (s Servers) Filter(name string) Servers {
    filtered := make(Servers, 0)

    for _, server := range s {
        if strings.Contains(server.Name, name) {
            filtered = append(filtered, server)
        }

    }

    return filtered
}

And now let us filter servers with the Foo string:

func main() {
    servers := ListServers()
    servers = servers.Filter("Foo")
    fmt.Printf("servers %+v\n", servers)
}

Voila! See how your code just simplified? You want to check if the servers are healthy? Or add a DB record for each of the server? No problem just add those new methods:

func (s Servers) Check() 
func (s Servers) AddRecord() 
func (s Servers) Len()
...

9. withContext wrapper functions

Sometimes you do repetitive stuff for every function, like locking/unlocking, initializing a new local context, preparing initial variables, etc.. An example would be:

func foo() {
    mu.Lock()
    defer mu.Unlock()

    // foo related stuff
}

func bar() {
    mu.Lock()
    defer mu.Unlock()

    // bar related stuff
}

func qux() {
    mu.Lock()
    defer mu.Unlock()

    // qux related stuff
}

If you want to change one thing, you need to go and change them all in other places. If its common task the best thing is to create a withContext function. This function takes a function as an argument and calls it with the given context:

func withLockContext(fn func()) {
    mu.Lock
    defer mu.Unlock()

    fn()
}

Then just refactor your initial functions to make use of this context wrapper:

func foo() {
    withLockContext(func() {
        // foo related stuff
    })
}

func bar() {
    withLockContext(func() {
        // bar related stuff
    })
}

func qux() {
    withLockContext(func() {
        // qux related stuff
    })
}

Don't just think of a locking context. The best use case for this is a DB connection or a DB context. Let's slightly change our withContext function:

func withDBContext(fn func(db DB)) error {
    // get a db connection from the connection pool
    dbConn := NewDB()

    return fn(dbConn)
}

As you see now it gets a connection, passes it to the given function and returns the error of the function call. Now all you do is:

func foo() {
    withDBContext(func(db *DB) error {
        // foo related stuff
    })
}

func bar() {
    withDBContext(func(db *DB) error {
        // bar related stuff
    })
}

func qux() {
    withDBContext(func(db *DB) error {
        // qux related stuff
    })
}

You changed to mind to use a different approach, like making some pre initialization stuff? No problem, just add them into withDBContext and you are good to go. This also works perfect for tests.

This approach has the disadvantage that it pushes out the indentation and makes it harder to read. Again seek always the simplest solution.

10. Add setter, getters for map access

If you are using maps heavily for retrieving and adding use always getters and setters around your map. By using getters and setters you can encapsulate the logic to their respective functions. The most common error made here is concurrent access. Say you have this in one goroutine:

m["foo"] = bar

And this on another:

delete(m, "foo")

What happens? Most of you are already familiar to race conditions like this. Basically this is a simple race condition because maps are not thread safe by default. But you can easily protect them with mutexes:

mu.Lock()
m["foo"] = "bar"
mu.Unlock()

And:

mu.Lock()
delete(m, "foo")
mu.Unlock()

Suppose you are using this map in other places. You need to go and put everywhere mutexes! However you can avoid this entirely by using getter and setter functions:

func Put(key, value string) {
    mu.Lock()
    m[key] = value
    mu.Unlock()
}
func Delete(key string) {
    mu.Lock()
    delete(m, key)
    mu.Unlock()
}

An improvement over this procedure would be using an interface. You could completely hide the implementation. Just use a simple, well defined interface and let the package users use them:

type Storage interface {
    Delete(key string)
    Get(key string) string
    Put(key, value string)
}

This is just an example but you get the idea. It doesn't matter what you use for the underlying implementation. What matters is the usage itself and an interface simplifies and solves lots of the bugs you'll encounter if you expose your internal data structures.

Having said that, sometimes an interface is just and overkill because you might have a need to lock several variables at once. Know you application well and apply this improvement only if you have a need for it.

Conclusion

Abstractions are not always good. Sometimes the most simplest thing is just the way you're doing it already. Having said that, don't try to make your code smarter. Go is by nature a simple language, in most cases it has only one way to do something. The power comes from this simplicity and it is one of the reasons why it's scaling so well on the human level.

Use these techniques if you really need them. For example converting a []Server to Servers is another abstraction, do it only if you have a valid reason for it. But some of the techniques like starting iotas with 1 could be used always. Again always strike in favor of simplicity.

A special thanks to Cihangir Savas, Andrew Gerrand, Ben Johnson and Damian Gryski for their valuable feedback and suggestions.

Recent Go projects and contributions

Below are some of my recent Go projects (that I couldn't announce on my blog) and contributions on Github:

1.) github.com/fatih/color 

This is an ANSI Color package for Go. It's handy and has different ways of using the API. Most of the Color packages were not as idiomatic as I assumed. I decided to write another one :) 

2.) github.com/lxc/go-lxc

go-lxc is the official Go bindings for LXC (it was just released 1 month ago). The project is maintained by one of the LXC core developers caglar10ur. I decided to look at it and found some improvements to be made. First  I introduced a new error type and simplified the error handling. Secondly I've found that the function for obtaining all containers on the current system was aborting if one of the LXC's was failing. This was fixed by  this commit (which is by the way also the default behavior in the current stable C API). And lastly I've changed test package name to enable access to private identifiers.

3.) github.com/fatih/stopwatch

It's a basic stopwatch wrapped around the standard `time` package. It has some nice features to measure the time of a function call. 

4.) github.com/fatih/pool

Pool is a thread-safe connection pool for Go's net.Conn interface. You create a factory function which is responsible of creating new connection. After that you create a pool and pass this factory function. Now whenever you make a pool.Get() call, it goes and fetches one of the connections from the pool, if there isn't any connections available it goes and creates another one via the factory function. It's simple but useful.

Set - a simple, thread safe Set implementation for Golang

I've create a new Go package (set)  that allows to use all common Set operations. It's very easy to create  a set and use any necessary set operation. I implement it with the built-in map data structure (that means it's hash-based). And the best part about is that it's totally thread safe. You can use it concurrently .  

For more information look at the Github page: https://github.com/fatih/set

Thread safe Set data structure in Go

Below is a simple SET data structure with basic operations. Each operation is thread-safe and shows how to implement custom SET data structures for your own application. Our example below uses "string" as the key data type, however you can use an interface{} or any other basic or custom data type you want.  Also it's a great example of how to use a mutex from  the sync package. 

To test it run via play.golang.org or download and execute: "go run set.go"

ZeroMQ request/reply to a specific server via Router-Router broker in Go

Do you know that you can connect a REQ sockect to a specific  REP socket with a intermediate ZeroMQ ROUTER to ROUTER broker in middle of them? That way you can use this broker as a reverseproxy or for totally something else. 

First let create a REQ socket:

Here "1123581321" is the targetKey to that target REP socket will listen to. That means you can have mutliple REQ sockets and multiple REP sockets listening to different keys. 

Here is the main code that enables us this powerful routing. It is in form ROUTER-to-ROUTER. We sending a MultiPart message via the REQ socket that contains the REP identity. After parsing the this key (in our example this is "1123581321") we change the the ZeroMQ message framing and setting this key as the REP identity. Therefore the broker will send this message to our REP socket, instead of a default one (picked via the load-balancing that zmq makes automatically). The code is below:

And finally our REP on the backend is listening to the specific key "1123581321" that was used by our REQ client. Any other message will be discarded automatically via Zmq. The code is in form of:

As you see, usually a ROUTER-to-ROUTER is not recommended to use, however in this case you can have full distributed clients that connect directly to a backend client, instead of load-balancing algorithm. You can choose to whom you might want to send your REQ. 

One use case is a HTTP Reverse proxy over ZeroMQ sockets. That way you can easily proxy clients from whatever port you want (useful for firewall gardens). Albeit it's not perfect (it's not working in a streaming fashion), a prototype with GO's net/http works works quite problem less. Any suggestions and improvements are welcome :)

Use the same key content but different values in Go map

As you know the type map in Go is an unordered collection of key-value pairs. Each key is unique and has a value. However with a little trick we can use the same key content for different values. But how?

Say we have struct named User

type User struct {
    location string
    age      int
}

let us initialize a map with these struct as the key, where the key is of type *User and value of string:

users := make(map[*User]string, 0)

and finally let us add several users:

users[&User{"San Francisco", 31}] = "chris"
users[&User{"Istanbul", 27}] = "fatih"
users[&User{"Istanbul", 27}] = "gokmen"
users[&User{"Berlin", 23}] = "micheal"

The key here is that each pointer has a different memory address. Therefore it's totally legal to use them as a key in map. Because the underlying addresses are different we can add two different users with the same content.

One cevat here is, if you try to print (using fmt or log) *User, you'll get the pointers memory address. Like:

map[0x2101bc000:chris 0x2101bc020:fatih 0x2101bc040:gokmen 0x2101bc060:micheal]

We can change it by defining a String() method on the struct *User

func (user User) String() string {
    return fmt.Sprintf("(%s, %d)", user.location, user.age)
}

Now whenever you use something like fmt.Println(users), you'll get:

map[(San Francisco, 31):chris (Istanbul, 27):fatih (Istanbul, 27):gokmen (Berlin, 23):micheal]

How to call a function every N minutes in Go?

Go has a nice package called time that allows to handle time manipulations very easily. The time package has a struct called ticker, which delivers 'ticks' at intervals. This is very handy because you can create a ticker and run it in a separate go closure.

First import the time package

import "time"

then create a new ticker

ticker := time.NewTicker(time.Minute * 1)

and finally start the ticker in a separate go closure

go func() {
    for _ = range ticker.C {
        fmt.Printf("ticked at %v", time.Now())
    }
}()

The print statement above will print the current time every one minute. Just change the interval and function for your needs

Useful zsh features for path handling

Did you know that you could expand paths in zsh. Try to type this in your terminal:

cd /u/l/b

this will expand to:

cd /usr/local/bin
#or 
cd /usr/lib/...

and let's assume for a moment that you are in a wrong directoy (/usr/local/bin) and you want actually cd into /usr/local/share.

That's easy too, just type:

cd bin share

this will replace the bin string in previous path with share

Subvim - Vim customized to be like SublimeText

Subvim is a project to bring the best features from SublimeText2 into Vim. I've created Subvim because I love the features of SublimeText2 but couldn't leave my old habbits (using Vim).

The name Subvim was a suggestions from a user at freenode, #vim channel. It was first called Subvime, however I wanted something that would remind macvim or gvim. I tried to embed all the awesome features inside the project. For example you don't have to compile any external binary for autocompletion or using the goto definition feature. Also there are many little hacks that makes Subvim a pleasure to use. It works only on Mac OS X and a Linux version is on the way.

For a more detailed information look at the project page at github.com/fatih/subvim

How to detect a bad written Go library

If the function doesn't return an error type and the error handling is made by checking the returned variable. Check the examples below.

This is bad:

variable := foo()
if value != 0 {
    log.Println("there is an error")
}

Because we don't know on the first look on which type we should check the error, here it is int, however it might be a pointer or a string:

This is good:

variable, err := foo()
if err != nil {
    log.Println(err)
}

Because we know that err is a type of error, which can be nil or not. It doesn't matter which type variable has, we are not interested. We are not overloading the variable with an error context

As you see we are not interested on the error message too, because the type err already has the error message stored (get it with err.Error() ).