Fatih Arslan

My thoughts about Programming, Coffee, Bags and various other stuff

My journey about implementing an HCL parser in pure Go

One of my new side projects was about a scanner/parser family written in pure Go for the Hashicorp Configuration Language. I had two reasons for it, the first one was to have fun and learn the internals of Go’s own parser family (go/{ast,token,scanner,parser}) and the second one was to have a hclfmt command, just like gofmt, which would format an HCL file based on predefined set of rules. (The original HCL parser was generated with Yacc. It works totally fine, but it was not as flexible as compared to an handwritten parser)

I’ve successfully finished this project and announced it on Twitter last week:

Having the new parser family also allowed me to create the hclfmt command. It can be called from the command line or integrated into and editor (via a save-hook). This is how it looks in action:

I’ve learned a lot of things while implementing the packages. Some parts were really easy and fun, but some details really was challenging. Here are my thoughts:

The new HCL parser was received well by the community. Some of the Hashicorp employees wanted to talk with me about possible integration and replace it with their current parser.

In the end Mitchell himself (CEO of Hashicorp) contacted me about how we could move forward with this parser. I’ve decided to be open and we agreed on moving it to the official HCL repository: github.com/hashicorp/hcl This means that the new parser will be used by all related Hashicorp products (such as Terraform, Nomad, Otto, etc..) and by hundreds of thousands users.

All my commits were moved and merged to the official repo and a thanks statement was added on behalf on me. He also announced it from Twitter. In the end I’ve learned a lot and made a significant contribution to the open source community.

If you have any questions or feedback, please feel free to share them with me on Twitter: @fatih