Private Golang Registry in Artifactory

The backend of my latest project consists of several microservices written in Go. Each service is built in Jenkins and then deployed to the Openshift Cloud Platform. The one problem we have is that this Jenkins is not allowed to communicate with servers outside of the companie’s network.

Therefore, we had to find out how we manage the vendor packages for our Go microservices.

Our first approach was to create a vendor directory with all dependencies. This vendor package was added to each Git repository. The downside is that you have to administrate these vendor packages and that the download size of the Git repository increases drastically. Also, several services require the same dependencies, so you end up downloading the same module several times.

Our second approach was to centralize the vendor package in our microservice-commons package. This Git repository holds code which is used by several or all microservices (authentication, authorisation, database methods, etc.). What we did is to put all dependencies into a subdirectory of commons.

We had to adjust each microservice’s Dockerfile so that it fetches the code and its dependencies from the microservice-commons cache.

ENV REPO https://bitbucket.com/scm/abcd/microservice-commons.git
ENV BRANCH develop

RUN git clone --depth 1 --single-branch --branch $BRANCH $REPO

First, we set two variables representing the Git repository and the branch we want to clone. Finally, we make a shallow clone to get the microservice-commons repository into the Docker build.

To give you a full picture, this go.mod file exemplarily show how we integrate the microservice-commons code into each microservice. go.mod is a metadata file that describes a Go package. It contains the package’s module name and a list of its dependencies.

module abcd/gateway-service

go 1.13

require (
	github.com/labstack/echo v3.3.10+incompatible
	abcd/microservice-commons v0.0.0
)

replace abcd/microservice-commons => ../microservice-commons

This setup worked well for a while but we actually wanted a setup similar to our frontend project where dependencies are stored in Artifactory. The virtual repository aggregates packages from both local and remote repositories. It serves as a cache for the remote NPM registry npmjs.org.

This brings me to our third approach.

Private Go Registry

Set up your own private Go registry in Artifactory by following the steps as desribed in their official how-to-guide. A remote Go repository in Artifactory serves as a caching proxy for github.com, golang.org, or a Go repository in a different Artifactory instance.

Artifactory requires Go version 1.11.0 and above. To check what version of Go you have, run go version. Download the latest stable release from the official download site.

Similar to other package manager, go has its local package cache. For me, it is located at my user directory at \go\pkg\mod\cache\download. Whenever you run go mod download or go build, the dependencies are stored and looked up from there.

Local Go package download cache

Create an archive (zip) for each of the containing directories including sumdb. This directory contains checksum files for all modules which are usually retrieved automatically from sum.golang.org.

Go to Artifactory and click on Deploy in the upper right corner. Select one of the zip files. Select ” Deploy as Bundle Artifact” and click “Deploy”.

Deploy zip file to Artifactory

Repeat this for all zip files (sumdb and remote repository folders). Finally, the directory structure of your Artifactory repository should look something like this.

Artifactory repository structure after deployment

At last, you have to set the GOPROXY variable, so that go mod download and other commands know where you fetch the dependencies from.

export GOPROXY=https://artifactory.intra.my-company.com/artifactory/api/go/go-cache-virtual

In case you are using a Dockerfile for your CI build, also add the GOPROXY variable there.

ENV GOPROXY=https://artifactory.intra.my-company.com/artifactory/api/go/go-cache-virtual
ENV GONOSUMDB=off

Now, your build fetches the dependencies from your private repository and does not longer have to contact any remote regristries.