Go Package CI/CD with GitHub Actions

2021-05-15 — Go, CI/CD, GitHub Actions — 9 min read

In a previous post, I wrote about how I implemented CI/CD checks and autoreleases for the Python implementation of my random-standup program. I also developed some similar workflows for the Go implementation, so I thought I'd also write a Go-flavoured post about packaging CI/CD using GitHub Actions. This post may seem very familiar if you read that previous post - as I described in my comparison between the Go and Python implementations of this program, my CI/CD goals are the same: PR checks and autoreleases.

As I said before, I wanted to ensure that:

Each change I make to my program won't break existing functionality (Continuous Integration), and
Publishing a new release to pkg.go.dev is automatic (Continuous Delivery/Deployment).

GitHub provides a workflow automation feature called GitHub Actions. Essentially, you write your workflow configurations in a YAML file in your-repo/.github/workflows/, and they'll be executed on certain repository events.

Continuous Integration

This automation is relatively straightforward. I want to run the following workflows on each commit into the repository trunk and on each pull request into trunk:

Test syntax by running a linting check with golangci-lint - it's the best linter (actually, I suppose it's a meta-linter since it invokes several separate linters) available for Go and slaps your wrist if you slip into some well-known antipatterns.
Test functionality by running automated unit tests on the entire program. This is an extremely simple program, so I definitely overengineered its factoring into functions to make it easier to unit test.
Test build stability by attempting to build the program (but discarding the build artifact) across as many OS and arch combinations supported by Go. Of course, I don't expect that anyone would run my standup randomizer using Plan 9 on an ARM chip, but this was more of an exercise to learn about Go's cross-compilation capabilities.

Here's the full workflow.

Each commit to trunk

The trigger for this is declared at the top of the workflow file:

1on:
2  push:
3    branches: [main]
4  pull_request:
5    branches: [main]

Test syntax by checking formatting

First, we have to checkout the repository in GitHub Actions using GitHub's own checkout action. Then, we have to set up the Go version using GitHub's setup-go action. GitHub Actions has 3 different OSes available for their runners, each with various Go versions, but it's safest to explicitly specify which Go version will be used.

Finally, we can use golangci-lint's provided GitHub Action for linting - it runs golangci-lint on the workflow runner's clone of the repo and outputs an error code if any Go file in the repo fails rules of any linters in golangci-lint. Note that golangci-lint fails if the AST cannot be parsed (i.e. if there are any syntax errors), so it can also be used for checking syntax correctness, which itself is a good proxy for checking for merge conflict strings. We can fail-fast with any checks this way - there's no need to spin up a compilation and a go test invocation if there are syntax errors.

1jobs:
2  lint:
3    name: Lint files
4    runs-on: 'ubuntu-latest'
5    steps:
6      - uses: actions/[email protected]
7      - uses: actions/setup-go@v2
8        with:
9          go-version: '1.16.4'
10      - name: golangci-lint
11        uses: golangci/golangci-lint-[email protected]
12        with:
13          version: latest

Test Functionality

Again, we need to checkout the repo for this job and set up the Go version:

1name: Run tests
2    runs-on: 'ubuntu-latest'
3    needs: lint
4    steps:
5      - uses: actions/[email protected]
6      - uses: actions/setup-go@v2
7        with:
8          go-version: '1.16.4'
9      - run: go test -v -cover

Note that unlike Python, no setup is needed to install dependencies (go test automatically grabs dependencies defined in go.mod) or set up a virtual environment, so there's a lot less boilerplate in CI/CD.

Test build stability for different OSes and architectures

Go provides cross-compilation tooling for a wide variety of operating systems and architectures. Essentially, you can run a command like

1$ GOOS=plan9 GOARCH=arm go build

and the Go compiler will build a binary that will run on the OS specified in GOOS and the arch in GOARCH. To see the full list of GOOS and GOARCH options, run go tool dist list.

We want to verify build stability across this set, so we can set up a matrix build for different GOOS and GOARCH options using GitHub Actions:

1build:
2    runs-on: 'ubuntu-latest'
3    needs: test
4    strategy:
5      matrix:
6        goosarch:
7          - 'aix/ppc64'
8          - 'android/amd64'
9          - 'android/arm64'
10          - 'darwin/amd64'
11          - 'darwin/arm64'
12          - 'dragonfly/amd64'
13          # ...

This is defined in the jobs.<job_id>.strategy.matrix directive. I've added just 1 variable for every GOOS and GOARCH pairing (truncated for this blogpost - there are 39 pairs defined in my workflow file).

Internally, the steps are somewhat like:

GitHub Actions parses the directives for the job and sees there's a matrix strategy.
It spins up a separate runner for each matrix combination and defines the variables matrix.goosarch as the values for that combination.
It runs the job steps in each runner it spun up in Step 2.

You can see an example of how this matrix run looks like in the GitHub Actions console here (see all the goosarch values in the left sidebar). These matrix options are run in parallel by default, so the runtime of the job determined by the slowest matrix option. Note that if your repository is private, you will be charged Actions minutes for each separate build matrix option, with some hefty multipliers for macOS and Windows runners (1 macOS minute is 10 minutes of Actions credit, 1 Windows minute is 2 minutes of Actions credit as of May 2021).

We do our usual checkout and Go version setup, then some basic Bash string-splitting on the / character so we can set the GOOS and GOARCH environment variables separately from a single matrix option:

1- name: Get OS and arch info
2        run: |
3          GOOSARCH=${{matrix.goosarch}}
4          GOOS=${GOOSARCH%/*}
5          GOARCH=${GOOSARCH#*/}
6          BINARY_NAME=${{github.repository}}-$GOOS-$GOARCH
7          echo "BINARY_NAME=$BINARY_NAME" >> $GITHUB_ENV
8          echo "GOOS=$GOOS" >> $GITHUB_ENV
9          echo "GOARCH=$GOARCH" >> $GITHUB_ENV

Then, we simply run Go's go build subcommand, which creates the binary:

1- name: Build
2        run: |
3          go build -o "$BINARY_NAME" -v

Auto-merge

GitHub also allows pull requests to be merged automatically if branch protection rules are configured and if the pull request passes all required reviews and status checks. In the repo Settings > Branches > Branch Protection rules, I have a rule defined for main requiring all jobs in the build.yml workflow to pass before a branch can be merged into main.

Release automation

There are 2 parts to GitHub release automation:

Create the GitHub release using Git tags and add the build artifacts to it (workflow).
Publish the package to pkg.go.dev (workflow).

Create GitHub Release

We set up the workflow to trigger on push to a tag beginning with v:

1on:
2  push:
3    # Sequence of patterns matched against refs/tags
4    tags:
5      - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10

Then, we define our release job, running on Ubuntu (cheapest and fastest GitHub Actions runner environment):

1name: Create Release
2
3jobs:
4  autorelease:
5    name: Create Release
6    runs-on: 'ubuntu-latest'

I also set up the same GOOS and GOARCH build matrix as in build.yml - when we create the GitHub release, we'll build and upload the binaries as release assets.

Our first 2 steps are almost the same as our Build workflow for pushes and PRs to main: we checkout the repo and set up Go. Our checkout step is slightly different, though: we provide 0 to the fetch-depth input so we make a deep clone with all commits, not a shallow clone with just the most recent commit.

1steps:
2      - name: Checkout code
3        uses: actions/checkout@v2
4        with:
5          fetch-depth: 0

Go specifies module versions using version control tagging, so we don't need to parse any manifest files like we did with Python. So, we can do the same Bash string splitting as before and build the binary:

1- name: Get OS and arch info
2        run: |
3          GOOSARCH=${{matrix.goosarch}}
4          GOOS=${GOOSARCH%/*}
5          GOARCH=${GOOSARCH#*/}
6          BINARY_NAME=${{github.repository}}-$GOOS-$GOARCH
7          echo "BINARY_NAME=$BINARY_NAME" >> $GITHUB_ENV
8          echo "GOOS=$GOOS" >> $GITHUB_ENV
9          echo "GOARCH=$GOARCH" >> $GITHUB_ENV
10      - name: Build
11        run: |
12          go build -o "$BINARY_NAME" -v

The next step is to create some release notes. I keep a release template in the .github folder and append some gitlog output to it:

1- name: Release Notes
2        run: git log $(git describe HEAD~ --tags --abbrev=0)..HEAD --pretty='format:* %h %s%n  * %an <%ae>' --no-merges >> ".github/RELEASE-TEMPLATE.md"

That gnarly gitlog command is checking all commits since the last tag to HEAD. For each commit, it appends the commit hash, the commit message subject, the author name, and the author email to the release template.

Finally, we use a 3rd-party release creation Action for creating a release draft with the release notes and artifacts we just created:

1- name: Release with Notes
2        uses: softprops/action-gh-release@v1
3        with:
4          body_path: ".github/RELEASE-TEMPLATE.md"
5          draft: true
6          files: ${{env.BINARY_NAME}}
7        env:
8          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This creates a draft visible at https://github.com/jidicula/random-standup/releases. I modify the release announcements as needed, and publish the release.

Publishing to pkg.go.dev

The final step of the release process is to notify pkg.go.dev that there's a new version available for the module. Here's the full workflow.

This time, we trigger the workflow to run on a release being published (the last step of the previous workflow is manually publishing a release draft):

1on:
2  release:
3    types:
4      - published

We do the same checkout as before. Then, we simply run curl to the URL where the module is fetched from by go get:

1jobs:
2  bump-index:
3    runs-on: ubuntu-latest
4    steps:
5      - name: Checkout repo
6        uses: actions/[email protected]
7      - name: Ping endpoint
8        run: curl "https://proxy.golang.org/github.com/jidicula/random-standup/@v/$(git describe HEAD --tags --abbrev=0).info"

pkg.go.dev recommends this as one of the ways of adding a new module (or module version) to its index.

Putting it all together

So overall, working on this project would involve:

Make a PR for my changes.
Confirm auto-merge.
Repeeat Steps 1 and 2 until I'm ready to release.
Create a tag on main pointing to the version bump commit.
Push the tag to GitHub.
Wait for the Create Release run to finish.
Go to https://github.com/jidicula/random-standup/releases and modify the Announcements for the just-created release draft.
Publish the release.
Wait for the Publish run to finish.
Check pkg.go.dev for the updated package version.

If you have any questions or comments, email me at [email protected], find me on Twitter @jidiculous, or post a comment here.

Did you find this post useful? Buy me a beverage or sponsor me here!