I had been putting off some gnarly dependency upgrades for over a year and finally got around to it last week, after a few false starts of the npm variety. I eventually gave up and rebuilt from scratch using the gatsby new tool, then ported my customizations. After I finished my polishes to some assets, I noticed that the changes didn't seem to have applied on forcepush.tech. This wasn't very mysterious, as I've since put a Cloudflare cache in front of my site.

As a quick refresher, my website stack is:

• Gatsby site in a GitHub repo...
• deployed to Netlify on push to main...
• cached on Cloudflare edge with a roughly 2-hour TTL (which doesn't seem to be configurable at the free tier)

I found a button on my Cloudflare dashboard to purge the site cache manually, so I used it and my changes became visible. I noticed the API help tip too, which would allow a way to automate this each time my site builds on Netlify. The API endpoint is pretty straightforward: just

Copycopy code to clipboard
1POST https://api.cloudflare.com/client/v4/zones/:identifier/purge_cache

with headers for "Authorization: Bearer TOKEN", "Content-Type", and no data required in the request. The only problem that remained was how to automatically trigger it.

Design Goals

• Cache the latest deploy as soon as it completes
• Purge and rebuild the cache automatically
• Don't purge needlessly, because I assume that rebuilding and propagating an unchanged cache to Cloudflare's edge is compute-expensive
• Don't pay for anything

Options

In my Netlify deploy config, I had a couple of options for triggering the Cloudflare endpoint: I could either append a curl to my build command or I could use a post-deploy webhook, which sends a POST to an arbitrary URL.

The curl approach seemed a bit kludge-y since it has to be &&-chained with an already lengthy rm -rf public/jidicula-resume && npm run build command, and I didn't like the idea of using a single-line text field for a multi-command script. This also wouldn't strictly be a post-deploy cache purge, because this is the build command config that runs at the beginning of a deploy run, not the end.

Hitting the purge endpoint before the deploy begins opens up 2 failure modes that negate the benefits of automating a purge:

• the cache gets purged before deploy and the deploy subsequently fails: the cache is rebuilt using the last successful deploy -> no-change cache rebuild
• the cache gets purged before deploy and the deploy is successful, but slow: the cache is rebuilt before the deploy completes, so it still contains the last deploy's stale content -> no-change cache rebuild

These reasons left me with the post-deploy POST hook approach... unfortunately Netlify doesn't allow custom headers with its POST hook and can only authenticate via JWS, so it can't meet Cloudflare's purge_cache API specification.

To solve this POST mismatch, I searched and found Brian Li's blogpost about using a serverless cloud function as middleware: upon receiving a POST to its trigger endpoint, send a POST request to Cloudflare to the cache-purge endpoint. Of course, I opted to do it in Go instead of Python: it would have an even tinier memory footprint, better performance without any tuning, and the speedy compilation would yield a faster function build.

Implementation

Here's my Go implementation:

Copycopy code to clipboard
1package purger
2
3import (
4	"fmt "
5	"io"
6	"log"
7	"net/http"
8	"strings"
9
10	"github.com/GoogleCloudPlatform/functions-framework-go/functions"
11)
12
13func init() {
14	functions.HTTP("PurgeCache", purgeCache)
15}
16
17// httpError logs the error and returns an HTTP error message and code.
18func httpError(w http.ResponseWriter, err error, msg string, errorCode int) {
19	errorMsg := fmt.Sprintf("%s: %v", msg, err)
20	log.Printf("%s", errorMsg)
21	http.Error(w, errorMsg, errorCode)
22}
23
24func purgeCache(w http.ResponseWriter, r *http.Request) {
25
26	log.Printf("Received %s from %v", r.Method, r.RemoteAddr)
27	if r.Method == "POST" {
28		body, err := io.ReadAll(r.Body)
29		if err != nil {
30			httpError(w, err, "error reading POST body", http.StatusInternalServerError)
31			return
32		}
33		log.Printf("Request body: %s", body)
34	}
35	// Send POST request to Cloudflare
36	client := &http.Client{}
37
38	data := `{"purge_everything":true}`
39	req, err := http.NewRequest("POST",
40                                "https://api.cloudflare.com/client/v4/zones/ZONE_ID/purge_cache", 
41                                strings.NewReader(data))
42	if err != nil {
43		httpError(w, err, "error creating new Request", http.StatusInternalServerError)
44		return
45	}
46
47	req.Header.Add("Authorization", "Bearer CLOUDFLARE-API-TOKEN")
48	req.Header.Add("Content-Type", "application/json")
49
50	cloudflareResp, err := client.Do(req)
51	if err != nil {
52		httpError(w, err, "error sending POST request", http.StatusInternalServerError)
53		return
54	}
55	defer cloudflareResp.Body.Close()
56
57	// Pass cloudflare response to caller
58
59	cloudflareRespBody, err := io.ReadAll(cloudflareResp.Body)
60	if err != nil {
61		httpError(w, err, "error reading Cloudflare response", http.StatusInternalServerError)
62		return
63	}
64
65	if cloudflareResp.StatusCode != http.StatusOK {
66		msg := fmt.Sprintf("error non-200 status: %s", cloudflareRespBody)
67		httpError(w, nil, msg, http.StatusInternalServerError)
68		return
69	}
70
71	log.Printf("Cloudflare response: %s", cloudflareRespBody)
72	_, err = w.Write(cloudflareRespBody)
73	if err != nil {
74		httpError(w, err, "error sending response to client", http.StatusInternalServerError)
75		return
76	}
77}

• There's some Google Cloud Functions boilerplate required: the init() and its functions.HTTP call that registers the function that's invoked.
• The invoked function seems to require receiving http.ResponseWriter and *http.Request in its parameters (I messed around to see if they could be omitted, as the documentation for the 2nd-gen Cloud Functions isn't exactly complete).
  • As usual in Go, I use *http.Client and http.NewRequest() for adding custom headers to a HTTP request - the steps are to create Request with NewRequest and pass it to client to send.
• I use all the builtin error codes that I can for handling various failure modes and informing the caller that something went wrong. Also for convenience, I factored out the usual log.Printf() & http.Error() calls into a httpError() function.
• Regardless of Cloudflare's response, the function forwards it back to the caller.
• And as a final polish, I log where the request came from and its body if it's a POST (which should only come from Netlify). Google Cloud Functions can be HTTP-triggered with any of POST, PUT, GET, DELETE, or OPTIONS requests.

For the Google Cloud Function config, I used the 2nd-gen Cloud Functions since it uses the Artifact Registry for storing the function image, and Artifact Registry has a free tier (1st-gen Cloud Functions use the Container Registry, which costs money). Additional configs:

• Memory allocated: 128 MiB (the lowest option possible)
• Timeout: 60 seconds (default)
• Autoscaling: 0 instance minimum to 1 instance maximum (only need 1 run instance for a cache purge)
• Region: us-east4 (this is in North Virginia, the same place where AWS's major us-east region is - Netlify is hosted on AWS so hopefully this reduces some latency)

Downside

The main downside of this approach is that it relies on security through obscurity - the trigger endpoint has to be publicly exposed for Netlify to be able to POST to it. In the worst case, the endpoint could get slammed with malicious requests, but end result is probably ok - I limited the function to 1 invocation at a time, so this could act as a throttle. If malicious requests become a problem, I'll add an early-return check to the function preamble for the request origin or content, or I might even work up some check that Netlify's hook can authenticate against using JWS.

Summary

Overall, I'm pretty satisfied with this cache-purging solution - it meets all my design goals, and it's fast:

• ✅ Cache the latest deploy as soon as it completes
• ✅ Purge and rebuild the cache automatically
• ✅ Don't purge needlessly, because I assume that rebuilding and propagating an unchanged cache to Cloudflare's edge is compute-expensive
• ✅ Don't pay for anything

If you have any questions or comments, email me at [email protected], find me on Twitter @jidiculous, or post a comment here.

Did you find this post useful? Buy me a beverage or sponsor me here!