Published: , by Jonathan M. Hethey,reading time: ~6 minutes

Cache-Control with GO REST APIs

In the last post we looked at how to create a command line application that stores API responses in our database. In this post we'll refactor the CLI into a REST API and add the possibility to cache-bust an already stored response.

Let's have a look at how we refactored the code by looking at the functions we have split the linear flow into:

// look up in our database
func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {}
// store a location in our database
func storeLocationRecord(record GeoResponse) error {}
// make a request to the external API
func locationRequest(city string, countryCode string) (GeoResponse, error) {}

// combining the above into one handy function
func getCityLocation(city string, countryCode string) (GeoResponse, error) {
  var record GeoResponse
  record, lookupError := lookupLocationRecord(city, countryCode)
  if lookupError != nil {
    record, requestError := locationRequest(city, countryCode)
    if requestError != nil {
      fmt.Printf("request error: %+v", requestError)
      return record, requestError
    }
    storeLocationRecord(record)
    return record, nil
  }
  return record, nil
}

This should still fulfill our expectations of the sequence diagram we included in the previous post. First, check the database, otherwise call api, store result for the next request.

Our main function gets significantly simppler, but it also needs to accomodate some code to handle the HTTP request through fiber:

func main() {

  app := fiber.New()

  app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
    city := c.Params("city")
    countryCode := c.Params("country_code")

    record, _ := getCityLocation(city, countryCode)
    return c.JSON(record)
  })

  app.Listen(":3000")
}

Cache Busting via HTTP Headers

Sometimes you have entitled clients that don't trust your cache or they might even have a good reason for requesting the most fresh copy of some data you could provide to them.

Looking at http cache headers there are quite a few options that are useful to us!

The most fitting one is no-cache, where the client requests a response that has not been cached. We'll look for that header when parsing our response and change our appllication's flow based on that header to either skip the lookup in the database or not.

Let's parse some headers and see if we can make a decision if we should rely on our cached result or not:

cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

var noCache bool
// simplified, should iterate through all headers separated by `,`
if len(cacheControlHeader) != 0 {
  if cacheControlHeader[0] == "no-cache" {
    noCache = true
  }
}

There's probably some elegant or standardised way of doing that, but now we can pass it to the function that looks up a record to skip the database part:

if noCache {
  record, requestError := locationRequest(city, countryCode)
  if requestError != nil {
    fmt.Printf("request error: %+v", requestError)
    return record, requestError
  }
  storeLocationRecord(record)
  return record, nil
}
// rest of the function

Golang and Upsert in Mongodb

Going the extra mile would obviously be to update our function to store location records to storeOrUpdate and make sure we, in case of a cache bust, store the most up to date values as well!

filter := bson.M{
	"name":    record.City,
	"country": record.CountryCode,
}

update := bson.D{
	{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

opts := options.Update().SetUpsert(true)
_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)

Luckily upserts in mongodb are something that's been well supported for a while and does not actually require two database calls. Although the syntax for inserting OR updating is a bit more cumbersome, it has its advantages.

When we make a request to our local service now, depending on if we pass the header or not, we'll get different log messages from the web server, but same results from the JSON response.

First time we query the API:

# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]

Second time:

# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
database record: {City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}

Third time with no-cache header:

# curl command:
curl curl --header "Cache-Control: no-cache" localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]

The third request successfully ignored the database record and went straight to the external API, success! We've achieved what we wanted for now, next steps include:

  • repeat the process for the weather forecast
  • possibly skip the caching of the weather response, as weather changes frequenty

Here is the full refactored code below:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"reflect"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"github.com/gofiber/fiber/v2"
)

type GeoResponse struct {
	City        string  `bson:"name" json:"name"`
	CountryCode string  `bson:"country" json:"country"`
	Lat         float64 `bson:"lat" json:"lat"`
	Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
	serverAPI := options.ServerAPI(options.ServerAPIVersion1)
	opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

	client, err := mongo.Connect(context.TODO(), opts)
	if err != nil {
		log.Fatal(err)
	}

	err = client.Ping(context.TODO(), nil)
	mongoClient = client
	return err
}

func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    city,
		"country": countryCode,
	}

	var dbRecord GeoResponse

	findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

	if findError != nil {
		fmt.Println("database lookup error:", findError)
		return dbRecord, findError
	}

	fmt.Printf("database record: %+v \n", dbRecord)
	return dbRecord, nil
}

func storeLocationRecord(record GeoResponse) error {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    record.City,
		"country": record.CountryCode,
	}

	update := bson.D{
		{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

	opts := options.Update().SetUpsert(true)
	_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
	if dbUpsertError != nil {
		fmt.Println("db insert error:", dbUpsertError)
	}
	return dbUpsertError
}

func locationRequest(city string, countryCode string) (GeoResponse, error) {
	API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"
	res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
	if err != nil {
		log.Fatal(err)
	}

	responseBody, _ := io.ReadAll(res.Body)
	defer res.Body.Close()
	var parsedResponse []GeoResponse
	jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
	if jsonErr != nil {
		fmt.Println("json parsing error:", jsonErr)
		return parsedResponse[0], jsonErr
	}

	fmt.Printf("received from external api: %+v \n", parsedResponse)
	return parsedResponse[0], nil
}

func getCityLocation(city string, countryCode string, noCache bool) (GeoResponse, error) {
	var record GeoResponse
	if noCache {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	record, lookupError := lookupLocationRecord(city, countryCode)
	if lookupError != nil {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	return record, nil
}

func main() {
	dbError := connect_to_mongodb()
	if dbError != nil {
		fmt.Printf("database connection error: %+v", dbError)
	}

	app := fiber.New()

	app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
		city := c.Params("city")
		countryCode := c.Params("country_code")
		cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

		var noCache bool
		if len(cacheControlHeader) != 0 {
			if cacheControlHeader[0] == "no-cache" {
				noCache = true
			}
		}

		// simple validation of parameters
		cityRef := reflect.ValueOf(city)
		countryCodeRef := reflect.ValueOf(countryCode)
		if cityRef.Kind() != reflect.String || countryCodeRef.Kind() != reflect.String {
			return c.Status(400).SendString("parameter error")
		}

		// get location
		record, _ := getCityLocation(city, countryCode, noCache)
		return c.JSON(record)
	})

	app.Listen(":3000")
}
Tagged with: #go #golang #mongodb #gofiber

Thank you for reading! If you have any comments, additions or questions, please tweet or toot them at me!

Subscribe via rss, atom feed.json