Caching with Golang and mongodb

Sometimes we rely on third party APIs where every request costs us money or have some strict rate limits. To mitigate the financial impact of a project and to have a bit of fun, we can add some caching measures to not call other APIs before either a cache expires or to keep copies of external data sources forever.

Fun fact: The API we're using let's us query 1_000_000 times a month before charging us, but let's pretend it's really expensive and limited.

Example Application: Locations and Weather

One of my use cases was to report the weather for a specific location. There's a pretty good weather API at [openweathermap.org][openweathermap].

Reading the API documentation we'll need to make 2 calls for every weather request, since the endpoint that returns the weather deals with latitude and longitude.

1. we get a user input (probably a city) and need to find its geographical location in lat/lon /api/geocoding-api
2. the actual request to get the current weather /current

Humble Beginnings: CLI API Requests

Let's start with the simplest application we can think of to validate that a part of it works that we'll need later on. We'll make an API request to locate a city and print the results:

package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
)

func main() {

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh" // your api key here

  city := "Berlin"
  countryCode := "DE"

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }

  defer res.Body.Close()

  body, err := io.ReadAll(res.Body)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(string(body))
}

Running this code, we get the following output:

go run main.go

[
  {
    "name":"Berlin",
    "local_names": {"bg":"Берлин","io":"Berlin"},
    "lat": 52.5170365,
    "lon": 13.3888599,
    "country": "DE"
  }
]

Sweet! Now let's go ahead and store that response in a database!

Database Setup

The natural place to save some data we want to keep readily available would of course be the database of our choice.

Let's create a docker-compose file to make sure we can easily spin up a database during development and decide on which shape we want to save the data in. We probably want to keep the data we're interested in and the parameters we want to query by in separate fields.

We don't need to store the full API response or we can stringify the data we would only need sporadically.

# file: docker-compose.yml
version: "3.8"
services:
  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"

Next we're going to write the response we get from the fun weather api to our own database when we run the script:

package main

import (
  "context"
  "encoding/json"
  "fmt"
  "io"
  "log"
  "net/http"

  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)

type GeoResponse struct {
  City        string  `bson:"name" json:"name"`
  CountryCode string  `bson:"country" json:"country"`
  Lat         float64 `bson:"lat" json:"lat"`
  Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
  serverAPI := options.ServerAPI(options.ServerAPIVersion1)
  opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

  client, err := mongo.Connect(context.TODO(), opts)
  if err != nil {
    log.Fatal(err)
  }

  err = client.Ping(context.TODO(), nil)
  mongoClient = client
  return err
}

func main() {
  dbError := connect_to_mongodb()
  if dbError != nil {
    fmt.Printf("database connection error: %+v", dbError)
  }

  collection := mongoClient.Database("weatherdb").Collection("cities")

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"

  city := os.Args[1]
  countryCode os.Args[2]

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }


  // parse api response to json
  responseBody, _ := io.ReadAll(res.Body)
  defer res.Body.Close()
  var parsedResponse []GeoResponse
  jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
  if jsonErr != nil {
    fmt.Println("json parsing error:", jsonErr)
  }

  fmt.Printf("%+v", parsedResponse)
  // save first record in array to the database
  dbResult, dbInsertError := collection.InsertOne(context.TODO(), parsedResponse[0])
  if dbInsertError != nil {
    fmt.Println("db insert error:", dbInsertError)
  }

  fmt.Printf("%+v", dbResult)
}

after adding the mongodb dependency to our file, we can run the following commands to try it out:

# in a separate terminal window
docker-compose up

# then in our project:
go mod init
go mod tidy
go run main.go

and we should have a record in our mongodb database weather:

or via the mongodb shell:

use weather;
db.cities.find();

{
  _id: ObjectId('666b4a7ba2febc2c80e3ec5d'),
  name: 'Berlin',
  country: 'DE',
  lat: 52.5170365,
  lon: 13.3888599
}

The problem we now face is, that if we simply run the script again with the same city and country code, we simply would

1. call the weather api (again)
2. save an identical database record (again)

To avoid this we need to first look up in our database IF we have a record that matches something that makes this unique, for example the combination of city and country.

Fun fact: Technically these are not unique, just look at the amount of Springfields in the US, you will not be disappointed. We'll just pretend these are unique enough for now.

Check the DB before API call

Let's first check our database for information we might have retrieved already, like described in this sequence diagram:

sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: Nope
    our Program->>+WeatherAPI: Hey, what's the location of Berlin?
    WeatherAPI->>-our Program: "lat: 52.5170365, lon: 13.3888599"
    our Program->>+Database: "Berlin is at lat: 52.5170365, lon: 13.3888599"

Or in the case that we do indeed already have a database record we skip the whole weather api bit!

sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: "Yup, it's at: 52.5170365, lon: 13.3888599"

Golang mongodb.find and parse to struct

Mostly of what we need to add is a quick call to our database first and returning early from the main function:

filter := bson.M{
  "name":    city,
  "country": countryCode,
}

var dbRecord GeoResponse

findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

if findError != nil {
  fmt.Println("database lookup error:", findError)
}

if findError == nil {
  fmt.Printf("database record: %+v \n", dbRecord)
  return
}

Now if we run our program again, we should see this output, call no external API and not create any duplicate records in the database 🙌.

database record: {City:Berlin CountryCode:DE Lat:52.5170365 Lon:13.3888599}

In the next part we'll look at invalidating some records automatically (locations of city don't often change, but the weather does sometimes) and how to serve all of this through a REST API that fits our needs.