
In Golang, importing modules is easy, however it has some opinions on how to structure your project. You're not supposed to sort the files of your project into different directories, but to keep your files in the root folder of your repository OR split out functionality into other modules.

However, it's not a cop, so you can still use the `package` field and an import path to import modules from a sub-directory.

## Go Idiomatic Directory Example

Let's create an ice cream shop that makes use of an `icecream.go` model and some basic price calculation.

`main.go`

```go
package main

import (
	"fmt"
)

func main() {
	myIcecream := Icecream{Flavour: "vanilla", Scoops: 3, BasePrice: 2.99}
	fmt.Println(myIcecream.Price())
}
```

`icecream.go`

```go
package main

type Icecream struct {
	Flavour   string
	Scoops    int
	BasePrice float32
}

func (i Icecream) Price() float32 {
	return i.BasePrice * float32(i.Scoops)
}
```

**directory structure:**

```
.
├── main.go
└── icecream.go

1 directory, 2 files
```

We can confirm that this works by running `go run main.go icecream.go` and it should print: `8.97`.

## Importing Go Files from Sub-directory Example

If we should decide to structure our projects with sub-directories, we need to make some changes. First of all, we should make sure to have a `go.mod` file that states the name of our package, so let's run `go mod init`, which will by defautl take the name of the directory as the module name.

`go.mod`

```
module icecream-shop

go 1.23.1
```

Next we can create a directory, like `models` and move our `icecream.go` file there.

**directory structure:**

```
.
├── go.mod
├── main.go
└── models
    └── icecream.go

2 directories, 3 files
```

If we try to run our code with `go run .` now, we'll get the following error:

```
# icecream-shop
./main.go:8:16: undefined: Icecream
```

Alright, let's now try to import it, we need to remember that unlike in for example JavaScript, we don't import specific files but modules, which are directories:

`main.go`

```go
package main

import (
	"fmt"
	"icecream-shop/models"
)

func main() {
	myIcecream := models.Icecream{Flavour: "vanilla", Scoops: 3, BasePrice: 2.99}
	fmt.Println(myIcecream.Price())
}
```

Note:

* add `"icecream-shop/models"`
* change `Icecream` to `models.Icecream`

However, we get another error if we try to execute it with `go run .`:

```
main.go:5:2: import "icecream-shop/models" is a program, not an importable package
```

Lastly, we'll need to change the package name of the file we moved into the sub-directory, otherwise we'll not be able to import it and use it:

`icecream.go`

```go
package models

type Icecream struct {
	Flavour   string
	Scoops    int
	BasePrice float32
}

func (i Icecream) Price() float32 {
	return i.BasePrice * float32(i.Scoops)
}
```

Note:

* we changed `package main` to `package models`, signalling that this is its own module

## Summary

If you're structuring your project, evaluate carefully if the functionality you're moving around should in fact be its own package or repository.

If not, remember to use the correct import paths.


In Golang, importing modules is easy, however it has some opinions on how to
structure your project...

Golang: Import Module from Sub Directory


Caddy is a web server that boasts with easy configuration and great defaults, such as automatic HTTPS support by default. While I didn't need https, I was curious about how configuring some redirects and static file serving would go. You can find out more about caddy at [caddyserver.com][caddyserver.com].

If you want to get the **full source** for this repository, the link is at the bottom.

As so often I needed redirects, as I was moving one of my websites to become a static site. Inside my docker container I would have a rendered directory structure that has the paths of the html files and of course some assets.

While migrating away from a CMS, I knew that I would break some of the links on the website. Blog posts like `/peaches` would become `/blog/peaches`, so redirects were important to me, optimally in an external file that I can load into the main config.

## Caddy in the Dockerfile

The simplest way in which you can use caddy in a dockerfile to serve static assets might be this:

```docker
FROM alpine
EXPOSE 5000
RUN apk add --no-cache caddy
RUN mkdir -p /var/www/html
COPY pages/ /var/www/html/
WORKDIR /var/www/html
CMD ["caddy", "file-server", "--listen", ":5000"]
```

Given we have some html files in the `pages` directory like so:

```
pages
├── blog
│   ├── eggplant
│   │   └── index.html
│   ├── ice-cream
│   │   └── index.html
│   └── peaches
│       └── index.html
└── index.html
```

we can serve the files by building the Dockerfile and running it:

```bash
docker build . -t caddy_test
docker run -i -p 127.0.0.1:5000:5000 -t caddy_test
```

Now visiting [localhost:5000](http://localhost:5000) we can see our HTML files.

For me it looks like this with some very minimal example HTML:

![ladybug browser showing example page](/assets/2024/caddy-docker-index.png)

While this will serve our files, we're missing out on the redirects we need to apply.

## Load your Caddyfile

The common config file for a caddy server is the `Caddyfile`, so let's create that one:

```caddy
# file: caddy-config/Caddyfile
# global directive
{
  # turn off https, we're in a trusted environment
  auto_https off
}

:5000 {
  # import redirects
  import /var/app/caddy-config/redirects

  # set the root for our served files
  root * /var/www/html

  # Enable the static file server.
  file_server {
    precompressed zstd br gzip
  }
}
```

This one will also listen on port `5000` and through the `file_server` directive serve the files with some compression enabled.

We can place the file in a sub-directory like `caddy-config` if we don't want to clutter the root of our project.

Now to add redirects we could insert a few lines above the `root` directive, like so:

```caddy
redir /ice-cream /blog/ice-cream permanent
```

### Loading an External Redirects File

For the redirects, we're creating a separate file, which I chose to just call `redirects` without any specific file ending.

```caddy
# redirect to front page
redir /tag/* / permanent
redir /author/* / permanent

# old post structure from /[name] to /blog/[name]

# exact match
redir /ice-cream /blog/ice-cream permanent

# match path and path with trailing slash
redir /peaches /blog/peaches permanent
redir /peaches/ /blog/peaches permanent

# match prefix, every route starting with /eggplant will be redirected
redir /eggplant* /blog/eggplant permanent
```

Please note that depending on how you want your redirects to work you might either want to match explicit paths, or use some of the wildcards you can pick up in the [caddy matcher documentation](https://caddyserver.com/docs/caddyfile/matchers#path-matchers).

To check if everything works to our satisfaction, we can run `docker build` and `docker run` again:

```bash
docker build . -t caddy_test
docker run -i -p 127.0.0.1:5000:5000 -t caddy_test
```

We should see output similar to this:

```
# [...] redacted
Successfully tagged caddy_test:latest
2024/06/16 11:54:18.241 INFO    using provided configuration    {"config_file": "/var/app/caddy-config/Caddyfile", "config_adapter": "caddyfile"}
2024/06/16 11:54:18.241 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "/var/app/caddy-config/Caddyfile", "line": 1}
2024/06/16 11:54:18.242 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/06/16 11:54:18.242 WARN    http.auto_https automatic HTTPS is completely disabled for server       {"server_name": "srv0"}
2024/06/16 11:54:18.242 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0001f6200"}
2024/06/16 11:54:18.242 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/06/16 11:54:18.242 WARN    tls     unable to get instance ID; storage clean stamps will be incomplete      {"error": "open /root/.local/share/caddy/instance.uuid: no such file or directory"}
2024/06/16 11:54:18.242 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/06/16 11:54:18.242 INFO    serving initial configuration
2024/06/16 11:54:18.243 INFO    tls     cleaning storage unit   {"storage": "FileStorage:/root/.local/share/caddy"}
2024/06/16 11:54:18.243 INFO    tls     finished cleaning storage units
```

Now when visiting the example links we can see that we're being redirected 🙌!

![firefox showing redirect](/assets/2024/caddy-redirect-firefox.png)

## Summary

This was a very short introduction to redirects in caddy and how to load a config file with imported redirects from an external file. Thank you for reading! Let me know what you're using caddy for 👀!

Full source for this example can be found here: [github.com/JonathanMH/stream-kitchensink/tree/master/caddy-docker](https://github.com/JonathanMH/stream-kitchensink/tree/master/caddy-docker)


[caddyserver.com]: https://caddyserver.com/


Caddy is a web server that boasts with easy configuration and great defaults,
such as automatic HTT...

Caddy inside Docker with Redirects


In the last post we looked at [how to create a command line application that stores API responses in our database](/p/caching-with-golang-and-mongodb/). In this post we'll refactor the CLI into a REST API and add the possibility to cache-bust an already stored response.

Let's have a look at how we refactored the code by looking at the functions we have split the linear flow into:

```go
// look up in our database
func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {}
// store a location in our database
func storeLocationRecord(record GeoResponse) error {}
// make a request to the external API
func locationRequest(city string, countryCode string) (GeoResponse, error) {}

// combining the above into one handy function
func getCityLocation(city string, countryCode string) (GeoResponse, error) {
  var record GeoResponse
  record, lookupError := lookupLocationRecord(city, countryCode)
  if lookupError != nil {
    record, requestError := locationRequest(city, countryCode)
    if requestError != nil {
      fmt.Printf("request error: %+v", requestError)
      return record, requestError
    }
    storeLocationRecord(record)
    return record, nil
  }
  return record, nil
}
```

This should still fulfill our expectations of the sequence diagram we included in the previous post. First, check the database, otherwise call api, store result for the next request.

Our `main` function gets significantly simppler, but it also needs to accomodate some code to handle the HTTP request through fiber:

```go
func main() {

  app := fiber.New()

  app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
    city := c.Params("city")
    countryCode := c.Params("country_code")

    record, _ := getCityLocation(city, countryCode)
    return c.JSON(record)
  })

  app.Listen(":3000")
}
```

## Cache Busting via HTTP Headers

Sometimes you have entitled clients that don't trust your cache or they might even have a good reason for requesting the most fresh copy of some data you could provide to them.

Looking at [http cache headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) there are quite a few options that are useful to us!

The most fitting one is `no-cache`, where the client requests a response that has not been cached. We'll look for that header when parsing our response and change our appllication's flow based on that header to either skip the lookup in the database or not.

Let's parse some headers and see if we can make a decision if we should rely on our cached result or not:

```go
cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

var noCache bool
// simplified, should iterate through all headers separated by `,`
if len(cacheControlHeader) != 0 {
  if cacheControlHeader[0] == "no-cache" {
    noCache = true
  }
}
```

There's probably some elegant or standardised way of doing that, but now we can pass it to the function that looks up a record to skip the database part:

```go
if noCache {
  record, requestError := locationRequest(city, countryCode)
  if requestError != nil {
    fmt.Printf("request error: %+v", requestError)
    return record, requestError
  }
  storeLocationRecord(record)
  return record, nil
}
// rest of the function
```

## Golang and Upsert in Mongodb

Going the extra mile would obviously be to update our function to store location records to `storeOrUpdate` and make sure we, in case of a cache bust, store the most up to date values as well!

```go
filter := bson.M{
	"name":    record.City,
	"country": record.CountryCode,
}

update := bson.D{
	{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

opts := options.Update().SetUpsert(true)
_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
```

Luckily `upserts` in mongodb are something that's been well supported for a while and does not actually require two database calls. Although the syntax for inserting OR updating is a bit more cumbersome, it has its advantages.

When we make a request to our local service now, depending on if we pass the header or not, we'll get different log messages from the web server, but same results from the JSON response.

First time we query the API:

```bash
# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]
```

Second time:

```bash
# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
database record: {City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}
```

Third time with no-cache header:

```bash
# curl command:
curl curl --header "Cache-Control: no-cache" localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]
```

The third request successfully ignored the database record and went straight to the external API, success! We've achieved what we wanted for now, next steps include:

* repeat the process for the weather forecast
* possibly skip the caching of the weather response, as weather changes frequenty

Here is the full refactored code below:

```go
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"reflect"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"github.com/gofiber/fiber/v2"
)

type GeoResponse struct {
	City        string  `bson:"name" json:"name"`
	CountryCode string  `bson:"country" json:"country"`
	Lat         float64 `bson:"lat" json:"lat"`
	Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
	serverAPI := options.ServerAPI(options.ServerAPIVersion1)
	opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

	client, err := mongo.Connect(context.TODO(), opts)
	if err != nil {
		log.Fatal(err)
	}

	err = client.Ping(context.TODO(), nil)
	mongoClient = client
	return err
}

func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    city,
		"country": countryCode,
	}

	var dbRecord GeoResponse

	findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

	if findError != nil {
		fmt.Println("database lookup error:", findError)
		return dbRecord, findError
	}

	fmt.Printf("database record: %+v \n", dbRecord)
	return dbRecord, nil
}

func storeLocationRecord(record GeoResponse) error {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    record.City,
		"country": record.CountryCode,
	}

	update := bson.D{
		{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

	opts := options.Update().SetUpsert(true)
	_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
	if dbUpsertError != nil {
		fmt.Println("db insert error:", dbUpsertError)
	}
	return dbUpsertError
}

func locationRequest(city string, countryCode string) (GeoResponse, error) {
	API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"
	res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
	if err != nil {
		log.Fatal(err)
	}

	responseBody, _ := io.ReadAll(res.Body)
	defer res.Body.Close()
	var parsedResponse []GeoResponse
	jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
	if jsonErr != nil {
		fmt.Println("json parsing error:", jsonErr)
		return parsedResponse[0], jsonErr
	}

	fmt.Printf("received from external api: %+v \n", parsedResponse)
	return parsedResponse[0], nil
}

func getCityLocation(city string, countryCode string, noCache bool) (GeoResponse, error) {
	var record GeoResponse
	if noCache {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	record, lookupError := lookupLocationRecord(city, countryCode)
	if lookupError != nil {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	return record, nil
}

func main() {
	dbError := connect_to_mongodb()
	if dbError != nil {
		fmt.Printf("database connection error: %+v", dbError)
	}

	app := fiber.New()

	app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
		city := c.Params("city")
		countryCode := c.Params("country_code")
		cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

		var noCache bool
		if len(cacheControlHeader) != 0 {
			if cacheControlHeader[0] == "no-cache" {
				noCache = true
			}
		}

		// simple validation of parameters
		cityRef := reflect.ValueOf(city)
		countryCodeRef := reflect.ValueOf(countryCode)
		if cityRef.Kind() != reflect.String || countryCodeRef.Kind() != reflect.String {
			return c.Status(400).SendString("parameter error")
		}

		// get location
		record, _ := getCityLocation(city, countryCode, noCache)
		return c.JSON(record)
	})

	app.Listen(":3000")
}
```


In the last post we looked at [how to create a command line application that
stores API responses i...

Cache-Control with GO REST APIs


Sometimes we rely on third party APIs where every request costs us money or have some strict rate limits. To mitigate the financial impact of a project and to have a bit of fun, we can add some caching measures to not call other APIs before either a cache expires or to keep copies of external data sources forever.

**Fun fact:** The API we're using let's us query `1_000_000` times a month before charging us, but let's pretend it's really expensive and limited.

## Example Application: Locations and Weather

One of my use cases was to report the weather for a specific location. There's a pretty good weather API at [openweathermap.org][openweathermap].

Reading the API documentation we'll need to make **2 calls** for every weather request, since the endpoint that returns the weather deals with latitude and longitude.

1. we get a user input (probably a city) and need to find its geographical location in lat/lon [/api/geocoding-api](https://openweathermap.org/api/geocoding-api)
2. the actual request to get the current weather [/current](https://openweathermap.org/current)

### Humble Beginnings: CLI API Requests

Let's start with the simplest application we can think of to validate that a part of it works that we'll need later on. We'll make an API request to locate a city and print the results:

```go
package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
)

func main() {

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh" // your api key here

  city := "Berlin"
  countryCode := "DE"

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }

  defer res.Body.Close()

  body, err := io.ReadAll(res.Body)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(string(body))
}
```

Running this code, we get the following output:

```bash
go run main.go
```

```json
[
  {
    "name":"Berlin",
    "local_names": {"bg":"Берлин","io":"Berlin"},
    "lat": 52.5170365,
    "lon": 13.3888599,
    "country": "DE"
  }
]
```

Sweet! Now let's go ahead and store that response in a database!

### Database Setup

The natural place to save some data we want to keep readily available would of course be the database of our choice.

Let's create a docker-compose file to make sure we can easily spin up a database during development and decide on which shape we want to save the data in. We probably want to keep the data we're interested in and the parameters we want to query by in separate fields.

We don't need to store the full API response or we can stringify the data we would only need sporadically.

```yaml
# file: docker-compose.yml
version: "3.8"
services:
  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
```

Next we're going to write the response we get from the fun weather api to our own database when we run the script:

```go
package main

import (
  "context"
  "encoding/json"
  "fmt"
  "io"
  "log"
  "net/http"

  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)

type GeoResponse struct {
  City        string  `bson:"name" json:"name"`
  CountryCode string  `bson:"country" json:"country"`
  Lat         float64 `bson:"lat" json:"lat"`
  Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
  serverAPI := options.ServerAPI(options.ServerAPIVersion1)
  opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

  client, err := mongo.Connect(context.TODO(), opts)
  if err != nil {
    log.Fatal(err)
  }

  err = client.Ping(context.TODO(), nil)
  mongoClient = client
  return err
}

func main() {
  dbError := connect_to_mongodb()
  if dbError != nil {
    fmt.Printf("database connection error: %+v", dbError)
  }

  collection := mongoClient.Database("weatherdb").Collection("cities")

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"

  city := os.Args[1]
  countryCode os.Args[2]

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }


  // parse api response to json
  responseBody, _ := io.ReadAll(res.Body)
  defer res.Body.Close()
  var parsedResponse []GeoResponse
  jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
  if jsonErr != nil {
    fmt.Println("json parsing error:", jsonErr)
  }

  fmt.Printf("%+v", parsedResponse)
  // save first record in array to the database
  dbResult, dbInsertError := collection.InsertOne(context.TODO(), parsedResponse[0])
  if dbInsertError != nil {
    fmt.Println("db insert error:", dbInsertError)
  }

  fmt.Printf("%+v", dbResult)
}
```

after adding the mongodb dependency to our file, we can run the following commands to try it out:

```bash
# in a separate terminal window
docker-compose up

# then in our project:
go mod init
go mod tidy
go run main.go
```

and we should have a record in our mongodb database `weather`:

![screenshot of mongodb compass gui](/assets/2024/weatherdb-compass-mongodb.png)

or via the mongodb shell:

```mongodb
use weather;
db.cities.find();
```

```js
{
  _id: ObjectId('666b4a7ba2febc2c80e3ec5d'),
  name: 'Berlin',
  country: 'DE',
  lat: 52.5170365,
  lon: 13.3888599
}
```

The problem we now face is, that if we simply run the script again with the same city and country code, we simply would

1. call the weather api (again)
2. save an identical database record (again)

To avoid this we need to first look up in our database IF we have a record that matches something that makes this unique, for example the combination of `city` and `country`.

**Fun fact:** Technically these are not unique, just look at the amount of Springfields in the US, you will not be disappointed. We'll just pretend these are unique enough for now.


## Check the DB before API call

Let's first check our database for information we might have retrieved already, like described in this sequence diagram:

```mermaid
sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: Nope
    our Program->>+WeatherAPI: Hey, what's the location of Berlin?
    WeatherAPI->>-our Program: "lat: 52.5170365, lon: 13.3888599"
    our Program->>+Database: "Berlin is at lat: 52.5170365, lon: 13.3888599"
```

![mermaidjs sequence diagram](/assets/2024/weatherdb-mermaidjs-sequence.png)

Or in the case that we do indeed already have a database record we skip the whole weather api bit!

```mermaid
sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: "Yup, it's at: 52.5170365, lon: 13.3888599"
```

### Golang mongodb.find and parse to struct

Mostly of what we need to add is a quick call to our database first and returning early from the `main` function:

```go
filter := bson.M{
  "name":    city,
  "country": countryCode,
}

var dbRecord GeoResponse

findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

if findError != nil {
  fmt.Println("database lookup error:", findError)
}

if findError == nil {
  fmt.Printf("database record: %+v \n", dbRecord)
  return
}
```

Now if we run our program again, we should see this output, call no external API and not create any duplicate records in the database 🙌.

```bash
database record: {City:Berlin CountryCode:DE Lat:52.5170365 Lon:13.3888599}
```

![our program in action](/assets/2024/weatherdb-vhs.gif)

In the next part we'll look at invalidating some records automatically (locations of city don't often change, but the weather does sometimes) and how to serve all of this through a REST API that fits our needs.

[openweathermap.org]: https://openweathermap.org


Sometimes we rely on third party APIs where every request costs us money or have
some strict rate l...

Caching with Golang and mongodb


World of Warcraft probably is one of the most extendable video games to write addons for that alter the interface and provide lots of customisation  to the players. In this post we'll look at how to write a custom trigger for the weak auras framework to automate build a mini "chatbot" in order to provide explanations for my fellow raiders.

One of the popular low code options to display additional reminders as visual or auditive signals is called [WeakAuras][weak-auras-website], which are often used for group activities like raids where groups of players between 10 and 40 players try to defeat bosses together. In these scenarios some roles or specific players get personal responsibilities like moving to a specific place in the arena or using specific abilities.

If these personal responsiblities apart from their usual role (like hunters shooting arrows or healers giving everyone a bit divine light to top of some health pools) are not taken care of, it will kill the entire group and you need to try the entire fight again. With fights being up to 10 minutes, that can be quite frustrating for the group, so as a raid leader, you often want to remind the players in your group of the mechanics in the fight that is about to happen.

If you don't raid in a guild with the same people 1-3 times a week, you have to invite more or less random players, that are a bit of a wild card when it comes to their knowledge of the encounters, the game in general and if you don't require that they join a voice that they're basically playing the game by themselves apart from the little chat window that only occupies a small portion of their interface.

What I used to do while leading groups through some of the raid dungeons that change every few months is to explain lots of mechanics in chat, by simply typing them out in varying degrees of specificity depending on how capable I believe a group to be.

This quickly got to a very ridiculous amount of typing the same things, which I thought would be a fun thing to automate away and pick up a bit of LUA on a train ride, the language used for weak auras and wow addons in general. All we want is a little bit of chat automation to make our multiple weekly runs through a boss filled raid dungeon a little smoother!

## The Chatbot WeakAura

WeakAuras support a number of built-in reactions that can let you display something on screen, like when a player gains a positive effect, like when a mage can cast a spell instantly instead of a 5 second cast time after another spell. Reacting to chat messages luckily was a built in event listener, which was great, now we just needed to interpret the chat message and respond with the most important explanations for the upcoming boss fight.

![weak aura responding](/assets/2024/weak-aura-chat-window.jpg)

The events we want to listen to are these:

```
CHAT_MSG_PARTY, CHAT_MSG_PARTY_LEADER, CHAT_MSG_RAID, CHAT_MSG_RAID_LEADER, CHAT_MSG_INSTANCE_CHAT
```

Which will let me test the code inside a regular party (for 5 people) or use it in the raid party chat.

As a little bonus, I wanted to only have the *Heroic* difficulty mechanics explained if I'm currently actually in the heroic difficulty raid.

![weak aura with heroic text](/assets/2024/weak-auras-heroic.png)

As I had barely written a line of Lua in my life, I needed to start with a Hello World! example and work my way up from there. Our custom functionality will be implemented in a custom trigger, which lets us write lua code to process incoming events forwarded by the specified events.

Disclaimer: you can edit these while the game is running in a sort of "in addon code editor", but it gets stored as a stringified piece of code that gets evaluated every time your specified event happens. This can make debugging a bit cumbersome and you basically end up copy & pasting it in and out of the game editor into your a dedicated code editor. After changes you probably need to reload your interface by typing `/reload` in the wow chat.

This is what the built in code editor looks like:

![weak auras code editor](/assets/2024/weak-auras-code-editor.webp)
 
I learned from another weak aura that the input for the custom trigger function would look something like this:

```lua
function('CHAT_MSG_RAID', '!explain gnarlroot')
```

Developing the script outside the game client had some pitfalls, for example that World of Warcraft actually adds a couple of globally available functions (which don't exactly have an official documentation) via its API for the convenience of developing addons. One of these functions is `strsplit`, which we kind of stole from stackoverflow:

```lua
-- usually provided by wow API
function strsplit(sep, inputstr)
  if sep == nil then
     sep = "%s"
  end
  local t={}
  for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
     table.insert(t, str)
  end
  return t
end
```

Secondly we needed the functions that we were going to depend on that are exposed by WoW, which we just return multiple values from to mimic that we're in a raid group and if it's on the *Heroic* difficulty or not:

```lua
-- fake functions
function GetInstanceInfo()
  return '', 'raid', 1, '', '', '', '', '', ''
end

function GetDifficultyInfo(fakeArg1)
  return '', '', true, '', true, '', ''
end
```

* `GetInstanceInfo` documentation: [wowpedia.fandom.com/wiki/API_GetInstanceInfo](https://wowpedia.fandom.com/wiki/API_GetInstanceInfo)
* `GetDifficultyInfo` documentation: [wowpedia.fandom.com/wiki/API_GetDifficultyInfo](https://wowpedia.fandom.com/wiki/API_GetDifficultyInfo)

Lastly we need a utility function that checks if a specific string is contained in the chat message to respond with the right boss explanation:


```lua
function FindSubStr(pattern, message)
  if string.find(string.lower(message), string.lower(pattern)) then
    return true
  else
    return false
  end
end
```

Now, putting these together, we can develop our custom trigger lua code locally and run it:

```bash
lua explain.lua
```

```lua
function FindBoss(msgContent)
  if FindSubStr("Gnarlroot", msgContent) then
    return {"Touch all roots with the fire debuff!", "Heroic: drop puddles on the adds!"}
  end
  if FindSubStr("Igira", msgContent) then
    return {"SOAK zones from left to right", "TANKS must stack", "help the HEALERS with the heal absorb"}
  end
-- and so on
end

function Explain(event, message)
  -- START not required in final script
  local function strsplit(sep, inputstr)
    if sep == nil then
      sep = "%s"
    end
    local t={}
    for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
      table.insert(t, str)
    end
    return t
  end
  -- END not required in final script

  local _, _, channel = strsplit("_", event)
  local _,type,difficultyID,_,_,_,_,_,_ = GetInstanceInfo()
  local _,_,isHeroic,_,displayHeroic,_,_ = GetDifficultyInfo(difficultyID)
  local showHeroicText = type == "raid" and displayHeroic;

  local explanationLines = findBoss(message)

  for _, v in ipairs(explanationLines) do
    if (FindSubStr("heroic:", v)) then
      if (showHeroicText) then
        -- print heroic guide lines if actually in heroic raid
        print(v)
      else 
      end
    else
      -- print normal guide lines
      print(v)
    end
  end
end

Explain('CHAT_MSG_RAID', '!explain gnarlroot')
```

which will output:

```
Touch all roots with the fire debuff!
Heroic: drop puddles on the adds!
```


```lua
  local _, _, channel = strsplit("_", event)
```

This means we're getting the *channel* an event happened on and then the string of the message, which is great for us. Now we need to filter messages and not execute anything further if the message doesn't contain `!explain`
and then we need to filter for what boss we would like an explanation for

One of the first things I learned about Lua was that function casing matters and that variables by default are global, which just means some linter will yell at you if you don't prepend them with `local` to give them an explicit scope, which will be used in the full example below.

## Summary

Weak Auras are a great way to get started coding in your favourite video game to update the interface, remind you of boss tactics and shape your experience and the ones of the players around you.

The weak auras I usually use are more complex and maintained by players way more hard core than me, but this was a fun experiment and I got lots of positive feedback over the raids done with the help of the explainy chat messages!

## References

One of the weak auras I stole code from especially for the custom trigger is the "wichtige WeakAura" that reacts to a number of words, usually synonyms for male genitalia, and then every player with the weakaura installed posts an ASCII represanation of a phallus in varying lenghts, in order to compare the randomly generated lengths.

```
Bob: !abfahrt
Bob: B===D
Alice: B=====D
Pete: B==D
```

The documentation for the WoW APIs I found was rather spotty, but I did find a weakaura that flashes a message with the difficulty when entering a raid, so thanks to that one for letting me implement the *Heroic:* messages: [wago.io/TsamMXYKS](https://wago.io/TsamMXYKS)

You can find the complete custom trigger below if you want to write the next weak aura for the next raid tier and you do a lot of random raiding or the weak aura at wago.io: [wago.io/wJvNnrQLP](https://wago.io/wJvNnrQLP)

Full custom trigger code:

```lua
function(event, msg)

    local function findSubStr(pattern, msg)
        if string.find(string.lower(msg), string.lower(pattern)) then
            return true
        else
            return false
        end
    end

    local function findBoss(msgContent)
        local groupsText = "Groups 1,3,5 LEFT, 2,4,6 RIGHT"
        if findSubStr("Gnarlroot", msgContent) then
            return {"Touch all roots with the fire debuff!", "Heroic: drop puddles on the adds!", "Heroic: you must SOAK flame circle to burn roots"}
        end
        if findSubStr("Igira", msgContent) then
            return {"SOAK zones from left to right", "TANKS must stack", "help the HEALERS with the heal absorb", "Heroic: fire debuffed peeps emit balls, dodge them"}
        end
        if findSubStr("Volcoross", msgContent) thehttps://wago.io/wJvNnrQLPn
            return {groupsText, "SOAK zones!", "TANK must take all stacks, taunt on JAWS", "Heroic: The 'tail' emits hot juicy 'lava'"}
        end
        if findSubStr("Larodar", msgContent) then
            return {"FOCUS damage and healing on Treants and Roots",
                "TANKS run away from charge",
                "Heroic: Tank and Melee must CATCH flame balls at boss",
            "Heroic: Brambles need to be DOUSED with seed before you can attack them"}
        end
        if findSubStr("Council", msgContent) then
            return {"Help tanks soak the bear charge", "Duck: eat plants, then go to URSOC and press 2 if he rages", "KILL them at the same time", "Heroic: TANK swap is a frontal", "Heroic: Barreling Charge alternating through ranged/melee players, care for debuff"}
        end
        if findSubStr("Nymue", msgContent) then
            return {groupsText, "SOAK all plants", "Do NOT let adds cross lines"}
        end
        if findSubStr("Smolderon", msgContent) then
            return {"If you DO NOT have a red circle, SOAK with the tank!", "After knockback, collect 5 red balls!"}
        end
        if findSubStr("Tindral", msgContent) then
            return {"Stay close to boss for roots AOE", "Tanks take turns on the mushroom soak", "Stomp seeds", "BL on third platform"}
        end
        if findSubStr("Fyrakk", msgContent) then
            return {"go first left, then right on Dream rend", "Catch all balls in intermission", "focus small adds", "heal souls", "assign seed carrier order"}
        end
    end

    local _, _, channel = strsplit("_", event)
    local _,type,difficultyID,_,_,_,_,_,_ = GetInstanceInfo()
    local _,_,isHeroic,_,displayHeroic,_,_ = GetDifficultyInfo(difficultyID)
    local showHeroicText = type == "raid" and displayHeroic;
    
    if msg then
        -- print(type,isHeroic,displayHeroic)
        if findSubStr("!explain", msg) then
            local explanationLines = findBoss(msg)
            
            for _, v in ipairs(explanationLines) do
                if (findSubStr("heroic:", v)) then
                    if (showHeroicText) then
                        -- print heroic guide lines if actually in heroic raid
                        SendChatMessage(v, channel)
                    else 
                    end
                else
                    SendChatMessage(v, channel)
                end
            end
        end
    end
    
end
```

[weak-auras-website]: https://weakauras.wtf/


World of Warcraft probably is one of the most extendable video games to write
addons for that alter...

Making a Weak Auras Chat Automation


In this post we're going to have a look at how to set up a local grafana loki installation via docker-compose and generating some test data for our fresh installation via a log transport.

Prerequisites:

* docker
* docker-compose
* node.js

Grafana is a great tool for capturing and searching logs and it can be a bit of an overwhelming system to break into. There is also a new query language to learn if you want to query logs in grafana, called *logql*, which we'll be using in a bit. Grafana also has a lot of components that do different things or serve different purposes. We're interested in capturing logs and visualising them for this tutorial.

Let's create a directory like `grafana-playground` and start producing some logs!

## Producing Logs for Grafana

Logging often happens straight to `stdout`, which is the terminal output of the shell running your application. You can pick up these logs with promtail and send them to Grafana. However, you can also log straight from your application and send them to Grafana and many logging libraries will provide you with features to send your logs to grafana instead of the standard out. One of these libraries is [winston][winston] and the relevant transport for [winston-loki][winston-loki].

Let's go ahead and create a new directory called `log-producer` to emit some logs:

```
mkdir log-producer
cd log-producer
npm init -y
npm install winston winston-loki
touch log-test.js
```

Then we can run this file from the official examples of winston-loki:

```javascript
// log-test.js
const winston = require('winston')
const LokiTransport = require('winston-loki');
const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
  labels: { job: 'winston-loki-example' }
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const run = async () => {
  while (true) {
    logger.debug('I am a debug log')
    logger.info('This is a test, no need to panic')
    logger.error('Testing for errors')
    await wait(1000)
  }
}

run()
```

and it will output something like this:

```
{"level":"debug","message":"I am a debug log"}
{"level":"info","message":"This is a test, no need to panic"}
{"level":"error","message":"Testing for errors"}
```

which is cool, but when we close the terminal, these log messages will disappear into the void, so let's go ahead and start a local grafana instance to actually keep and browse these logs.

## Setting up Loki

The simplest way is to follow the [official instructions](https://grafana.com/docs/loki/latest/setup/install/docker/) for installing grafana locally with docker-compose:

```bash
wget https://raw.githubusercontent.com/grafana/loki/v2.9.1/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up
```

The command will download the up to date version of the docker-compose file and start promtail and grafana.

Having a look inside that file shows that grafana will run at port `3100` and the web interface will run at port `3000`.

```yaml
  image: grafana/loki:2.9.1https://github.com/sebhildebrandt/systeminformation
  ports:
    - "3100:3100"
# ...
      url: http://loki:3100
```

run `docker-compose up` and confirm that you actually get a nice splash screen of grafana at [localhost:3000](http://localhost:3000).

Now we can produce some logs to look at in Grafana!

## Using a Grafana Transport for Winston in Node:

Let's go ahead and run the script we placed in `log-producer` earlier again:

```
node log-test.js
```

Now we can go to the Grafana web interface, find *Explore* in the sidebar and start querying your logs.

Select `job` -> `winston-loki-example` in the label filters and hit `Run query`

![grafana query builder](/assets/2023/grafana-query-builder.png)

We sould be able to see a list of logs and a graph showing the split between `info` and `error` severity/level logs. If we also want to capture `debug` logs, we need to explicitly set the loglevel in the transport like so:

```diff
logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
+  level: 'debug',
  labels: { job: 'winston-loki-example' }
}))
```

![grafana log volume](/assets/2023/grafana-log-volume.png)

## Visualising your CPU temperature

For more exciting example data and a better dataset for visualisation, we can grab our CPU temperature and post the current temperature every second. After we start, we can also do something CPU intensive like transcoding video a video and observing the temperature changing.

You can use the [systeminformation package][systeminformation] or parse the output of `sensors`. Turns out `sensors -j` will give you an easily parseable JSON object of various temperature sensors on your Linux system. I had to parse the output from `sensors`, because the node package wasn't working for me. I'll still post logs every second and we can have a look at how we can turn the values into a timeseries graph or diagram in grafana. 🙌

My CPU temperature log producer:

```javascript
// temp.js
const util = require('node:util');
const exec = util.promisify(require('node:child_process').exec);

const winston = require('winston')
const LokiTransport = require("winston-loki");


const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  labels: { job: 'cpu-temp' },
  json: true,
  format: winston.format.json(),
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const cpuTemperature = async () => {
  const { stdout: sensorsOutput } = await exec('sensors -j');
  const sensorsJSON = JSON.parse(sensorsOutput);
  return sensorsJSON['k10temp-pci-00c3']['Tctl']['temp1_input'];
}

const run = async () => {
  while (true) {
    try {
      const temp = await cpuTemperature();
      logger.info({ message: 'got cpu load', temp })
    } catch (e) {
      logger.error({ message: 'could not read cpu load', e })
    }

    await wait(1000);
  }
}

run()
```

Now you should be able to filter after the job `cpu-temp` and see the logs come in.

If you can see the logs after switching the job field, we can click the "add to dashboard" button in the top of the interface, edit the panel and change the type to *Time series*.

![grafana time series](/assets/2023/grafana-time-series.png)

My query is

```logql
{job="cpu-temp"} | json | line_format `{{.temp}}`
```

![grafana parse json fields](/assets/2023/grafana-parse-json.png)

but you can also easily build it in the query builder. The important parts are:

* use the JSON formatter to let grafana use the individual fields of your JSON log messages
* specify the field that we want to plot in our timeseries graph

Lastly we need to add a transform to tell Grafana that our parsed JSON field is a `number` and not a `string`.

![grafana transform field type to number](/assets/2023/grafana-transform.png)

Now you should see a nice green line crawling across your graph. Let's make the data a little more exciting by doing something CPU intensive:


```bash
ffmpeg -i generic-legal-file.mp4 -c:v libvpx-vp9 -b:v 3M generic.webm
```

This leads me to get the nice graph right here and I can see the temperature be elevated for the duration of the video encoding:

![grafana time series cpu temperature](/assets/2023/grafana-cpu-temperature.png)

I hope this can serve as a quick beginners guide to grafana and the basics of visualisation. There's much more to write about and how awesome distributed tracing and logging is, but practical examples that require less than 50 lines of code are usually the best intro to get everyone to start playing with a thing!

[winston]: https://github.com/winstonjs/winston
[winston-loki]: https://github.com/JaniAnttonen/winston-loki
[systeminformation]: https://github.com/sebhildebrandt/systeminformation


In this post we're going to have a look at how to set up a local grafana loki
installation via dock...

Visualising Data with Grafana Loki From Scratch


The Dungeons and Dragons movie was a really light-hearted treat and not as deep or epic as other movies this year, but still pretty entertaining. Some epic special effects, a party of adventurers LITERALLY recruiting people by class and abilities on a quest for loot.

Some of monsters were also hilarous, it's like somebody wrote the brief with a few funny remarks to make everyone laugh at the meeting:

> Yeah and give me a monster that's like those round animals from that guy on youtube! Yeah pretty fat, exactly! Nono, it's still dangerous and an actual threat to the party!

Anyways, if you're into DnD or any generic fantasy stuff, you will have a bunch of chuckles watching this one.


The Dungeons and Dragons movie was a really light-hearted treat and not as deep
or epic as other mo...

Watched: Dungeons & Dragons: Honor Among Thieves


Everything Everywhere All at Once was a great movie to see at the cinema. The fantastic mash-up between Matrix style reality warping multiverse action and a family struggle of first generation immigrants with beaurocracy and what they really wanted to do with their lives.

The absurity and how the movie doesn't take its own visual expression too seriously, made it a special piece. I had mostly remained unspoilered/untrailered and I can only recommend watching it without any prior exposure.

I had a really good time and can definitely recommend watching it for both the fun experience and the message.


Everything Everywhere All at Once was a great movie to see at the cinema. The
fantastic mash-up bet...

Watched: Everything Everywhere All at Once


If you want a truck load of free classic books for your kindle or other e-reader you can download them for free from the fantastic [Project Gutenberg][gutenberg] which collects lots public domain works as ebooks. If a book is not available in the desired format, it can be easily converted to any e-reader.

I purchased a kindle and it's been a surprisingly good experience. I've resisted buying another device to read, but e-ink displays are actually nicer on the eyes and also nicer to read in dark environments like the bedroom without illuminating the entire room with an LED display.

On top of that e-ink displays also perform much better in bright environments like... a beach. Who are we kidding, I'm not going to be at a beach, but occasionally in a not well sun-shielded train compartment.

## Download Free Ebooks for your Kindle 💾

I can only recommend browsing around [Project Gutenberg][gutenberg] bit, because there's lots of absolute gold and things that are probably either on everybodies reading list or something you can very easily fall asleep to (no offense Nietzsche).

For this example I have downloaded is *The Legend of Sleepy Hollow*.

![Sleepy Hollow ebook](/assets/2023/project-gutenberg-book-download.webp)

Huge respect to the project, which has amassed over 70000 public domain books in a convenient online library.

## Convert epub to a kindle compatible format (azw3) 🖋️

If the Kindle download option doesn't work or if your e-reader does not support the epub format, you can convert the downloaded `.epub` with a fun and open source library software called [Calibre][calibre]. You can use your favourite package manager on Linux or `brew install calibre` on MacOS.

When you import a book your Calibre screen should look a little like this:

![calibre overview list](/assets/2023/calibre-list-screen.webp)

After importing a book you can right click and convert into dozens of formats, my kindle paperwhite (5) prefers the `AZW3` format.

![calibre export format kindle azw3](/assets/2023/calibre-convert-export-kindle.webp)

## Download Free Ebooks onto the Kindle Paperwhite 📚

To my surprise the kindle works just like a USB stick, as a generic USB mass storage volume and you can move the downloaded and converted ebook into a directory called Downloads where the files from the amazon store are also saved.

This is very convenient for me, since I usually have some computer with a USB connection nearby before I decide reading a book.

![move ebook to kindle via usb](/assets/2023/kindle-transfer-converted-book.webp)

After unplugging the e-reader and accessing the library you should see the newly acquired book in the list of books. If it doesn't show up, you need to have a look at which formats your e-reader supports.

Now you should be able to enjoy the books on your kindle without any issue. For me all features worked as well, taking notes, chapter progress, page numbers and re-flowing pages with changing font sizes.

![photo of a kindle with the downloaded book](/assets/2023/kindle-gutenberg-photo.webp)

I still need to test which e-readers actually just straight up support .epub, I assume some of the more indie ones just do. If you know any fun trivia or great ebook tricks, get in touch and spill the beans! I'm new to this 😉.

The Onyx Boox Poke 3 has been tested and supports `epub` out of the box.

[gutenberg]: https://www.gutenberg.org/
[calibre]: https://calibre-ebook.com/


If you want a truck load of free classic books for your kindle or other e-reader
you can download t...

Free ebooks for the Kindle Paperwhite with Project Gutenberg


In this post we're going to see how we can build a Mastodon bot with GO that posts pre-planned post once per day. It will support attachments and respective alt texts to cover visually impaired people as well.

The library we're going to use to talk to the mastodon API is [go-mastodon][go-mastodon], which supports all the features we need for uploading assets and attaching them to our post.

## Preparing a Post/Toot

To keep our future posts nice and organised, we'll drop them into directories that have a `$year/$date.toml` structure. We'll use [toml][toml] for our content since it doesn't rely on indentation like yaml and lets us create key/value pairs without much trouble.

The first post ever posted by the [Game Trivia Bot][gametrivia bot] looks like this:

```toml
text = "The official Twitter account for Fall Guys released a \"human for scale\" illustration featuring their playable characters. Fall Guys are all 183cm (6ft) by the way."
tags = ['FallGuys']
assets = ['fallguys-scale.jpeg', 'fallguys-eyes.png']
assetsAlts = ['A Fall Guy next to human', 'Fall Guy eyes and skull']
credits = ['https://www.artstation.com/tudormorris']
```

We have the body `text`, the `tags` which will appear like `#FallGuys`, the `assets` (image files) and the alt text for those image files: `assetsAlts`. Lastly we also have `credits` for the image we're posting because we obviously want our followers to explore the cool creators behind visuals, talks or screenshots.

For our example we will simplify the format a little bit and only support one media attachment per post, like some of the famous bird site accounts that post a fox or an opossum every hour.

Now we also need a function to load the post and to parse it into a data type that we can work with in our program.

Let's create a file called `posts.go` to parse an example toml file:

```go
package main

import (
  "fmt"
  "os"
  "time"

  "github.com/BurntSushi/toml"
)

type Post struct {
  Text       string
  Tags       []string
  Asset      string
  AssetAlt   string
  Credits    string
}

func LoadPost() (Post, error) {
  var post Post
  postFileName := "posts/example.toml"

  _, fileExistErr := os.Stat("./" + postFileName)

  if fileExistErr == nil {
    fmt.Printf("File exists, processing\n")
  } else {
    fmt.Printf("File does not exist\n")
    return post, fileExistErr
  }

  _, tomlError := toml.DecodeFile(postFileName, &post)
  if tomlError != nil {
    fmt.Printf("TOML file reading/decoding error: %v\n", tomlError)
  }

  return post, nil
}
```

To load a post for a specific day, we would have to change the `postFileName` to something like:

```go
  currentTime := time.Now()
  postFileName := fmt.Sprintf("date-posts/%s/%s.toml", currentTime.Format("2006"), currentTime.Format("2006-01-02"))
```

which would allow you to use a directory structure like this:

```
├── date-posts
│   ├── 2022
│   │   ├── 2022-12-01.toml
│   ├── 2023
│   │   ├── 2023-09-23.toml
│   └── drafts
```

We'll get back to actually using the parsed post later.

## Getting the Mastodon API key

Mastodon bots are their own accounts, so you should check with your instance policies if bot accounts are allowed and then register an account with the desired username. Then, in the web interface you need to create an app with the right scopes. We'll need "write" at least, but check whatever you need for your bot.

![mastodon application scopes](/assets/2023/mastodon-api-keys.webp)

find your client key and client secret:

![client key and secret](/assets/2023/mastodon-application-bot-scopes.webp)

We'll also need to tell the bot which server it lives on like `streamers.social` or `mastodon.social`.


## Loading API Configuration

We need to pass the respective credentials to our bot and when we deploy it we probably also want to read it from the environment and not from a file committed to git, so let's create our `main.go` file and a `config.go` file to load our configuration either from a `.env` file in development or from the ENV variables in production.

main.go:

```go
package main

import (
  "fmt"
  "log"
)

var config map[string]string

func main() {

  envs, error := GetConfig()

  if error != nil {
    log.Fatalf("Error loading .env or ENV: %v", error)
  }

  fmt.Printf("%v", envs)
}
```

config.go

```go
package main

import (
  "fmt"
  "os"

  "github.com/joho/godotenv"
)

func readConfigFromENV() map[string]string {
  var envs = make(map[string]string)

  envs["MASTODON_SERVER"] = os.Getenv("MASTODON_SERVER")
  envs["APP_CLIENT_ID"] = os.Getenv("APP_CLIENT_ID")
  envs["APP_CLIENT_SECRET"] = os.Getenv("APP_CLIENT_SECRET")
  envs["APP_USER"] = os.Getenv("APP_USER")
  envs["APP_PASSWORD"] = os.Getenv("APP_PASSWORD")

  return envs
}

func GetConfig() (map[string]string, error) {

  if os.Getenv("APP_ENV") != "production" {
    envs, error := godotenv.Read(".env")
    if error != nil {
      fmt.Println("Error loading .env file")
    }
    return envs, error
  } else {
    envs := readConfigFromENV()
    return envs, nil
  }
}
```

and lastly this is the content of our `.env` file which sould also be listed in your `.gitignore` file:

```
MASTODON_SERVER="https://streamers.social"
APP_CLIENT_ID="2atM1d_KxIswqRn7uvusTe"
APP_CLIENT_SECRET="siuI0GDyOv9xD_htlQ5Tu"
APP_USER="gametrivia@jonathanmh.com"
APP_PASSWORD="super secret password"
```

You will need to replace the values with the ones you got from the web interface and a strong randomly generated password for your bot account.

Now if we run

```bash
go mod init
go mod tidy
go run .
```

We should see the following output that confirms that our config is being loaded correctly:

```
map[APP_CLIENT_ID:2atM1d_KxIswqRn7uvusTe APP_CLIENT_SECRET:siuI0GDyOv9xD_htlQ5Tu APP_PASSWORD:super secret password APP_USER:gametrivia@jonathanmh.com MASTODON_SERVER:https://streamers.social]
```

## Using the Mastodon API

This is great, we've moved our config loading to a file outside of our `main.go` to keep it a bit tidier and now we'll move on to making sure our API connection works as intended. In the examples of go-mastodon we can steal the following snippet:

```go
  c := mastodon.NewClient(&mastodon.Config{
    Server:       "https://mstdn.jp",
    ClientID:     "client-id",
    ClientSecret: "client-secret",
  })
  err := c.Authenticate(context.Background(), "your-email", "your-password")
  if err != nil {
    log.Fatal(err)
  }
  timeline, err := c.GetTimelineHome(context.Background(), nil)
  if err != nil {
    log.Fatal(err)
  }
  for i := len(timeline) - 1; i >= 0; i-- {
    fmt.Println(timeline[i])
  }
```

and we can amend it to use our config like this:

```go
  c := mastodon.NewClient(&mastodon.Config{
    Server:       envs["MASTODON_SERVER"],
    ClientID:     envs["APP_CLIENT_ID"],
    ClientSecret: envs["APP_CLIENT_SECRET"],
  })

  err := c.Authenticate(context.Background(), envs["APP_USER"], envs["APP_PASSWORD"])
```

remember to also import `"github.com/mattn/go-mastodon"` and then we can run:

```
go mod tidy
go run .
```

Which should no emit any errors or output since your bots timeline is empty. If you have manually posted anything, the posts should be listed in your terminal. If your credentials are wrong of if there are any bugs in your code, it should show something like the following error:

```
2023/01/22 17:41:18 bad authorization: 401 Unauthorized: invalid_client
exit status 1
```

## Posting to the Mastodon API

Finally, we'll post our example post, let's create a file called `posts/example.toml` and add a picture called `posts/owl.jpg` and try to post it to the timeline.

example.toml:

```
text = "The majestic owl, owling about in its natural habitat."
tags = ['owl', 'nature', 'photography']
asset = 'owl.jpg'
assetAlt = 'an owl'
credits = 'https://unsplash.com/photos/miEWeTPYsFE'
```

Now we amend the main.go file to include the following:

```go
  post, err := LoadPost()
  if err != nil {
    log.Fatal(err)
  }

  var media mastodon.Media
  file, err := os.Open("./posts/" + post.Asset)
  if err != nil {
    log.Fatal(err)
  }

  media = mastodon.Media{File: file, Description: post.AssetAlt}
  attachment, err := c.UploadMediaFromMedia(context.Background(), &media)
  var attachmentIDs []mastodon.ID

  attachmentIDs = append(attachmentIDs, attachment.ID)

  finalText := post.Text + "\n"

  for i := 0; i < len(post.Tags); i++ {
    finalText = finalText + "#" + post.Tags[i] + " "
  }

  finalText = finalText + "\n" + "Credits: " + post.Credits

  toot := mastodon.Toot{
    Status:   finalText,
    MediaIDs: attachmentIDs,
  }

  fmt.Printf("About to publish: %#v\n", toot)
```

In this snippet we first upload the attachment for the post and save the attachment ID, then we alter the text and append each hashtag prefixed with a `#` and lastly add the credits.

Then we provide both to the `mastodon.Toot` type struct, made available by the library.

If you receive no error, you can add the final lines to post to your bot account:

```go
status, err := c.PostStatus(context.Background(), &toot)

if err != nil {
  log.Fatalf("%#v\n", err)
}
```

Now you should be able to see your post on your bot account:

![bot account published post](/assets/2023/mastodon-bot-toot.webp)

To post periodically I use a cronjob to run the application inside a docker container in production once per day. This takes a bit of planning, but it deploys automatically to dokku on merges to the main branch, which I will cover in the future.

Thank you so much for reading, [drop me a link on mastodon](https://streamers.social/@jonathanmh) to which wonderful bots you're creating!

[go-mastodon]: https://github.com/mattn/go-mastodon
[gametrivia bot]: https://streamers.social/@gametrivia


In this post we're going to see how we can build a Mastodon bot with GO that
posts pre-planned post...