
Email verification can be tricky, both to actually get right and for users to deal with, when engineers get over-zealous with their validation requirements.

## What Does an Email Address Look Like?

The simple answer, that spring into most minds could be a short list of requirements:

* it has an `@` and then there's a `.` somewhere
* there's something in front of and after the `@` and behind the `.`

That's almost correct, but even governments struggle with email addresses that don't comply with some made up requirement that top level domains are something like `.co.uk` or `.de`. They can be pretty long these days, but lots of email validations will fail them.

Also `jonathan@gov.co.uk` has two dots in it, so our initial statement would already only be a half-truth.

Luckily there are some specifications that describe what an email address CAN look like, like [RFC: 5322][RFC: 5322] and extensions to it like [RFC: 6854][RFC: 6854]. However, many engineers don't have the luxory of looking up specs while implementing a feature and grab the nearest software package available to accomodate their needs or further clamp down requirements (aka violate the spec) to fulfill some sort of business requirement.

This, a weird talk on YouTube and some reddit comments also taught me that apparently `cow@[dead::beef]` is a valid, spec compliant email address. 🤯

1. `cow` \<- this is fine
2. `[]` possible part of an IP address notation?
3. `dead::beef` valid characters in an IPv6 address (0-9,a-z)

Note: It's up to you, running a business, to disregard specs purposefully, but usually I bet it's done by accident. 

## How to Verify that a String is an Email Address?

There's plenty of packages for various programming languages to ensure this, an often followed approach is to verify compliance of an email address via regular expressions. There's even a website that gathered some regular expressions for different languages (and even SQL dialects) to figure out if a user input is an email address: [emailregex.com][emailregex.com], which is helpful.

I often work with Node.js and there's a LOT of email validation packages on npm. The disappointing thing is that they barely state which RFC spec they target.

Some of them do other interesting things though, like for example the [deep-email-validator][deep-email-validator]

> * Validates email was not generated by disposable email service using disposable-email-domains.
> * Validates MX records are present on DNS.
> * Validates SMTP server is running.
> * Validates mailbox exists on SMTP server.

This was new to me, that fateful night that I stumbled down the email validation rabbit hole. How smart to check the DNS resolution for DNS records and check in the with SMTP server!

Other packages focus solely on the external components of DNS and SMTP like [email-existence][email-existence] while others focues on only the user provided information like [validator.js][validator.js], which actually has a lot of great test cases [in their github repository](https://github.com/validatorjs/validator.js/blob/master/test/validators.test.js)

## Is This a Throwaway Email Account?

There are projects to track which email service providers generate temporary mail boxes for users in order to remain anonymous or creatively experience a number of consecutive trial accounts like [disposable-email-domains](https://github.com/ivolo/disposable-email-domains/). Given how quickly you can register domains and point them at the same service, I would take this list and its accuracy with a grain of salt, since it's prone to go out of date quickly and some of the discarded domains might actually belong to someone being a good netizen.

## Third Party Providers and Email Sending APIs

Companies in the email business also make up creative definitions of when an email address should be considered valid, like [Agillic](https://support.agillic.com/hc/en-gb/articles/360007361511-Understanding-How-Valid-Recipients-are-Measured):

> VALID_EMAIL = Not false - Checks if the email address has previously bounced (1 Hard bounce or 1.000 soft bounces)

This is basically custom to their infrastructure and how they curate their clients data sets over time, which they probably have some experience with. The field name however, might be a little misleading without having their glossary of definitions open on the side.

Some email API providers also create some listicles that have questionable reasoning behind their recommendations, like this post on mailtrap:

> with Deep email validator, an MIT-certified dependency package

I'm sure they meant MIT-licensed, but come on, who proof reads these?

Obviously it's a post to pitch their own services, as we can see later in the article:

> As you can see, for my email service, I use Mailtrap Email Sending, which offers robust email-sending capabilities. 

which brings me closer to my use-case.

### So Jonathan, How DID You Get Into This Rabbithole?

There was a [reddit post that caught my attention](https://www.reddit.com/r/golang/comments/1geoosa/updates_on_the_email_verifier_project_in_go/), which had some interesting critique of the project, calling out possible errors in the code and structure of the project. I gave in to curiosity and started reading the spec and started lifting test cases from various sources like wikipedia, reddit comments and various node libraries, which is where I discovered the questionable state of test cases.

I will applaud the creator of the post for putting themself out there to get some feedback (or github stars) and I'm happy they did, otherwise it would probably have taken me a little longer to get a better understanding of the topic myself!

Even worse, implementing spec compliant email validation might produce some very very unexpected results for engineers and users. Lots of email service providers do not allow users to create local parts (the stuff in front of the `@`) that would theoretically be supported in the spec.

![gmail does only allow certain characters in emails](/assets/2024/gmail-allowed-letters.png)

My email use cases are usually that **something** happens and I need to send someone an email. Like they sign up for a web project and need to click a link to give me some reasonable indicator that they're an individual with a mailbox.

If I would get targeted in some ridiculous attack, someone might get the idea to just tank my email domain reputation by signing up 100_000 bogus accounts that makes **ME** look like a spammer, for attempting to send the "welcome" emails, because I didn't check if an email exists in the first place. This would be unfortunate for me and might just fast track any of my future emails to a spam folder or even worse, get my account blocked from sending.

## Honourable Mention: Golang

One of the excellent things I found about GO is that it comes with a *reasonably* accurate function for parsing email addresses [net/mail ParseAddress](https://pkg.go.dev/net/mail#ParseAddress). It's very conveniently allowing you to parse strings like `Jonathan M. Hethey <public@jonathanmh.com>` into an Address struct:

```go
type Address struct {
  Name    string // "Jonathan M. Hethey"
  Address string // "public@jonathanmh.com"
}
```

This is incredibly useful as a baseline for checking user input and you can layer additional methods on top (like DNS and SMTP probes).

Also, there's a go package which lets you mock the DNS resolver if you want to write some tests for your DNS resolution feature: [go-mockdns](https://github.com/foxcpp/go-mockdns), another feature I was missing from the published GO project.

I haven't had the chance to look into mocking an SMTP server yet, but that sounds like another rabbit hole for another day.

## Summary

1. If I provide an open source library for validation/verification I will note which god damn spec I'm targeting 🤓
2. If I provide functionality for talking to different protocols, I will mock my tests that don't break if someone registers the domain I use in my tests. 🙃
3. If I have elaborate test cases I will strongly consider putting them in a portable format to make them easier for other libraries to adopt. 📖

[RFC: 5322]: https://datatracker.ietf.org/doc/html/rfc5322
[RFC: 6854]: https://datatracker.ietf.org/doc/html/rfc6854
[emailregex.com]: https://emailregex.com/
[deep-email-validator]: https://github.com/mfbx9da4/deep-email-validator
[validator.js]: https://github.com/validatorjs/validator.js/
[email-existence]: https://github.com/scippio/email-existence/


Email verification can be tricky, both to actually get right and for users to
deal with, when engin...

Email Verification and Validation


In Golang, importing modules is easy, however it has some opinions on how to structure your project. You're not supposed to sort the files of your project into different directories, but to keep your files in the root folder of your repository OR split out functionality into other modules.

However, it's not a cop, so you can still use the `package` field and an import path to import modules from a sub-directory.

## Go Idiomatic Directory Example

Let's create an ice cream shop that makes use of an `icecream.go` model and some basic price calculation.

`main.go`

```go
package main

import (
	"fmt"
)

func main() {
	myIcecream := Icecream{Flavour: "vanilla", Scoops: 3, BasePrice: 2.99}
	fmt.Println(myIcecream.Price())
}
```

`icecream.go`

```go
package main

type Icecream struct {
	Flavour   string
	Scoops    int
	BasePrice float32
}

func (i Icecream) Price() float32 {
	return i.BasePrice * float32(i.Scoops)
}
```

**directory structure:**

```
.
├── main.go
└── icecream.go

1 directory, 2 files
```

We can confirm that this works by running `go run main.go icecream.go` and it should print: `8.97`.

## Importing Go Files from Sub-directory Example

If we should decide to structure our projects with sub-directories, we need to make some changes. First of all, we should make sure to have a `go.mod` file that states the name of our package, so let's run `go mod init`, which will by defautl take the name of the directory as the module name.

`go.mod`

```
module icecream-shop

go 1.23.1
```

Next we can create a directory, like `models` and move our `icecream.go` file there.

**directory structure:**

```
.
├── go.mod
├── main.go
└── models
    └── icecream.go

2 directories, 3 files
```

If we try to run our code with `go run .` now, we'll get the following error:

```
# icecream-shop
./main.go:8:16: undefined: Icecream
```

Alright, let's now try to import it, we need to remember that unlike in for example JavaScript, we don't import specific files but modules, which are directories:

`main.go`

```go
package main

import (
	"fmt"
	"icecream-shop/models"
)

func main() {
	myIcecream := models.Icecream{Flavour: "vanilla", Scoops: 3, BasePrice: 2.99}
	fmt.Println(myIcecream.Price())
}
```

Note:

* add `"icecream-shop/models"`
* change `Icecream` to `models.Icecream`

However, we get another error if we try to execute it with `go run .`:

```
main.go:5:2: import "icecream-shop/models" is a program, not an importable package
```

Lastly, we'll need to change the package name of the file we moved into the sub-directory, otherwise we'll not be able to import it and use it:

`icecream.go`

```go
package models

type Icecream struct {
	Flavour   string
	Scoops    int
	BasePrice float32
}

func (i Icecream) Price() float32 {
	return i.BasePrice * float32(i.Scoops)
}
```

Note:

* we changed `package main` to `package models`, signalling that this is its own module

## Summary

If you're structuring your project, evaluate carefully if the functionality you're moving around should in fact be its own package or repository.

If not, remember to use the correct import paths.


In Golang, importing modules is easy, however it has some opinions on how to
structure your project...

Golang: Import Module from Sub Directory


Caddy is a web server that boasts with easy configuration and great defaults, such as automatic HTTPS support by default. While I didn't need https, I was curious about how configuring some redirects and static file serving would go. You can find out more about caddy at [caddyserver.com][caddyserver.com].

If you want to get the **full source** for this repository, the link is at the bottom.

As so often I needed redirects, as I was moving one of my websites to become a static site. Inside my docker container I would have a rendered directory structure that has the paths of the html files and of course some assets.

While migrating away from a CMS, I knew that I would break some of the links on the website. Blog posts like `/peaches` would become `/blog/peaches`, so redirects were important to me, optimally in an external file that I can load into the main config.

## Caddy in the Dockerfile

The simplest way in which you can use caddy in a dockerfile to serve static assets might be this:

```docker
FROM alpine
EXPOSE 5000
RUN apk add --no-cache caddy
RUN mkdir -p /var/www/html
COPY pages/ /var/www/html/
WORKDIR /var/www/html
CMD ["caddy", "file-server", "--listen", ":5000"]
```

Given we have some html files in the `pages` directory like so:

```
pages
├── blog
│   ├── eggplant
│   │   └── index.html
│   ├── ice-cream
│   │   └── index.html
│   └── peaches
│       └── index.html
└── index.html
```

we can serve the files by building the Dockerfile and running it:

```bash
docker build . -t caddy_test
docker run -i -p 127.0.0.1:5000:5000 -t caddy_test
```

Now visiting [localhost:5000](http://localhost:5000) we can see our HTML files.

For me it looks like this with some very minimal example HTML:

![ladybug browser showing example page](/assets/2024/caddy-docker-index.png)

While this will serve our files, we're missing out on the redirects we need to apply.

## Load your Caddyfile

The common config file for a caddy server is the `Caddyfile`, so let's create that one:

```caddy
# file: caddy-config/Caddyfile
# global directive
{
  # turn off https, we're in a trusted environment
  auto_https off
}

:5000 {
  # import redirects
  import /var/app/caddy-config/redirects

  # set the root for our served files
  root * /var/www/html

  # Enable the static file server.
  file_server {
    precompressed zstd br gzip
  }
}
```

This one will also listen on port `5000` and through the `file_server` directive serve the files with some compression enabled.

We can place the file in a sub-directory like `caddy-config` if we don't want to clutter the root of our project.

Now to add redirects we could insert a few lines above the `root` directive, like so:

```caddy
redir /ice-cream /blog/ice-cream permanent
```

### Loading an External Redirects File

For the redirects, we're creating a separate file, which I chose to just call `redirects` without any specific file ending.

```caddy
# redirect to front page
redir /tag/* / permanent
redir /author/* / permanent

# old post structure from /[name] to /blog/[name]

# exact match
redir /ice-cream /blog/ice-cream permanent

# match path and path with trailing slash
redir /peaches /blog/peaches permanent
redir /peaches/ /blog/peaches permanent

# match prefix, every route starting with /eggplant will be redirected
redir /eggplant* /blog/eggplant permanent
```

Please note that depending on how you want your redirects to work you might either want to match explicit paths, or use some of the wildcards you can pick up in the [caddy matcher documentation](https://caddyserver.com/docs/caddyfile/matchers#path-matchers).

To check if everything works to our satisfaction, we can run `docker build` and `docker run` again:

```bash
docker build . -t caddy_test
docker run -i -p 127.0.0.1:5000:5000 -t caddy_test
```

We should see output similar to this:

```
# [...] redacted
Successfully tagged caddy_test:latest
2024/06/16 11:54:18.241 INFO    using provided configuration    {"config_file": "/var/app/caddy-config/Caddyfile", "config_adapter": "caddyfile"}
2024/06/16 11:54:18.241 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "/var/app/caddy-config/Caddyfile", "line": 1}
2024/06/16 11:54:18.242 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/06/16 11:54:18.242 WARN    http.auto_https automatic HTTPS is completely disabled for server       {"server_name": "srv0"}
2024/06/16 11:54:18.242 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0001f6200"}
2024/06/16 11:54:18.242 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/06/16 11:54:18.242 WARN    tls     unable to get instance ID; storage clean stamps will be incomplete      {"error": "open /root/.local/share/caddy/instance.uuid: no such file or directory"}
2024/06/16 11:54:18.242 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/06/16 11:54:18.242 INFO    serving initial configuration
2024/06/16 11:54:18.243 INFO    tls     cleaning storage unit   {"storage": "FileStorage:/root/.local/share/caddy"}
2024/06/16 11:54:18.243 INFO    tls     finished cleaning storage units
```

Now when visiting the example links we can see that we're being redirected 🙌!

![firefox showing redirect](/assets/2024/caddy-redirect-firefox.png)

## Summary

This was a very short introduction to redirects in caddy and how to load a config file with imported redirects from an external file. Thank you for reading! Let me know what you're using caddy for 👀!

Full source for this example can be found here: [github.com/JonathanMH/stream-kitchensink/tree/master/caddy-docker](https://github.com/JonathanMH/stream-kitchensink/tree/master/caddy-docker)


[caddyserver.com]: https://caddyserver.com/


Caddy is a web server that boasts with easy configuration and great defaults,
such as automatic HTT...

Caddy inside Docker with Redirects


In the last post we looked at [how to create a command line application that stores API responses in our database](/p/caching-with-golang-and-mongodb/). In this post we'll refactor the CLI into a REST API and add the possibility to cache-bust an already stored response.

Let's have a look at how we refactored the code by looking at the functions we have split the linear flow into:

```go
// look up in our database
func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {}
// store a location in our database
func storeLocationRecord(record GeoResponse) error {}
// make a request to the external API
func locationRequest(city string, countryCode string) (GeoResponse, error) {}

// combining the above into one handy function
func getCityLocation(city string, countryCode string) (GeoResponse, error) {
  var record GeoResponse
  record, lookupError := lookupLocationRecord(city, countryCode)
  if lookupError != nil {
    record, requestError := locationRequest(city, countryCode)
    if requestError != nil {
      fmt.Printf("request error: %+v", requestError)
      return record, requestError
    }
    storeLocationRecord(record)
    return record, nil
  }
  return record, nil
}
```

This should still fulfill our expectations of the sequence diagram we included in the previous post. First, check the database, otherwise call api, store result for the next request.

Our `main` function gets significantly simppler, but it also needs to accomodate some code to handle the HTTP request through fiber:

```go
func main() {

  app := fiber.New()

  app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
    city := c.Params("city")
    countryCode := c.Params("country_code")

    record, _ := getCityLocation(city, countryCode)
    return c.JSON(record)
  })

  app.Listen(":3000")
}
```

## Cache Busting via HTTP Headers

Sometimes you have entitled clients that don't trust your cache or they might even have a good reason for requesting the most fresh copy of some data you could provide to them.

Looking at [http cache headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) there are quite a few options that are useful to us!

The most fitting one is `no-cache`, where the client requests a response that has not been cached. We'll look for that header when parsing our response and change our appllication's flow based on that header to either skip the lookup in the database or not.

Let's parse some headers and see if we can make a decision if we should rely on our cached result or not:

```go
cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

var noCache bool
// simplified, should iterate through all headers separated by `,`
if len(cacheControlHeader) != 0 {
  if cacheControlHeader[0] == "no-cache" {
    noCache = true
  }
}
```

There's probably some elegant or standardised way of doing that, but now we can pass it to the function that looks up a record to skip the database part:

```go
if noCache {
  record, requestError := locationRequest(city, countryCode)
  if requestError != nil {
    fmt.Printf("request error: %+v", requestError)
    return record, requestError
  }
  storeLocationRecord(record)
  return record, nil
}
// rest of the function
```

## Golang and Upsert in Mongodb

Going the extra mile would obviously be to update our function to store location records to `storeOrUpdate` and make sure we, in case of a cache bust, store the most up to date values as well!

```go
filter := bson.M{
	"name":    record.City,
	"country": record.CountryCode,
}

update := bson.D{
	{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

opts := options.Update().SetUpsert(true)
_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
```

Luckily `upserts` in mongodb are something that's been well supported for a while and does not actually require two database calls. Although the syntax for inserting OR updating is a bit more cumbersome, it has its advantages.

When we make a request to our local service now, depending on if we pass the header or not, we'll get different log messages from the web server, but same results from the JSON response.

First time we query the API:

```bash
# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]
```

Second time:

```bash
# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
database record: {City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}
```

Third time with no-cache header:

```bash
# curl command:
curl curl --header "Cache-Control: no-cache" localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]
```

The third request successfully ignored the database record and went straight to the external API, success! We've achieved what we wanted for now, next steps include:

* repeat the process for the weather forecast
* possibly skip the caching of the weather response, as weather changes frequenty

Here is the full refactored code below:

```go
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"reflect"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"github.com/gofiber/fiber/v2"
)

type GeoResponse struct {
	City        string  `bson:"name" json:"name"`
	CountryCode string  `bson:"country" json:"country"`
	Lat         float64 `bson:"lat" json:"lat"`
	Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
	serverAPI := options.ServerAPI(options.ServerAPIVersion1)
	opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

	client, err := mongo.Connect(context.TODO(), opts)
	if err != nil {
		log.Fatal(err)
	}

	err = client.Ping(context.TODO(), nil)
	mongoClient = client
	return err
}

func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    city,
		"country": countryCode,
	}

	var dbRecord GeoResponse

	findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

	if findError != nil {
		fmt.Println("database lookup error:", findError)
		return dbRecord, findError
	}

	fmt.Printf("database record: %+v \n", dbRecord)
	return dbRecord, nil
}

func storeLocationRecord(record GeoResponse) error {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    record.City,
		"country": record.CountryCode,
	}

	update := bson.D{
		{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

	opts := options.Update().SetUpsert(true)
	_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
	if dbUpsertError != nil {
		fmt.Println("db insert error:", dbUpsertError)
	}
	return dbUpsertError
}

func locationRequest(city string, countryCode string) (GeoResponse, error) {
	API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"
	res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
	if err != nil {
		log.Fatal(err)
	}

	responseBody, _ := io.ReadAll(res.Body)
	defer res.Body.Close()
	var parsedResponse []GeoResponse
	jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
	if jsonErr != nil {
		fmt.Println("json parsing error:", jsonErr)
		return parsedResponse[0], jsonErr
	}

	fmt.Printf("received from external api: %+v \n", parsedResponse)
	return parsedResponse[0], nil
}

func getCityLocation(city string, countryCode string, noCache bool) (GeoResponse, error) {
	var record GeoResponse
	if noCache {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	record, lookupError := lookupLocationRecord(city, countryCode)
	if lookupError != nil {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	return record, nil
}

func main() {
	dbError := connect_to_mongodb()
	if dbError != nil {
		fmt.Printf("database connection error: %+v", dbError)
	}

	app := fiber.New()

	app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
		city := c.Params("city")
		countryCode := c.Params("country_code")
		cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

		var noCache bool
		if len(cacheControlHeader) != 0 {
			if cacheControlHeader[0] == "no-cache" {
				noCache = true
			}
		}

		// simple validation of parameters
		cityRef := reflect.ValueOf(city)
		countryCodeRef := reflect.ValueOf(countryCode)
		if cityRef.Kind() != reflect.String || countryCodeRef.Kind() != reflect.String {
			return c.Status(400).SendString("parameter error")
		}

		// get location
		record, _ := getCityLocation(city, countryCode, noCache)
		return c.JSON(record)
	})

	app.Listen(":3000")
}
```


In the last post we looked at [how to create a command line application that
stores API responses i...

Cache-Control with GO REST APIs


Sometimes we rely on third party APIs where every request costs us money or have some strict rate limits. To mitigate the financial impact of a project and to have a bit of fun, we can add some caching measures to not call other APIs before either a cache expires or to keep copies of external data sources forever.

**Fun fact:** The API we're using let's us query `1_000_000` times a month before charging us, but let's pretend it's really expensive and limited.

## Example Application: Locations and Weather

One of my use cases was to report the weather for a specific location. There's a pretty good weather API at [openweathermap.org][openweathermap].

Reading the API documentation we'll need to make **2 calls** for every weather request, since the endpoint that returns the weather deals with latitude and longitude.

1. we get a user input (probably a city) and need to find its geographical location in lat/lon [/api/geocoding-api](https://openweathermap.org/api/geocoding-api)
2. the actual request to get the current weather [/current](https://openweathermap.org/current)

### Humble Beginnings: CLI API Requests

Let's start with the simplest application we can think of to validate that a part of it works that we'll need later on. We'll make an API request to locate a city and print the results:

```go
package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
)

func main() {

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh" // your api key here

  city := "Berlin"
  countryCode := "DE"

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }

  defer res.Body.Close()

  body, err := io.ReadAll(res.Body)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(string(body))
}
```

Running this code, we get the following output:

```bash
go run main.go
```

```json
[
  {
    "name":"Berlin",
    "local_names": {"bg":"Берлин","io":"Berlin"},
    "lat": 52.5170365,
    "lon": 13.3888599,
    "country": "DE"
  }
]
```

Sweet! Now let's go ahead and store that response in a database!

### Database Setup

The natural place to save some data we want to keep readily available would of course be the database of our choice.

Let's create a docker-compose file to make sure we can easily spin up a database during development and decide on which shape we want to save the data in. We probably want to keep the data we're interested in and the parameters we want to query by in separate fields.

We don't need to store the full API response or we can stringify the data we would only need sporadically.

```yaml
# file: docker-compose.yml
version: "3.8"
services:
  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
```

Next we're going to write the response we get from the fun weather api to our own database when we run the script:

```go
package main

import (
  "context"
  "encoding/json"
  "fmt"
  "io"
  "log"
  "net/http"

  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)

type GeoResponse struct {
  City        string  `bson:"name" json:"name"`
  CountryCode string  `bson:"country" json:"country"`
  Lat         float64 `bson:"lat" json:"lat"`
  Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
  serverAPI := options.ServerAPI(options.ServerAPIVersion1)
  opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

  client, err := mongo.Connect(context.TODO(), opts)
  if err != nil {
    log.Fatal(err)
  }

  err = client.Ping(context.TODO(), nil)
  mongoClient = client
  return err
}

func main() {
  dbError := connect_to_mongodb()
  if dbError != nil {
    fmt.Printf("database connection error: %+v", dbError)
  }

  collection := mongoClient.Database("weatherdb").Collection("cities")

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"

  city := os.Args[1]
  countryCode os.Args[2]

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }


  // parse api response to json
  responseBody, _ := io.ReadAll(res.Body)
  defer res.Body.Close()
  var parsedResponse []GeoResponse
  jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
  if jsonErr != nil {
    fmt.Println("json parsing error:", jsonErr)
  }

  fmt.Printf("%+v", parsedResponse)
  // save first record in array to the database
  dbResult, dbInsertError := collection.InsertOne(context.TODO(), parsedResponse[0])
  if dbInsertError != nil {
    fmt.Println("db insert error:", dbInsertError)
  }

  fmt.Printf("%+v", dbResult)
}
```

after adding the mongodb dependency to our file, we can run the following commands to try it out:

```bash
# in a separate terminal window
docker-compose up

# then in our project:
go mod init
go mod tidy
go run main.go
```

and we should have a record in our mongodb database `weather`:

![screenshot of mongodb compass gui](/assets/2024/weatherdb-compass-mongodb.png)

or via the mongodb shell:

```mongodb
use weather;
db.cities.find();
```

```js
{
  _id: ObjectId('666b4a7ba2febc2c80e3ec5d'),
  name: 'Berlin',
  country: 'DE',
  lat: 52.5170365,
  lon: 13.3888599
}
```

The problem we now face is, that if we simply run the script again with the same city and country code, we simply would

1. call the weather api (again)
2. save an identical database record (again)

To avoid this we need to first look up in our database IF we have a record that matches something that makes this unique, for example the combination of `city` and `country`.

**Fun fact:** Technically these are not unique, just look at the amount of Springfields in the US, you will not be disappointed. We'll just pretend these are unique enough for now.


## Check the DB before API call

Let's first check our database for information we might have retrieved already, like described in this sequence diagram:

```mermaid
sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: Nope
    our Program->>+WeatherAPI: Hey, what's the location of Berlin?
    WeatherAPI->>-our Program: "lat: 52.5170365, lon: 13.3888599"
    our Program->>+Database: "Berlin is at lat: 52.5170365, lon: 13.3888599"
```

![mermaidjs sequence diagram](/assets/2024/weatherdb-mermaidjs-sequence.png)

Or in the case that we do indeed already have a database record we skip the whole weather api bit!

```mermaid
sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: "Yup, it's at: 52.5170365, lon: 13.3888599"
```

### Golang mongodb.find and parse to struct

Mostly of what we need to add is a quick call to our database first and returning early from the `main` function:

```go
filter := bson.M{
  "name":    city,
  "country": countryCode,
}

var dbRecord GeoResponse

findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

if findError != nil {
  fmt.Println("database lookup error:", findError)
}

if findError == nil {
  fmt.Printf("database record: %+v \n", dbRecord)
  return
}
```

Now if we run our program again, we should see this output, call no external API and not create any duplicate records in the database 🙌.

```bash
database record: {City:Berlin CountryCode:DE Lat:52.5170365 Lon:13.3888599}
```

![our program in action](/assets/2024/weatherdb-vhs.gif)

In the next part we'll look at invalidating some records automatically (locations of city don't often change, but the weather does sometimes) and how to serve all of this through a REST API that fits our needs.

[openweathermap.org]: https://openweathermap.org


Sometimes we rely on third party APIs where every request costs us money or have
some strict rate l...

Caching with Golang and mongodb


World of Warcraft probably is one of the most extendable video games to write addons for that alter the interface and provide lots of customisation  to the players. In this post we'll look at how to write a custom trigger for the weak auras framework to automate build a mini "chatbot" in order to provide explanations for my fellow raiders.

One of the popular low code options to display additional reminders as visual or auditive signals is called [WeakAuras][weak-auras-website], which are often used for group activities like raids where groups of players between 10 and 40 players try to defeat bosses together. In these scenarios some roles or specific players get personal responsibilities like moving to a specific place in the arena or using specific abilities.

If these personal responsiblities apart from their usual role (like hunters shooting arrows or healers giving everyone a bit divine light to top of some health pools) are not taken care of, it will kill the entire group and you need to try the entire fight again. With fights being up to 10 minutes, that can be quite frustrating for the group, so as a raid leader, you often want to remind the players in your group of the mechanics in the fight that is about to happen.

If you don't raid in a guild with the same people 1-3 times a week, you have to invite more or less random players, that are a bit of a wild card when it comes to their knowledge of the encounters, the game in general and if you don't require that they join a voice that they're basically playing the game by themselves apart from the little chat window that only occupies a small portion of their interface.

What I used to do while leading groups through some of the raid dungeons that change every few months is to explain lots of mechanics in chat, by simply typing them out in varying degrees of specificity depending on how capable I believe a group to be.

This quickly got to a very ridiculous amount of typing the same things, which I thought would be a fun thing to automate away and pick up a bit of LUA on a train ride, the language used for weak auras and wow addons in general. All we want is a little bit of chat automation to make our multiple weekly runs through a boss filled raid dungeon a little smoother!

## The Chatbot WeakAura

WeakAuras support a number of built-in reactions that can let you display something on screen, like when a player gains a positive effect, like when a mage can cast a spell instantly instead of a 5 second cast time after another spell. Reacting to chat messages luckily was a built in event listener, which was great, now we just needed to interpret the chat message and respond with the most important explanations for the upcoming boss fight.

![weak aura responding](/assets/2024/weak-aura-chat-window.jpg)

The events we want to listen to are these:

```
CHAT_MSG_PARTY, CHAT_MSG_PARTY_LEADER, CHAT_MSG_RAID, CHAT_MSG_RAID_LEADER, CHAT_MSG_INSTANCE_CHAT
```

Which will let me test the code inside a regular party (for 5 people) or use it in the raid party chat.

As a little bonus, I wanted to only have the *Heroic* difficulty mechanics explained if I'm currently actually in the heroic difficulty raid.

![weak aura with heroic text](/assets/2024/weak-auras-heroic.png)

As I had barely written a line of Lua in my life, I needed to start with a Hello World! example and work my way up from there. Our custom functionality will be implemented in a custom trigger, which lets us write lua code to process incoming events forwarded by the specified events.

Disclaimer: you can edit these while the game is running in a sort of "in addon code editor", but it gets stored as a stringified piece of code that gets evaluated every time your specified event happens. This can make debugging a bit cumbersome and you basically end up copy & pasting it in and out of the game editor into your a dedicated code editor. After changes you probably need to reload your interface by typing `/reload` in the wow chat.

This is what the built in code editor looks like:

![weak auras code editor](/assets/2024/weak-auras-code-editor.webp)
 
I learned from another weak aura that the input for the custom trigger function would look something like this:

```lua
function('CHAT_MSG_RAID', '!explain gnarlroot')
```

Developing the script outside the game client had some pitfalls, for example that World of Warcraft actually adds a couple of globally available functions (which don't exactly have an official documentation) via its API for the convenience of developing addons. One of these functions is `strsplit`, which we kind of stole from stackoverflow:

```lua
-- usually provided by wow API
function strsplit(sep, inputstr)
  if sep == nil then
     sep = "%s"
  end
  local t={}
  for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
     table.insert(t, str)
  end
  return t
end
```

Secondly we needed the functions that we were going to depend on that are exposed by WoW, which we just return multiple values from to mimic that we're in a raid group and if it's on the *Heroic* difficulty or not:

```lua
-- fake functions
function GetInstanceInfo()
  return '', 'raid', 1, '', '', '', '', '', ''
end

function GetDifficultyInfo(fakeArg1)
  return '', '', true, '', true, '', ''
end
```

* `GetInstanceInfo` documentation: [wowpedia.fandom.com/wiki/API_GetInstanceInfo](https://wowpedia.fandom.com/wiki/API_GetInstanceInfo)
* `GetDifficultyInfo` documentation: [wowpedia.fandom.com/wiki/API_GetDifficultyInfo](https://wowpedia.fandom.com/wiki/API_GetDifficultyInfo)

Lastly we need a utility function that checks if a specific string is contained in the chat message to respond with the right boss explanation:


```lua
function FindSubStr(pattern, message)
  if string.find(string.lower(message), string.lower(pattern)) then
    return true
  else
    return false
  end
end
```

Now, putting these together, we can develop our custom trigger lua code locally and run it:

```bash
lua explain.lua
```

```lua
function FindBoss(msgContent)
  if FindSubStr("Gnarlroot", msgContent) then
    return {"Touch all roots with the fire debuff!", "Heroic: drop puddles on the adds!"}
  end
  if FindSubStr("Igira", msgContent) then
    return {"SOAK zones from left to right", "TANKS must stack", "help the HEALERS with the heal absorb"}
  end
-- and so on
end

function Explain(event, message)
  -- START not required in final script
  local function strsplit(sep, inputstr)
    if sep == nil then
      sep = "%s"
    end
    local t={}
    for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
      table.insert(t, str)
    end
    return t
  end
  -- END not required in final script

  local _, _, channel = strsplit("_", event)
  local _,type,difficultyID,_,_,_,_,_,_ = GetInstanceInfo()
  local _,_,isHeroic,_,displayHeroic,_,_ = GetDifficultyInfo(difficultyID)
  local showHeroicText = type == "raid" and displayHeroic;

  local explanationLines = findBoss(message)

  for _, v in ipairs(explanationLines) do
    if (FindSubStr("heroic:", v)) then
      if (showHeroicText) then
        -- print heroic guide lines if actually in heroic raid
        print(v)
      else 
      end
    else
      -- print normal guide lines
      print(v)
    end
  end
end

Explain('CHAT_MSG_RAID', '!explain gnarlroot')
```

which will output:

```
Touch all roots with the fire debuff!
Heroic: drop puddles on the adds!
```


```lua
  local _, _, channel = strsplit("_", event)
```

This means we're getting the *channel* an event happened on and then the string of the message, which is great for us. Now we need to filter messages and not execute anything further if the message doesn't contain `!explain`
and then we need to filter for what boss we would like an explanation for

One of the first things I learned about Lua was that function casing matters and that variables by default are global, which just means some linter will yell at you if you don't prepend them with `local` to give them an explicit scope, which will be used in the full example below.

## Summary

Weak Auras are a great way to get started coding in your favourite video game to update the interface, remind you of boss tactics and shape your experience and the ones of the players around you.

The weak auras I usually use are more complex and maintained by players way more hard core than me, but this was a fun experiment and I got lots of positive feedback over the raids done with the help of the explainy chat messages!

## References

One of the weak auras I stole code from especially for the custom trigger is the "wichtige WeakAura" that reacts to a number of words, usually synonyms for male genitalia, and then every player with the weakaura installed posts an ASCII represanation of a phallus in varying lenghts, in order to compare the randomly generated lengths.

```
Bob: !abfahrt
Bob: B===D
Alice: B=====D
Pete: B==D
```

The documentation for the WoW APIs I found was rather spotty, but I did find a weakaura that flashes a message with the difficulty when entering a raid, so thanks to that one for letting me implement the *Heroic:* messages: [wago.io/TsamMXYKS](https://wago.io/TsamMXYKS)

You can find the complete custom trigger below if you want to write the next weak aura for the next raid tier and you do a lot of random raiding or the weak aura at wago.io: [wago.io/wJvNnrQLP](https://wago.io/wJvNnrQLP)

Full custom trigger code:

```lua
function(event, msg)

    local function findSubStr(pattern, msg)
        if string.find(string.lower(msg), string.lower(pattern)) then
            return true
        else
            return false
        end
    end

    local function findBoss(msgContent)
        local groupsText = "Groups 1,3,5 LEFT, 2,4,6 RIGHT"
        if findSubStr("Gnarlroot", msgContent) then
            return {"Touch all roots with the fire debuff!", "Heroic: drop puddles on the adds!", "Heroic: you must SOAK flame circle to burn roots"}
        end
        if findSubStr("Igira", msgContent) then
            return {"SOAK zones from left to right", "TANKS must stack", "help the HEALERS with the heal absorb", "Heroic: fire debuffed peeps emit balls, dodge them"}
        end
        if findSubStr("Volcoross", msgContent) thehttps://wago.io/wJvNnrQLPn
            return {groupsText, "SOAK zones!", "TANK must take all stacks, taunt on JAWS", "Heroic: The 'tail' emits hot juicy 'lava'"}
        end
        if findSubStr("Larodar", msgContent) then
            return {"FOCUS damage and healing on Treants and Roots",
                "TANKS run away from charge",
                "Heroic: Tank and Melee must CATCH flame balls at boss",
            "Heroic: Brambles need to be DOUSED with seed before you can attack them"}
        end
        if findSubStr("Council", msgContent) then
            return {"Help tanks soak the bear charge", "Duck: eat plants, then go to URSOC and press 2 if he rages", "KILL them at the same time", "Heroic: TANK swap is a frontal", "Heroic: Barreling Charge alternating through ranged/melee players, care for debuff"}
        end
        if findSubStr("Nymue", msgContent) then
            return {groupsText, "SOAK all plants", "Do NOT let adds cross lines"}
        end
        if findSubStr("Smolderon", msgContent) then
            return {"If you DO NOT have a red circle, SOAK with the tank!", "After knockback, collect 5 red balls!"}
        end
        if findSubStr("Tindral", msgContent) then
            return {"Stay close to boss for roots AOE", "Tanks take turns on the mushroom soak", "Stomp seeds", "BL on third platform"}
        end
        if findSubStr("Fyrakk", msgContent) then
            return {"go first left, then right on Dream rend", "Catch all balls in intermission", "focus small adds", "heal souls", "assign seed carrier order"}
        end
    end

    local _, _, channel = strsplit("_", event)
    local _,type,difficultyID,_,_,_,_,_,_ = GetInstanceInfo()
    local _,_,isHeroic,_,displayHeroic,_,_ = GetDifficultyInfo(difficultyID)
    local showHeroicText = type == "raid" and displayHeroic;
    
    if msg then
        -- print(type,isHeroic,displayHeroic)
        if findSubStr("!explain", msg) then
            local explanationLines = findBoss(msg)
            
            for _, v in ipairs(explanationLines) do
                if (findSubStr("heroic:", v)) then
                    if (showHeroicText) then
                        -- print heroic guide lines if actually in heroic raid
                        SendChatMessage(v, channel)
                    else 
                    end
                else
                    SendChatMessage(v, channel)
                end
            end
        end
    end
    
end
```

[weak-auras-website]: https://weakauras.wtf/


World of Warcraft probably is one of the most extendable video games to write
addons for that alter...

Making a Weak Auras Chat Automation


In this post we're going to have a look at how to set up a local grafana loki installation via docker-compose and generating some test data for our fresh installation via a log transport.

Prerequisites:

* docker
* docker-compose
* node.js

Grafana is a great tool for capturing and searching logs and it can be a bit of an overwhelming system to break into. There is also a new query language to learn if you want to query logs in grafana, called *logql*, which we'll be using in a bit. Grafana also has a lot of components that do different things or serve different purposes. We're interested in capturing logs and visualising them for this tutorial.

Let's create a directory like `grafana-playground` and start producing some logs!

## Producing Logs for Grafana

Logging often happens straight to `stdout`, which is the terminal output of the shell running your application. You can pick up these logs with promtail and send them to Grafana. However, you can also log straight from your application and send them to Grafana and many logging libraries will provide you with features to send your logs to grafana instead of the standard out. One of these libraries is [winston][winston] and the relevant transport for [winston-loki][winston-loki].

Let's go ahead and create a new directory called `log-producer` to emit some logs:

```
mkdir log-producer
cd log-producer
npm init -y
npm install winston winston-loki
touch log-test.js
```

Then we can run this file from the official examples of winston-loki:

```javascript
// log-test.js
const winston = require('winston')
const LokiTransport = require('winston-loki');
const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
  labels: { job: 'winston-loki-example' }
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const run = async () => {
  while (true) {
    logger.debug('I am a debug log')
    logger.info('This is a test, no need to panic')
    logger.error('Testing for errors')
    await wait(1000)
  }
}

run()
```

and it will output something like this:

```
{"level":"debug","message":"I am a debug log"}
{"level":"info","message":"This is a test, no need to panic"}
{"level":"error","message":"Testing for errors"}
```

which is cool, but when we close the terminal, these log messages will disappear into the void, so let's go ahead and start a local grafana instance to actually keep and browse these logs.

## Setting up Loki

The simplest way is to follow the [official instructions](https://grafana.com/docs/loki/latest/setup/install/docker/) for installing grafana locally with docker-compose:

```bash
wget https://raw.githubusercontent.com/grafana/loki/v2.9.1/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up
```

The command will download the up to date version of the docker-compose file and start promtail and grafana.

Having a look inside that file shows that grafana will run at port `3100` and the web interface will run at port `3000`.

```yaml
  image: grafana/loki:2.9.1https://github.com/sebhildebrandt/systeminformation
  ports:
    - "3100:3100"
# ...
      url: http://loki:3100
```

run `docker-compose up` and confirm that you actually get a nice splash screen of grafana at [localhost:3000](http://localhost:3000).

Now we can produce some logs to look at in Grafana!

## Using a Grafana Transport for Winston in Node:

Let's go ahead and run the script we placed in `log-producer` earlier again:

```
node log-test.js
```

Now we can go to the Grafana web interface, find *Explore* in the sidebar and start querying your logs.

Select `job` -> `winston-loki-example` in the label filters and hit `Run query`

![grafana query builder](/assets/2023/grafana-query-builder.png)

We sould be able to see a list of logs and a graph showing the split between `info` and `error` severity/level logs. If we also want to capture `debug` logs, we need to explicitly set the loglevel in the transport like so:

```diff
logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
+  level: 'debug',
  labels: { job: 'winston-loki-example' }
}))
```

![grafana log volume](/assets/2023/grafana-log-volume.png)

## Visualising your CPU temperature

For more exciting example data and a better dataset for visualisation, we can grab our CPU temperature and post the current temperature every second. After we start, we can also do something CPU intensive like transcoding video a video and observing the temperature changing.

You can use the [systeminformation package][systeminformation] or parse the output of `sensors`. Turns out `sensors -j` will give you an easily parseable JSON object of various temperature sensors on your Linux system. I had to parse the output from `sensors`, because the node package wasn't working for me. I'll still post logs every second and we can have a look at how we can turn the values into a timeseries graph or diagram in grafana. 🙌

My CPU temperature log producer:

```javascript
// temp.js
const util = require('node:util');
const exec = util.promisify(require('node:child_process').exec);

const winston = require('winston')
const LokiTransport = require("winston-loki");


const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  labels: { job: 'cpu-temp' },
  json: true,
  format: winston.format.json(),
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const cpuTemperature = async () => {
  const { stdout: sensorsOutput } = await exec('sensors -j');
  const sensorsJSON = JSON.parse(sensorsOutput);
  return sensorsJSON['k10temp-pci-00c3']['Tctl']['temp1_input'];
}

const run = async () => {
  while (true) {
    try {
      const temp = await cpuTemperature();
      logger.info({ message: 'got cpu load', temp })
    } catch (e) {
      logger.error({ message: 'could not read cpu load', e })
    }

    await wait(1000);
  }
}

run()
```

Now you should be able to filter after the job `cpu-temp` and see the logs come in.

If you can see the logs after switching the job field, we can click the "add to dashboard" button in the top of the interface, edit the panel and change the type to *Time series*.

![grafana time series](/assets/2023/grafana-time-series.png)

My query is

```logql
{job="cpu-temp"} | json | line_format `{{.temp}}`
```

![grafana parse json fields](/assets/2023/grafana-parse-json.png)

but you can also easily build it in the query builder. The important parts are:

* use the JSON formatter to let grafana use the individual fields of your JSON log messages
* specify the field that we want to plot in our timeseries graph

Lastly we need to add a transform to tell Grafana that our parsed JSON field is a `number` and not a `string`.

![grafana transform field type to number](/assets/2023/grafana-transform.png)

Now you should see a nice green line crawling across your graph. Let's make the data a little more exciting by doing something CPU intensive:


```bash
ffmpeg -i generic-legal-file.mp4 -c:v libvpx-vp9 -b:v 3M generic.webm
```

This leads me to get the nice graph right here and I can see the temperature be elevated for the duration of the video encoding:

![grafana time series cpu temperature](/assets/2023/grafana-cpu-temperature.png)

I hope this can serve as a quick beginners guide to grafana and the basics of visualisation. There's much more to write about and how awesome distributed tracing and logging is, but practical examples that require less than 50 lines of code are usually the best intro to get everyone to start playing with a thing!

[winston]: https://github.com/winstonjs/winston
[winston-loki]: https://github.com/JaniAnttonen/winston-loki
[systeminformation]: https://github.com/sebhildebrandt/systeminformation


In this post we're going to have a look at how to set up a local grafana loki
installation via dock...

Visualising Data with Grafana Loki From Scratch


The Dungeons and Dragons movie was a really light-hearted treat and not as deep or epic as other movies this year, but still pretty entertaining. Some epic special effects, a party of adventurers LITERALLY recruiting people by class and abilities on a quest for loot.

Some of monsters were also hilarous, it's like somebody wrote the brief with a few funny remarks to make everyone laugh at the meeting:

> Yeah and give me a monster that's like those round animals from that guy on youtube! Yeah pretty fat, exactly! Nono, it's still dangerous and an actual threat to the party!

Anyways, if you're into DnD or any generic fantasy stuff, you will have a bunch of chuckles watching this one.


The Dungeons and Dragons movie was a really light-hearted treat and not as deep
or epic as other mo...

Watched: Dungeons & Dragons: Honor Among Thieves


Everything Everywhere All at Once was a great movie to see at the cinema. The fantastic mash-up between Matrix style reality warping multiverse action and a family struggle of first generation immigrants with beaurocracy and what they really wanted to do with their lives.

The absurity and how the movie doesn't take its own visual expression too seriously, made it a special piece. I had mostly remained unspoilered/untrailered and I can only recommend watching it without any prior exposure.

I had a really good time and can definitely recommend watching it for both the fun experience and the message.


Everything Everywhere All at Once was a great movie to see at the cinema. The
fantastic mash-up bet...

Watched: Everything Everywhere All at Once


If you want a truck load of free classic books for your kindle or other e-reader you can download them for free from the fantastic [Project Gutenberg][gutenberg] which collects lots public domain works as ebooks. If a book is not available in the desired format, it can be easily converted to any e-reader.

I purchased a kindle and it's been a surprisingly good experience. I've resisted buying another device to read, but e-ink displays are actually nicer on the eyes and also nicer to read in dark environments like the bedroom without illuminating the entire room with an LED display.

On top of that e-ink displays also perform much better in bright environments like... a beach. Who are we kidding, I'm not going to be at a beach, but occasionally in a not well sun-shielded train compartment.

## Download Free Ebooks for your Kindle 💾

I can only recommend browsing around [Project Gutenberg][gutenberg] bit, because there's lots of absolute gold and things that are probably either on everybodies reading list or something you can very easily fall asleep to (no offense Nietzsche).

For this example I have downloaded is *The Legend of Sleepy Hollow*.

![Sleepy Hollow ebook](/assets/2023/project-gutenberg-book-download.webp)

Huge respect to the project, which has amassed over 70000 public domain books in a convenient online library.

## Convert epub to a kindle compatible format (azw3) 🖋️

If the Kindle download option doesn't work or if your e-reader does not support the epub format, you can convert the downloaded `.epub` with a fun and open source library software called [Calibre][calibre]. You can use your favourite package manager on Linux or `brew install calibre` on MacOS.

When you import a book your Calibre screen should look a little like this:

![calibre overview list](/assets/2023/calibre-list-screen.webp)

After importing a book you can right click and convert into dozens of formats, my kindle paperwhite (5) prefers the `AZW3` format.

![calibre export format kindle azw3](/assets/2023/calibre-convert-export-kindle.webp)

## Download Free Ebooks onto the Kindle Paperwhite 📚

To my surprise the kindle works just like a USB stick, as a generic USB mass storage volume and you can move the downloaded and converted ebook into a directory called Downloads where the files from the amazon store are also saved.

This is very convenient for me, since I usually have some computer with a USB connection nearby before I decide reading a book.

![move ebook to kindle via usb](/assets/2023/kindle-transfer-converted-book.webp)

After unplugging the e-reader and accessing the library you should see the newly acquired book in the list of books. If it doesn't show up, you need to have a look at which formats your e-reader supports.

Now you should be able to enjoy the books on your kindle without any issue. For me all features worked as well, taking notes, chapter progress, page numbers and re-flowing pages with changing font sizes.

![photo of a kindle with the downloaded book](/assets/2023/kindle-gutenberg-photo.webp)

I still need to test which e-readers actually just straight up support .epub, I assume some of the more indie ones just do. If you know any fun trivia or great ebook tricks, get in touch and spill the beans! I'm new to this 😉.

The Onyx Boox Poke 3 has been tested and supports `epub` out of the box.

[gutenberg]: https://www.gutenberg.org/
[calibre]: https://calibre-ebook.com/


If you want a truck load of free classic books for your kindle or other e-reader
you can download t...

Free ebooks for the Kindle Paperwhite with Project Gutenberg


I often ask people to get a blog, mostly because they tell me about something amazing that they just shared with me.

Lots of these people are developers, but the article I'm linking here pretty much applies to every field or interest. Don't skip the embedded video [The Craft of Writing Effectively](https://www.youtube.com/watch?v=vtIzMaLkCaM) either, because it has some absolute gold in it.

> Why did your teachers want you to explain stuff?
> Because they wanted to know whether you understood it1
> You guys don't know how to explain stuff.
> You explain stuff under the model of demonstrating to somebody that you understand it. [...]
> Noone cares about the inside of your head

The article itself has some excellent visualisations as well and I want to highlight why personal blogs can span an incredibly wide spectrum of thoughts and style.

![blog spectrum](/assets/2024/personal-blog-spectrum.avif)

You have no obligation to anyone to conform to a specific style, you can think freely and try things out across a very broad spectrum about any topic, to practise clarity, to play with obscurity, be a clown or write an obituary.

All this can be altruisticly useful to others, but most of all, self-servingly to you.


I often ask people to get a blog, mostly because they tell me about something
amazing that they jus...

Latest Blog Posts: