
Envoy is a very powerful tool for routing requests to your infrastructure. Compared to Nginx also supports gRPC, more advanced request matching and load balancing. Both can be used as a reverse proxy, but envoy has a truck load of features and very naturally integrates many services across different hostnames and deployment setups.

The official envoy github organisation sports a plethora of great examples as well, which makes up for their very dry and counter-intuitive documentation: [github.com/envoyproxy/examples/][envoy-examples]

## Configuring an API Gateway with Docker Compose

In this example I want to build a simple setup of:

1. One envoy proxy acting as an api gateway
2. The first service (users-api) which will be exposed by the gateway
2. The second service (photos-api), only called by the users-api service

Our directory structure looks as follows:

```
├── docker-compose.yaml
├── gateway
│   └── envoy.yaml
├── private-pictures
│   ├── Dockerfile
│   └── main.go
└── users-api
    ├── Dockerfile
    └── main.go
```

Here's what we want to happen when a user requests our envoy proxy:

![](/assets/2025/envoy-apigateway.png)

{/* ```
flowchart TD
    A[User] -->|makes request| B(Envoy)
    B --> C{match route to service}
    C -->|/users/| D[Users API]
    C -->|all others| E[400 not found]
``` */}

Let's start with the docker compose file:

```yaml
# docker-compose.yaml
services:
  gateway:
    image: envoyproxy/envoy:v1.31-latest
    entrypoint: "/usr/local/bin/envoy"
    command:
      - "--config-path /etc/envoy/envoy.yaml"
    volumes:
      - ./gateway/envoy.yaml:/etc/envoy/envoy.yaml
    ports:
      # this will be our externally exposed port
      - "3000:3000"
      # optional, envoy admin ui
      - "9901:9901"
  users-api:
    build: ./users-api/
    # ports:
    #   - "3001:3333"
  photos-api:
    build: ./photos-api/
    # ports:
    #   - "3002:3333"
```

The most important parts of the config are that we're defining our envoy image and mount the config file via these lines:

```yaml
command:
  - "--config-path /etc/envoy/envoy.yaml"
volumes:
  - ./gateway/envoy.yaml:/etc/envoy/envoy.yaml
```

Also we're defining our services, which each consist of the tiniest HTTP API and their own docker-compose file.

```yaml
users-api:
build: ./users-api/
# ports:
#   - "3001:3333" host_port:container_port
```

If you want to access the services individually, you can uncomment these lines and access the users api service at `localhost:3001`. Eventually we only want these services to be accessed VIA our envoy proxy though.

## Public and Private Services

When architecting your infrastructure (past a certain comlexity) you will probably have some microservices that talk to each other but never have to respond to a user directly. This scenario is very common in a microservices architecture, but can occur in any.

In this example we only want to expose one of our services via envoy, which then calls the seconds service. `users-api` is allowed to ask `photos-api`, but there will be no directly exposed route to the `photos-api` in the envoy config.

Docker compose will not expose services hosted, but it will allow the containerss to access each other via their exposed ports and since they're all separte containers, they will have distinct IP addresses on the host machine.

The first service, our `users-api` contains the following code:

```go
// path: ./users-api/main.go
package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
)

func getUser(w http.ResponseWriter, r *http.Request) {
  debug := "service: user, request at /user"
  fmt.Println(debug)

  // calls other service, skips gateway
  // not a dynamic param, unimportant to the example
  resp, err := http.Get("http://photos-api:3333/pictures/user_id")	if err != nil {
    log.Fatalln(err)
  }
  body, err := io.ReadAll(resp.Body)
  if err != nil {
    log.Fatalln(err)
  }
  bodyString := string(body)
  log.Println(bodyString)

  io.WriteString(w, fmt.Sprintf("%v\n", debug))
  io.WriteString(w, bodyString)
}

func main() {
  http.HandleFunc("/user", getUser)

  err := http.ListenAndServe(":3333", nil)
  if err != nil {
    log.Fatal(err)
  }
}
```

The service we don't directly expose is almost identical, except that it handles HTTP requests at a different path:

```go
// path: ./photos-api/main.go
func getPhoto(w http.ResponseWriter, r *http.Request) {
  debug := "service: private-pictures, request at /pictures/user_id"
  fmt.Println(debug)
  io.WriteString(w, "no photos found for user_id")
}

func main() {
  http.HandleFunc("/pictures/user_id", getPhoto)
  // ...
}
```

Again, note that both go services can serve on the same HTTP port, as they get individual IPs / host names for their docker containers.

### Service Dockerfiles

We'll create some very minimalistic Dockerfiles for our services to have them run out GO backend services:

```Dockerfile
# same for users-api and photos-api
FROM golang:1.24-alpine
WORKDIR /app
COPY . .
RUN go build ./main.go
EXPOSE 3333
ENTRYPOINT ["./main"]
```

## Envoy Configuration for Matching Routes

Note: The full envoy config will be available at the bottom of the post, because it's quite verbose.

Envoy lets you match routes in a LOT of different ways, in our config we're going to match all hostnames at the route prefix `/user`, so all requets where the path starts with `/user` will be proxied to the specified service.


```yaml
routes:
  - match:
      # the incoming route of the request example.com/user
      prefix: "/user"
    route:
      # this is the reference to the service you want to route the request to
      cluster: users-api-envoy
  - match:
      # all other requets will result in an empty response with status code 400
      prefix: "/"
    direct_response:
      status: 400
```

To make envoy aware of the service on your network, you will need to add a cluster, we will only define one endpoint address as we're not trying to load balance or failover anything of the sort right now.

```yaml
clusters:
  name: users-api-envoy
    # ...
  load_assignment:
    # this will be referenced in the route config
    cluster_name: users-api-envoy
      - endpoint:
        address:
        socket_address:
            # service name from your docker compose file
            address: users-api
            port_value: 3333
```

You can define as many of these services as you want, depending on how many applications you need to route through the envoy proxy.

## Summary

When we run `docker compose up` we should see both the services and the envoy proxy start, when requesting the exposed endpoint with

```bash
curl -i localhost:3000/user
```

We should see the following output in the terminal executing the curl command:

```
HTTP/1.1 200 OK
date: Sat, 19 Apr 2025 17:14:53 GMT
content-length: 59
content-type: text/plain; charset=utf-8
x-envoy-upstream-service-time: 7
server: envoy

service: user, request at /user
no photos found for user_id
```

and this log in the docker compose output:

```
users-api-1   | service: user, request at /user
photos-api-1  | service: private-pictures, request at /pictures/user_id
users-api-1   | 2025/04/19 17:14:53 no photos found for user_id
```

This is handy for debugging routing issues and making sure that the users request is correctly handled 🙌.

Other requests should by default get a `400` status code.

## Complete Envoy Config 🖨️

Here's the full envoy config for this example for the minimal api gateway example.

```yaml
static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 3000 # map this port in docker-compose file
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/user"
                          route:
                            cluster: users-api-envoy
                        - match:
                            prefix: "/"
                          direct_response:
                            status: 400
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: users-api-envoy
      connect_timeout: 0.25s
      type: strict_dns
      lb_policy: round_robin
      load_assignment:
        cluster_name: users-api-envoy
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: users-api # as defined in your docker-compose file
                      port_value: 3333

admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901
```

[envoy-examples]: https://github.com/envoyproxy/examples/


Envoy is a very powerful tool for routing requests to your infrastructure.
Compared to Nginx also s...

Envoy as an API Gateway


Golang introduced easier path parameter based routing as a vanilla language feature with the 1.22 release, which is incredibly handy if you don't have a lot of routes or if a generic http framework does not make sense for your use case.

To create a parameterised router in Go all you need is the standard library and we'll have a look at a minimalist example below.

## Path Parameters for Routing

Many webservices adhere to RESTful principles where you categorise your routes into resources and let your clients specify which resource to GET, update (PUT, PATCH), create (POST) or DELETE via the URL.

So if I want to see my signed up users I would issue a GET request to `/users` and if I want to see a specific user I would do the same with `/users/1`, the `1` being the relevant path parameter.

## ServeMux in Golang

To start routing within your GO app, let's create a route and handler for the `/hello-world` route:

```go
package main

import (
	"net/http"
)

func main() {
	mux := http.NewServeMux()

	mux.HandleFunc("/hello-world", helloHandler)

	http.ListenAndServe(":3666", mux)
}

func helloHandler(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("Hello World!"))
}
```

This will, when run via `go run main.go` start a server you can access at `localhost:3666/hello-world` and you should see the text `Hello World`.

So far so good, we have a web server and now we want to add a route for the users with a dynamic path parameter. Next we need to add a new route that specifies the `{id}` as a


```diff
	mux.HandleFunc("/hello-world", helloHandler)
+	mux.HandleFunc("/users/{id}", userHandler)
```

Next we're going to add the handler function for the users, which will check if a user exists in our database or not.

```go
func userHandler(w http.ResponseWriter, r *http.Request) {
	userId := r.PathValue("id")
	fmt.Println("got userId:", userId)
	knownUserIds := []string{"1", "2", "3"}
	if slices.Contains(knownUserIds, userId) {
		w.Write([]byte(fmt.Sprintf("user with id %s has been found!", userId)))
		return
	}
	w.WriteHeader(http.StatusNotFound)
	w.Write([]byte(fmt.Sprintf("user with id %s has not been found!", userId)))
}
```

You can test these routes by accessing `localhost:3666/users/1` and `localhost:3666/users/4`
which should give you different output based on if a user is present in our `knownUserIds` slice.

To retrieve the path parameter value we use `userId := r.PathValue("id")`, which needs to specify the same string that we defined in the route (`{id}`) omitting the curly braces.

Note that the data type is a string and that you might need to type-cast it if you're trying to compare it to numbers.

Next we're faking a database by creating a slice of strings with fake user IDs and check if the requested user ID is among the test values.

```go
knownUserIds := []string{"1", "2", "3"}
if slices.Contains(knownUserIds, userId) {
	w.Write([]byte(fmt.Sprintf("user with id %s has been found!", userId)))
	return
}
```

This also makes sure the response is written back to the client.

For all user IDs that can NOT be found, you can use `WriteHeader` to assign the correct HTTP status code of `404`:

```go
w.WriteHeader(http.StatusNotFound)
```

before responding with the relevant text response.

Please note that error handling has largely been omitted for brevity! Web serve responsibly in the wild 😉.


Golang introduced easier path parameter based routing as a vanilla language
feature with the 1.22 r...

Vanilla GO Path Parameters


Making HTTP requests via TOR is simple to achieve in GO and can be useful if you would otherwise hit IP based rate limits or if you simply want to avoid getting your server mistakenly blocked for automating some task like web scraping. Connecting via the TOR network to interact with `.onion` URLs and thereby access content exclusive to TOR.

Please note that you might experience being blocked regardless as lots of TOR exit nodes are listed on the TOR projects website.

In Go you can set a proxy for your HTTP request client, which will be used for all GET, POST, etc requests, in our case will be a TOR `socks5` proxy.

## Setting up a TOR Proxy

If you don't have a TOR proxy set up locally or if you want to run it on a separate machine in your infrastructure, you yan make use of `docker-compose` from the [tor-privoxy repository on github](https://github.com/dockage/tor-privoxy/)

This will get you a docker-compose.yml:

```yaml
services:
  tor-privoxy:
    restart: always
    image: dockage/tor-privoxy:latest
    ports:
      - "9050:9050" # Tor proxy
      - "9051:9051" # Tor control port
      - "8118:8118" # Privoxy (optional)
```

running `docker-compose up` in the directory where your docker-compose.yml is stored, you will start the proxy and expose the port, `9050` being the important port to us.

You can test the proxy via curl:

```bash
curl --proxy socks5h://localhost:9050 'https://check.torproject.org/api/ip'
```

which should output something like:

```json
{"IsTor":true,"IP":"192.42.116.217"}
```

Great, we have a TOR socks5 proxy! Let's look at how to use it in Golang

## Using the GO ProxyURL for TOR connections

Let's look at the below minimum viable go http request using TOR:

```go
package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
  "net/url"
)

func main() {
  // this is needs to target the same port as the TOR proxy is exposed on
  proxyURL, _ := url.Parse("socks5://127.0.0.1:9050")

  // these two lines are all that is needed for the GO requests to be proxied
  httpTransport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
  httpClient := &http.Client{Transport: httpTransport}

  response, httpError := httpClient.Get("https://api.ipify.org?format=json")
  if httpError != nil {
    log.Fatal("error during get request", httpError)
  }
  defer response.Body.Close()

  body, err := io.ReadAll(response.Body)
  if err != nil {
    log.Fatal("error reading response body", err)
  }
  fmt.Println(response.StatusCode, string(body))
  // output: 200 {"ip":"192.42.116.217"}
}

```

Now instead of inlining the configuration of the HTTP client, it would be far more useful if we could use this across multiple requests in our project, so we probably want to move the HTTP client proxy configuration out of our `main.go` file:

```diff
- httpTransport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
- httpClient := &http.Client{Transport: httpTransport}
+ httpClient := GetClient()
```

and create a new file called `client.go`:

```go
package main

import (
  "net/http"
  "net/url"
)

func GetClient() *http.Client {
  proxyURL, _ := url.Parse("socks5://127.0.0.1:9050")
  httpTransport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
  httpClient := &http.Client{Transport: httpTransport}
  return httpClient
}
```

which will allow us to use it anywhere in your project.

On top of using a proxy, you probably want to set a meaningful user agent instead of `Go-http-client/2.0` to make sure people can contact you (or mimic a legic browser user agent).

Let me know if this post helped you or what you need a TOR proxy for on the socials below 😉


Making HTTP requests via TOR is simple to achieve in GO and can be useful if you
would otherwise hi...

Golang Proxy HTTP Requests via TOR


Email verification can be tricky, both to actually get right and for users to deal with, when engineers get over-zealous with their validation requirements.

## What Does an Email Address Look Like?

The simple answer, that spring into most minds could be a short list of requirements:

* it has an `@` and then there's a `.` somewhere
* there's something in front of and after the `@` and behind the `.`

That's almost correct, but even governments struggle with email addresses that don't comply with some made up requirement that top level domains are something like `.co.uk` or `.de`. They can be pretty long these days, but lots of email validations will fail them.

Also `jonathan@gov.co.uk` has two dots in it, so our initial statement would already only be a half-truth.

Luckily there are some specifications that describe what an email address CAN look like, like [RFC: 5322][RFC: 5322] and extensions to it like [RFC: 6854][RFC: 6854]. However, many engineers don't have the luxory of looking up specs while implementing a feature and grab the nearest software package available to accomodate their needs or further clamp down requirements (aka violate the spec) to fulfill some sort of business requirement.

This, a weird talk on YouTube and some reddit comments also taught me that apparently `cow@[dead::beef]` is a valid, spec compliant email address. 🤯

1. `cow` \<- this is fine
2. `[]` possible part of an IP address notation?
3. `dead::beef` valid characters in an IPv6 address (0-9,a-z)

Note: It's up to you, running a business, to disregard specs purposefully, but usually I bet it's done by accident. 

## How to Verify that a String is an Email Address?

There's plenty of packages for various programming languages to ensure this, an often followed approach is to verify compliance of an email address via regular expressions. There's even a website that gathered some regular expressions for different languages (and even SQL dialects) to figure out if a user input is an email address: [emailregex.com][emailregex.com], which is helpful.

I often work with Node.js and there's a LOT of email validation packages on npm. The disappointing thing is that they barely state which RFC spec they target.

Some of them do other interesting things though, like for example the [deep-email-validator][deep-email-validator]

> * Validates email was not generated by disposable email service using disposable-email-domains.
> * Validates MX records are present on DNS.
> * Validates SMTP server is running.
> * Validates mailbox exists on SMTP server.

This was new to me, that fateful night that I stumbled down the email validation rabbit hole. How smart to check the DNS resolution for DNS records and check in the with SMTP server!

Other packages focus solely on the external components of DNS and SMTP like [email-existence][email-existence] while others focues on only the user provided information like [validator.js][validator.js], which actually has a lot of great test cases [in their github repository](https://github.com/validatorjs/validator.js/blob/master/test/validators.test.js)

## Is This a Throwaway Email Account?

There are projects to track which email service providers generate temporary mail boxes for users in order to remain anonymous or creatively experience a number of consecutive trial accounts like [disposable-email-domains](https://github.com/ivolo/disposable-email-domains/). Given how quickly you can register domains and point them at the same service, I would take this list and its accuracy with a grain of salt, since it's prone to go out of date quickly and some of the discarded domains might actually belong to someone being a good netizen.

## Third Party Providers and Email Sending APIs

Companies in the email business also make up creative definitions of when an email address should be considered valid, like [Agillic](https://support.agillic.com/hc/en-gb/articles/360007361511-Understanding-How-Valid-Recipients-are-Measured):

> VALID_EMAIL = Not false - Checks if the email address has previously bounced (1 Hard bounce or 1.000 soft bounces)

This is basically custom to their infrastructure and how they curate their clients data sets over time, which they probably have some experience with. The field name however, might be a little misleading without having their glossary of definitions open on the side.

Some email API providers also create some listicles that have questionable reasoning behind their recommendations, like this post on mailtrap:

> with Deep email validator, an MIT-certified dependency package

I'm sure they meant MIT-licensed, but come on, who proof reads these?

Obviously it's a post to pitch their own services, as we can see later in the article:

> As you can see, for my email service, I use Mailtrap Email Sending, which offers robust email-sending capabilities. 

which brings me closer to my use-case.

### So Jonathan, How DID You Get Into This Rabbithole?

There was a [reddit post that caught my attention](https://www.reddit.com/r/golang/comments/1geoosa/updates_on_the_email_verifier_project_in_go/), which had some interesting critique of the project, calling out possible errors in the code and structure of the project. I gave in to curiosity and started reading the spec and started lifting test cases from various sources like wikipedia, reddit comments and various node libraries, which is where I discovered the questionable state of test cases.

I will applaud the creator of the post for putting themself out there to get some feedback (or github stars) and I'm happy they did, otherwise it would probably have taken me a little longer to get a better understanding of the topic myself!

Even worse, implementing spec compliant email validation might produce some very very unexpected results for engineers and users. Lots of email service providers do not allow users to create local parts (the stuff in front of the `@`) that would theoretically be supported in the spec.

![gmail does only allow certain characters in emails](/assets/2024/gmail-allowed-letters.png)

My email use cases are usually that **something** happens and I need to send someone an email. Like they sign up for a web project and need to click a link to give me some reasonable indicator that they're an individual with a mailbox.

If I would get targeted in some ridiculous attack, someone might get the idea to just tank my email domain reputation by signing up 100_000 bogus accounts that makes **ME** look like a spammer, for attempting to send the "welcome" emails, because I didn't check if an email exists in the first place. This would be unfortunate for me and might just fast track any of my future emails to a spam folder or even worse, get my account blocked from sending.

## Honourable Mention: Golang

One of the excellent things I found about GO is that it comes with a *reasonably* accurate function for parsing email addresses [net/mail ParseAddress](https://pkg.go.dev/net/mail#ParseAddress). It's very conveniently allowing you to parse strings like `Jonathan M. Hethey <public@jonathanmh.com>` into an Address struct:

```go
type Address struct {
  Name    string // "Jonathan M. Hethey"
  Address string // "public@jonathanmh.com"
}
```

This is incredibly useful as a baseline for checking user input and you can layer additional methods on top (like DNS and SMTP probes).

Also, there's a go package which lets you mock the DNS resolver if you want to write some tests for your DNS resolution feature: [go-mockdns](https://github.com/foxcpp/go-mockdns), another feature I was missing from the published GO project.

I haven't had the chance to look into mocking an SMTP server yet, but that sounds like another rabbit hole for another day.

## Summary

1. If I provide an open source library for validation/verification I will note which god damn spec I'm targeting 🤓
2. If I provide functionality for talking to different protocols, I will mock my tests that don't break if someone registers the domain I use in my tests. 🙃
3. If I have elaborate test cases I will strongly consider putting them in a portable format to make them easier for other libraries to adopt. 📖

[RFC: 5322]: https://datatracker.ietf.org/doc/html/rfc5322
[RFC: 6854]: https://datatracker.ietf.org/doc/html/rfc6854
[emailregex.com]: https://emailregex.com/
[deep-email-validator]: https://github.com/mfbx9da4/deep-email-validator
[validator.js]: https://github.com/validatorjs/validator.js/
[email-existence]: https://github.com/scippio/email-existence/


Email verification can be tricky, both to actually get right and for users to
deal with, when engin...

Email Verification and Validation


In Golang, importing modules is easy, however it has some opinions on how to structure your project. You're not supposed to sort the files of your project into different directories, but to keep your files in the root folder of your repository OR split out functionality into other modules.

However, it's not a cop, so you can still use the `package` field and an import path to import modules from a sub-directory.

## Go Idiomatic Directory Example

Let's create an ice cream shop that makes use of an `icecream.go` model and some basic price calculation.

`main.go`

```go
package main

import (
	"fmt"
)

func main() {
	myIcecream := Icecream{Flavour: "vanilla", Scoops: 3, BasePrice: 2.99}
	fmt.Println(myIcecream.Price())
}
```

`icecream.go`

```go
package main

type Icecream struct {
	Flavour   string
	Scoops    int
	BasePrice float32
}

func (i Icecream) Price() float32 {
	return i.BasePrice * float32(i.Scoops)
}
```

**directory structure:**

```
.
├── main.go
└── icecream.go

1 directory, 2 files
```

We can confirm that this works by running `go run main.go icecream.go` and it should print: `8.97`.

## Importing Go Files from Sub-directory Example

If we should decide to structure our projects with sub-directories, we need to make some changes. First of all, we should make sure to have a `go.mod` file that states the name of our package, so let's run `go mod init`, which will by defautl take the name of the directory as the module name.

`go.mod`

```
module icecream-shop

go 1.23.1
```

Next we can create a directory, like `models` and move our `icecream.go` file there.

**directory structure:**

```
.
├── go.mod
├── main.go
└── models
    └── icecream.go

2 directories, 3 files
```

If we try to run our code with `go run .` now, we'll get the following error:

```
# icecream-shop
./main.go:8:16: undefined: Icecream
```

Alright, let's now try to import it, we need to remember that unlike in for example JavaScript, we don't import specific files but modules, which are directories:

`main.go`

```go
package main

import (
	"fmt"
	"icecream-shop/models"
)

func main() {
	myIcecream := models.Icecream{Flavour: "vanilla", Scoops: 3, BasePrice: 2.99}
	fmt.Println(myIcecream.Price())
}
```

Note:

* add `"icecream-shop/models"`
* change `Icecream` to `models.Icecream`

However, we get another error if we try to execute it with `go run .`:

```
main.go:5:2: import "icecream-shop/models" is a program, not an importable package
```

Lastly, we'll need to change the package name of the file we moved into the sub-directory, otherwise we'll not be able to import it and use it:

`icecream.go`

```go
package models

type Icecream struct {
	Flavour   string
	Scoops    int
	BasePrice float32
}

func (i Icecream) Price() float32 {
	return i.BasePrice * float32(i.Scoops)
}
```

Note:

* we changed `package main` to `package models`, signalling that this is its own module

## Summary

If you're structuring your project, evaluate carefully if the functionality you're moving around should in fact be its own package or repository.

If not, remember to use the correct import paths.


In Golang, importing modules is easy, however it has some opinions on how to
structure your project...

Golang: Import Module from Sub Directory


Caddy is a web server that boasts with easy configuration and great defaults, such as automatic HTTPS support by default. While I didn't need https, I was curious about how configuring some redirects and static file serving would go. You can find out more about caddy at [caddyserver.com][caddyserver.com].

If you want to get the **full source** for this repository, the link is at the bottom.

As so often I needed redirects, as I was moving one of my websites to become a static site. Inside my docker container I would have a rendered directory structure that has the paths of the html files and of course some assets.

While migrating away from a CMS, I knew that I would break some of the links on the website. Blog posts like `/peaches` would become `/blog/peaches`, so redirects were important to me, optimally in an external file that I can load into the main config.

## Caddy in the Dockerfile

The simplest way in which you can use caddy in a dockerfile to serve static assets might be this:

```docker
FROM alpine
EXPOSE 5000
RUN apk add --no-cache caddy
RUN mkdir -p /var/www/html
COPY pages/ /var/www/html/
WORKDIR /var/www/html
CMD ["caddy", "file-server", "--listen", ":5000"]
```

Given we have some html files in the `pages` directory like so:

```
pages
├── blog
│   ├── eggplant
│   │   └── index.html
│   ├── ice-cream
│   │   └── index.html
│   └── peaches
│       └── index.html
└── index.html
```

we can serve the files by building the Dockerfile and running it:

```bash
docker build . -t caddy_test
docker run -i -p 127.0.0.1:5000:5000 -t caddy_test
```

Now visiting [localhost:5000](http://localhost:5000) we can see our HTML files.

For me it looks like this with some very minimal example HTML:

![ladybug browser showing example page](/assets/2024/caddy-docker-index.png)

While this will serve our files, we're missing out on the redirects we need to apply.

## Load your Caddyfile

The common config file for a caddy server is the `Caddyfile`, so let's create that one:

```caddy
# file: caddy-config/Caddyfile
# global directive
{
  # turn off https, we're in a trusted environment
  auto_https off
}

:5000 {
  # import redirects
  import /var/app/caddy-config/redirects

  # set the root for our served files
  root * /var/www/html

  # Enable the static file server.
  file_server {
    precompressed zstd br gzip
  }
}
```

This one will also listen on port `5000` and through the `file_server` directive serve the files with some compression enabled.

We can place the file in a sub-directory like `caddy-config` if we don't want to clutter the root of our project.

Now to add redirects we could insert a few lines above the `root` directive, like so:

```caddy
redir /ice-cream /blog/ice-cream permanent
```

### Loading an External Redirects File

For the redirects, we're creating a separate file, which I chose to just call `redirects` without any specific file ending.

```caddy
# redirect to front page
redir /tag/* / permanent
redir /author/* / permanent

# old post structure from /[name] to /blog/[name]

# exact match
redir /ice-cream /blog/ice-cream permanent

# match path and path with trailing slash
redir /peaches /blog/peaches permanent
redir /peaches/ /blog/peaches permanent

# match prefix, every route starting with /eggplant will be redirected
redir /eggplant* /blog/eggplant permanent
```

Please note that depending on how you want your redirects to work you might either want to match explicit paths, or use some of the wildcards you can pick up in the [caddy matcher documentation](https://caddyserver.com/docs/caddyfile/matchers#path-matchers).

To check if everything works to our satisfaction, we can run `docker build` and `docker run` again:

```bash
docker build . -t caddy_test
docker run -i -p 127.0.0.1:5000:5000 -t caddy_test
```

We should see output similar to this:

```
# [...] redacted
Successfully tagged caddy_test:latest
2024/06/16 11:54:18.241 INFO    using provided configuration    {"config_file": "/var/app/caddy-config/Caddyfile", "config_adapter": "caddyfile"}
2024/06/16 11:54:18.241 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "/var/app/caddy-config/Caddyfile", "line": 1}
2024/06/16 11:54:18.242 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/06/16 11:54:18.242 WARN    http.auto_https automatic HTTPS is completely disabled for server       {"server_name": "srv0"}
2024/06/16 11:54:18.242 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0001f6200"}
2024/06/16 11:54:18.242 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/06/16 11:54:18.242 WARN    tls     unable to get instance ID; storage clean stamps will be incomplete      {"error": "open /root/.local/share/caddy/instance.uuid: no such file or directory"}
2024/06/16 11:54:18.242 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/06/16 11:54:18.242 INFO    serving initial configuration
2024/06/16 11:54:18.243 INFO    tls     cleaning storage unit   {"storage": "FileStorage:/root/.local/share/caddy"}
2024/06/16 11:54:18.243 INFO    tls     finished cleaning storage units
```

Now when visiting the example links we can see that we're being redirected 🙌!

![firefox showing redirect](/assets/2024/caddy-redirect-firefox.png)

## Summary

This was a very short introduction to redirects in caddy and how to load a config file with imported redirects from an external file. Thank you for reading! Let me know what you're using caddy for 👀!

Full source for this example can be found here: [github.com/JonathanMH/stream-kitchensink/tree/master/caddy-docker](https://github.com/JonathanMH/stream-kitchensink/tree/master/caddy-docker)


[caddyserver.com]: https://caddyserver.com/


Caddy is a web server that boasts with easy configuration and great defaults,
such as automatic HTT...

Caddy inside Docker with Redirects


In the last post we looked at [how to create a command line application that stores API responses in our database](/p/caching-with-golang-and-mongodb/). In this post we'll refactor the CLI into a REST API and add the possibility to cache-bust an already stored response.

Let's have a look at how we refactored the code by looking at the functions we have split the linear flow into:

```go
// look up in our database
func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {}
// store a location in our database
func storeLocationRecord(record GeoResponse) error {}
// make a request to the external API
func locationRequest(city string, countryCode string) (GeoResponse, error) {}

// combining the above into one handy function
func getCityLocation(city string, countryCode string) (GeoResponse, error) {
  var record GeoResponse
  record, lookupError := lookupLocationRecord(city, countryCode)
  if lookupError != nil {
    record, requestError := locationRequest(city, countryCode)
    if requestError != nil {
      fmt.Printf("request error: %+v", requestError)
      return record, requestError
    }
    storeLocationRecord(record)
    return record, nil
  }
  return record, nil
}
```

This should still fulfill our expectations of the sequence diagram we included in the previous post. First, check the database, otherwise call api, store result for the next request.

Our `main` function gets significantly simppler, but it also needs to accomodate some code to handle the HTTP request through fiber:

```go
func main() {

  app := fiber.New()

  app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
    city := c.Params("city")
    countryCode := c.Params("country_code")

    record, _ := getCityLocation(city, countryCode)
    return c.JSON(record)
  })

  app.Listen(":3000")
}
```

## Cache Busting via HTTP Headers

Sometimes you have entitled clients that don't trust your cache or they might even have a good reason for requesting the most fresh copy of some data you could provide to them.

Looking at [http cache headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) there are quite a few options that are useful to us!

The most fitting one is `no-cache`, where the client requests a response that has not been cached. We'll look for that header when parsing our response and change our appllication's flow based on that header to either skip the lookup in the database or not.

Let's parse some headers and see if we can make a decision if we should rely on our cached result or not:

```go
cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

var noCache bool
// simplified, should iterate through all headers separated by `,`
if len(cacheControlHeader) != 0 {
  if cacheControlHeader[0] == "no-cache" {
    noCache = true
  }
}
```

There's probably some elegant or standardised way of doing that, but now we can pass it to the function that looks up a record to skip the database part:

```go
if noCache {
  record, requestError := locationRequest(city, countryCode)
  if requestError != nil {
    fmt.Printf("request error: %+v", requestError)
    return record, requestError
  }
  storeLocationRecord(record)
  return record, nil
}
// rest of the function
```

## Golang and Upsert in Mongodb

Going the extra mile would obviously be to update our function to store location records to `storeOrUpdate` and make sure we, in case of a cache bust, store the most up to date values as well!

```go
filter := bson.M{
	"name":    record.City,
	"country": record.CountryCode,
}

update := bson.D{
	{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

opts := options.Update().SetUpsert(true)
_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
```

Luckily `upserts` in mongodb are something that's been well supported for a while and does not actually require two database calls. Although the syntax for inserting OR updating is a bit more cumbersome, it has its advantages.

When we make a request to our local service now, depending on if we pass the header or not, we'll get different log messages from the web server, but same results from the JSON response.

First time we query the API:

```bash
# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]
```

Second time:

```bash
# curl command:
curl localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
database record: {City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}
```

Third time with no-cache header:

```bash
# curl command:
curl curl --header "Cache-Control: no-cache" localhost:3000/location/Hildesheim/DE
# output: {"name":"Hildesheim","country":"DE","lat":52.1521636,"lon":9.9513046}

# webserver output:
received from external api: [{City:Hildesheim CountryCode:DE Lat:52.1521636 Lon:9.9513046}]
```

The third request successfully ignored the database record and went straight to the external API, success! We've achieved what we wanted for now, next steps include:

* repeat the process for the weather forecast
* possibly skip the caching of the weather response, as weather changes frequenty

Here is the full refactored code below:

```go
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"reflect"

	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"

	"github.com/gofiber/fiber/v2"
)

type GeoResponse struct {
	City        string  `bson:"name" json:"name"`
	CountryCode string  `bson:"country" json:"country"`
	Lat         float64 `bson:"lat" json:"lat"`
	Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
	serverAPI := options.ServerAPI(options.ServerAPIVersion1)
	opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

	client, err := mongo.Connect(context.TODO(), opts)
	if err != nil {
		log.Fatal(err)
	}

	err = client.Ping(context.TODO(), nil)
	mongoClient = client
	return err
}

func lookupLocationRecord(city string, countryCode string) (GeoResponse, error) {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    city,
		"country": countryCode,
	}

	var dbRecord GeoResponse

	findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

	if findError != nil {
		fmt.Println("database lookup error:", findError)
		return dbRecord, findError
	}

	fmt.Printf("database record: %+v \n", dbRecord)
	return dbRecord, nil
}

func storeLocationRecord(record GeoResponse) error {
	collection := mongoClient.Database("weatherdb").Collection("cities")

	filter := bson.M{
		"name":    record.City,
		"country": record.CountryCode,
	}

	update := bson.D{
		{Key: "$set", Value: bson.D{{Key: "name", Value: record.City}, {Key: "country", Value: record.CountryCode}, {Key: "lat", Value: record.Lat}, {Key: "lon", Value: record.Lon}}}}

	opts := options.Update().SetUpsert(true)
	_, dbUpsertError := collection.UpdateOne(context.TODO(), filter, update, opts)
	if dbUpsertError != nil {
		fmt.Println("db insert error:", dbUpsertError)
	}
	return dbUpsertError
}

func locationRequest(city string, countryCode string) (GeoResponse, error) {
	API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"
	res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
	if err != nil {
		log.Fatal(err)
	}

	responseBody, _ := io.ReadAll(res.Body)
	defer res.Body.Close()
	var parsedResponse []GeoResponse
	jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
	if jsonErr != nil {
		fmt.Println("json parsing error:", jsonErr)
		return parsedResponse[0], jsonErr
	}

	fmt.Printf("received from external api: %+v \n", parsedResponse)
	return parsedResponse[0], nil
}

func getCityLocation(city string, countryCode string, noCache bool) (GeoResponse, error) {
	var record GeoResponse
	if noCache {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	record, lookupError := lookupLocationRecord(city, countryCode)
	if lookupError != nil {
		record, requestError := locationRequest(city, countryCode)
		if requestError != nil {
			fmt.Printf("request error: %+v", requestError)
			return record, requestError
		}
		storeLocationRecord(record)
		return record, nil
	}
	return record, nil
}

func main() {
	dbError := connect_to_mongodb()
	if dbError != nil {
		fmt.Printf("database connection error: %+v", dbError)
	}

	app := fiber.New()

	app.Get("/location/:city/:country_code", func(c *fiber.Ctx) error {
		city := c.Params("city")
		countryCode := c.Params("country_code")
		cacheControlHeader := c.GetReqHeaders()["Cache-Control"]

		var noCache bool
		if len(cacheControlHeader) != 0 {
			if cacheControlHeader[0] == "no-cache" {
				noCache = true
			}
		}

		// simple validation of parameters
		cityRef := reflect.ValueOf(city)
		countryCodeRef := reflect.ValueOf(countryCode)
		if cityRef.Kind() != reflect.String || countryCodeRef.Kind() != reflect.String {
			return c.Status(400).SendString("parameter error")
		}

		// get location
		record, _ := getCityLocation(city, countryCode, noCache)
		return c.JSON(record)
	})

	app.Listen(":3000")
}
```


In the last post we looked at [how to create a command line application that
stores API responses i...

Cache-Control with GO REST APIs


Sometimes we rely on third party APIs where every request costs us money or have some strict rate limits. To mitigate the financial impact of a project and to have a bit of fun, we can add some caching measures to not call other APIs before either a cache expires or to keep copies of external data sources forever.

**Fun fact:** The API we're using let's us query `1_000_000` times a month before charging us, but let's pretend it's really expensive and limited.

## Example Application: Locations and Weather

One of my use cases was to report the weather for a specific location. There's a pretty good weather API at [openweathermap.org][openweathermap].

Reading the API documentation we'll need to make **2 calls** for every weather request, since the endpoint that returns the weather deals with latitude and longitude.

1. we get a user input (probably a city) and need to find its geographical location in lat/lon [/api/geocoding-api](https://openweathermap.org/api/geocoding-api)
2. the actual request to get the current weather [/current](https://openweathermap.org/current)

### Humble Beginnings: CLI API Requests

Let's start with the simplest application we can think of to validate that a part of it works that we'll need later on. We'll make an API request to locate a city and print the results:

```go
package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
)

func main() {

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh" // your api key here

  city := "Berlin"
  countryCode := "DE"

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }

  defer res.Body.Close()

  body, err := io.ReadAll(res.Body)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(string(body))
}
```

Running this code, we get the following output:

```bash
go run main.go
```

```json
[
  {
    "name":"Berlin",
    "local_names": {"bg":"Берлин","io":"Berlin"},
    "lat": 52.5170365,
    "lon": 13.3888599,
    "country": "DE"
  }
]
```

Sweet! Now let's go ahead and store that response in a database!

### Database Setup

The natural place to save some data we want to keep readily available would of course be the database of our choice.

Let's create a docker-compose file to make sure we can easily spin up a database during development and decide on which shape we want to save the data in. We probably want to keep the data we're interested in and the parameters we want to query by in separate fields.

We don't need to store the full API response or we can stringify the data we would only need sporadically.

```yaml
# file: docker-compose.yml
version: "3.8"
services:
  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
```

Next we're going to write the response we get from the fun weather api to our own database when we run the script:

```go
package main

import (
  "context"
  "encoding/json"
  "fmt"
  "io"
  "log"
  "net/http"

  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)

type GeoResponse struct {
  City        string  `bson:"name" json:"name"`
  CountryCode string  `bson:"country" json:"country"`
  Lat         float64 `bson:"lat" json:"lat"`
  Lon         float64 `bson:"lon" json:"lon"`
}

var mongoClient *mongo.Client

func connect_to_mongodb() error {
  serverAPI := options.ServerAPI(options.ServerAPIVersion1)
  opts := options.Client().ApplyURI("mongodb://localhost:27017").SetServerAPIOptions(serverAPI)

  client, err := mongo.Connect(context.TODO(), opts)
  if err != nil {
    log.Fatal(err)
  }

  err = client.Ping(context.TODO(), nil)
  mongoClient = client
  return err
}

func main() {
  dbError := connect_to_mongodb()
  if dbError != nil {
    fmt.Printf("database connection error: %+v", dbError)
  }

  collection := mongoClient.Database("weatherdb").Collection("cities")

  API_KEY := "bbikfunzdxcshrcqbtigmkzmifdojrsh"

  city := os.Args[1]
  countryCode os.Args[2]

  res, err := http.Get(fmt.Sprintf("http://api.openweathermap.org/geo/1.0/direct?q=%s,%s&appid=%s", city, countryCode, API_KEY))
  if err != nil {
    log.Fatal(err)
  }


  // parse api response to json
  responseBody, _ := io.ReadAll(res.Body)
  defer res.Body.Close()
  var parsedResponse []GeoResponse
  jsonErr := json.Unmarshal([]byte(responseBody), &parsedResponse)
  if jsonErr != nil {
    fmt.Println("json parsing error:", jsonErr)
  }

  fmt.Printf("%+v", parsedResponse)
  // save first record in array to the database
  dbResult, dbInsertError := collection.InsertOne(context.TODO(), parsedResponse[0])
  if dbInsertError != nil {
    fmt.Println("db insert error:", dbInsertError)
  }

  fmt.Printf("%+v", dbResult)
}
```

after adding the mongodb dependency to our file, we can run the following commands to try it out:

```bash
# in a separate terminal window
docker-compose up

# then in our project:
go mod init
go mod tidy
go run main.go
```

and we should have a record in our mongodb database `weather`:

![screenshot of mongodb compass gui](/assets/2024/weatherdb-compass-mongodb.png)

or via the mongodb shell:

```mongodb
use weather;
db.cities.find();
```

```js
{
  _id: ObjectId('666b4a7ba2febc2c80e3ec5d'),
  name: 'Berlin',
  country: 'DE',
  lat: 52.5170365,
  lon: 13.3888599
}
```

The problem we now face is, that if we simply run the script again with the same city and country code, we simply would

1. call the weather api (again)
2. save an identical database record (again)

To avoid this we need to first look up in our database IF we have a record that matches something that makes this unique, for example the combination of `city` and `country`.

**Fun fact:** Technically these are not unique, just look at the amount of Springfields in the US, you will not be disappointed. We'll just pretend these are unique enough for now.


## Check the DB before API call

Let's first check our database for information we might have retrieved already, like described in this sequence diagram:

```mermaid
sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: Nope
    our Program->>+WeatherAPI: Hey, what's the location of Berlin?
    WeatherAPI->>-our Program: "lat: 52.5170365, lon: 13.3888599"
    our Program->>+Database: "Berlin is at lat: 52.5170365, lon: 13.3888599"
```

![mermaidjs sequence diagram](/assets/2024/weatherdb-mermaidjs-sequence.png)

Or in the case that we do indeed already have a database record we skip the whole weather api bit!

```mermaid
sequenceDiagram
    our Program->>+Database: Do we have a location for Berlin?
    Database-->>-our Program: "Yup, it's at: 52.5170365, lon: 13.3888599"
```

### Golang mongodb.find and parse to struct

Mostly of what we need to add is a quick call to our database first and returning early from the `main` function:

```go
filter := bson.M{
  "name":    city,
  "country": countryCode,
}

var dbRecord GeoResponse

findError := collection.FindOne(context.TODO(), filter).Decode(&dbRecord)

if findError != nil {
  fmt.Println("database lookup error:", findError)
}

if findError == nil {
  fmt.Printf("database record: %+v \n", dbRecord)
  return
}
```

Now if we run our program again, we should see this output, call no external API and not create any duplicate records in the database 🙌.

```bash
database record: {City:Berlin CountryCode:DE Lat:52.5170365 Lon:13.3888599}
```

![our program in action](/assets/2024/weatherdb-vhs.gif)

In the next part we'll look at invalidating some records automatically (locations of city don't often change, but the weather does sometimes) and how to serve all of this through a REST API that fits our needs.

[openweathermap.org]: https://openweathermap.org


Sometimes we rely on third party APIs where every request costs us money or have
some strict rate l...

Caching with Golang and mongodb


World of Warcraft probably is one of the most extendable video games to write addons for that alter the interface and provide lots of customisation  to the players. In this post we'll look at how to write a custom trigger for the weak auras framework to automate build a mini "chatbot" in order to provide explanations for my fellow raiders.

One of the popular low code options to display additional reminders as visual or auditive signals is called [WeakAuras][weak-auras-website], which are often used for group activities like raids where groups of players between 10 and 40 players try to defeat bosses together. In these scenarios some roles or specific players get personal responsibilities like moving to a specific place in the arena or using specific abilities.

If these personal responsiblities apart from their usual role (like hunters shooting arrows or healers giving everyone a bit divine light to top of some health pools) are not taken care of, it will kill the entire group and you need to try the entire fight again. With fights being up to 10 minutes, that can be quite frustrating for the group, so as a raid leader, you often want to remind the players in your group of the mechanics in the fight that is about to happen.

If you don't raid in a guild with the same people 1-3 times a week, you have to invite more or less random players, that are a bit of a wild card when it comes to their knowledge of the encounters, the game in general and if you don't require that they join a voice that they're basically playing the game by themselves apart from the little chat window that only occupies a small portion of their interface.

What I used to do while leading groups through some of the raid dungeons that change every few months is to explain lots of mechanics in chat, by simply typing them out in varying degrees of specificity depending on how capable I believe a group to be.

This quickly got to a very ridiculous amount of typing the same things, which I thought would be a fun thing to automate away and pick up a bit of LUA on a train ride, the language used for weak auras and wow addons in general. All we want is a little bit of chat automation to make our multiple weekly runs through a boss filled raid dungeon a little smoother!

## The Chatbot WeakAura

WeakAuras support a number of built-in reactions that can let you display something on screen, like when a player gains a positive effect, like when a mage can cast a spell instantly instead of a 5 second cast time after another spell. Reacting to chat messages luckily was a built in event listener, which was great, now we just needed to interpret the chat message and respond with the most important explanations for the upcoming boss fight.

![weak aura responding](/assets/2024/weak-aura-chat-window.jpg)

The events we want to listen to are these:

```
CHAT_MSG_PARTY, CHAT_MSG_PARTY_LEADER, CHAT_MSG_RAID, CHAT_MSG_RAID_LEADER, CHAT_MSG_INSTANCE_CHAT
```

Which will let me test the code inside a regular party (for 5 people) or use it in the raid party chat.

As a little bonus, I wanted to only have the *Heroic* difficulty mechanics explained if I'm currently actually in the heroic difficulty raid.

![weak aura with heroic text](/assets/2024/weak-auras-heroic.png)

As I had barely written a line of Lua in my life, I needed to start with a Hello World! example and work my way up from there. Our custom functionality will be implemented in a custom trigger, which lets us write lua code to process incoming events forwarded by the specified events.

Disclaimer: you can edit these while the game is running in a sort of "in addon code editor", but it gets stored as a stringified piece of code that gets evaluated every time your specified event happens. This can make debugging a bit cumbersome and you basically end up copy & pasting it in and out of the game editor into your a dedicated code editor. After changes you probably need to reload your interface by typing `/reload` in the wow chat.

This is what the built in code editor looks like:

![weak auras code editor](/assets/2024/weak-auras-code-editor.webp)
 
I learned from another weak aura that the input for the custom trigger function would look something like this:

```lua
function('CHAT_MSG_RAID', '!explain gnarlroot')
```

Developing the script outside the game client had some pitfalls, for example that World of Warcraft actually adds a couple of globally available functions (which don't exactly have an official documentation) via its API for the convenience of developing addons. One of these functions is `strsplit`, which we kind of stole from stackoverflow:

```lua
-- usually provided by wow API
function strsplit(sep, inputstr)
  if sep == nil then
     sep = "%s"
  end
  local t={}
  for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
     table.insert(t, str)
  end
  return t
end
```

Secondly we needed the functions that we were going to depend on that are exposed by WoW, which we just return multiple values from to mimic that we're in a raid group and if it's on the *Heroic* difficulty or not:

```lua
-- fake functions
function GetInstanceInfo()
  return '', 'raid', 1, '', '', '', '', '', ''
end

function GetDifficultyInfo(fakeArg1)
  return '', '', true, '', true, '', ''
end
```

* `GetInstanceInfo` documentation: [wowpedia.fandom.com/wiki/API_GetInstanceInfo](https://wowpedia.fandom.com/wiki/API_GetInstanceInfo)
* `GetDifficultyInfo` documentation: [wowpedia.fandom.com/wiki/API_GetDifficultyInfo](https://wowpedia.fandom.com/wiki/API_GetDifficultyInfo)

Lastly we need a utility function that checks if a specific string is contained in the chat message to respond with the right boss explanation:


```lua
function FindSubStr(pattern, message)
  if string.find(string.lower(message), string.lower(pattern)) then
    return true
  else
    return false
  end
end
```

Now, putting these together, we can develop our custom trigger lua code locally and run it:

```bash
lua explain.lua
```

```lua
function FindBoss(msgContent)
  if FindSubStr("Gnarlroot", msgContent) then
    return {"Touch all roots with the fire debuff!", "Heroic: drop puddles on the adds!"}
  end
  if FindSubStr("Igira", msgContent) then
    return {"SOAK zones from left to right", "TANKS must stack", "help the HEALERS with the heal absorb"}
  end
-- and so on
end

function Explain(event, message)
  -- START not required in final script
  local function strsplit(sep, inputstr)
    if sep == nil then
      sep = "%s"
    end
    local t={}
    for str in string.gmatch(inputstr, "([^"..sep.."]+)") do
      table.insert(t, str)
    end
    return t
  end
  -- END not required in final script

  local _, _, channel = strsplit("_", event)
  local _,type,difficultyID,_,_,_,_,_,_ = GetInstanceInfo()
  local _,_,isHeroic,_,displayHeroic,_,_ = GetDifficultyInfo(difficultyID)
  local showHeroicText = type == "raid" and displayHeroic;

  local explanationLines = findBoss(message)

  for _, v in ipairs(explanationLines) do
    if (FindSubStr("heroic:", v)) then
      if (showHeroicText) then
        -- print heroic guide lines if actually in heroic raid
        print(v)
      else 
      end
    else
      -- print normal guide lines
      print(v)
    end
  end
end

Explain('CHAT_MSG_RAID', '!explain gnarlroot')
```

which will output:

```
Touch all roots with the fire debuff!
Heroic: drop puddles on the adds!
```


```lua
  local _, _, channel = strsplit("_", event)
```

This means we're getting the *channel* an event happened on and then the string of the message, which is great for us. Now we need to filter messages and not execute anything further if the message doesn't contain `!explain`
and then we need to filter for what boss we would like an explanation for

One of the first things I learned about Lua was that function casing matters and that variables by default are global, which just means some linter will yell at you if you don't prepend them with `local` to give them an explicit scope, which will be used in the full example below.

## Summary

Weak Auras are a great way to get started coding in your favourite video game to update the interface, remind you of boss tactics and shape your experience and the ones of the players around you.

The weak auras I usually use are more complex and maintained by players way more hard core than me, but this was a fun experiment and I got lots of positive feedback over the raids done with the help of the explainy chat messages!

## References

One of the weak auras I stole code from especially for the custom trigger is the "wichtige WeakAura" that reacts to a number of words, usually synonyms for male genitalia, and then every player with the weakaura installed posts an ASCII represanation of a phallus in varying lenghts, in order to compare the randomly generated lengths.

```
Bob: !abfahrt
Bob: B===D
Alice: B=====D
Pete: B==D
```

The documentation for the WoW APIs I found was rather spotty, but I did find a weakaura that flashes a message with the difficulty when entering a raid, so thanks to that one for letting me implement the *Heroic:* messages: [wago.io/TsamMXYKS](https://wago.io/TsamMXYKS)

You can find the complete custom trigger below if you want to write the next weak aura for the next raid tier and you do a lot of random raiding or the weak aura at wago.io: [wago.io/wJvNnrQLP](https://wago.io/wJvNnrQLP)

Full custom trigger code:

```lua
function(event, msg)

    local function findSubStr(pattern, msg)
        if string.find(string.lower(msg), string.lower(pattern)) then
            return true
        else
            return false
        end
    end

    local function findBoss(msgContent)
        local groupsText = "Groups 1,3,5 LEFT, 2,4,6 RIGHT"
        if findSubStr("Gnarlroot", msgContent) then
            return {"Touch all roots with the fire debuff!", "Heroic: drop puddles on the adds!", "Heroic: you must SOAK flame circle to burn roots"}
        end
        if findSubStr("Igira", msgContent) then
            return {"SOAK zones from left to right", "TANKS must stack", "help the HEALERS with the heal absorb", "Heroic: fire debuffed peeps emit balls, dodge them"}
        end
        if findSubStr("Volcoross", msgContent) thehttps://wago.io/wJvNnrQLPn
            return {groupsText, "SOAK zones!", "TANK must take all stacks, taunt on JAWS", "Heroic: The 'tail' emits hot juicy 'lava'"}
        end
        if findSubStr("Larodar", msgContent) then
            return {"FOCUS damage and healing on Treants and Roots",
                "TANKS run away from charge",
                "Heroic: Tank and Melee must CATCH flame balls at boss",
            "Heroic: Brambles need to be DOUSED with seed before you can attack them"}
        end
        if findSubStr("Council", msgContent) then
            return {"Help tanks soak the bear charge", "Duck: eat plants, then go to URSOC and press 2 if he rages", "KILL them at the same time", "Heroic: TANK swap is a frontal", "Heroic: Barreling Charge alternating through ranged/melee players, care for debuff"}
        end
        if findSubStr("Nymue", msgContent) then
            return {groupsText, "SOAK all plants", "Do NOT let adds cross lines"}
        end
        if findSubStr("Smolderon", msgContent) then
            return {"If you DO NOT have a red circle, SOAK with the tank!", "After knockback, collect 5 red balls!"}
        end
        if findSubStr("Tindral", msgContent) then
            return {"Stay close to boss for roots AOE", "Tanks take turns on the mushroom soak", "Stomp seeds", "BL on third platform"}
        end
        if findSubStr("Fyrakk", msgContent) then
            return {"go first left, then right on Dream rend", "Catch all balls in intermission", "focus small adds", "heal souls", "assign seed carrier order"}
        end
    end

    local _, _, channel = strsplit("_", event)
    local _,type,difficultyID,_,_,_,_,_,_ = GetInstanceInfo()
    local _,_,isHeroic,_,displayHeroic,_,_ = GetDifficultyInfo(difficultyID)
    local showHeroicText = type == "raid" and displayHeroic;
    
    if msg then
        -- print(type,isHeroic,displayHeroic)
        if findSubStr("!explain", msg) then
            local explanationLines = findBoss(msg)
            
            for _, v in ipairs(explanationLines) do
                if (findSubStr("heroic:", v)) then
                    if (showHeroicText) then
                        -- print heroic guide lines if actually in heroic raid
                        SendChatMessage(v, channel)
                    else 
                    end
                else
                    SendChatMessage(v, channel)
                end
            end
        end
    end
    
end
```

[weak-auras-website]: https://weakauras.wtf/


World of Warcraft probably is one of the most extendable video games to write
addons for that alter...

Making a Weak Auras Chat Automation


In this post we're going to have a look at how to set up a local grafana loki installation via docker-compose and generating some test data for our fresh installation via a log transport.

Prerequisites:

* docker
* docker-compose
* node.js

Grafana is a great tool for capturing and searching logs and it can be a bit of an overwhelming system to break into. There is also a new query language to learn if you want to query logs in grafana, called *logql*, which we'll be using in a bit. Grafana also has a lot of components that do different things or serve different purposes. We're interested in capturing logs and visualising them for this tutorial.

Let's create a directory like `grafana-playground` and start producing some logs!

## Producing Logs for Grafana

Logging often happens straight to `stdout`, which is the terminal output of the shell running your application. You can pick up these logs with promtail and send them to Grafana. However, you can also log straight from your application and send them to Grafana and many logging libraries will provide you with features to send your logs to grafana instead of the standard out. One of these libraries is [winston][winston] and the relevant transport for [winston-loki][winston-loki].

Let's go ahead and create a new directory called `log-producer` to emit some logs:

```
mkdir log-producer
cd log-producer
npm init -y
npm install winston winston-loki
touch log-test.js
```

Then we can run this file from the official examples of winston-loki:

```javascript
// log-test.js
const winston = require('winston')
const LokiTransport = require('winston-loki');
const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
  labels: { job: 'winston-loki-example' }
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const run = async () => {
  while (true) {
    logger.debug('I am a debug log')
    logger.info('This is a test, no need to panic')
    logger.error('Testing for errors')
    await wait(1000)
  }
}

run()
```

and it will output something like this:

```
{"level":"debug","message":"I am a debug log"}
{"level":"info","message":"This is a test, no need to panic"}
{"level":"error","message":"Testing for errors"}
```

which is cool, but when we close the terminal, these log messages will disappear into the void, so let's go ahead and start a local grafana instance to actually keep and browse these logs.

## Setting up Loki

The simplest way is to follow the [official instructions](https://grafana.com/docs/loki/latest/setup/install/docker/) for installing grafana locally with docker-compose:

```bash
wget https://raw.githubusercontent.com/grafana/loki/v2.9.1/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up
```

The command will download the up to date version of the docker-compose file and start promtail and grafana.

Having a look inside that file shows that grafana will run at port `3100` and the web interface will run at port `3000`.

```yaml
  image: grafana/loki:2.9.1https://github.com/sebhildebrandt/systeminformation
  ports:
    - "3100:3100"
# ...
      url: http://loki:3100
```

run `docker-compose up` and confirm that you actually get a nice splash screen of grafana at [localhost:3000](http://localhost:3000).

Now we can produce some logs to look at in Grafana!

## Using a Grafana Transport for Winston in Node:

Let's go ahead and run the script we placed in `log-producer` earlier again:

```
node log-test.js
```

Now we can go to the Grafana web interface, find *Explore* in the sidebar and start querying your logs.

Select `job` -> `winston-loki-example` in the label filters and hit `Run query`

![grafana query builder](/assets/2023/grafana-query-builder.png)

We sould be able to see a list of logs and a graph showing the split between `info` and `error` severity/level logs. If we also want to capture `debug` logs, we need to explicitly set the loglevel in the transport like so:

```diff
logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
+  level: 'debug',
  labels: { job: 'winston-loki-example' }
}))
```

![grafana log volume](/assets/2023/grafana-log-volume.png)

## Visualising your CPU temperature

For more exciting example data and a better dataset for visualisation, we can grab our CPU temperature and post the current temperature every second. After we start, we can also do something CPU intensive like transcoding video a video and observing the temperature changing.

You can use the [systeminformation package][systeminformation] or parse the output of `sensors`. Turns out `sensors -j` will give you an easily parseable JSON object of various temperature sensors on your Linux system. I had to parse the output from `sensors`, because the node package wasn't working for me. I'll still post logs every second and we can have a look at how we can turn the values into a timeseries graph or diagram in grafana. 🙌

My CPU temperature log producer:

```javascript
// temp.js
const util = require('node:util');
const exec = util.promisify(require('node:child_process').exec);

const winston = require('winston')
const LokiTransport = require("winston-loki");


const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  labels: { job: 'cpu-temp' },
  json: true,
  format: winston.format.json(),
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const cpuTemperature = async () => {
  const { stdout: sensorsOutput } = await exec('sensors -j');
  const sensorsJSON = JSON.parse(sensorsOutput);
  return sensorsJSON['k10temp-pci-00c3']['Tctl']['temp1_input'];
}

const run = async () => {
  while (true) {
    try {
      const temp = await cpuTemperature();
      logger.info({ message: 'got cpu load', temp })
    } catch (e) {
      logger.error({ message: 'could not read cpu load', e })
    }

    await wait(1000);
  }
}

run()
```

Now you should be able to filter after the job `cpu-temp` and see the logs come in.

If you can see the logs after switching the job field, we can click the "add to dashboard" button in the top of the interface, edit the panel and change the type to *Time series*.

![grafana time series](/assets/2023/grafana-time-series.png)

My query is

```logql
{job="cpu-temp"} | json | line_format `{{.temp}}`
```

![grafana parse json fields](/assets/2023/grafana-parse-json.png)

but you can also easily build it in the query builder. The important parts are:

* use the JSON formatter to let grafana use the individual fields of your JSON log messages
* specify the field that we want to plot in our timeseries graph

Lastly we need to add a transform to tell Grafana that our parsed JSON field is a `number` and not a `string`.

![grafana transform field type to number](/assets/2023/grafana-transform.png)

Now you should see a nice green line crawling across your graph. Let's make the data a little more exciting by doing something CPU intensive:


```bash
ffmpeg -i generic-legal-file.mp4 -c:v libvpx-vp9 -b:v 3M generic.webm
```

This leads me to get the nice graph right here and I can see the temperature be elevated for the duration of the video encoding:

![grafana time series cpu temperature](/assets/2023/grafana-cpu-temperature.png)

I hope this can serve as a quick beginners guide to grafana and the basics of visualisation. There's much more to write about and how awesome distributed tracing and logging is, but practical examples that require less than 50 lines of code are usually the best intro to get everyone to start playing with a thing!

[winston]: https://github.com/winstonjs/winston
[winston-loki]: https://github.com/JaniAnttonen/winston-loki
[systeminformation]: https://github.com/sebhildebrandt/systeminformation


In this post we're going to have a look at how to set up a local grafana loki
installation via dock...

Visualising Data with Grafana Loki From Scratch


I often ask people to get a blog, mostly because they tell me about something amazing that they just shared with me.

Lots of these people are developers, but the article I'm linking here pretty much applies to every field or interest. Don't skip the embedded video [The Craft of Writing Effectively](https://www.youtube.com/watch?v=vtIzMaLkCaM) either, because it has some absolute gold in it.

> Why did your teachers want you to explain stuff?
> Because they wanted to know whether you understood it1
> You guys don't know how to explain stuff.
> You explain stuff under the model of demonstrating to somebody that you understand it. [...]
> Noone cares about the inside of your head

The article itself has some excellent visualisations as well and I want to highlight why personal blogs can span an incredibly wide spectrum of thoughts and style.

![blog spectrum](/assets/2024/personal-blog-spectrum.avif)

You have no obligation to anyone to conform to a specific style, you can think freely and try things out across a very broad spectrum about any topic, to practise clarity, to play with obscurity, be a clown or write an obituary.

All this can be altruisticly useful to others, but most of all, self-servingly to you.


I often ask people to get a blog, mostly because they tell me about something
amazing that they jus...

Latest Blog Posts: