
In this post we're going to have a look at how to set up a local grafana loki installation via docker-compose and generating some test data for our fresh installation via a log transport.

Prerequisites:

* docker
* docker-compose
* node.js

Grafana is a great tool for capturing and searching logs and it can be a bit of an overwhelming system to break into. There is also a new query language to learn if you want to query logs in grafana, called *logql*, which we'll be using in a bit. Grafana also has a lot of components that do different things or serve different purposes. We're interested in capturing logs and visualising them for this tutorial.

Let's create a directory like `grafana-playground` and start producing some logs!

## Producing Logs for Grafana

Logging often happens straight to `stdout`, which is the terminal output of the shell running your application. You can pick up these logs with promtail and send them to Grafana. However, you can also log straight from your application and send them to Grafana and many logging libraries will provide you with features to send your logs to grafana instead of the standard out. One of these libraries is [winston][winston] and the relevant transport for [winston-loki][winston-loki].

Let's go ahead and create a new directory called `log-producer` to emit some logs:

```
mkdir log-producer
cd log-producer
npm init -y
npm install winston winston-loki
touch log-test.js
```

Then we can run this file from the official examples of winston-loki:

```javascript
// log-test.js
const winston = require('winston')
const LokiTransport = require('winston-loki');
const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
  labels: { job: 'winston-loki-example' }
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const run = async () => {
  while (true) {
    logger.debug('I am a debug log')
    logger.info('This is a test, no need to panic')
    logger.error('Testing for errors')
    await wait(1000)
  }
}

run()
```

and it will output something like this:

```
{"level":"debug","message":"I am a debug log"}
{"level":"info","message":"This is a test, no need to panic"}
{"level":"error","message":"Testing for errors"}
```

which is cool, but when we close the terminal, these log messages will disappear into the void, so let's go ahead and start a local grafana instance to actually keep and browse these logs.

## Setting up Loki

The simplest way is to follow the [official instructions](https://grafana.com/docs/loki/latest/setup/install/docker/) for installing grafana locally with docker-compose:

```bash
wget https://raw.githubusercontent.com/grafana/loki/v2.9.1/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up
```

The command will download the up to date version of the docker-compose file and start promtail and grafana.

Having a look inside that file shows that grafana will run at port `3100` and the web interface will run at port `3000`.

```yaml
  image: grafana/loki:2.9.1https://github.com/sebhildebrandt/systeminformation
  ports:
    - "3100:3100"
# ...
      url: http://loki:3100
```

run `docker-compose up` and confirm that you actually get a nice splash screen of grafana at [localhost:3000](http://localhost:3000).

Now we can produce some logs to look at in Grafana!

## Using a Grafana Transport for Winston in Node:

Let's go ahead and run the script we placed in `log-producer` earlier again:

```
node log-test.js
```

Now we can go to the Grafana web interface, find *Explore* in the sidebar and start querying your logs.

Select `job` -> `winston-loki-example` in the label filters and hit `Run query`

![grafana query builder](/assets/2023/grafana-query-builder.png)

We sould be able to see a list of logs and a graph showing the split between `info` and `error` severity/level logs. If we also want to capture `debug` logs, we need to explicitly set the loglevel in the transport like so:

```diff
logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  json: true,
+  level: 'debug',
  labels: { job: 'winston-loki-example' }
}))
```

![grafana log volume](/assets/2023/grafana-log-volume.png)

## Visualising your CPU temperature

For more exciting example data and a better dataset for visualisation, we can grab our CPU temperature and post the current temperature every second. After we start, we can also do something CPU intensive like transcoding video a video and observing the temperature changing.

You can use the [systeminformation package][systeminformation] or parse the output of `sensors`. Turns out `sensors -j` will give you an easily parseable JSON object of various temperature sensors on your Linux system. I had to parse the output from `sensors`, because the node package wasn't working for me. I'll still post logs every second and we can have a look at how we can turn the values into a timeseries graph or diagram in grafana. 🙌

My CPU temperature log producer:

```javascript
// temp.js
const util = require('node:util');
const exec = util.promisify(require('node:child_process').exec);

const winston = require('winston')
const LokiTransport = require("winston-loki");


const logger = winston.createLogger()

logger.add(new winston.transports.Console({
  format: winston.format.json(),
  level: 'debug'
}))

logger.add(new LokiTransport({
  host: 'http://127.0.0.1:3100',
  labels: { job: 'cpu-temp' },
  json: true,
  format: winston.format.json(),
}))

const wait = (duration) => new Promise(resolve => {
  setTimeout(resolve, duration)
})

const cpuTemperature = async () => {
  const { stdout: sensorsOutput } = await exec('sensors -j');
  const sensorsJSON = JSON.parse(sensorsOutput);
  return sensorsJSON['k10temp-pci-00c3']['Tctl']['temp1_input'];
}

const run = async () => {
  while (true) {
    try {
      const temp = await cpuTemperature();
      logger.info({ message: 'got cpu load', temp })
    } catch (e) {
      logger.error({ message: 'could not read cpu load', e })
    }

    await wait(1000);
  }
}

run()
```

Now you should be able to filter after the job `cpu-temp` and see the logs come in.

If you can see the logs after switching the job field, we can click the "add to dashboard" button in the top of the interface, edit the panel and change the type to *Time series*.

![grafana time series](/assets/2023/grafana-time-series.png)

My query is

```logql
{job="cpu-temp"} | json | line_format `{{.temp}}`
```

![grafana parse json fields](/assets/2023/grafana-parse-json.png)

but you can also easily build it in the query builder. The important parts are:

* use the JSON formatter to let grafana use the individual fields of your JSON log messages
* specify the field that we want to plot in our timeseries graph

Lastly we need to add a transform to tell Grafana that our parsed JSON field is a `number` and not a `string`.

![grafana transform field type to number](/assets/2023/grafana-transform.png)

Now you should see a nice green line crawling across your graph. Let's make the data a little more exciting by doing something CPU intensive:


```bash
ffmpeg -i generic-legal-file.mp4 -c:v libvpx-vp9 -b:v 3M generic.webm
```

This leads me to get the nice graph right here and I can see the temperature be elevated for the duration of the video encoding:

![grafana time series cpu temperature](/assets/2023/grafana-cpu-temperature.png)

I hope this can serve as a quick beginners guide to grafana and the basics of visualisation. There's much more to write about and how awesome distributed tracing and logging is, but practical examples that require less than 50 lines of code are usually the best intro to get everyone to start playing with a thing!

[winston]: https://github.com/winstonjs/winston
[winston-loki]: https://github.com/JaniAnttonen/winston-loki
[systeminformation]: https://github.com/sebhildebrandt/systeminformation


In this post we're going to have a look at how to set up a local grafana loki
installation via dock...

Visualising Data with Grafana Loki From Scratch


The Dungeons and Dragons movie was a really light-hearted treat and not as deep or epic as other movies this year, but still pretty entertaining. Some epic special effects, a party of adventurers LITERALLY recruiting people by class and abilities on a quest for loot.

Some of monsters were also hilarous, it's like somebody wrote the brief with a few funny remarks to make everyone laugh at the meeting:

> Yeah and give me a monster that's like those round animals from that guy on youtube! Yeah pretty fat, exactly! Nono, it's still dangerous and an actual threat to the party!

Anyways, if you're into DnD or any generic fantasy stuff, you will have a bunch of chuckles watching this one.


The Dungeons and Dragons movie was a really light-hearted treat and not as deep
or epic as other mo...

Watched: Dungeons & Dragons: Honor Among Thieves


Everything Everywhere All at Once was a great movie to see at the cinema. The fantastic mash-up between Matrix style reality warping multiverse action and a family struggle of first generation immigrants with beaurocracy and what they really wanted to do with their lives.

The absurity and how the movie doesn't take its own visual expression too seriously, made it a special piece. I had mostly remained unspoilered/untrailered and I can only recommend watching it without any prior exposure.

I had a really good time and can definitely recommend watching it for both the fun experience and the message.


Everything Everywhere All at Once was a great movie to see at the cinema. The
fantastic mash-up bet...

Watched: Everything Everywhere All at Once


If you want a truck load of free classic books for your kindle or other e-reader you can download them for free from the fantastic [Project Gutenberg][gutenberg] which collects lots public domain works as ebooks. If a book is not available in the desired format, it can be easily converted to any e-reader.

I purchased a kindle and it's been a surprisingly good experience. I've resisted buying another device to read, but e-ink displays are actually nicer on the eyes and also nicer to read in dark environments like the bedroom without illuminating the entire room with an LED display.

On top of that e-ink displays also perform much better in bright environments like... a beach. Who are we kidding, I'm not going to be at a beach, but occasionally in a not well sun-shielded train compartment.

## Download Free Ebooks for your Kindle 💾

I can only recommend browsing around [Project Gutenberg][gutenberg] bit, because there's lots of absolute gold and things that are probably either on everybodies reading list or something you can very easily fall asleep to (no offense Nietzsche).

For this example I have downloaded is *The Legend of Sleepy Hollow*.

![Sleepy Hollow ebook](/assets/2023/project-gutenberg-book-download.webp)

Huge respect to the project, which has amassed over 70000 public domain books in a convenient online library.

## Convert epub to a kindle compatible format (azw3) 🖋️

If the Kindle download option doesn't work or if your e-reader does not support the epub format, you can convert the downloaded `.epub` with a fun and open source library software called [Calibre][calibre]. You can use your favourite package manager on Linux or `brew install calibre` on MacOS.

When you import a book your Calibre screen should look a little like this:

![calibre overview list](/assets/2023/calibre-list-screen.webp)

After importing a book you can right click and convert into dozens of formats, my kindle paperwhite (5) prefers the `AZW3` format.

![calibre export format kindle azw3](/assets/2023/calibre-convert-export-kindle.webp)

## Download Free Ebooks onto the Kindle Paperwhite 📚

To my surprise the kindle works just like a USB stick, as a generic USB mass storage volume and you can move the downloaded and converted ebook into a directory called Downloads where the files from the amazon store are also saved.

This is very convenient for me, since I usually have some computer with a USB connection nearby before I decide reading a book.

![move ebook to kindle via usb](/assets/2023/kindle-transfer-converted-book.webp)

After unplugging the e-reader and accessing the library you should see the newly acquired book in the list of books. If it doesn't show up, you need to have a look at which formats your e-reader supports.

Now you should be able to enjoy the books on your kindle without any issue. For me all features worked as well, taking notes, chapter progress, page numbers and re-flowing pages with changing font sizes.

![photo of a kindle with the downloaded book](/assets/2023/kindle-gutenberg-photo.webp)

I still need to test which e-readers actually just straight up support .epub, I assume some of the more indie ones just do. If you know any fun trivia or great ebook tricks, get in touch and spill the beans! I'm new to this 😉.

The Onyx Boox Poke 3 has been tested and supports `epub` out of the box.

[gutenberg]: https://www.gutenberg.org/
[calibre]: https://calibre-ebook.com/


If you want a truck load of free classic books for your kindle or other e-reader
you can download t...

Free ebooks for the Kindle Paperwhite with Project Gutenberg


In this post we're going to see how we can build a Mastodon bot with GO that posts pre-planned post once per day. It will support attachments and respective alt texts to cover visually impaired people as well.

The library we're going to use to talk to the mastodon API is [go-mastodon][go-mastodon], which supports all the features we need for uploading assets and attaching them to our post.

## Preparing a Post/Toot

To keep our future posts nice and organised, we'll drop them into directories that have a `$year/$date.toml` structure. We'll use [toml][toml] for our content since it doesn't rely on indentation like yaml and lets us create key/value pairs without much trouble.

The first post ever posted by the [Game Trivia Bot][gametrivia bot] looks like this:

```toml
text = "The official Twitter account for Fall Guys released a \"human for scale\" illustration featuring their playable characters. Fall Guys are all 183cm (6ft) by the way."
tags = ['FallGuys']
assets = ['fallguys-scale.jpeg', 'fallguys-eyes.png']
assetsAlts = ['A Fall Guy next to human', 'Fall Guy eyes and skull']
credits = ['https://www.artstation.com/tudormorris']
```

We have the body `text`, the `tags` which will appear like `#FallGuys`, the `assets` (image files) and the alt text for those image files: `assetsAlts`. Lastly we also have `credits` for the image we're posting because we obviously want our followers to explore the cool creators behind visuals, talks or screenshots.

For our example we will simplify the format a little bit and only support one media attachment per post, like some of the famous bird site accounts that post a fox or an opossum every hour.

Now we also need a function to load the post and to parse it into a data type that we can work with in our program.

Let's create a file called `posts.go` to parse an example toml file:

```go
package main

import (
  "fmt"
  "os"
  "time"

  "github.com/BurntSushi/toml"
)

type Post struct {
  Text       string
  Tags       []string
  Asset      string
  AssetAlt   string
  Credits    string
}

func LoadPost() (Post, error) {
  var post Post
  postFileName := "posts/example.toml"

  _, fileExistErr := os.Stat("./" + postFileName)

  if fileExistErr == nil {
    fmt.Printf("File exists, processing\n")
  } else {
    fmt.Printf("File does not exist\n")
    return post, fileExistErr
  }

  _, tomlError := toml.DecodeFile(postFileName, &post)
  if tomlError != nil {
    fmt.Printf("TOML file reading/decoding error: %v\n", tomlError)
  }

  return post, nil
}
```

To load a post for a specific day, we would have to change the `postFileName` to something like:

```go
  currentTime := time.Now()
  postFileName := fmt.Sprintf("date-posts/%s/%s.toml", currentTime.Format("2006"), currentTime.Format("2006-01-02"))
```

which would allow you to use a directory structure like this:

```
├── date-posts
│   ├── 2022
│   │   ├── 2022-12-01.toml
│   ├── 2023
│   │   ├── 2023-09-23.toml
│   └── drafts
```

We'll get back to actually using the parsed post later.

## Getting the Mastodon API key

Mastodon bots are their own accounts, so you should check with your instance policies if bot accounts are allowed and then register an account with the desired username. Then, in the web interface you need to create an app with the right scopes. We'll need "write" at least, but check whatever you need for your bot.

![mastodon application scopes](/assets/2023/mastodon-api-keys.webp)

find your client key and client secret:

![client key and secret](/assets/2023/mastodon-application-bot-scopes.webp)

We'll also need to tell the bot which server it lives on like `streamers.social` or `mastodon.social`.


## Loading API Configuration

We need to pass the respective credentials to our bot and when we deploy it we probably also want to read it from the environment and not from a file committed to git, so let's create our `main.go` file and a `config.go` file to load our configuration either from a `.env` file in development or from the ENV variables in production.

main.go:

```go
package main

import (
  "fmt"
  "log"
)

var config map[string]string

func main() {

  envs, error := GetConfig()

  if error != nil {
    log.Fatalf("Error loading .env or ENV: %v", error)
  }

  fmt.Printf("%v", envs)
}
```

config.go

```go
package main

import (
  "fmt"
  "os"

  "github.com/joho/godotenv"
)

func readConfigFromENV() map[string]string {
  var envs = make(map[string]string)

  envs["MASTODON_SERVER"] = os.Getenv("MASTODON_SERVER")
  envs["APP_CLIENT_ID"] = os.Getenv("APP_CLIENT_ID")
  envs["APP_CLIENT_SECRET"] = os.Getenv("APP_CLIENT_SECRET")
  envs["APP_USER"] = os.Getenv("APP_USER")
  envs["APP_PASSWORD"] = os.Getenv("APP_PASSWORD")

  return envs
}

func GetConfig() (map[string]string, error) {

  if os.Getenv("APP_ENV") != "production" {
    envs, error := godotenv.Read(".env")
    if error != nil {
      fmt.Println("Error loading .env file")
    }
    return envs, error
  } else {
    envs := readConfigFromENV()
    return envs, nil
  }
}
```

and lastly this is the content of our `.env` file which sould also be listed in your `.gitignore` file:

```
MASTODON_SERVER="https://streamers.social"
APP_CLIENT_ID="2atM1d_KxIswqRn7uvusTe"
APP_CLIENT_SECRET="siuI0GDyOv9xD_htlQ5Tu"
APP_USER="gametrivia@jonathanmh.com"
APP_PASSWORD="super secret password"
```

You will need to replace the values with the ones you got from the web interface and a strong randomly generated password for your bot account.

Now if we run

```bash
go mod init
go mod tidy
go run .
```

We should see the following output that confirms that our config is being loaded correctly:

```
map[APP_CLIENT_ID:2atM1d_KxIswqRn7uvusTe APP_CLIENT_SECRET:siuI0GDyOv9xD_htlQ5Tu APP_PASSWORD:super secret password APP_USER:gametrivia@jonathanmh.com MASTODON_SERVER:https://streamers.social]
```

## Using the Mastodon API

This is great, we've moved our config loading to a file outside of our `main.go` to keep it a bit tidier and now we'll move on to making sure our API connection works as intended. In the examples of go-mastodon we can steal the following snippet:

```go
  c := mastodon.NewClient(&mastodon.Config{
    Server:       "https://mstdn.jp",
    ClientID:     "client-id",
    ClientSecret: "client-secret",
  })
  err := c.Authenticate(context.Background(), "your-email", "your-password")
  if err != nil {
    log.Fatal(err)
  }
  timeline, err := c.GetTimelineHome(context.Background(), nil)
  if err != nil {
    log.Fatal(err)
  }
  for i := len(timeline) - 1; i >= 0; i-- {
    fmt.Println(timeline[i])
  }
```

and we can amend it to use our config like this:

```go
  c := mastodon.NewClient(&mastodon.Config{
    Server:       envs["MASTODON_SERVER"],
    ClientID:     envs["APP_CLIENT_ID"],
    ClientSecret: envs["APP_CLIENT_SECRET"],
  })

  err := c.Authenticate(context.Background(), envs["APP_USER"], envs["APP_PASSWORD"])
```

remember to also import `"github.com/mattn/go-mastodon"` and then we can run:

```
go mod tidy
go run .
```

Which should no emit any errors or output since your bots timeline is empty. If you have manually posted anything, the posts should be listed in your terminal. If your credentials are wrong of if there are any bugs in your code, it should show something like the following error:

```
2023/01/22 17:41:18 bad authorization: 401 Unauthorized: invalid_client
exit status 1
```

## Posting to the Mastodon API

Finally, we'll post our example post, let's create a file called `posts/example.toml` and add a picture called `posts/owl.jpg` and try to post it to the timeline.

example.toml:

```
text = "The majestic owl, owling about in its natural habitat."
tags = ['owl', 'nature', 'photography']
asset = 'owl.jpg'
assetAlt = 'an owl'
credits = 'https://unsplash.com/photos/miEWeTPYsFE'
```

Now we amend the main.go file to include the following:

```go
  post, err := LoadPost()
  if err != nil {
    log.Fatal(err)
  }

  var media mastodon.Media
  file, err := os.Open("./posts/" + post.Asset)
  if err != nil {
    log.Fatal(err)
  }

  media = mastodon.Media{File: file, Description: post.AssetAlt}
  attachment, err := c.UploadMediaFromMedia(context.Background(), &media)
  var attachmentIDs []mastodon.ID

  attachmentIDs = append(attachmentIDs, attachment.ID)

  finalText := post.Text + "\n"

  for i := 0; i < len(post.Tags); i++ {
    finalText = finalText + "#" + post.Tags[i] + " "
  }

  finalText = finalText + "\n" + "Credits: " + post.Credits

  toot := mastodon.Toot{
    Status:   finalText,
    MediaIDs: attachmentIDs,
  }

  fmt.Printf("About to publish: %#v\n", toot)
```

In this snippet we first upload the attachment for the post and save the attachment ID, then we alter the text and append each hashtag prefixed with a `#` and lastly add the credits.

Then we provide both to the `mastodon.Toot` type struct, made available by the library.

If you receive no error, you can add the final lines to post to your bot account:

```go
status, err := c.PostStatus(context.Background(), &toot)

if err != nil {
  log.Fatalf("%#v\n", err)
}
```

Now you should be able to see your post on your bot account:

![bot account published post](/assets/2023/mastodon-bot-toot.webp)

To post periodically I use a cronjob to run the application inside a docker container in production once per day. This takes a bit of planning, but it deploys automatically to dokku on merges to the main branch, which I will cover in the future.

Thank you so much for reading, [drop me a link on mastodon](https://streamers.social/@jonathanmh) to which wonderful bots you're creating!

[go-mastodon]: https://github.com/mattn/go-mastodon
[gametrivia bot]: https://streamers.social/@gametrivia


In this post we're going to see how we can build a Mastodon bot with GO that
posts pre-planned post...

Building a Mastodon bot with GO


Gemini is a smaller than HTTP protocol which only supports a smaller than Markdown syntax and supports virtually no media but ASCII art.

I like the concept and even though it's very niche, I wondered:

> How hard can it be? I have all my content as .md(x) files now anyways, surely it's going to be a quick one!

I was slightly unprepared for some of the pitfalls that of course partly are a result of me using [dokku][dokku] for deploying a lot of my web projects. Dokku by default only lets you handle http and https connections, but there's a cool plugin that lets you handle `gemini://` or other generic TCP connections.

## Preparing Content for Extraction ⚒️

I'm running [Next.js][nextjs] for my main blog for a few weeks now and I really like it. I can write markdown, push to a git remote and have an article published in a workflow that fits me well. On top of that I can write JavaScript components that let me do interactive stuff like [breaking down texts with virtual textmarkers](/p/building-a-text-annotation-component-with-react/) or [blurhash playgrounds](/tools/blurhash).

The metadata for blog posts is saved in frontmatter on top of the file, easy to extract from small scripts, which I made use of for the gemlog mirror.

First I created a new directory, let's say my blog lives in `jonathanmh.com` and my gmi mirror at `gmi-jonathanmh.com`. My content is not actually served based on file path of the `.mdx` file, because I wanted to sort my posts into years like `2021`, `2022` and so on, so I have a `slug` field in my frontmatter that tells Next.js the path.

This is my hacked together content extraction script, cut down a little bit for cases that might not be super relevant for you:

```JavaScript
const fs = require('fs');
const path = require('path');
const glob = require('glob');
const matter = require('gray-matter');
const DateTime = require("luxon").DateTime;

const contentPath = path.resolve(__dirname, '../', '../', 'jonathanmh.com', 'posts');

const files = glob.sync(`${contentPath}/**/*.mdx`, {
  realpath: true
});

let posts = files.map(postFileName => {
  const markdownWithMeta = fs.readFileSync(postFileName, 'utf-8');
  const { data: frontMatter, content } = matter(markdownWithMeta);

  return {
    frontMatter,
    content,
    isDraft: frontMatter.isDraft,
    slug: frontMatter.slug,
    baseName: path.basename(postFileName, '.mdx')
  }
});

posts = posts.sort(function (postA, postB) {
  const postADate = DateTime.fromSQL(postA.frontMatter.date);
  const postBDate = DateTime.fromSQL(postB.frontMatter.date);
  return postBDate.toMillis() - postADate.toMillis();
});

// posts for the index page
const indexPosts = [];

let indexTemplate = fs.readFileSync(path.resolve(__dirname, '../', 'templates', 'index.gmi'), 'utf-8');

for (let i = 0; i < posts.length; i++) {
  const post = posts[i];
  if (i < 10) {
    indexPosts.push(post);
  }

  const destinationPath = path.resolve(__dirname, '../', 'mdContent', `${post.slug}.mdx`);
  const content = `# ${post.frontMatter.title}\n\n${post.frontMatter.date}, by: Jonathan M. Hethey\n\n${post.content}`;
  console.log(`writing ${post.slug}.mdx`);
  fs.writeFileSync(destinationPath, content, 'utf-8');
}

// instead of using a templating engine we're going to stick stuff in this string
let indexPostsString = '';

indexPosts.forEach(post => {
  indexPostsString += `=> ${post.frontMatter.slug}.gmi ${post.frontMatter.title} - (${post.frontMatter.date})\n`
});

console.log('buliding index.gmi')
indexTemplate = indexTemplate.replace('[PLACEHOLDER]', indexPostsString)
const indexDestinationPath = path.resolve(__dirname, '../', 'content', `index.gmi`);

fs.writeFileSync(indexDestinationPath, indexTemplate, 'utf-8');
```

The most important parts are:

We're finding the directory where all the mdx posts are:

```JavaScript
const contentPath = path.resolve(__dirname, '../', '../', 'jonathanmh.com', 'posts');
```

We're not just copying the file, but we're writing `title` and `content` to a new mdx file!

```JavaScript
const content = `# ${post.frontMatter.title}\n\n${post.frontMatter.date}, by: Jonathan M. Hethey\n\n${post.content}`;
```

We're building a gemtext string from a template and writing the last 10 blog posts to it:

```JavaScript
indexPosts.forEach(post => {
	indexPostsString += `=> ${post.frontMatter.slug}.gmi ${post.frontMatter.title} - (${post.frontMatter.date})\n`
});

console.log('buliding index.gmi')
indexTemplate = indexTemplate.replace('[PLACEHOLDER]', indexPostsString)
```

with my `index.gmi` template looking like this:

```gmi
# JonathanMH.com

This is the gemini presence of jonathanmh.com

## Recent Posts:

[PLACEHOLDER]

=> https://jonathanmh.com For an http version of this site

=> https://streamers.social/@jonathanmh I'm on Mastodon

=> Full Archive /archive.gmi
```

This is pretty crude, but does what we want `¯\_(ツ)_/¯`.

Now we still need to convert our markdown to actual gemtext, for which I use [md2gemtext][md2gemtext], which I've paired with the copy script in a small bash file:

```bash
#!/bin/bash

mkdir -p mdContent
node scripts/copyContent.js

for document in mdContent/*.mdx; do
	echo converting $document
	md2gemtext $document content/$(basename $document .mdx).gmi
done
```

Now we have a `content` directory full of our previously markdown, now gmi posts, hell yeah! Alright, now for the deployment!

## Deploying Gemini or other NON HTTP to Dokku 🐳

The Dockerfile for this one was pretty simple, because all the work happens before the server starts, we're using [agate][agate] to serve the files:

```Dockerfile
FROM rust:latest

RUN cargo install agate

WORKDIR /usr/src/blog
COPY content content

EXPOSE 1965
CMD ["agate", "--hostname=jonathanmh.com", "--addr=0.0.0.0:1965"]
```

I wanted my gemini blog to appear at `gemini://jonathanmh.com` and not at a subdomain which is why I added another dokku app and pointed my domain at it and only on port 1965, where as the main site takes care of ports `80` and `443`. For that we also need the `nginx-stream` plugin, since the default plugin only handles http and https:

```bash
sudo dokku plugin:install https://github.com/josegonzalez/dokku-nginx-stream.git
sudo dokku apps:create gmi-jonathanmh-com
sudo dokku domains:add gmi-jonathanm.com jonathanmh.com
sudo dokku proxy:ports:remove 80 gmi-jonathanmh.com
sudo dokku proxy:ports-add gmi-jonathanmh.com tcp:1965:1965
```

If you get an error message like:

```
nginx: [emerg] "stream" directive is duplicate in /etc/nginx/nginx.conf:88
nginx: configuration file /etc/nginx/nginx.conf test failed
```

You can fix your config by opening `/etc/nginx/nginx.conf` and removing this duplicate block in the end:

```nginx
stream {
  include /etc/nginx/app-stream/*.conf;
}
```

This sometimes happens when two plugins try to install the same block and don't check for the presence of a block.

After that you can reload the main nginx config:

```bash
nginx -t
systemctl reload nginx
```

After deploying your app you should be able to see your gemlog at `gemini://yourdomain.tld`, YES! Also very cool that I can point one domain at different applications that respond on different ports, thank you dokku!

## Persisting Gemini Certificates 🔐

What I didn't account for was that `agate` does generate a new certificate and that if a gemini browser previously accessed a page with a different cert, it will show an error page. To persist the certificates we need to mount a volume into our app at the correct directory, so that the certificate gets created the first time the container runs and then loads the same ones afterwards:

```bash
dokku storage:ensure-directory gmi-jonathanmh-agate-certs
dokku storage:mount gmi-jonathanmh.com /var/lib/dokku/data/storage/gmi-jonathanmh-agate-certs:/usr/src/blog/.certificates
# output
=====> gmi-jonathanmh.com storage information
  Storage build mounts:
  Storage deploy mounts:
    -v /var/lib/dokku/data/storage/gmi-jonathanmh-agate-certs:/usr/src/blog/.certificates
  Storage run mounts:
    -v /var/lib/dokku/data/storage/gmi-jonathanmh-agate-certs:/usr/src/blog/.certificates 
dokku ps:restart gmi-jonathanmh.com
```

If you want to make sure it worked, you can run `dokku ps:restart gmi-jonathanmh.com` and then reload the page in your gemini browser. If you get not certificate warning, it is still using the same cert, even after the container being restarted.

## Summary

This was a fun one and it's really nice to see that my content migration paid off. Having an uncomplicated way to access the text part of the content available without joining over 30 tables and being able to parse the metadate with a single NPM package is absolute gold.

Improvements I still want to make is to finish converting my legacy content to markdown, because there are a few stray HTML tags here and there sticking out in the gemlog, which I need to fix.

In the meantime, get a gemini client like [Lagrange][lagrange] and check out [gemini://jonathanmh.com][gemini://jonathanmh.com]

[dokku]: https://dokku.com/
[agate]: https://github.com/mbrubeck/agate
[md2gemtext]: https://docs.rs/md2gemtext/latest/md2gemtext/
[lagrange]: https://github.com/skyjake/lagrange
[gemini://jonathanmh.com]: gemini://jonathanmh.com


Gemini is a smaller than HTTP protocol which only supports a smaller than
Markdown syntax and suppo...

Mirroring Next.js Blog to Gemini


Blurhash is an algorithm to save a tiny version of an image while the high resolution version is being fetched or for NSFW content warnings.

I first stumbled across [blurhash][blurhash] through Mastodon a few years back, but they're not the only ones using it, apparently Wolt themselves and Signal use it as well. In short, it's a string that looks like a very blurry image, which can be scaled seemlessly.

```
eKR2}TXUXo4TtS_M%LxatRad?wMxVs.8WBDONHIpRPX9%gt7n$NGj?
```

By now there's lots of implementations for both en- and decoders [listed on their github repo](https://github.com/woltapp/blurhash/), which makes it really easy to implement.

## Why should I use a Blurhash?

For me, there are two good reasons, performance and more fine-grained control for the users.

### Performance

If you're able to render a blurhash before a high resolution image is loaded, it gives you something to render into the placeholder element that will tell the visitor that something will be there soon and you can mitigate [Cumulative Layout Shift](https://web.dev/cls/) (CLS), which will improve the overall performance of your page load speed.

This may seem counter-intuitive, but rendering an image with unknown dimensions at the top of the page while the page loads will likely cause the browser to re-calculate the position of all elemnents that occur after it, which we want to avoid.

An element that you could use instead of implementing images directly on a blog could look like this:

```html
<div
  className="blurhash-image"
  data-blurhash="eKR2}TXUXo4TtS_M%LxatRad?wMxVs.8WBDONHIpRPX9%gt7n$NGj?"
  data-img-src="/assets/example-dog.jpg"
  style="width: 400px; height: 200px;"
>
</div>
```

and with a JavaScript library like [fast-blurhash](https://github.com/mad-gooze/fast-blurhash) you can pick up the `data-blurhash` value and render it to a canvas element when the page or component loads. When your real asset is available you can swap it out.

If you want to keep the initial page load as small as possible, just wait for the element to come into view through an [intersectionObserver](https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API). You can start loading the image asset if it's close to the scroll depth of the user, remove the `canvas` element and substitute it with the `img` tag.

One of the things crucial for performance is to know the dimensions of the element beforehand, so just not loading the images at first has a limited effect.

### Additional Control for your Users

Another reason to use a blurhash instead just rendering the video is to let users decide if they can or want to view a specific type of content in their feed right now. If you're scrolling through your timeline at work, you maybe want to display the recent upload of your favourite nude photographer as a blurry box and only reveal the actual shot after an extra click, like for example Reddit and Mastodon encourage:

![reddit nsfw mode](/assets/2022/analog-nsfw-blurhash-combined.webp)

Source: [r/analog post](https://www.reddit.com/r/analog/comments/z5dajl/la_sunrise_contax_g2_35mm_portra400/)

Technically reddit does not use blurhash, but, at the time of writing, a combination of a processed jpg, which their servers blur, and a [CSS blur][css blur] filter.

Mastodon on the other hand uses the blurhash algorithm which saves some bandwith compared to having an actual image file, but emits less immediate detail:

![Mastodon NSFW mode](/assets/2022/mastodon-blurhash.webp)

Source: [Mastodon post](https://mastodon.art/@Saltybretzel/109396720818864733)

## Try it out

If you want to try out how your blurhash previews would look, you can play a bit on the [blurhash playground][blurhash playground]. It's still a bit rough around the edges, but it supports different aspect ratios in contrast to the official one website.

The images are not leaving your browser, but are processed by the TypeScript library in that very browser window.

[blurhash]: https://blurha.sh/
[css blur]: https://developer.mozilla.org/docs/Web/CSS/filter-function/blur
[blurhash playground]: /tools/blurhash


Blurhash is an algorithm to save a tiny version of an image while the high
resolution version is be...

What is a Blurhash?


Let's check out how to host a [gemini][gemini] capsule based on the content you have in [hugo][hugo] already!

For this little tutorial, we'll use the Hugo [Getting Started guide](https://gohugo.io/getting-started/quick-start/) and create a small blog with exactly one blog post and then start serving it over `gemini://`.

The markdwon to gemtext conversion and serving will be done through some GO packages, so remember to install [go][go].

## Setting up Hugo

Firstly we'll set up hugo, either through your package manager or through the GO toolchain. Open up a terminal and run the default tutorial instructions:

```bash
hugo new site quickstart
cd quickstart
git init
git submodule add https://github.com/theNewDynamic/gohugo-theme-ananke themes/ananke
echo "theme = 'ananke'" >> config.toml
hugo server
```

Now we also need some example content, so we'll add `my-first-post.md` to the `content` directory:

```markdown
---
title: "My First Post"
date: 2022-11-20T09:03:20-08:00
---
## Introduction

This is **bold** text, and this is *emphasized* text.

Visit the [Hugo](https://gohugo.io) website!
```

now we should see the post show up at `localhost:1313`. So far so good!

## Serving Gemini from Hugo Content

There are three related projects all by [n0x1m][n0x1m] that are going to enable us to serve gemfiles from the existing hugo content. What they did is to grab some hugo code from a few versions ago to generate an identical path tree and they also serve it over gemini, perfect, right?

With the go toolchain installed you can run:

```
go install github.com/n0x1m/hugoext@latest
go install github.com/n0x1m/md2gmi@latest
go install github.com/n0x1m/gmifs@latest
```

Now we can run:

```
hugoext -ext gmi -pipe md2gmi
```

and it will build the `.gmi` file output in the `public` directory `\o/`.

Next, to serve the content we can run:

```
gmifs -autoindex
```

and visit: `gemini://localhost`

What we should see in a gemini browser is this:

![](/assets/2022/gmifs-serve-hugo.png)

Brilliant, we can conclude it works!

Now we just need to deploy it somewhere and point a domain at it.

## Deploying hugoext / gmifs with Docker

I'm used to Docker at this point and I don't have the nerve to learn 100 different hosting providers own config syntax, so we'll write a small Dockerfile.

One of the providers I've been wanting to try out is [fly.io][fly.io], but you can obviously deploy anywhere where Docker images are supported, like heroku, AWS, Azure or whatever you can think of.

```Dockerfile
FROM golang:1.19-alpine

# install GO dependencies
RUN go install github.com/n0x1m/hugoext@latest
RUN go install github.com/n0x1m/md2gmi@latest
RUN go install github.com/n0x1m/gmifs@latest

RUN mkdir -p /var/site
WORKDIR /var/site

COPY content /var/site

EXPOSE 1965

CMD ["gmifs", "-autoindex", "-autocertvalidity=666"]
```

In the `CMD` line we add `"-autocertvalidity=666"`

because by default our certificate the self-signed cert would only be valid for 1 day:

```
-autocertvalidity int
        valid days when using a gmifs provisioned certificate (default 1)
```

Now let's build it with `docker build . -t hugo-gmi` and run with:

```bash
docker run --init -p 1965:1965 -t hugo-gmi
```

which maps the exposed port to the same post on the host system. If you get an error, it's because you still have `gmifs` running on the same port in another terminal.

Accessing `gemini://localhost` should now again show you your website.

### Deploying Docker Image to fly.io

If you want to use [fly.io][fly.io], head on over and create an account first, you'll need to punch in your credit card if you want to deploy.

You'll get some things for free, but be vary of excess bandwidth you might have to pay for:

> Up to 3 shared-cpu-1x 256mb VMs
> 3GB persistent volume storage (total)
> 160GB outbound data transfer 

First we need to install the [flyctl][flyctl], for me that is:

```bash
curl -L https://fly.io/install.sh | sh
```

Then you will want to authenticate your flyctl to your fly.io account, so you run:

```bash
flyctl auth login
```

which will open your browser and let you authenticate the request, after that you're logged in on your system.

Next, to trigger the interactive dialogue for deployment, we run the following in our `quickstart` directory or where your gmi-hugo blog is located.

```bash
flytcl launch
```

We answer a few questions, but the defaults are pretty good. We need to answer with `y` when it comes to if we actually want to deploy.

![flyctl region selection](/assets/2022/flyctl-region.png)

Now we can watch some terminal output of our image being built in production, after it's done we can go to our fly.io dashboard, find our app and access the random subdomain with our gemini browser!

![fly.io dashboard and lagrange](/assets/2022/fly-dashboard-lagrange.png)

Now you just need to grab your DNS settings and point `gmi.yourdomain.com` at this IP address or host to make sure you can be reached at the desired address!

## Closing Thoughts

I got incredibly lucky finding the awesome projects thata are gmifs and hugoext because they were exactly my use case. Trying out fly.io was very pleasant and really perfect for tiny projects and niche blogs!

Your deploy process might vary, you might already have a Dockerfile and might just want to copy your content directory to a different project directory that will only have the dockerfile above to push to a separate remote.

For my own blog I had to use a slightly different approach to get my content in shape to be served through `gemini://jonathanmh.com`, because I don't personally use hugo.

[fly.io]: https://fly.io
[n0x1m]: https://github.com/n0x1m
[go]: https://go.dev/
[hugo]: https://gohugo.io/
[gemini]: https://jonathanmh.com/p/getting-started-with-gemini/


Let's check out how to host a [gemini][gemini] capsule based on the content you
have in [hugo][hugo...

Gemini Mirror from Hugo


Let's have a look at how to start a minimal web server on Windows to make your frontend development easier. This is useful for especially vanilla JavaScript development or if you want to make sure your asset loading works as it would on the server.

Lots of tutorials either focus on one underlying programming language only or they suggest installing more complex servers like nginx or apache through XAMPP which is usually a lot more configuration than you need to do.

We'll use [Chocolatey][chocolatey] to install either node, ruby, python or php which all come with a small web server built in, that you can start on any port to serve your frontend project from the directory you're in with Powershell. You can use any other terminal you have installed, it's just a default on Windows, feel free to use your preferred one.

## The Example Project 😎

Our example Project will have 3 files:

* index.html
* index.js
* games.json

which will mimick a site that loads additional data from a server, which in our case is just a JSON file with some structured data in it, a list of our favourite games!

index.html:

```html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Jonathan's Gaming Site</title>
</head>
<body>
  <h1>A list of my favourite games!</h1>
  <div id="games"></div>
    <h2>debug:</h2>
  <pre id="debug"></pre>
  <script src="index.js"></script>
</body>
</html>
```

* `<script src="index.js"></script>` loads our `index.js`

games.json:

```json
[
  {
    "name": "Warhammer 40k: Darktide",
    "developers": [
      "Fatshark"
    ],
    "publishers": [
      "Fatshark"
    ],
    "released": {
      "year": 2022
    }
  },
  {
    "name": "Valheim",
    "developers": [
      "Iron Gate Studio"
    ],
    "publishers": [
      "Coffee Stain Publishing"
    ],
    "released": {
      "year": 2021
    }
  },
  {
    "name": "Deep Rock Galactic",
    "developers": [
      "Ghost Ship Games"
    ],
    "publishers": [
      "Coffee Stain Publishing"
    ],
    "released": {
      "year": 2020
    } 
  }
]
```

This is our structured data source, you could also imagine that this is a network request to an API.


index.js:

```javascript
console.log('index.js starts executing!')

fetch('./games.json')
  .then((response) => response.json())
  .then((data) => {
    const debugElement = document.querySelector('#games');
    const outputElement = document.querySelector('#games');

    debugElement.innerHTML = JSON.stringify(data)

    let ul = document.createElement("ul")

    data.forEach(element => {
      let li = document.createElement('li')
      li.innerText = `${element.name} (${element.released.year})`
      ul.appendChild(li);
    });

    outputElement.appendChild(ul);
  });
```

* `fetch` lets us load data from another file or server response
* `element.appendChild()` lets us put newly created HTML elements into existing ones

If we were to just double-click our `index.html` from a file manager, it would load the JavaScript file just fine, but as soon as our JavaScript tries to make a network request, it would show us an error in the browsers JavaScript console, since the browser will not let JavaScript make a "network" request to the local file system.

This puts our pages in a bit of a broken state, as you can see here:

![](/assets/2022/non-cors-file-path.png)

The CORS error in Firefox:

```
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at file:///home/jonathan/projects/simple-servers/games.json. (Reason: CORS request not http).
Uncaught (in promise) TypeError: NetworkError when attempting to fetch resource.
    <anonymous> file:///home/jonathan/projects/simple-servers/index.js:3
```

The CORS error in Chrome:

```
Access to fetch at 'file:///home/jonathan/projects/simple-servers/games.json' from origin 'null' has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes: http, data, isolated-app, chrome-extension, chrome, https, chrome-untrusted.
GET file:///home/jonathan/projects/simple-servers/games.json net::ERR_FAILED
(anonymous) @ index.js:7
Uncaught (in promise) TypeError: Failed to fetch
    at index.js:3:1
```

These are not the only contexts for these errors, you can also mitigate them with server headers if you're not trying to load something through the `file://` protocol.

My usual default solution, because I always have Node.js installed would be:

```
npx serve .
# or
npm install -g serve
serve .
```

but it also works with lots of other languages and runtimes, which we're checking out in the next section.

The *working* version of the test site is supposed to look like this:

![](/assets/2022/cors-simple-server-success.png)

## Installing Dependencies with Chocolatey 🍫

You don't **need** to do this step, you can also install [node.js][nodejs] or any of the options from the official websites, it's just more convenient to me to have a package manager like `brew` on MacOS or `apt` or `pacman` on Linux available on Windows as well.

If you open your Powershell as an Administrator after installing [Chocolatey][chocolatey], you can for exmaple install ruby to run a server.

![](/assets/2022/powershell-as-admin.png)

```Powershell
choco install ruby
# cd C:\Users\jonathan\simple-server
ruby -run -e httpd . -p 8080
# if: webrick is not found. You may need to `gem install webrick` to install webrick.
# gem install webrick
```

and you can visit your tiny ruby server at `http://localhost:8080`.

Now your Powershell output should look something like this:

![](/assets/2022/powershell-ruby-server.png)

This also works for python:

```Powershell
choco install python3
# cd C:\Users\jonathan\simple-server
python -m http.server 8080
```

You can even run this straight from within VScode or any other editor that has any terminal integration, you can bring it up with `CTRL+J`:

![](/assets/2022/vscode-python-server.png)

This also works in Sublime and a few others, but I would recommend getting in the habit of having a terminal open in the directory where you're editing code because it's a very common practise in lots of languages.

## Adding Python3 to Path on Windows 🐍

I had to add the Python3 installed through Chocolatey to my Path manually, which was fairly simple if you find your "Advanced System Settings":

![](/assets/2022/windows-powershell-python-path.webp)

You'll need to add a new line to the Environment which is the path to where `python.exe` exists, so we need to check where chocolatey installs it.

## Summary 💡

Now we have a super simple web server that we only run when we want to, so not as a continous background service, that runs on a port of our choosing like 3000 or 8080 and it lets us develop websites much closer to real world conditions than only using the file system like back in the old Frontpage days 🙌.

Further Reading:

* [MDN: local dev server](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/set_up_a_local_testing_server)

[chocolatey]: https://chocolatey.org/
[nodejs]: https://nodejs.org/


Let's have a look at how to start a minimal web server on Windows to make your
frontend development...

Simple Webserver on Windows


In this post I'll illustrate how to pay next-to-nothing for email sending by using the Mailgun FLEX **plan**, if you complete the maze of design dark patterns, that is the mailgun web application.

I was recently in need of a developer friendly email plan to facilitate signup confirmation and password reset emails for a project and one of the services that sprung to mind was [mailgun][mailgun]! 

Obviously not the only mail sending service on the market, but they do in fact offer a pretty good price! It's 1000 mails free and after that about 1 USD per 1000 mails, which is great for not having to host your own email infrastructure that would probably get insta-spam blocked by the big email players anyways.

[The honorable Mr. Wixfort](https://wixfort.de/), when we were debugging a server issue, alerted me that I was in fact on a trial that would charge me 35$ a month for about 50 emails (instead of the first 5000) free ones, because my sending volume is VERY low, so I started digging around for alternatives.

I was sure there was some cheap low volume option, but I couldn't see anything on their pricing page:

![mailgun plans](/assets/2022/mailgun-plans.png)

As I was about to cancel my mailgun account I did find the option to pay as you go with 1000 free emails sends a month! The shady thing is that you need to complete the following steps first:

1. Sign up on a free trial for the Foundation Plan
1. Enter your credit card information
1. Complete cell number verification
1. UNSUBSCRIBE from the trial (this is not actually relevant for you if you just want to send transactional emails, your domain stays verified as well)

I have illustrated the steps below, starting in the billing interface:

![](/assets/2022/mailgun-billing.png)

Unsubscribe

![](/assets/2022/mailgun-flex.png)

Next

![](/assets/2022/mailgun-flex-plan.png)

Finished! You're now "unsubscribed", yet on a "pay as you go" PLAN and you can still send emails.

This must have been the worst UX experience I've had in recent times and the business desire here STRONGLY outweighs the user experience by several orders of magnitude.

You can read about the [flex plan in the mailgun help center](https://help.mailgun.com/hc/en-us/articles/203068914-What-Are-the-Differences-Between-the-Free-and-Flex-Plans-), which doesn't exactly show you how you get there, but at least it tells you how to set a global sending limit for your account.

[mailgun]: https://mailgun.com



In this post I'll illustrate how to pay next-to-nothing for email sending by
using the Mailgun FLEX...