Nginx 410 maps, matching and external files

In order to bulk retire multiple URLs of your web site at once, you can use nginx maps to have the webserver reply with a 410 http code, which means: GONE and not coming back.

This can be common when migrating away from a specific CMS like WordPress that builds a bunch of taxonomy pages that you will no longer be supporting or similar.

Also, this method will work for a any sort of matching you can do, but we’ll focus on 410 and a custom error page, since this is what I had to implement recently.

Lastly we will also make nginx load a custom 410 error page, instead of the boring default.

The full code can be found in the repository: https://github.com/JonathanMH/nginx-410-map

Continue reading “Nginx 410 maps, matching and external files”

Nginx and External Config Files for Redirects

Nginx is my favourite webserver and that’s mostly because I like the config file format and how many features it brings for my needs like reverse proxy for node apps, static file serving and cache headers for SPAs and rewriting APIs into specific paths of a URL.

I’m currently working on a project that will need a few redirects to weed out typos in URLs, to deprecate a number of them (which will just yield a 410 http error code) and otherwise will mostly be static content.

This is why I thought it would be a great idea to find the most convenient way for me to create a lot of redirects in nginx and to have them as part of the project, in code version control and not on some server where they will be forgotten about until something goes wrong.

Continue reading “Nginx and External Config Files for Redirects”

OCR in GO for EFT screenshots (part 1)

In the beginning, there was an idea, that we would catalogue the names of our adversaries, that we triumphed over in Escape from Tarkov.

We shot and looted, carried out dogtags, rarely but eagerly and kept our little kill journal on the shared website.

Then we broke our shared deployment process and my commit permissions were revoked, so my dog tags piled up in my inventory. At the end of the wipe (season) I had killed about 750 PMCs and the process of keeping a log grew tedious.

A new solution needed birthing. I thought to myself:

How hard can OCR / image recognition really be?

Well. There were some unexpected pitfalls on the way.

Finding an OCR library

Turns out there’s a few OCR libraries out there and technically some of those are overkill for our purpose, since a lot of them are trained towards understanding languages and words of a language. We just need strings extracted from a screenshot, which will be things like SexHaver420 or IfakEnjoyer and abbreviations like USEC, so we’re not in need of actually understanding language. We just have a picture that we need to turn into machine readable strings.

I felt like GO would be a good choice, because I can easily build a web app to upload images and either get back a set of strings or even through the github api open automatic pull requests with the data extracted, so I found a go ocr server that I could dump images into and see how well it would do (ran well from the included dockerfile).

After cloning the git repo you can use docker to spin up a local webserver with which you can upload images and test how much of a text the gosseract OCR library can detect (which you get as a JSON response).

Now we’re getting stuff like this from our unedited screenshot:

"i PS Ws) i Cap) St\n. 4 Se CROC ki: =0) Headshot (SVDS, 725m)"

Cleaning up the input image

When I uploaded my first screenshot, it was all a mess, since the screenshots have a lot of additional information we don’t need. Turns out lots of OCR is done on images with a white background (we don’t have that) and with very high contrast fonts (we don’t have that), we also had lots of empty space that didn’t actually contain the text we wanted, so I decided to try some image processing with Imagemagick/Graphicsmagick).

We’re going to:

  1. crop the image
  2. invert the colours
  3. increase the contrast

and see how that goes

sudo apt-get install libgraphicsmagick1-dev

There’s usually great amounts of examples when using any of those utilities and lots of libraries for any backend runtime. I knew I needed something like this GIMP/Photoshop feature (Levels) to increase the contrast of the screenshot.

The official documentation of the graphicsmagic CLI states:

>  -level <black_point>{,<gamma>}{,<white_point>}{%}

	adjust the level of image contrast

	Give one, two or three values delimited with commas: black-point, gamma, white-point (e.g. 10,1.0,250 or 2%,0.5,98%). adjust the level of image contrast
Give one, two or three values delimited with commas: black-point, gamma, white-point (e.g. 10,1.0,250 or 2%,0.5,98%).

Great, so I could just set black and white points in percentages? That sounds like a convenient option!

Turns out the GO library doesn’t let you do that:

func (mw *MagickWand) LevelImage(blackPoint, gamma, whitePoint float64) error

Source

This one just wants three floating point numbers between 0 and the maximum quantum value. Turns out the maximum quantum value is determined by what bit-depth you image has and it is not (necessarily) a number between 0 and 255 and the function to figure out the max quantum value is still on the TODO of the library last released in 2017 ¯\_(ツ)_/¯.

Anyways, for my testing I just assumed a max quantum value of 65356, which worked out much better:

// make image black/white
mw.SetImageColorspace(gmagick.COLORSPACE_GRAY)
// invert
mw.NegateImage(false)
// increase contrast
assumed_quantum_max := 65356.0
blackPoint := assumed_quantum_max / 2.3
gamma := .2
whitePoint := assumed_quantum_max / .5
mw.LevelImage(blackPoint, gamma, whitePoint)

This code will invert (negate) and level the image and make the lowest half (ish) black, make most of the greys black and make everything else white.

This image performs much better and with some additional cropping we actually get a decent output from our OCR server:

{
"result": "# TIME PLAYER LVL FACTION STATUS\n\n1 dotHaku 39 USEC Headshat (SVOS, 7@.5m)\n\n2 CementMixerr 12. -USEC Killed (SVDS, thorax, 29m)\n3 Jilpe 11: USEC Killed (SVDS, thorax, 214m)\n4 Kocherga -- GUARD Headshat (SVDS, 9.8m)\n\n5 Zimniy -- GUARD Killed (SVDOS, thorax, 16.8m)\n6 Svinec -- GUARD Killed (SVDS, thorax, 4.1m)\n7 Maksim Minskiy --  SCAV Headshot (Mk 16, 99m)\n\n8 Yastreb -- RAIDER Headshot (Mk 16, 785m)",
    "version": "0.2.0"
}

which is a lot better than the earlier output, but it still could do with some improvement, since I’m pretty sure it’s supposed to be Headshot instead of Headshat.

For the cropping we kind of just vaguely measured the x and y points in GIMP and it works out for my resolution screenshots (2560 x 1440).

mw.CropImage(1300, 800, 600, 200)

The Result

So far we’ve made progress on kind of optimising an image from:

to:

which is pretty cool.

If you want to try it yourself, this would be the full code for now:

go.mod

module doggy-tagger

go 1.18

require github.com/gographics/gmagick v1.0.0

main.go

package main

import (
    &quot;flag&quot;

    &quot;github.com/gographics/gmagick&quot;
)

func cropAndLevel(sourceImage string, outputImage string) {
    mw := gmagick.NewMagickWand()
    defer mw.Destroy()
    mw.ReadImage(sourceImage)

    // crop
    mw.CropImage(1300, 800, 600, 200)

    // make image black/white
    mw.SetImageColorspace(gmagick.COLORSPACE_GRAY)
    // invert
    mw.NegateImage(false)

    // increase contrast
    assumed_quantum_max := 65356.0
    blackPoint := assumed_quantum_max / 2.3
    gamma := .2
    whitePoint := assumed_quantum_max / .5
    mw.LevelImage(blackPoint, gamma, whitePoint)

    mw.WriteImage(outputImage)
}

func main() {
    f := flag.String(&quot;from&quot;, &quot;&quot;, &quot;original image file ...&quot;)
    t := flag.String(&quot;to&quot;, &quot;&quot;, &quot;target file ...&quot;)
    flag.Parse()

    gmagick.Initialize()
    defer gmagick.Terminate()

    cropAndLevel(*f, *t)
}

and can be run by: go run . -from example-in.png -to example-out.png

Now, the next step will be to turn see how well we can tune this and gosseract to consistently pass a set of example images and later on to automate bulk-processing end of raid screenshots and just spitting out lists of names. Let’s see when we get to it, but so far it’s been a lot of fun!

Music Monday: Devil Special (Dover, Bridge City Sinners)

Welcome to this episode of Music Monday! Today we’ll have two bands that in one way or another have a popular song regarding the devil! The Rolling Stones aren’t part of that list, I’m guessing you know their take on him.

The devil today means something very different to different groups of the population. Depending on how much or how you believe in a big religion, he’s a very real fallen angel that “makes you do bad things” and got you kicked out of paradise. I guess the semantic drift that the word has is that it means either some true, often immoral, desire or temptation that either we or society doesn’t approve of giving in to.

Continue reading “Music Monday: Devil Special (Dover, Bridge City Sinners)”

Mocking HTTP Connections in Golang Tests (scraping and API)

In this article we’re going to have a look at how to mock http connections while running your tests on your golang application.

Since we do not want to call the real API or crawl an actual site to make sure that our application works correctly, we want to make sure that our application works within our defined parameters.

Continue reading “Mocking HTTP Connections in Golang Tests (scraping and API)”

How to Fix: Adobe Premiere JPG / Still Image Glitch

This week I tried to get a still image into one of my videos, but they showed some weird artifacts.

I use Adobe Premiere Pro CC (2020) for most of my editing and it was really bothersome to figure this one out. In fact I don’t really consider this solved, but only “worked around”.

Continue reading “How to Fix: Adobe Premiere JPG / Still Image Glitch”

Updating PHP 5 (or any other) Apps with Docker

PHP5, released in 2014, has reached its end of life in 2018, but many hosters continued to support it. If you have any custom PHP applications, you might still be running them on a server that has a PHP5 runtime.

The upgrade path for these applications might not be obvious, but I found a way that should make it easier, so in this article we’re going to have a look at running PHP simultaneously with PHP5 and PHP7 using docker and docker-compose.

Continue reading “Updating PHP 5 (or any other) Apps with Docker”

How to set up an SSH Honeypot (low interaction) and live stream it

I recently found a streamer on twitch that had an interesting title:

Watch people trying to hack my computer

and the stream was just an endless list of unsuccessful login attempts to the bespoken server. In the description it said:

Can you teach me how to do this? No.

So naturally I had to do a bit of research and try out streaming some failed SSH login attempts into one of my servers. In this post we’ll have a look at how to set up your own honeypot with pshitt and if you’re interested you can stream the attempts to twitch as well 😉

Continue reading “How to set up an SSH Honeypot (low interaction) and live stream it”

Showing Indentation, Spaces, Tabs (Invisibles) in Various Editors

Invisibles are the characters that are in your editor anyways, but they are usually not displayed with anything but empty space or simply completely invisible, like line breaks. Showing invisibles in Vi or Vim can easily be done by changing the vim config with the list and listchars properties. Other editors have config files or an easy to use UI like Atom or Visual Studio Code.

Continue reading “Showing Indentation, Spaces, Tabs (Invisibles) in Various Editors”