Nginx 410 maps, matching and external files

In order to bulk retire multiple URLs of your web site at once, you can use nginx maps to have the webserver reply with a 410 http code, which means: GONE and not coming back.

This can be common when migrating away from a specific CMS like WordPress that builds a bunch of taxonomy pages that you will no longer be supporting or similar.

Also, this method will work for a any sort of matching you can do, but we’ll focus on 410 and a custom error page, since this is what I had to implement recently.

Lastly we will also make nginx load a custom 410 error page, instead of the boring default.

The full code can be found in the repository: https://github.com/JonathanMH/nginx-410-map

Continue reading “Nginx 410 maps, matching and external files”

Nginx and External Config Files for Redirects

Nginx is my favourite webserver and that’s mostly because I like the config file format and how many features it brings for my needs like reverse proxy for node apps, static file serving and cache headers for SPAs and rewriting APIs into specific paths of a URL.

I’m currently working on a project that will need a few redirects to weed out typos in URLs, to deprecate a number of them (which will just yield a 410 http error code) and otherwise will mostly be static content.

This is why I thought it would be a great idea to find the most convenient way for me to create a lot of redirects in nginx and to have them as part of the project, in code version control and not on some server where they will be forgotten about until something goes wrong.

Continue reading “Nginx and External Config Files for Redirects”

OCR in GO for EFT screenshots (part 1)

In the beginning, there was an idea, that we would catalogue the names of our adversaries, that we triumphed over in Escape from Tarkov.

We shot and looted, carried out dogtags, rarely but eagerly and kept our little kill journal on the shared website.

Then we broke our shared deployment process and my commit permissions were revoked, so my dog tags piled up in my inventory. At the end of the wipe (season) I had killed about 750 PMCs and the process of keeping a log grew tedious.

A new solution needed birthing. I thought to myself:

How hard can OCR / image recognition really be?

Well. There were some unexpected pitfalls on the way.

Finding an OCR library

Turns out there’s a few OCR libraries out there and technically some of those are overkill for our purpose, since a lot of them are trained towards understanding languages and words of a language. We just need strings extracted from a screenshot, which will be things like SexHaver420 or IfakEnjoyer and abbreviations like USEC, so we’re not in need of actually understanding language. We just have a picture that we need to turn into machine readable strings.

I felt like GO would be a good choice, because I can easily build a web app to upload images and either get back a set of strings or even through the github api open automatic pull requests with the data extracted, so I found a go ocr server that I could dump images into and see how well it would do (ran well from the included dockerfile).

After cloning the git repo you can use docker to spin up a local webserver with which you can upload images and test how much of a text the gosseract OCR library can detect (which you get as a JSON response).

Now we’re getting stuff like this from our unedited screenshot:

"i PS Ws) i Cap) St\n. 4 Se CROC ki: =0) Headshot (SVDS, 725m)"

Cleaning up the input image

When I uploaded my first screenshot, it was all a mess, since the screenshots have a lot of additional information we don’t need. Turns out lots of OCR is done on images with a white background (we don’t have that) and with very high contrast fonts (we don’t have that), we also had lots of empty space that didn’t actually contain the text we wanted, so I decided to try some image processing with Imagemagick/Graphicsmagick).

We’re going to:

  1. crop the image
  2. invert the colours
  3. increase the contrast

and see how that goes

sudo apt-get install libgraphicsmagick1-dev

There’s usually great amounts of examples when using any of those utilities and lots of libraries for any backend runtime. I knew I needed something like this GIMP/Photoshop feature (Levels) to increase the contrast of the screenshot.

The official documentation of the graphicsmagic CLI states:

>  -level <black_point>{,<gamma>}{,<white_point>}{%}

	adjust the level of image contrast

	Give one, two or three values delimited with commas: black-point, gamma, white-point (e.g. 10,1.0,250 or 2%,0.5,98%). adjust the level of image contrast
Give one, two or three values delimited with commas: black-point, gamma, white-point (e.g. 10,1.0,250 or 2%,0.5,98%).

Great, so I could just set black and white points in percentages? That sounds like a convenient option!

Turns out the GO library doesn’t let you do that:

func (mw *MagickWand) LevelImage(blackPoint, gamma, whitePoint float64) error

Source

This one just wants three floating point numbers between 0 and the maximum quantum value. Turns out the maximum quantum value is determined by what bit-depth you image has and it is not (necessarily) a number between 0 and 255 and the function to figure out the max quantum value is still on the TODO of the library last released in 2017 ¯\_(ツ)_/¯.

Anyways, for my testing I just assumed a max quantum value of 65356, which worked out much better:

// make image black/white
mw.SetImageColorspace(gmagick.COLORSPACE_GRAY)
// invert
mw.NegateImage(false)
// increase contrast
assumed_quantum_max := 65356.0
blackPoint := assumed_quantum_max / 2.3
gamma := .2
whitePoint := assumed_quantum_max / .5
mw.LevelImage(blackPoint, gamma, whitePoint)

This code will invert (negate) and level the image and make the lowest half (ish) black, make most of the greys black and make everything else white.

This image performs much better and with some additional cropping we actually get a decent output from our OCR server:

{
"result": "# TIME PLAYER LVL FACTION STATUS\n\n1 dotHaku 39 USEC Headshat (SVOS, 7@.5m)\n\n2 CementMixerr 12. -USEC Killed (SVDS, thorax, 29m)\n3 Jilpe 11: USEC Killed (SVDS, thorax, 214m)\n4 Kocherga -- GUARD Headshat (SVDS, 9.8m)\n\n5 Zimniy -- GUARD Killed (SVDOS, thorax, 16.8m)\n6 Svinec -- GUARD Killed (SVDS, thorax, 4.1m)\n7 Maksim Minskiy --  SCAV Headshot (Mk 16, 99m)\n\n8 Yastreb -- RAIDER Headshot (Mk 16, 785m)",
    "version": "0.2.0"
}

which is a lot better than the earlier output, but it still could do with some improvement, since I’m pretty sure it’s supposed to be Headshot instead of Headshat.

For the cropping we kind of just vaguely measured the x and y points in GIMP and it works out for my resolution screenshots (2560 x 1440).

mw.CropImage(1300, 800, 600, 200)

The Result

So far we’ve made progress on kind of optimising an image from:

to:

which is pretty cool.

If you want to try it yourself, this would be the full code for now:

go.mod

module doggy-tagger

go 1.18

require github.com/gographics/gmagick v1.0.0

main.go

package main

import (
    &quot;flag&quot;

    &quot;github.com/gographics/gmagick&quot;
)

func cropAndLevel(sourceImage string, outputImage string) {
    mw := gmagick.NewMagickWand()
    defer mw.Destroy()
    mw.ReadImage(sourceImage)

    // crop
    mw.CropImage(1300, 800, 600, 200)

    // make image black/white
    mw.SetImageColorspace(gmagick.COLORSPACE_GRAY)
    // invert
    mw.NegateImage(false)

    // increase contrast
    assumed_quantum_max := 65356.0
    blackPoint := assumed_quantum_max / 2.3
    gamma := .2
    whitePoint := assumed_quantum_max / .5
    mw.LevelImage(blackPoint, gamma, whitePoint)

    mw.WriteImage(outputImage)
}

func main() {
    f := flag.String(&quot;from&quot;, &quot;&quot;, &quot;original image file ...&quot;)
    t := flag.String(&quot;to&quot;, &quot;&quot;, &quot;target file ...&quot;)
    flag.Parse()

    gmagick.Initialize()
    defer gmagick.Terminate()

    cropAndLevel(*f, *t)
}

and can be run by: go run . -from example-in.png -to example-out.png

Now, the next step will be to turn see how well we can tune this and gosseract to consistently pass a set of example images and later on to automate bulk-processing end of raid screenshots and just spitting out lists of names. Let’s see when we get to it, but so far it’s been a lot of fun!

Mocking HTTP Connections in Golang Tests (scraping and API)

In this article we’re going to have a look at how to mock http connections while running your tests on your golang application.

Since we do not want to call the real API or crawl an actual site to make sure that our application works correctly, we want to make sure that our application works within our defined parameters.

Continue reading “Mocking HTTP Connections in Golang Tests (scraping and API)”

Updating PHP 5 (or any other) Apps with Docker

PHP5, released in 2014, has reached its end of life in 2018, but many hosters continued to support it. If you have any custom PHP applications, you might still be running them on a server that has a PHP5 runtime.

The upgrade path for these applications might not be obvious, but I found a way that should make it easier, so in this article we’re going to have a look at running PHP simultaneously with PHP5 and PHP7 using docker and docker-compose.

Continue reading “Updating PHP 5 (or any other) Apps with Docker”

Cookies, sessionStorage, localStorage. What’s the difference?!

This post is about different storage possibilities in your browser. There are cookies, and different kinds of storage that can be accessed through JavaScript APIs in your client side code. They’re used for authentication, tracking tools like Google Analytics and a lot of other stuff, so let’s have a look at how they work!

Continue reading “Cookies, sessionStorage, localStorage. What’s the difference?!”

Web Components 101: Hello World

Web Components are a way to encapsulate functionality into little self-contained parts that can be reused and adapted like you’re used to from frontend libraries like React, Vue, Angular or Ember. The special thing about Web components is that they work straight in your browser.

When it comes to support, a lot of features are supported in all major browsers, as can bee checked out at caniuse.com.

As you might have seen, web components usually are understood as several sub-parts, mostly:

Continue reading “Web Components 101: Hello World”

Unit Tests with Rust Tutorial 101

Writing unit tests is heavily integrated into rust without additional dependencies or external test runners. You can write unit tests inside the file that has the testable code or use a separate tests directory, let’s see how we do that.

You don’t need to have any prior rust knowledge, but you will have to have rust installed, which most easily is done by using rustup.

Writing Your First Test

When creating a rust project with cargo new and we’ll need a function that takes arguments and returns something, so let’s go for a simple sum function in main.rs:

fn sum(a: i32, b: i32) -> i32 {
    return a + b;
}

fn main(){
    println!("12 + 1 is {}", sum(12,1));
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn sum_test() {
        assert_eq!(345+5, sum(345, 5));
    }
}

Now for the interesting parts in the bottom of the file

  • #[cfg(test)] <- the test section attribute (see conditional compilation)
  • #[test] <- the test attribute for the individual test
  • assert_eq! macro to compare the two values

This is the simplest way to implement a unit test in rust, the next step is to look at how to move your functions and tests into another separate file.

Running cargo test should output something like:

    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running target/debug/deps/rust_test_101-e2cf395340fbef91

running 1 test
test tests::sum_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Adding the following has the correct types, but it would cause the return value to be larger than i32 allows:

#[test]
#[should_panic]
fn greater_than_i32() {
    sum(2147483647,1); // this would be bigger than i32 allows
}

By using #[should_panic] we tell the test run to expect the following function to fail. This catches errors that usually crop up with:

thread 'main' panicked at 'attempt to add with overflow', src/main.rs:2:12

Now this is the easiest possible way to get started with writing tests in Rust, let me know if you want to know more in future posts! Don’t forget to check out the Rust Book Chapter on Testing as well.

What is an Interface?

An interface is something I first stumbled upon when I started learning compiled languages. I had read the term a couple of times before, but really only with C, Java and lately Go an interface actually means something to me that I can remember.

I’ve used the abbreviation API about one thousand times in my life, which stands for Application programming interface, usually I mean web APIs, where I get some JSON to feed it to my frontend or something, but that’s already a very specific use case.

As with many words in computer science, interface can mean many things, but depending on context it means a very specific thing. To quote wikipedia:

In computing, an interface is a shared boundary across which two or more separate components of a computer system exchange information.

and more specifically regarding programming:

the term interface is used to define an abstract type that contains no data but defines behaviours as method signatures.

So an interface is a bit like a puzzle piece. You can’t just attach your code or your function anywhere, but you need to find the right part that’s sticking out.

Implementing / Satisfying Intefaces

To implement or to satisfy an interface means to comply with the rules set by the blueprint. In object oriented languages such as C# or Java, interfaces are abstract classes, that just set some requirements for which functions must be included and which types they can receive or return.

Since Golang is not an object oriented language, there are no classes to implement, but we can still satisfy an interface with our type and it will be checked automatically by the compiler. Let’s pretend we are programming a series of vending machines and instead of a <Product> type, they’re supposed to return a string:

package main

import "fmt"

type VendingMachine interface {
    ReturnItem() string
}

type Cola struct {
}

func (d Cola) ReturnItem() string {
    return "you get a cooled cola"
}

func main() {
    var vend1 VendingMachine
    vend1 = Cola{}
    fmt.Println(vend1.ReturnItem())
}

You can see that type VendingMachine interface creates the first blueprint interface and the Cola struct now will need to satisfy the interface by also having a ReturnItem function.

We test this by creating a variable vend1 of type VendingMachine and assign a Cola{} struct to it.

The expected output is: you get a cooled cola

If we remove the following code:

func (d Cola) ReturnItem() string {
    return "you get a cooled cola"
}

We get the error:

# command-line-arguments
./main.go:21:8: cannot use Cola literal (type Cola) as type VendingMachine in assignment:
        Cola does not implement VendingMachine (missing ReturnItem method)

Thanks Go! Helpful! This way we can’t forget the essential ReturnItem function in our VendingMachine and make our customers really angry at us, because no products come out after they payed.

Similarly, if we change the return type of the Cola ReturnItem to int:

./main.go:13:9: cannot use "you get a cooled cola" (type string) as type int in return argument
./main.go:25:8: cannot use Cola literal (type Cola) as type VendingMachine in assignment:
        Cola does not implement VendingMachine (wrong type for ReturnItem method)
                have ReturnItem() int
                want ReturnItem() string

This makes sure that we return the correct type as well, because the humans or possibly robots interfacing with our vending machine will expect strings, not integers to come out.

Lastly, when coming from other languages you might try to implement a go interface by using the interface to create a type instance, maybe like this:

vend1 := VendingMachine{Cola{}

However, since the implements checking is done by the compiler and abstracted away, you’ll only get an error about a composite literal:

./main.go:17:25: invalid type for composite literal: VendingMachine

Summary

Interfaces are handy tools to ensure that your application can have consistently grouped functionalities with different values, even in a non OOP language like Go.

Real world examples of interfaces are the Readers and Writers in Go are IO parts like the Writer or Reader which you can try out and extend for some simple file system operations if you’re interested.