In order to bulk retire multiple URLs of your web site at once, you can use nginx maps to have the webserver reply with a 410 http code, which means: GONE and not coming back.
This can be common when migrating away from a specific CMS like WordPress that builds a bunch of taxonomy pages that you will no longer be supporting or similar.
Also, this method will work for a any sort of matching you can do, but we’ll focus on 410 and a custom error page, since this is what I had to implement recently.
Lastly we will also make nginx load a custom 410 error page, instead of the boring default.
Nginx is my favourite webserver and that’s mostly because I like the config file format and how many features it brings for my needs like reverse proxy for node apps, static file serving and cache headers for SPAs and rewriting APIs into specific paths of a URL.
I’m currently working on a project that will need a few redirects to weed out typos in URLs, to deprecate a number of them (which will just yield a 410 http error code) and otherwise will mostly be static content.
This is why I thought it would be a great idea to find the most convenient way for me to create a lot of redirects in nginx and to have them as part of the project, in code version control and not on some server where they will be forgotten about until something goes wrong.
In the beginning, there was an idea, that we would catalogue the names of our adversaries, that we triumphed over in Escape from Tarkov.
We shot and looted, carried out dogtags, rarely but eagerly and kept our little kill journal on the shared website.
Then we broke our shared deployment process and my commit permissions were revoked, so my dog tags piled up in my inventory. At the end of the wipe (season) I had killed about 750 PMCs and the process of keeping a log grew tedious.
A new solution needed birthing. I thought to myself:
How hard can OCR / image recognition really be?
Well. There were some unexpected pitfalls on the way.
Finding an OCR library
Turns out there’s a few OCR libraries out there and technically some of those are overkill for our purpose, since a lot of them are trained towards understanding languages and words of a language. We just need strings extracted from a screenshot, which will be things like SexHaver420 or IfakEnjoyer and abbreviations like USEC, so we’re not in need of actually understanding language. We just have a picture that we need to turn into machine readable strings.
I felt like GO would be a good choice, because I can easily build a web app to upload images and either get back a set of strings or even through the github api open automatic pull requests with the data extracted, so I found a go ocr server that I could dump images into and see how well it would do (ran well from the included dockerfile).
After cloning the git repo you can use docker to spin up a local webserver with which you can upload images and test how much of a text the gosseract OCR library can detect (which you get as a JSON response).
Now we’re getting stuff like this from our unedited screenshot:
"i PS Ws) i Cap) St\n. 4 Se CROC ki: =0) Headshot (SVDS, 725m)"
Cleaning up the input image
When I uploaded my first screenshot, it was all a mess, since the screenshots have a lot of additional information we don’t need. Turns out lots of OCR is done on images with a white background (we don’t have that) and with very high contrast fonts (we don’t have that), we also had lots of empty space that didn’t actually contain the text we wanted, so I decided to try some image processing with Imagemagick/Graphicsmagick).
We’re going to:
crop the image
invert the colours
increase the contrast
and see how that goes
sudo apt-get install libgraphicsmagick1-dev
There’s usually great amounts of examples when using any of those utilities and lots of libraries for any backend runtime. I knew I needed something like this GIMP/Photoshop feature (Levels) to increase the contrast of the screenshot.
The official documentation of the graphicsmagic CLI states:
> -level <black_point>{,<gamma>}{,<white_point>}{%}
adjust the level of image contrast
Give one, two or three values delimited with commas: black-point, gamma, white-point (e.g. 10,1.0,250 or 2%,0.5,98%). adjust the level of image contrast
Give one, two or three values delimited with commas: black-point, gamma, white-point (e.g. 10,1.0,250 or 2%,0.5,98%).
Great, so I could just set black and white points in percentages? That sounds like a convenient option!
This one just wants three floating point numbers between 0 and the maximum quantum value. Turns out the maximum quantum value is determined by what bit-depth you image has and it is not (necessarily) a number between 0 and 255 and the function to figure out the max quantum value is still on the TODO of the library last released in 2017 ¯\_(ツ)_/¯.
Anyways, for my testing I just assumed a max quantum value of 65356, which worked out much better:
which is a lot better than the earlier output, but it still could do with some improvement, since I’m pretty sure it’s supposed to be Headshot instead of Headshat.
For the cropping we kind of just vaguely measured the x and y points in GIMP and it works out for my resolution screenshots (2560 x 1440).
mw.CropImage(1300, 800, 600, 200)
The Result
So far we’ve made progress on kind of optimising an image from:
to:
which is pretty cool.
If you want to try it yourself, this would be the full code for now:
go.mod
module doggy-tagger
go 1.18
require github.com/gographics/gmagick v1.0.0
and can be run by: go run . -from example-in.png -to example-out.png
Now, the next step will be to turn see how well we can tune this and gosseract to consistently pass a set of example images and later on to automate bulk-processing end of raid screenshots and just spitting out lists of names. Let’s see when we get to it, but so far it’s been a lot of fun!
Welcome to this episode of Music Monday! Today we’ll have two bands that in one way or another have a popular song regarding the devil! The Rolling Stones aren’t part of that list, I’m guessing you know their take on him.
The devil today means something very different to different groups of the population. Depending on how much or how you believe in a big religion, he’s a very real fallen angel that “makes you do bad things” and got you kicked out of paradise. I guess the semantic drift that the word has is that it means either some true, often immoral, desire or temptation that either we or society doesn’t approve of giving in to.
In this post we’re going to have a close look at the Exway Flex with the belt-driven Riot drive train. I am in no way sponsored or affiliated and bought the board myself.
In this article we’re going to have a look at how to mock http connections while running your tests on your golang application.
Since we do not want to call the real API or crawl an actual site to make sure that our application works correctly, we want to make sure that our application works within our defined parameters.
This week I tried to get a still image into one of my videos, but they showed some weird artifacts.
I use Adobe Premiere Pro CC (2020) for most of my editing and it was really bothersome to figure this one out. In fact I don’t really consider this solved, but only “worked around”.
PHP5, released in 2014, has reached its end of life in 2018, but many hosters continued to support it. If you have any custom PHP applications, you might still be running them on a server that has a PHP5 runtime.
The upgrade path for these applications might not be obvious, but I found a way that should make it easier, so in this article we’re going to have a look at running PHP simultaneously with PHP5 and PHP7 using docker and docker-compose.
I recently found a streamer on twitch that had an interesting title:
Watch people trying to hack my computer
and the stream was just an endless list of unsuccessful login attempts to the bespoken server. In the description it said:
Can you teach me how to do this? No.
So naturally I had to do a bit of research and try out streaming some failed SSH login attempts into one of my servers. In this post we’ll have a look at how to set up your own honeypot with pshitt and if you’re interested you can stream the attempts to twitch as well 😉
Invisibles are the characters that are in your editor anyways, but they are usually not displayed with anything but empty space or simply completely invisible, like line breaks. Showing invisibles in Vi or Vim can easily be done by changing the vim config with the list and listchars properties. Other editors have config files or an easy to use UI like Atom or Visual Studio Code.