Tracing or Preventing HTTP Redirects in Golang

In this post we're going to have a look at how to have a close look at HTTP requests processed by Go and how to prevent automatic redirection through 301, 302 or similar redirects. This is useful both for investigating tracking links from twitter, buffer, bit.ly or email marketing, or to make sure your tools to generate them actually work ;) For a demo, check out the Redirect Checker Tool

Note: You're expected to have go 1.7, if you don't some features of the net/http package might be unavailable to you. You can check this by typing go version in your terminal. Sample output: go version go1.7.3 linux/amd64.

Making an HTTP request in Go

Let's have a look at how to create a simple HTTP request in Go and read the status code.

// file: http-request.go
package main

import(
  "fmt"
  "net/http"
)

func main(){
  resp, err := http.Get("http://www.jonathanmh.com/")

  if err != nil {
    fmt.Println(err)
  }

  fmt.Println("StatusCode:", resp.StatusCode)
  fmt.Println(resp.Request.URL)
}

Output:

StatusCode: 200
http://jonathanmh.com/

Note that the resp.Request.URL is not the one we defined for our GET request, but that it has changed. That's because I'm redirecting users accessing www.jonathanmh.com to jonathanmh.com. Go does the sane thing and just follows along. In most cases, that's what we want, not so if we want to build a tool like the mighty Redirect Checker Tool! For a case like this, we want to know every step of the way and the status codes for every request.

Making an HTTP request in Go not Follow Redirects

In order to make Go NOT follow redirects, we need to create our own http.Client instance with a custom function for CheckRedirect. This will enable us to return from the request before it follows a redirect and again give us the HTTP status code and the address it's trying to access.

// file: http-nofollow-request.go
package main

import(
  "fmt"
  "net/http"
)

func main(){
  client := &http.Client{
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
      return http.ErrUseLastResponse
  } }

  resp, err := client.Get("http://www.jonathanmh.com")

  if err != nil {
    fmt.Println(err)
  }

  fmt.Println("StatusCode:", resp.StatusCode)
  fmt.Println(resp.Request.URL)
}

Output:

StatusCode: 301
http://www.jonathanmh.com

Great! This time we didn't get the forwarded URL, but only the one we were requesting, including the status code, which is perfect! Well, it's almost perfect. Unfortunately now our program finishes happily after executing the first request and doesn't at all tell us about where it was supposed to be redirected to, which is a bummer.

Looping through HTTP redirects in Go

Okay, last step, we're doing to have a look at how to loop through redirects with Go. First of all we should set a max limit for request (so nobody can send us in an infinite loop), also we need to create a condition for when we're going to be done.

Let's say we assume that if we get an HTTP status code of 200, we're happy and we'll stop trying to do anything else.

// file: http-redir-loop.go
package main

import(
  "fmt"
  "net/http"
)

func main(){
  myURL := "http://www.jonathanmh.com"
  nextURL := myURL
  var i int
  for i < 100 {
    client := &http.Client{
      CheckRedirect: func(req *http.Request, via []*http.Request) error {
        return http.ErrUseLastResponse
    } }

    resp, err := client.Get(nextURL)

    if err != nil {
      fmt.Println(err)
    }

    fmt.Println("StatusCode:", resp.StatusCode)
    fmt.Println(resp.Request.URL)

    if resp.StatusCode == 200 {
      fmt.Println("Done!")
      break
    } else {
      nextURL = resp.Header.Get("Location")
      i += 1
    }
  }
}

Output:

StatusCode: 301
http://www.jonathanmh.com
StatusCode: 200
http://jonathanmh.com/
Done!

This example is pretty much like the last one, except that we have some variables for the URLs, like myURL and nextURL. At the end of the loop, if the status code is not 200 - OK, we set the nextURL to resp.Header.Get("Location") which is the HTTP header field for redirect targets. You can by the way also look at those headers if you turn to the Network tab of your Chrome or Firefox Developer Tools and tick the box Preserve Log.

That's it! You now know how to do some pretty cool things with net/http in Go. All you need to do now to make it available to the public (if you want to) is to wrap it in an http API, run it on a server and build a small user interface for it, like I've done on the Redirect Checker Tool ;)

Further Reading and Sources

One blog post can not rule them all, maybe what I explained isn't what you wanted at all so go have a look at the following links:

Let me know what you think and if this post was useful to you!

Tagged with: #go #golang #http

Thank you for reading! If you have any comments, additions or questions, please tweet or toot them at me!