Golang Proxy HTTP Requests via TOR

Making HTTP requests via TOR is simple to achieve in GO and can be useful if you would otherwise hit IP based rate limits or if you simply want to avoid getting your server mistakenly blocked for automating some task like web scraping. Connecting via the TOR network to interact with .onion URLs and thereby access content exclusive to TOR.

Please note that you might experience being blocked regardless as lots of TOR exit nodes are listed on the TOR projects website.

In Go you can set a proxy for your HTTP request client, which will be used for all GET, POST, etc requests, in our case will be a TOR socks5 proxy.

Setting up a TOR Proxy

If you don't have a TOR proxy set up locally or if you want to run it on a separate machine in your infrastructure, you yan make use of docker-compose from the tor-privoxy repository on github

This will get you a docker-compose.yml:

services:
  tor-privoxy:
    restart: always
    image: dockage/tor-privoxy:latest
    ports:
      - "9050:9050" # Tor proxy
      - "9051:9051" # Tor control port
      - "8118:8118" # Privoxy (optional)

running docker-compose up in the directory where your docker-compose.yml is stored, you will start the proxy and expose the port, 9050 being the important port to us.

You can test the proxy via curl:

curl --proxy socks5h://localhost:9050 'https://check.torproject.org/api/ip'

which should output something like:

{"IsTor":true,"IP":"192.42.116.217"}

Great, we have a TOR socks5 proxy! Let's look at how to use it in Golang

Using the GO ProxyURL for TOR connections

Let's look at the below minimum viable go http request using TOR:

package main

import (
  "fmt"
  "io"
  "log"
  "net/http"
  "net/url"
)

func main() {
  // this is needs to target the same port as the TOR proxy is exposed on
  proxyURL, _ := url.Parse("socks5://127.0.0.1:9050")

  // these two lines are all that is needed for the GO requests to be proxied
  httpTransport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
  httpClient := &http.Client{Transport: httpTransport}

  response, httpError := httpClient.Get("https://api.ipify.org?format=json")
  if httpError != nil {
    log.Fatal("error during get request", httpError)
  }
  defer response.Body.Close()

  body, err := io.ReadAll(response.Body)
  if err != nil {
    log.Fatal("error reading response body", err)
  }
  fmt.Println(response.StatusCode, string(body))
  // output: 200 {"ip":"192.42.116.217"}
}

Now instead of inlining the configuration of the HTTP client, it would be far more useful if we could use this across multiple requests in our project, so we probably want to move the HTTP client proxy configuration out of our main.go file:

- httpTransport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
- httpClient := &http.Client{Transport: httpTransport}
+ httpClient := GetClient()

and create a new file called client.go:

package main

import (
  "net/http"
  "net/url"
)

func GetClient() *http.Client {
  proxyURL, _ := url.Parse("socks5://127.0.0.1:9050")
  httpTransport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}
  httpClient := &http.Client{Transport: httpTransport}
  return httpClient
}

which will allow us to use it anywhere in your project.

On top of using a proxy, you probably want to set a meaningful user agent instead of Go-http-client/2.0 to make sure people can contact you (or mimic a legic browser user agent).

Let me know if this post helped you or what you need a TOR proxy for on the socials below 😉