Published: 2020-08-02, by Jonathan M. Hethey,reading time: ~4 minutes

Mocking HTTP Connections in Golang Tests (scraping and API)

In this article we're going to have a look at how to mock http connections while running your tests on your golang application.

Since we do not want to call the real API or crawl an actual site to make sure that our application works correctly, we want to make sure that our application works within our defined parameters.

There's a great module that can help us with the task of mocking HTTP responses for tests called httpmock

HTTP mocks for web scraping

Let's say we have a component in our application that will do some web scraping, so we might use something like goquery.

In the below example we'll use a simple function that visits a website and extracts the content of the <title> tag.

filename: scrape.go

package main
import (
    "log"

    "github.com/PuerkitoBio/goquery"
)

func ScrapeTitle(websiteURL string) (title string) {
    doc, err := goquery.NewDocument(websiteURL)
    if err != nil {
        log.Fatalln("could not load website for scraping")
    }
    siteTitle := doc.Find("title").Contents().Text()
    return siteTitle
}

Now if we are to write a unit test for that, we can do that as follows:

filename: scrape_test.go

package main

import (
    "fmt"
    "testing"
)

func TestScrapeTitle(t *testing.T) {
    fmt.Println("TestScrapeTitle")
    siteTitle := ScrapeTitle("http://jonathanmh.com")
    if siteTitle != "Hi, I'm JonathanMH" {
        t.Error("title incorrect")
    }
    fmt.Println(siteTitle)
}

In the test we run the function and compare the title we expect with the title that was scraped by the function.

Now the problem with this test is, that when ever we run go test it will actually go to my website and read the title. This means two things:

Our tests will be slower and more error prone than they could be
I can never change my website title without changing the tests for this project
Most important: We introduced a dependency outside our control for our program that doesn't have any relation to it

To fix this we commonly use mocks, a way of faking http responses, but to actually have the exchange of information happen on the computer where the tests are run, without having to rely on an external webserver or API backend to be available.

HTTP mocks for API requests

In Golang we can use httpmock to intercept any http requests made and pin the responses in our tests. This way we can verify that our program works correctly, without having to actually send a requests over the network.

To install httpmock we can add a go.mod file:

module go-http-mock

go 1.13

require github.com/PuerkitoBio/goquery v1.5.1
require github.com/jarcoal/httpmock v1.0.5

and running go mod download.

Rewriting our scrape_test.go would look like this:

package main

import (
    "fmt"
    "testing"

    "github.com/jarcoal/httpmock"
)

func TestScrapeTitle(t *testing.T) {
    fmt.Println("TestScrapeTitle")
    myMockPage := `<!DOCTYPE html><html lang=en-US class=no-js><head><title>What does the Otter say?</title></head></html>`

    httpmock.Activate()
    httpmock.RegisterResponder("GET", "https://jonathanmh.com",
        httpmock.NewStringResponder(200, myMockPage))

    siteTitle := ScrapeTitle("https://jonathanmh.com")
    httpmock.DeactivateAndReset()

    if siteTitle != "What does the Otter say?" {
        t.Error("<title> was not correctly extracted")
    }

    fmt.Println(siteTitle)
}

after which we can run go test and it should produce the following output:

TestScrapeTitle
What does the Otter say?
PASS
ok      go-http-mock    0.003s

Let's go over the most important changes ot the file:

myMockPage :=... sets up our example response, a piece of plain text that our function will parse into a HTML and look for the title
httpmock.Activate() activates the mocking, before this no requests can be intercepted
httpmock.RegisterResponder() defines the METHOD and the URL, so GET or POST and an address at which we fake an http response
httpmock.NewStringResponder will need a status code and a string to respond with instead of what actually lives at that URL
httpmock.DeactivateAndReset() stops mocking responses for the rest of the test

If you instead want to mock an API response you can use something like this:

httpmock.NewStringResponder(200, `{"id": "507f1f77bcf86cd799439011", "name": "Golang and other Adventures"}`))

That's it! Our client consuming the string should take care of the JSON parsing.

If you're familiar with mocking http connections in node.js you may have heard of the nock library, which is pretty popular when building JavaScript projects.

Hope you enjoyed this little post about mocking in GO, let me know what you're building in the comments!