HTTP and TLS handling with the Go HTTP client

This is the first part in a series about building the web security scanner WebSecurity.is. The scanner is implemented as a web service in Go and is designed to test web apps in 4 categories of security - HTTP, HTML5, connection (SSL/TLS) and Google Safe Browsing. I’ve previously written how it can help secure web apps.

HTTP requests with Go

We start off with the most basic part, how to scan web sites on demand in Go and analyze the results. If you are familiar with Go, this part sounds easy, owing to Go’s feature-rich standard library. And for the most part that’s correct. However, there are a few gotchas and just a little complexity involved.

Go’s HTTP client is contained in the net/http package. Using it is as simple as

resp, err := http.Get("http://example.com/")

But this will fail to scan some websites and may wait indefinitely for an answer since there is no timeout. When running an interactive scanner, this is a no-go.

To take more control of the HTTP request, we create our own client. We also create our own transport (more on that later) and request to set custom headers.

timeout := time.Duration(30 * time.Second)
tr := &http.Transport{
	MaxIdleConnsPerHost: 10,
}

client := http.Client{
	Transport: tr,
	Timeout:   timeout,
}

req, _ := http.NewRequest("GET", "http://example.com/", nil)
req.Header.Set("Referer", "My totally awesome scanner!")

resp, err := client.Do(req)

Handling TLS errors

When you scan a lot of servers, you are bound to run into some issues. Most of the ones which have tripped up Go’s HTTP client are due to some quirky TLS settings on the server. Some are due to non-standard server TLS implementations or misconfigurations but there have also been a few issues with Go’s TLS handling. WebSecurity.is only runs reliably on Go version 1.7 or higher, where many issues have been fixed. But what issues might we run into and how can we handle misbehaving servers?

First off, we might get an error “connection reset by peer”. In most cases, this is resolved by capping TLS at version 1.1 and retrying:

trTls11 := &http.Transport{
	MaxIdleConnsPerHost: 10,
	TLSClientConfig: &tls.Config{MaxVersion: tls.VersionTLS11},
}
client.Transport = trTls11
resp, err := client.Do(req)

We might also run into websites with invalid or untrusted certificates, resulting in an error message starting off with “x509: […]”. Since the scanner is intended to work around this, we make note of the error and then turn off certificate verification and retry the request.

trSkipVerify := &http.Transport{
	MaxIdleConnsPerHost: 10,
	TLSClientConfig: &tls.Config {
		MaxVersion: tls.VersionTLS11,
		InsecureSkipVerify: true,
	},
}
client.Transport = trSkipVerify
resp, err := client.Do(req)

Note: This is appropriate for the security scanner use case. For most other purposes, you definitely don’t want to skip certificate verification! If you don’t understand what turning off verification means, don’t blindly copy this code.

Checking for HTTP to HTTPS redirect

The web is moving to HTTPS, which is a good thing. But websites will still see a lot of HTTP requests and should be able to handle them properly by redirecting to HTTPS. We can check for this with the following code.

//CheckHTTPSRedirect checks if HTTP requests are redirected to HTTPS on the same host
//This is one of the requirements for HSTS preload list inclusion
func CheckHTTPSRedirect(url *url.URL) bool {
	req, _ := http.NewRequest("GET", url.String(), nil)
	client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
		return http.ErrUseLastResponse
	}
	req.Close = true

	resp, err := client.Do(req)	
	if err == nil && (resp.StatusCode == 301 || resp.StatusCode == 302) {
		headers := resp.Header
		response, ok := headers["Location"]
		if ok {
			redirURL, err := url.Parse(response[0])
			if err == nil {
				if redirURL.Host == url.Host && redirURL.Scheme == "https" {					
					//HTTP redirects to HTTPS
					return true
				}
			}
		}
	}
	//HTTP does not redirect to HTTPS, or does so improperly
	return false
}

---

Photo credit: Bolungarvik harbour, Iceland. Own photo..

• ← Previous Post
• Next Post →

---