ioutil.ReadAll(response.Body) blocks forever - Golang
go, http
Solution
I suspect your problem is that you try to read the response body even if there's an error:
if err != nil {
fmt.Println(err)
}
You should either have an `else` after this, or you should `return` or `continue` or something else. Your `ReadAll()` line is undefined behavior.
(If you originally copied this from the `Get()` example code, note that it includes a `log.Fatalf()` in the error leg, which terminates the program.)
I suspect that, as you say, occasionally you are getting a network error for one reason or another. Are you checking the output for the result of the `Println()`? The way you've done it, I could imagine it easily getting buried in the output.
As @twotwotwo notes, this URL returns a redirect to the same URL with a trailing slash. `Get()` will automatically handle this for you, so that's not the problem. By default curl does not follow redirects while wget does. You can see the header information by passing `-i` to curl.
Other things to verify:
Make sure your `defer` is actually being called. Remember, `defer` is called at the end of the function, not the end of the current scope. So if you're in a loop (as you mention), you will just accumulate `defer` blocks and never actually close these responses.
If the server in fact never closes the connection, then `io.ReadAll()` will never return. This is a feature. If you want a timeout, you need to handle that yourself. You should be able to test this hypothesis with tools like `curl`. For some solutions, see:
- golang http timeout and goroutines accumulation
- How to set timeout for http.Get() requests in Golang?
- `http.Transport.ResponseHeaderTimeout`
Problem
``` tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, } client := &http.Client{Transport: tr} response, err := client.Get(link) if err != nil { fmt.Println(err) } defer response.Body.Close() //block forever at the next line content, _ = ioutil.ReadAll(response.Body) ``` The above is my code to read content from a webpage which resides in a loop. I found sometimes the line `ioutil.ReadAll(response.Body)` will block forever. This happens randomly, however, it almost always happens on this webpage: `http://xkcd.com/55` . It's very interesting that when I do `curl http://xkcd.com/55`, it returns nothing, however, `wget http://xkcd.com/55` returns the whole webpage.