How can I download a single file from multiple locations via HTTP?

curl, download, http, wget

Solution

Assuming this is a programming question (given that this is StackOverflow) I am going to explain how instead of just linking to a download accelerator that takes advantage of this.

What is needed in terms of the server to do this?

- A server that supports `Range` HTTP header.

- A server that allows for concurrent connections. It is possible to support `Range` while not allowing multiple simultaneous connection by using either endpoint or IP based restrictions server side. For this reason, I recommend you set up a simple test server instead of downloading from a file sharing site while testing this.

What is the `Range` Header?

Data transmission over HTTP is sent in order starting from the beginning of the file if the `Range` header is not set. The first byte of the file on the server will be the first byte of the HTTP response and the last byte of the file on the server will be the last byte of the HTTP response. The `Range` header allows you to specify where the bytes should start sending from allowing you to "skip" the beginning of the response.

Actual Answer Example

Our Situation

The response is plain text. The response content is just one word "StackOverflow!!" encoding ASCII, meaning each character is one byte. Therefore, the `Content-Length` header's value is 15 octets (another term for bytes).

We are going to download this file using 3 requests. For the sake of this example, we are going to say it will be 3 times faster but you should realize that this method will make downloads slower for very small files. This is because HTTP headers must be sent with each request as well as the 3-way handshake. We will also assume that the server supports HEAD requests and that the `Content-Length` header is sent with the download response. Finally, this request will be preformed using GET for reasons of HEAD requests. However, there are workarounds for POST.

Juicy Details

First, perform an HTTP HEAD request. Take the "Content-Length" header and divide that value by the amount of concurrent parallel connections you wish to make. For this example, the `Content-Length` is 15 and we wish to make 3 connections so the divided value will be 5.

Now preform the amount of requests you wished to preform parallel. With each request, set the `Range` header to "Range: bytes=" followe by how many requests have already been made times the divided value found above. Then append "-" followed by the value you just determined plus the divided value. For this example, each request should have the header set as followed.

- `Range: bytes=0-5`

- `Range: bytes=5-10`

- `Range: bytes=10-15`

The response of each of these requests should be

- Stack

- Overf

- low!!

In essence, we are just conforming to Range specification (section 3.12 of RFC 2616) as well as Byte Range specification (section 14.35 of RFC 2616).

Finally, append the bytes of each request to form the final response data.

Disclaimer: I've never actually tried this but it should work in theory

Problem

Original source