Using Libcurl to send very large strings to a server using REST API?
c++, http, libcurl, networking, rest
Solution
This is easily done with libcurl, but you should let libcurl do the entire request and you should just feed it the data to send little by little. The post-callback.c example on the curl web site is almost exactly what you're asking for.
The key is to set the correct header and then use the read callback.
Problem
I am trying to send very large strings (up to 16GB worth) to a cloud server (similar to AWS) for storage as a file using the REST API. I would like to stream the string in smaller chunks to the server, and the string chunks will be generated on-the-fly. From what I can tell, setting the Transfer Encoding to "chunked" will help me do just that. I have started writing a C++ program to do this, using the libcurl library. Here is a relevant snippet with simplified test strings: ``` headers = curl_slist_append(headers, "Accept: text/plain"); headers = curl_slist_append(headers, "Content-Type: binary/octet-stream"); headers = curl_slist_append(headers, "Transfer-Encoding: chunked"); curl_easy_setopt(curl, CURLOPT_VERBOSE, true); curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT"); curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(curl, CURLOPT_URL, "http://192.168.0.1:8080/namespace/test/test"); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "5/r/nHELLO/r/n"); curl_res = curl_easy_perform(curl); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "3/r/nBYE/r/n"); curl_res = curl_easy_perform(curl); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "0/r/n"); curl_res = curl_easy_perform(curl); ``` From what I understand about chunked transfers, each chunk should begin with the size of the string expressed as a hexadecimal number, followed by the actual string. When a zero is encountered for the size, then the server knows that the transfer is complete (http://en.wikipedia.org/wiki/Chunked_transfer_encoding). However, when I look at the contents of the file after I execute the above, all I see is '0' instead of 'HELLOBYE'. So not only is it overwriting the file at each chunk transfer, it is also interpreting the size of the string as part of the string. So my question boils down to: - How can I get it to interpret the first part of my chunk transfers as the size of the string? - How can I get it to "concatenate" sequential chunks transfers?