Indy MIME decoding of Multipart/Form-Data Requests returns trailing CR/LF

Question

Indy 10.6 revision 5128 seems to contain a change which breaks the code below for HTTP form uploads. The received data contains two additional bytes at the end, a CR/LF pair. Reading through the changed code lines between 5127 and 5128 did not bring me to the root cause. I will try to debug it when I find the time and post the result here (but maybe somebody is faster). Here is a stand-alone demo application which shows a HTML upload form at `http://127.0.0.1:8080` ``` program IndyMultipartUploadDemo; {$APPTYPE CONSOLE} uses IdHTTPServer, IdCustomHTTPServer, IdContext, IdSocketHandle, IdGlobal, IdMessageCoder, IdGlobalProtocols, IdMessageCoderMIME, IdMultiPartFormData, SysUtils, Classes; type TMimeHandler = procedure(var VDecoder: TIdMessageDecoder; var VMsgEnd: Boolean; const Response: TIdHTTPResponseInfo) of object; TMyServer = class(TIdHTTPServer) private procedure ProcessMimePart(var VDecoder: TIdMessageDecoder; var VMsgEnd: Boolean; const Response: TIdHTTPResponseInfo); function…

Accepted Answer

The current SVN revision is 5203, so you are a little behind in updates.

I tested your code as-is using revision 5203 in XE2 with IE11.

I uploaded a test `.pas` file, and it is 53 bytes larger in the `upload` folder. I can confirm that the raw `PostStream` data prior to decoding is correct.

Yes, I do see an extra CRLF at the end of the file, and that is related to the way `TIdMessageDecoderMIME` decodes non-binary non-base64/QP-encoded data (which your example does not). It reads the data line-by-line, decoding each line as it goes, writing decoded lines to the destination stream with a new line break when not using a binary transfer encoding. That logic does not take into account that a line break in front of a MIME boundary belongs to the boundary, not to the data preceding the boundary. The MIME spec is quite clear on that, but Indy does not yet take that into account for non-base64 data.

The rest of the differences in file size are all related to non-ASCII characters being converted to sequences of `$3F` bytes, including the UTF-8 BOM. This is because the `PostStream` data is being decoded as 7bit ASCII in `TIdMessageDecoderMIME.ReadBody()` because there is no `Content-Transfer-Encoding` header being sent with the file data, so Indy is defaulting to ASCII due to this statement in RFC 2045 Section 6.1:

"Content-Transfer-Encoding: 7BIT" is assumed if the Content-Transfer-Encoding header field is not present.

However, Section 6.4 states the following, which seems to contradict 6.1:

Any entity with an unrecognized Content-Transfer-Encoding must be treated as if it has a Content-Type of "application/octet-stream", regardless of what the Content-Type header field actually says.

`ReadBody()` handles both cases, however 6.1 is checked first, so Indy assumes a `7bit` encoding, which then nullifies its handling of 6.4 since `7bit` is not an unrecognized encoding. Unless one is to assume that a missing `Content-Transfer-Encoding` should be treated as an unrecognized encoding, which Indy currently does not.

The actual `Content-Type` of the upload is `application/octet-stream`, which implies an 8bit encoding. When I update `ReadBody()` to treat `application/octet-stream` as `8bit` instead of `7bit` when applying Section 6.1, all of the problems go away:

```
if LContentTransferEncoding = '' then begin
  // RLebeau 04/08/2014: According to RFC 2045 Section 6.1:
  // "Content-Transfer-Encoding: 7BIT" is assumed if the
  // Content-Transfer-Encoding header field is not present."
  if IsHeaderMediaType(LContentType, 'application/mac-binhex40') then begin  {Do not Localize}
    LContentTransferEncoding := 'binhex40'; {do not localize}
  end

// START FIX!!
  else if IsHeaderMediaType(LContentType, 'application/octet-stream') then begin  {Do not Localize}
    LContentTransferEncoding := '8bit'; {do not localize}
  end
  // END FIX!!

else begin
    LContentTransferEncoding := '7bit'; {do not localize}
  end;
end
```

The uploaded file is the correct file size, bytes are decoded and written correctly without non-ASCII sequences being converted to `$3F` sequences, and there is no extra CRLF at the end of the file.

I will have to investigate the issue further to see if there is a better way to handle this descrepency. I have opened tickets in Indy's issue trackers for it. In the meantime, you have a workaround if you patch your copy of Indy.

Indy MIME decoding of Multipart/Form-Data Requests returns trailing CR/LF

Solution

Problem