Bash: Remove headers from HTTP response

Question

If I have some text containing HTTP headers and body, eg: ``` HTTP/1.1 200 OK Cache-Control: public, max-age=38 Content-Type: text/html; charset=utf-8 Expires: Fri, 22 Nov 2013 06:15:01 GMT Last-Modified: Fri, 22 Nov 2013 06:14:01 GMT Vary: * X-Frame-Options: SAMEORIGIN Date: Fri, 22 Nov 2013 06:14:22 GMT My website Hello world! ``` and this text is being piped in from a command, how can I remove the headers to leave just the body? (Within the headers, ` ` is used as the line break. ` ` marks the end of the headers and the start of the body.) Here's what I've tried (`...` indicates any command such as `cat` or `curl` which will output some HTTP headers and body to stdout): sed My first idea was to do substitution with `sed`, to remove everything before the first occurrence of ` `: ``` ... | sed 's|^.*? ||' ``` But this doesn't work, mainly because `sed` only operates on individual lines…

Accepted Answer

sed can do this:

```
sed '1,/^$/d' data.txt
```

This command deletes everything starting from line 1, and ending at the first occurrence of an empty line (`^$`). This works if you have `
` as a newline character. If you have `
` as a newline character, you can use `dos2unix` and `unix2dos` to convert them back and forth or you can add the `` character to the sed regex:

```
sed '1,/^$/d' data.txt
```

However, the last line will only work if you have `
` as a newline character, to make it work on both types of newlines, you can use:

```
sed '1,/^\{0,1\}$/d' data.txt
```

Here we are looking for an empty line with either 0 or 1 `` characters.

Bash: Remove headers from HTTP response

Solution

Problem