What do the ENT_HTML5, ENT_HTML401, ... modifiers on html_entity_decode do?

html, html-entities, htmlspecialchars, php

Solution

I started wondering what behavior these constants have when I saw these constants at the htmlspecialchars page. The documentation was rubbish, so I started digging in the source code of PHP.

Basically, these constants affect whether certain entities are encoded or not (or decoded for `html_entity_decode`). The most obvious effect is whether the apostrophe (`'`) is encoded to `'` (for `ENT_HTML401`) or `'` (for others). Similarly, it determines whether `'` is decoded or not when using `html_entity_decode`. (`'` is always decoded).

All usages can be found in ext/standard/html.c and its header file. From ext/standard/html.h:

#define ENT_HTML_DOC_HTML401            0
#define ENT_HTML_DOC_XML1                       16
#define ENT_HTML_DOC_XHTML                      32
#define ENT_HTML_DOC_HTML5                      (16|32)

(replace `ENT_HTML_DOC_` by `ENT_` to get their PHP constant names)

I started looking for all occurrences of these constants, and can share the following on the behaviour of the `ENT_*` constants:

- It affects which numeric entities will be decoded or not. For example, `` gets decoded to an unreadable/invalid character for `ENT_HTML401`, and `ENT_XHTML` and `ENT_XML1`. For `ENT_HTML5` however, this is considered an invalid character and hence it stays ``. (C function unicode_cp_is_allowed)

- With `ENT_SUBSTITUTE` enabled, invalid code unit sequences for a specified character set are replaced with `�`. (does not depend on document type!)

- With `ENT_DISALLOWED` enabled, code points that are disallowed for the specified document type are replaced with `�`. (does not depend on charset!)

- With `ENT_IGNORE`, the same invalid code unit sequences from `ENT_SUBSTITUTE` are removed and no replacement is done (depends on choice of "document type", e.g. `ENT_HTML5`)

- Disallow `
` for `ENT_HTML5` (line 976)

- `ENT_XHTML` shares the entity map with `ENT_HTML401`. The only difference is that `'` will be converted to an apostrophe with `ENT_XHTML` while `ENT_HTML401` does not convert it (see this line)

- `ENT_HTML401` and `ENT_XHTML` use exactly the same entity map (minus the difference from the previous point). `ENT_HTML5` uses its own map. Others (currently `ENT_XML1`) have a very limited decoding map (`>`, `&`, `<`, `'`, `"` and their numeric equivalents). (see C function unescape_inverse_map)

- Note for the previous point: when only a few entities must be escaped (think of `htmlspecialchars`), all entities map will use the same one as `ENT_XML1`, except for `ENT_HTML401`. That one will not use `'`, but `'`.

That covers almost everything. I am not going to list all entity differences, instead I would like to point at https://github.com/php/php-src/tree/php-5.4.11/ext/standard/html_tables for some text files that contain the mappings for each type.

What ENT_* should I use for htmlspecialchars?

When using `htmlspecialchars` with ENT_COMPAT (default) or ENT_NOQUOTES, it does not matter which one you pick (see below). I saw some answers here on SO that boils down to this:

<input value="<?php echo htmlspecialchars($str, ENT_HTML5);?>" >

This is insecure. It will override the default value `ENT_HTML401 | ENT_COMPAT` which has as difference that HTML5 entities are used, but also that quotes are not escaped anymore! In addition, this is redundant code. The entities that have to be encoded by `htmlspecialchars` are the same for all `ENT_HTML401`, `ENT_HTML5`, etc.

Just use `ENT_COMPAT` or `ENT_QUOTES` instead. The latter also works when you use apostrophes for attributes (`value='foo'`). If you only have two arguments for `htmlspecialchars`, do not include the argument at all since it is the default (`ENT_HTML401` is 0, remember?).

When you want to print something on the page (between tags, not attributes), it does not matter at all which one you pick as it will have equal effect. It is even sufficient to use `ENT_NOQUOTES | ENT_HTML401` which equals to the numeric value `0`.

See also below, about ENT_SUBTITUTE and ENT_DISALLOWED.

What ENT_* should I use for htmlentities?

If your text editor or database is so crappy that you cannot include non-US-ASCII characters (e.g. UTF-8), you can use htmlentities. Otherwise, save some bytes and use htmlspecialchars instead (see above).

Whether you need to use `ENT_HTML401`, `ENT_HTML5` or something else depends on how your page is served. When you have a HTML5 page (`<!doctype html>`), use `ENT_HTML5`. XHTML or XML? Use the corresponding `ENT_XHTML` or `ENT_XML1`. With no doctype or plain ol' HTML4, use `ENT_HTML401` (which is the default when omitted).

Should I use ENT_DISALLOWED, ENT_IGNORE or ENT_SUBSTITUTE?

By default, byte sequences that are invalid for the given character set are removed. To have a `�` in place of an invalid byte sequence, specify `ENT_SUBSTITUTE`. (note that `&#FFFD;` is shown for non-UTF-8 charsets). When you specify `ENT_IGNORE` though, these characters are not shown even if you specified `ENT_SUBSTITUTE`.

Invalid characters for a document type are substituted by the same replacement character (or its entity) above when `ENT_DISALLOWED` is specified. This happens regardless of having `ENT_IGNORE` set (which has nothing to do with invalid chars for doctypes).

Problem

Since php 5.4 html_entity_decode introduces four new flags, with a minimal explanation ``` ENT_HTML401 Handle code as HTML 4.01. ENT_XML1 Handle code as XML 1. ENT_XHTML Handle code as XHTML. ENT_HTML5 Handle code as HTML 5. ``` I want to understand what are they for. In which cases are they significant? My guess, (but may I be wrong) is that any different standard, encodes some unusual chars but any other don't, so in order to respect that, they are here. My research: htmlentities has the same minimal explanation, with no examples too. I have googled with no luck.

Original source