Correct character encoding
character-encoding, php
Solution
HTML entities have two goals:
- Escape characters that have a special meaning in HTML, such as angle quotes, so they can be used as literals.
- Display characters that are not supported by the character set you are using, such as the euro symbol in an ISO-8859-1 document.
They are not exactly an encoding tool.
If you want to convert from one charset into another one, I suggest you use iconv(). However, you must know both the source and the target charset. The source charset should be mentioned in the Content-Type response header and the target charset is something you decided when you started the site (although in your case it looks like UTF-8 is the most reasonable option).
Problem
I'm currently scraping a website for various pieces of textual data (with permission, of course). The issue I'm seeing is that certain characters aren't correctly encoded in the process. This is particularly prominent with apostrophes ('): leading to characters such as: . Currently, I use the following code to convert various HTML entities from the scraped data: ``` htmlentities($content, ENT_COMPAT, 'UTF-8', FALSE) ``` Is there a better way to handle this sort of thing?