How to detect string byte encoding?
byte, encoding, python, string, unicode
Solution
if your files either in `cp1252` and `utf-8`, then there is an easy way.
import logging
def force_decode(string, codecs=['utf8', 'cp1252']):
for i in codecs:
try:
return string.decode(i)
except UnicodeDecodeError:
pass
logging.warn("cannot decode url %s" % ([string]))
for item in os.listdir(rootPath):
#Convert to Unicode
if isinstance(item, str):
item = force_decode(item)
print item
otherwise, there is a charset detect lib.
Python - detect charset and convert to utf-8
https://pypi.python.org/pypi/chardet
Problem
I've got about 1000 filenames read by `os.listdir()`, some of them are encoded in UTF8 and some are CP1252. I want to decode all of them to Unicode for further processing in my script. Is there a way to get the source encoding to correctly decode into Unicode? Example: ``` for item in os.listdir(rootPath): #Convert to Unicode if isinstance(item, str): item = item.decode('cp1252') # or item = item.decode('utf-8') print item ```