How to detect malformed UTF characters

character-encoding, perl, utf-8

Solution

Consider Python. It allows to extend codecs with user-defined error handlers, so you can replace undecodable bytes with anything you want.

import codecs
codecs.register_error('spacer', lambda ex: (u' ', ex.start + 1))
s = 'spam\xb0\xc0eggs\xd0bacon'.decode('utf8', 'spacer')
print s.encode('utf8')

This prints:

spam  eggs bacon

Problem

I want to detect and replace malformed UTF-8 characters with blank space using a Perl script while loading the data using SQL*Loader. How can I do this?

Original source