Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8
ruby, ruby-2.0, ruby-on-rails, ruby-on-rails-4
Solution
In Ruby 2.0 the `encode` method is a no-op when encoding a string to its current encoding:
Please note that conversion from an encoding `enc` to the same encoding `enc` is a no-op, i.e. the receiver is returned without any changes, and no exceptions are raised, even if there are invalid bytes.
This changed in 2.1, which also added the `scrub` method as an easier way to do this.
If you are unable to upgrade to 2.1, you’ll have to encode into a different encoding and back in order to remove invalid bytes, something like:
if ! s.valid_encoding?
s = s.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
end
Problem
I see this a lot and haven't figured out a graceful solution. If user input contains invalid byte sequences, I need to be able to have it not raise an exception. For example: ``` # @raw_response comes from user and contains invalid UTF-8 # for example: @raw_response = "\xBF" regex.match(@raw_response) ArgumentError: invalid byte sequence in UTF-8 ``` Numerous similar questions have been asked and the result appears to be encoding or force encoding the string. Neither of these work for me however: ``` regex.match(@raw_response.force_encoding("UTF-8")) ArgumentError: invalid byte sequence in UTF-8 ``` or ``` regex.match(@raw_response.encode("UTF-8", :invalid=>:replace, :replace=>"?")) ArgumentError: invalid byte sequence in UTF-8 ``` Is this a bug with Ruby 2.0.0 or am I missing something? What is strange is it appear to be encoding correctly, but match continues to raise an exception: ``` @raw_response.encode("UTF-8", :invalid=>:replace, :replace=>"?").encoding => #<Encoding:UTF-8> ```