Ruby and MySQL UTF-8 characters
encoding, mysql, ruby, sequel, utf-8
Solution
Try using the `mysql2` adapter instead of the `mysql` adapter, as I believe the `mysql2` driver handles encodings while the `mysql` driver does not.
Problem
I'm switching a Sinatra application from SQLite3 to MySQL. For some reason, which I can't understand, when I extract data from the MySQL using Ruby and Sequel the characters appear in 8-BIT ASCII instead of UTF-8. The deployment environment is a FreeBSD 9.1 and MySQL 5.6.12, with a system-wide ruby19 installed from FreeBSD ports. RVM ruby-2.0p247 produces the same result though. My `my.cnf` is the following: ``` # The following options will be passed to all MySQL clients [client] default-character-set=utf8 #password = your_password port = 3306 socket = /tmp/mysql.sock # Here follows entries for some specific programs # The MySQL server [mysqld] port = 3306 socket = /tmp/mysql.sock skip-external-locking key_buffer_size = 128M max_allowed_packet = 1M table_open_cache = 256 sort_buffer_size = 1M read_buffer_size = 1M read_rnd_buffer_size = 2M myisam_sort_buffer_size = 32M thread_cache_size = 4 query_cache_size= 8M # Try number of CPU's*2 for thread_concurrency thread_concurrency = 2 # encoding issues character-set-server=utf8 collation-server=utf8_general_ci log-bin=mysql-bin binlog_format=mixed server-id = 1 [mysqldump] quick max_allowed_packet = 16M [mysql] no-auto-rehash safe-updates [myisamchk] key_buffer_size = 64M sort_buffer_size = 64M read_buffer = 1M write_buffer = 1M [mysqlhotcopy] interactive-timeout ``` All my files use the shebang line along with the UTF-8 encoding like this script I use to test entries: ``` #!/usr/bin/env ruby # encoding: UTF-8 require 'sequel' msql = Sequel.connect(adapter: 'mysql', host: 'localhost', database: 'metrosignage', user: 'atma', password: 'toola697', encoding: 'utf8') b = msql[:drama_addressbook] b.each do |entry| p entry # p entry[:city].force_encoding("utf-8") end ``` If I use `entry[:city].force_encoding("utf-8")` the output is correct, Greek UTF-8 characters are displayed fine. However I don't understand why I can't extract UTF-8 directly. The table I'm reading the data is created using the following SQL: ``` CREATE TABLE `drama_addressbook` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(255) DEFAULT NULL, `address` varchar(255) DEFAULT NULL, `address_no` int(11) DEFAULT NULL, `address_description` varchar(255) DEFAULT NULL, `phone` varchar(255) DEFAULT NULL, `city` varchar(255) DEFAULT NULL, `country` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=30 DEFAULT CHARSET=utf8; ``` So the database is UTF-8 and the data is UTF-8. My questions are: - Am I doing something wrong? - Why does Ruby need `force_encoding`?