This is the second post of my series “Do you speak UTF-8 ?”. Here you can find the first article on this topic.
This article covers how Ruby 1.9 handles encoding internally and which tooling it provides for encoding issues. Prior to Ruby 1.9 a String was just a sequence of bytes. Calling the method size() returned the size of this byte array, not the character count. In Ruby 1.9 the Encoding
As you can see in this example: You may ask for the encoding of a String. size() returns the actual character count. bytesize() gives you the actual number of bytes. How these two representations differ can be seen if you compare the codepoints and the bytes.
german_string = "muÃ
So this looks fine. But what to do if you want to change the encoding of a String eg. to write it to a file in a specific format. To change the encoding of a String you can use String#encode .true if one of the characters exceeds the ASCII -7Encoding::CompatibilityError.
As mentioned in the prior post: text files do not have that meta information. There is no information about the encoding stored along with the file. The same restriction applies to source files. The effect of this issue can be seen if you change the encoding of a source file that includes non ASCII strings inside a text editor.
But not only text editors have to know the encoding of strings to store the correct byte sequence to disk. The Ruby interpreter has to know how to decode the read bytes. To add this meta information Ruby 1.9 allows to specify this information as a comment in the first line of the source code:
# encoding : utf-8
# ... or ...
# -*- encoding : utf-8 -*-
class ApplicationController < ActionController::Base
GERMAN_STRING = "muÃ
Speaking about files. How do I specify the encoding of a file that I want to read from disk? File.open() allows you to assign the encoding that will be used to read and encode the file data. In this example the read string will have the encoding specified at open() .
file = File.open('iso-8859-1.txt', 'r:iso-8859-1')
puts file.external_encoding.name # >> ISO-8859-1
file_content = file.read
puts file_content.encoding.name # >> ISO-8859-1
file = File.open('iso-8859-1.txt', 'r:iso-8859-1:utf-8')
puts file.external_encoding.name # >> ISO-8859-1
file_content = file.read
puts file_content.encoding.name # >> UTF-8In the second part of the above code example you can see that there is the possibility to specify the internal encoding of this file operation. The read string will be transcoded to that given format.
Doing this makes life really easier. For most applications it is a best practice to work with just one internal encoding. Preferably ‘utf-8′.
Sources: