Recode Windows-1252 characters as UTF-8

You are dealing with "\u008a" or "\u009a" strings in your database. They get rendered correctly as "Š" or "š" in a browser, but you cannot display them in a command line console. Ruby says that they are "valid UTF-8" encoding.

In reality, those are windows-1252 encoded string that were mis-interpreted as UTF-8, and as such they get mapped to the Unicode Latin-1 Supplement Block.

Luckily, characters from 0080 to 009F, spanning the whole windows-1252 encoding, are non-printable in Unicode, so it's perfectly safe to assume those are just wrongly interpreted windows-1252 characters, to be able to match and recode them.

Use this function to recode them to proper UTF-8:

def recode_windows_1252_to_utf8(string)
  string.gsub(/[\u0080-\u009F]/) {|x| x.getbyte(1).chr.
    force_encoding('windows-1252').encode('utf-8') }
end

Here we are stripping the first byte of the (wrong) encoding utf-8 encoding (0xc2), creating a new single-character string with the second byte and telling Ruby it's windows-1252, and letting Ruby itself do the encoding to utf-8.