Displaying Utf8 Stings In Ubuntu's Terminal With A Python Script

In my Python script running at the command line on Ubuntu, it's selecting UTF8-encoded content from a MySQL database. Then, I want to print the string to the console. The displayed

Solution 1:

It is very strongly recommended that you not use "?" as a replacement char. Just set your output encoding to UTF-8 and be done with it.

for s in("stdin","stdout","stderr"): 
   setattr(sys, s, io.TextIOWrapper(getattr(sys, s).detach(), encoding="utf8"))

Alternately, set your PYTHONIOENCODING envariable to utf8 so that python stops guessing about the output encoding.

Either approach is infinitely much better than manually encoding, which is stupid.

If you refuse to upgrade to Python3, I also recommend

from __future__ import unicode_literals

to banish all that stupid u'...' stuff.

Lately I’ve starting all my Python progams like this:

#!/usr/bin/env python3.2# -*- coding: UTF-8 -*-from __future__ import print_function
from __future__ import unicode_literals

import re
import sys
import os

ifnot (("PYTHONIOENCODING"in os.environ)
            and
        re.search("^utf-?8$", os.environ["PYTHONIOENCODING"], re.I)):
    sys.stderr.write(sys.argv[0] + ": Please set your PYTHONIOENCODING envariable to utf8\n")
    sys.exit(1)

import unicodedata
if unicodedata.unidata_version < "6.0.0":
    print("WARNING: Your old UCD is out of date, expected at least 6.0.0 but got", 
           unicodedata.unidata_version)

wide_enough = (sys.maxunicode >= 0x10FFFF)
ifnot wide_enough:
    print("WARNING: Narrow build detected, your Python lacks full Unicode support!!")

Solution 2:

You can get the current encoding of STDOUT like this:

>>>import sys>>>sys.stdout.encoding
UTF-8

Then encode your Unicode string accordingly:

>>> u"Ä"u'\xc4'>>> sys.stdout.write(u"Ä".encode(enc, 'replace'))

The 'replace' avoids UnicodeEncodeErrors, when a character is not representable in the terminal's encoding. This will then be replaced with a question mark.

Solution 3:

The input encoding of the text (utf-8 here) does not matter. You should convert utf8-bytestring into Unicode as soon as possible then to print the text:

print(unicode_text)

• do not encode the text into utf8 before printing
• do not modify sys.stdout to encode the text using utf8 for you

The output encoding is specified by the environment that runs your script based on locale settings (LANG, LC_CTYPE, LC_ALL) or PYTHONIOENCODING envvar. Do not output utf8 unconditionally.

For example, if the locale is not set; you could specify it explicitly:

$ LANG=en_US.utf8 python your_script.py

Make sure your terminal is capable of showing the corresponding Unicode characters (fonts, the corresponding locales (locale -a) are installed).

In other words, to fix the output, fix the environment e.g., configure your locale settings to use C.UTF-8 by default.