ks_c-5601-1987 is used by microsoft when it really means cp949

[bitdancer]

BPO	18625
Nosy	@malemburg, @warsaw, @hyeshik, @ezio-melotti, @bitdancer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-01.23:54:04.002>
labels = ['type-feature', 'expert-email', 'expert-unicode']
title = 'ks_c-5601-1987 is used by microsoft when it really means cp949'
updated_at = <Date 2013-08-02.08:23:53.892>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2013-08-02.08:23:53.892>
actor = 'lemburg'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode', 'email']
creation = <Date 2013-08-01.23:54:04.002>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 18625
keywords = []
message_count = 2.0
messages = ['194139', '194163']
nosy_count = 5.0
nosy_names = ['lemburg', 'barry', 'hyeshik.chang', 'ezio.melotti', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18625'
versions = ['Python 3.4']

Linked PRs

• gh-62825: Fix encoding aliases "KS_C_5601-1987", "KS X 1001", etc #150933
• [3.15] gh-62825: Fix encoding aliases "KS_C_5601-1987", "KS X 1001", etc (GH-150933) #150946

[bitdancer]

When Microsoft handles Korean text, it uses its own code page, cp949, which is a superset of ks_c-5601-1987. But when talking to the rest of the world, it claims that the character set name is ks_c-5601-1987. This means that text claimed to be in ks_c-5601-1987 in email messages (and probably on web pages) can't always be decoded using the codec that ks_c-5601-1987 maps to (euc_kr). [*]

This problem shows up in the real world in email. If non euc_kr characters are used, email will try blow up when trying to decode the ostensibly ks_c-5601-1987 text. (I'm not sure if it will also blow up trying to encode it, I'm not sure what characters the two codecs cover.)

I'm not sure what the best solution is, but one possibility would be to add a "fixup" table to email that would cause it to decode ostensibly ks_c-5601-1987 text using the cp949 codec. Since cp949 is a superset, this should at least solve the input side.

[*] Some relevant standards discussion:

http://lists.w3.org/Archives/Public/ietf-charsets/2001AprJun/0033.html

From this, it isn't clear why we map ks_c-5601-1987 to euc_kr, since they at least appear to be different codecs. I haven't looked at the relevant RFCs to see what the differences are, though.

[malemburg]

The alias was added by Hye-Shik Chang:

http://hg.python.org/cpython-fullhistory/annotate/887ce39f95f2/Lib/encodings/aliases.py#198

I've added him to the nosy list.

If the alias don't match, we'd have to add a codec for the mismatching encoding to maintain compatibility (provided the mismatching encoding is still in use).

[mikethetexan]

Is there any update on this bug?

[serhiy-storchaka]

What is the relation between KS_C_5601-1987, EUC-KR and cp949? In the IANA registry KS_C_5601-1987 and EUC-KR are two different encodings, and cp949 is not registered. In Python, ks_c_5601_1987 is an alias of euc_kr. Should not it be an alias of cp949 instead?

[corona10]

@serhiy-storchaka
After re-investigating this issue, I think ks_c_5601_1987 should be treated as an alias of cp949, considering the historical background that both ks_c_5601_1987 and cp949 were introduced by Microsoft.

[serhiy-storchaka]

As I understand it, KS_C_5601-1987 is simply a set of characters with assigned indexes, like Unicode for East Asia before Unicode. EUC-KR and CP949 are different schemes to encode these indexes in bytes, like UCS2 and UTF16 for Unicode. Originally EUC-KR was limited, and CP949 extended it. Microsoft calls CP949 simply KS_C_5601-1987, like somebody can call UTF-8 or UCS2/UTF-16 simply Unicode.

In Python, EUC-KR is extended. Both EUC-KR and CP949 are able to encode the same 17176 characters. But the result of encoding is different.

>>> '\uac02'.encode('euc-kr')
b'\xa4\xd4\xa4\xa1\xa4\xbf\xa4\xa2'
>>> '\uac02'.encode('cp949')
b'\x81A'

CP949 can decode what was encoded in EUC-KR, but the result is different from the original. EUC-KR can fail trying to decode what was encoded in CP949.

>>> '\uac02'.encode('euc-kr').decode('cp949')
'ㅤㄱㅏㄲ'
>>> '\uac02'.encode('cp949').decode('euc-kr')
UnicodeDecodeError: 'euc_kr' codec can't decode byte 0x81 in position 0: illegal multibyte sequence
decoding with 'euc-kr' codec failed

[malemburg]

According to the Wikipedia page (https://en.wikipedia.org/wiki/KS_X_1001), KS X 1001 is the new name for KS C 5601. If you look at the implementation, both cp949 and eur_kr reference KS X 1001, and that's no surprise, since the KS X 1001 character set seems to be a partial encoding only covering the Hangul and Hanja chars, while cp949 and eur_kr cover additional chars.

They both use different ways of encoding the characters from KS X 1001, though. The cp949 codec seems to be closer to the original KS X 1001 encoding from looking at the code, so I think switching these aliases:

    'ksc5601'            : 'euc_kr',
    'ks_c_5601'          : 'euc_kr',
    'ks_c_5601_1987'     : 'euc_kr',
    'ksx1001'            : 'euc_kr',
    'ks_x_1001'          : 'euc_kr',

over to 'cp949' is a more correct approach.

That said, I'm no expert in these encodings and don't know how the codecs actually work. I can just look at the code and what it does.

Here's the relevant part from cp949:

        OUTBYTE1((code >> 8) | 0x80);
        if (code & 0x8000)
            OUTBYTE2(code & 0xFF); /* MSB set: CP949 */
        else
            OUTBYTE2((code & 0xFF) | 0x80); /* MSB unset: ks x 1001 */

If the MSB is not set, CP949 will look like KS X 1001.