Re: First beta version of the Debian SGML/XML HOWTO
On Wednesday 10 November 1999, at 22 h 5, the keyboard of Taketoshi Sano
<xlj06203@nifty.ne.jp> wrote:
> I have commited. Please use this for web as one of translations if possible.
I have a small problem, see hereunder.
> > PS: what encoding will you use for a Japanese text?
> > I'm not sure all the XML tools will support it.
>
> It is written in EUC-JP.
Why not UTF-8? I remember a discussion/flamewar on debian-devel a few months
ago about wether Debian programs should or should not support multi-byte
options, Unicode, etc.
> I ask the status of tools to Kenshi Muto, and he answered me in Japanese.
> I translated his answer as best as I can.
>
> jade is 8bit through, so EUC-JP text can be handled by jade. libxml seems
> to have no problem also.
XML::Parser, the Perl module which is used to implement pre-processing
(conditional sections) does not know this encoding:
% perl ../condition.pl 2.2 ../howto.ja.db > howto.ja.db
Couldn't open encmap euc-jp.enc:
No such file or directory
at /usr/lib/perl5/5.005/i386-linux/XML/Parser.pm line 174
The documentation says it knows natively only UTF-8, ISO-8859-1, UTF-16, and
US-ASCII, but it can be extended with the XML::Encoding module and some of
these encodings are provided. I included at the end a message saying why
euc-jp is not there.
In the meantime, I used x-euc-jp-unicode. It seems to work. Look at
<http://www.debian.org/~bortz/SGML-HOWTO/potato/howto.ja.html> and, if it is
fine, I'll commit.
> When converting EUC-JP text into html, the information with othercredit
> element would not come on HTML. The name of translater also would not
> be shown on HTML. He suspect stylesheet problem about othercredit element.
He is right. It is a common problem with the DSSSL Modular Stylesheets,
specially with articles. I'll try to add that.
> And character entity such as é in document causes trouble if
My fault.
Mapping files for Japanese encodings
1998 12/25
Fuji Xerox Information Systems
MURATA Makoto
1. Overview
This version of XML::Parser and XML::Encoding does not come with map files for
the charset "Shift_JIS" and the charset "euc-jp". Unfortunately, each of these
charsets has more than one mapping. None of these mappings are
considered as authoritative.
Therefore, we have come to believe that it is dangerous to provide map files
for these charsets. Rather, we introduce several private charsets and map
files for these private charsets. If IANA, Unicode Consoritum, and JIS
eventually reach a consensus, we will be able to provide map files for
"Shift_JIS" and "euc-jp".
2. Different mappings from existing charsets to Unicode
1) Different mappings in JIS X0221 and Unicode
The mapping between JIS X0208:1990 and Unicode 1.1 and the mapping
between JIS X0212:1990 and Unicode 1.1 are published from Unicode
consortium. They are available at
ftp://ftp.unicode.org/Public/MAPPINGS/EASTASIA/JIS/JIS0208.TXT and
ftp://ftp.unicode.org/Public/MAPPINGS/EASTASIA/JIS/JIS0212.TXT,
respectively.) These mapping files have a note as below:
# The kanji mappings are a normative part of ISO/IEC 10646. The
# non-kanji mappings are provisional, pending definition of
# official mappings by Japanese standards bodies.
Unfortunately, the non-kanji mappings in the Japanese standard for ISO 10646/1,
namely JIS X 0221:1995, is different from the Unicode Consortium mapping since
0x213D of JIS X 0208 is mapped to U+2014 (em dash) rather than U+2015
(horizontal bar). Furthermore, JIS X 0221 clearly says that the mapping is
informational and non-normative. As a result, some companies (e.g., Microsoft and
Apple) have introduced slightly different mappings. Therefore, neither the
Unicode consortium mapping nor the JIS X 0221 mapping are considered as
authoritative.
2) Shift-JIS
This charset is especially problematic, since its definition has been unclear
since its inception.
The current registration of the charset "Shift_JIS" is as below:
>Name: Shift_JIS (preferred MIME name)
>MIBenum: 17
>Source: A Microsoft code that extends csHalfWidthKatakana to include
> kanji by adding a second byte when the value of the first
> byte is in the ranges 81-9F or E0-EF.
>Alias: MS_Kanji
>Alias: csShiftJIS
First, this does not reference to the mapping "Shift-JIS to Unicode"
published by the Unicode consortium (available at
ftp://ftp.unicode.org/Public/MAPPINGS/EASTASIA/JIS/SHIFTJIS.TXT).
Second, "kanji" in this registration can be interepreted in different ways.
Does this "kanji" reference to JIS X0208:1978, JIS X0208:1983, or JIS
X0208:1990(== JIS X0208:1997)? These three standards are *incompatible* with
each other. Moreover, we can even argue that "kanji" refers to JIS X0212 or
ideographic characters in other countries.
Third, each company has extended Shift JIS. For example, Microsoft introduced
OEM extensions (NEC extensionsand IBM extensions).
Forth, Shift JIS uses JIS X0201, which is almost upper-compatible with US-ASCII
but is not quite. 5C and 7E of JIS X 0201 are different from backslash and
tilde, respectively. However, many programming languages (e.g., Java)
ignore this difference and assumes that 5C and 7E of Shift JIS are backslash
and tilde.
3. Proposed charsets and mappings
As a tentative solution, we introduce two private charsets for EUC-JP and four
priviate charsets for Shift JIS.
1) EUC-JP
We have two charsets, namely "x-eucjp-unicode" and "x-eucjp-jisx0221". Their
difference is only one code point. The mapping for the former is based
on the Unicode Consortium mapping, while the latter is based on the JIS X0221
mapping.
2) Shift JIS
We have four charsets, namely x-sjis-unicode, x-sjis-jisx0221,
x-sjis-jdk117, and x-sjis-cp932.
The mapping for the charset x-sjis-unicode is the one published by the Unicode
consortium. The mapping for x-sjis-jisx0221 is almost equivalent to
x-sjis-unicode, but 0x213D of JIS X 0208 is mapped to U+2014 (em dash) rather
than U+2015. The charset x-sjis-jdk117 is again almost equivalent to
x-sjis-unicode, but 0x5C and 0x7E of JIS X0201 are mapped to backslash and
tilde.
The charset x-sjis-cp932 is used by Microsoft Windows, and its mapping is
published from the Unicode Consortium (available at:
ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.txt). The
coded character set for this charset includes NEC-extensions and
IBM-extensions. 0x5C and 0x7E of JIS X0201 are mapped to backslash and tilde;
0x213D is mapped to U+2015; and 0x2140, 0x2141, 0x2142, and 0x215E of JIS X
0208 are mapped to compatibility characters.
Makoto
Fuji Xerox Information Systems
Tel: +81-44-812-7230 Fax: +81-44-812-7231
E-mail: murata@apsdc.ksp.fujixerox.co.jp
Reply to: