From owner-ports-jp@jp.FreeBSD.org Fri Jul  9 08:48:44 2004
Received: (from daemon@localhost)
	by castle.jp.FreeBSD.org (8.11.6p2+3.4W/8.11.3) id i68Nmi308570;
	Fri, 9 Jul 2004 08:48:44 +0900 (JST)
	(envelope-from owner-ports-jp@jp.FreeBSD.org)
Received: from t-mta3.odn.ne.jp (mfep3.odn.ne.jp [143.90.131.181])
	by castle.jp.FreeBSD.org (8.11.6p2+3.4W/8.11.3) with ESMTP/inet id i68NmiI08565
	for <ports-jp@jp.FreeBSD.org>; Fri, 9 Jul 2004 08:48:44 +0900 (JST)
	(envelope-from delmonta@ht.sakura.ne.jp)
Received: from ht.sakura.ne.jp ([211.121.42.245]) by t-mta3.odn.ne.jp
          with ESMTP
          id <20040708234844115.RZXX.153421.t-mta3.odn.ne.jp@mta3.odn.ne.jp>
          for <ports-jp@jp.FreeBSD.org>; Fri, 9 Jul 2004 08:48:44 +0900
Message-ID: <40EDDD42.36244A30@ht.sakura.ne.jp>
From: IIJIMA Hiromitsu <delmonta@ht.sakura.ne.jp>
Organization: DENNOU GEDOU GAKKAI, N. D. D. // FABRICA UTILITATIS
X-Mailer: Mozilla 4.78 [ja] (Win98; U)
X-Accept-Language: ja,en,zh-TW,zh,zh-CN,de,es,ko
MIME-Version: 1.0
To: ports-jp@jp.FreeBSD.org
References: <40EA670B.C2ECB55A@ht.sakura.ne.jp>
				 <040706180525.M0104221@pelsia.ensure-tech.co.jp> <87acyclezo.fsf@roddy.acest.tutrp.tut.ac.jp>
Content-Type: text/plain; charset=iso-2022-jp
Content-Transfer-Encoding: 7bit
Reply-To: ports-jp@jp.FreeBSD.org
Precedence: list
Date: Fri, 09 Jul 2004 08:48:18 +0900
X-Sequence: ports-jp 14856
Subject: [ports-jp 14856] Re: converters/libiconv and net/samba3
Sender: owner-ports-jp@jp.FreeBSD.org
X-Originator: delmonta@ht.sakura.ne.jp
X-Distribute: distribute version 2.1 (Alpha) patchlevel 24e+040628

$B$$$$$8$^$G$9!#(B

> > # ports-jp $B$K!VF($2$FMh$^$7$?!W$C$F$$$&$N$O$b$A$m$s%"%j(B:-).
> > # $B$=$l$K$h$C$FK\2H$N(BML$B$G$N5DO@$KCmL\$7$^$7$g$&(B. $B$H$$$&$3$H(B
> > # $B$G(B http://lists.freebsd.org/pipermail/freebsd-gnome/2004-July/007401.html
> > # $B$G;O$^$k%9%l%C%I$K$7$+$HCmL\(B! (^^v.
> 
> $B$H$$$&$3$H$G!":F$SD)$s$G$_$^$7$?$,!"(BFrom $B$r:>>N$9$k$N$rK:$l$^$7$?!#(B^^;

$B%U%)%m!<$,$D$$$F$^$9$M!#(B

> Btw, are you guys pretty sure you problem comes form libiconv?
...
> Why I asking because I already saw false reports 
> about libiconv problems when people tried to convert windows client 
> encoding to samba's host encoding and this is not always possible.

$B$@$=$&$G$9!#(B
$B$=$b$=$bJQ49$G$-$J$$;vNc$J$N$K(B libiconv $B$N$;$$$@$H<gD%$5$l$?%1!<%9$,J#?t(B
$B$"$C$?!"$H!#(B

> I have 
> few japanese windows workstations here and if you like can check what's 
> wrong with them. Just give me a simple instructions how to reproduce a 
> problem in this case.

$B:#$N(B libiconv $B$GIT6q9g$N5/$-$k<BNc$r5s$2$m!"$H!#(B

				$B!y(B

$B$"$H$N%U%)%m!<$G$-$=$&$J$H$3$m$O!"$A$g$C$H=q$$$F$_$^$7$?!#K\2H$KD>@\=P$=(B
$B$&$+$H;W$C$?$s$G$9$,!"$3$l$GG<F@$7$F$b$i$($k$+$=$l$H$b5U8z2L$K$J$k$+H=CG(B
$B$,$D$+$J$$$N$G!"$H$j$"$($:!"$3$3(B ports-jp $B$KN.$7$^$9!#LdBj$J$$$h$&$@$C$?(B
$B$i$=$N$^$^N.$7$A$c$C$F$/$@$5$$!#!d!dCf<#$5$s!"$[$+3'MM(B

$B!t$"$H!"$b$71QJ8K!$N4V0c$$$r8+$D$1$?$iD>$7$F$+$i=P$7$F$/$@$5$$(B :-)

----
> If Microsoft called some hacked Shift_JIS version Shift_JIS 
> it doesn't make it valid for the rest of the world.

Absolutely right, in a context in which JIS-Unicode mapping does matter.
In such a context, we cannot, and will not, call CP932 as Shift_JIS.

But in practice, we have traditionally treated official Shift_JIS,
Microsoft CP932, Apple Japanese set, Sun Java's "SJIS" encoding, etc.
as identical to each other, whenever we do not use extra characters
added by each vendor (for example, codepoints circa 0x85??-0x87??).

Historically, JIS X0208 was born in 1978 and soon 'shift' encoding
(what we now call Shift_JIS) and 8th-bit-on encoding (what we now call
EUC) were invented. Of course there was no Unicode at that time and
JIS did not describe precisely how each symbol is used.

For instance, Shift_JIS 0x8166 (now all vendors map to U+2019 RIGHT
SINGLE QUOTATION MARK) played two roles: right quotation mark and
apostrophe. Despite that we now have U+FF07 FULLWIDTH APOSTROPHE!

It is in 1997 that JIS X0208 was reformed to identify each symbol by
English name and UTF codepoint, and it was too late. I heard that
a draft was proposed in 1994, but virtually nobody knew it.

So each vendor had designed its own JIS-Unicode mapping in the way that
he/she thought is the best. Microsoft CP932 and Apple Japanese set are
a few examples of such mappings, with addition of extra characters.

# Apple Japanese set is available at:
# http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/JAPANESE.TXT


See Shift_JIS codepoint 0x81CA. Most tables map this symbol to U+00AC
NOT SIGN, but Microsoft does to U+FFE2, to avoid conflict with OS/2
single-byte NOT SIGN.


> For instance you can not have 1:1 mapping between cp932 and eucJP.

You are right, if you applied the rule strictly.

So we do not employ Unicode but we map CP932 or Shift_JIS characters to
EUC-JP in this simple calculation, with no conversion table:

	Either MS, Apple, or Shift_JIS    EUC-JP
	-----------------------------------------------
	0x00-0x7F (ASCII)              -> 0x00-0x7F
	0xA1-0xDF (JIS X0201 Katakana) -> 0x8EA1-0x8EDF

	0x8140-0x819E (except 0x817F)  -> 0xA1A1-0xA1FE
	0x819F-0x81FC                  -> 0xA2A1-0xA2FE
	0x8240-0x829E (ditto)          -> 0xA3A1-0xA3FE
	0x829F-0x82FC                  -> 0xA4A1-0xA4FE
		:
	0x9F9F-0x9FFC                  -> 0xDEA1-0xDEFE

	0xE040-0xE09E (ditto)          -> 0xDFA1-0xDFFE
		:
	0xEF9F-0xEFFC                  -> 0xFEFE-0xFEFE

	0x80 and 0xFD-0xFF (Apple only)-> not supported

	(CP932 only)
	0xF040-0xF9FC (private area)   -> not supported
	0xFA40-0xFC4B (IBM extention)  -> few converters support them,
			but there are two ways:
			(a) every character here has its duplicate within
			the range of 0x8140-0xEFFC (namely 0x87?? and
			0xED40-0xEEFC) for historical reason, so it can
			be unified to its counterpart, though this breaks
			round-trip conversion.
			(b) most characters here (perhaps all) are included
			in JIS X0212 (specified by 'ESC $ ( D' in 7-bit
			encodings and prefixed by 0x8F in EUC-JP), so
			you can convert them to X0212 characters if your
			applications support X0212.

Extra characters by Microsoft or Apple are mapped to EUC-JP undefined
codepoints, but we either use a font that supports such extras or totally
eliminate these characters.

----

========================================================================
$BHSEh(B $B9@8w(B / $B$G$k$b$s$?!&$$$$$8$^(B   http://www.ht.sakura.ne.jp/~delmonta/
IIJIMA Hiromitsu, aka Delmonta           mailto:delmonta@ht.sakura.ne.jp
