[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: charsets in debian/control

On Tuesday 07 December 2004 10:40 am, Richard Atterer wrote:
> No, you do not have to do this. You can keep working with "char", the
> changes when switching to UTF-8 will mostly have to deal with the fact that
> one Unicode character is represented by more than one char. This means that
> you need to use a different strlen function, take care only to chop strings
> of char at character boundaries, ensure that input strings are actually
> valid UTF-8, etc.

  This might work for programs that relatively blindly manipulate character 
strings and can pass them off to the terminal for processing.  In fact, 
aptitude does a *lot* of processing and formatting of strings internally.  
That means, for instance: splitting strings into words and paragraphs, 
truncating strings, finding out how wide strings are.

  More importantly, it also makes significant (and increasing) use of strings 
annotated with the terminal attributes of each character (think colors, 
bold/reverse video, etc).  Needless to day, it performs all of the above 
operations on those strings as well.

  All of these are impacted by extended charsets: for instance, you need to 
use a different function to find whitespace, and combining characters with 
their attributes requires the use of a structure where an integer previously 
sufficed.  That's not to mention finding the length of a string, which is 
necessary to perform most types of layout.

    The changes that are necessary are at least:

  At a minimum, the class used for formatted strings will have to be 
re-targeted to support either formatted wide strings or formatted utf8 
strings.  If wide characters or are not used internally, it is also necessary 
to audit every occurrence of s.size() and check whether the length-in-memory 
or the length-in-characters of the string is being queried.  If neither wide 
characters nor a utf8-specialized basic_string are used, it is necessary to 
audit every string constructor (which might cut a substring) and make sure 
that it doesn't play havoc with utf8 codings.  Every use of isspace() and 
friends will have to be replaced with Unicode-aware equivalents.

  And that's just the problems I can think of off the top of my head.

  It's also necessary to use a completely different set of terminal i/o 
routines, but this is pretty much expected.

  None of these problems are insurmountable, of course, and I know pretty much 
how to solve must of them.  However, it's also true that none of them exist 
*at all* when using iso-8859-1, which is why I object to the comment that 
it's no harder to handle utf8 than iso-8859-1.  (in fact, if your terminal 
speaks iso-8859-1, aptitude will handle it just fine without any changes)


/------------------- Daniel Burrows <dburrows@debian.org> ------------------\
|              Hi, I'm a .signature virus!                                  |
|              Copy me into your .signature to help me spread!              |
\---------------- The Turtle Moves! -- http://www.lspace.org ---------------/

Attachment: pgpGbO_pCHqSG.pgp
Description: PGP signature

Reply to: