[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: UTF-8 in jessie

On Mon, Aug 12, 2013 at 12:50:35PM +0200, Vincent Lefevre wrote:
> On 2013-08-12 02:51:52 +0200, Adam Borowski wrote:
> > Detecting non-UTF files is easy:
> > * false positives are impossible
> > * false negatives are extremely unlikely: combinations of letters that would
> >   happen to match a valid utf character don't happen naturally, and even if
> >   they did, every single combination in the file tested would need to match
> >   valid utf.
> Not that unlikely, and it is rather annoying that Firefox (and
> therefore Iceweasel) gets this wrong due to an ambiguity with TIS-620.
> IMHO, in case of ambiguity, UTF-8 should always be preferred by
> default (applications could have options to change the preferences).

That's the opposite of what I'm talking about: it is hard to reliably detect
ancient encodings, because they tend to assign a character to every possible
bit stream.  On the other hand, only certain combinations of bytes with the
8th bit set are valid UTF-8, and thus it is possible to detect UTF-8 with
good accuracy.  It is obviously trivial to fool such detection deliberately,
but such combinations don't happen in real languages, and thus if something
validates as UTF-8, it is safe to assume it indeed is.
> > On the other hand, detecting text files is hard.
> Deciding whether a file is a text file may be hard even for a human.
> What about text files with ANSI control sequences?

Same as, say, a Word97 document: not text for my purposes.  It might be
just coloured plain text, but there is no generic way to handle that.
Binary formats go more into subgoal 1 of my proposal: arbitrary Unicode
input that matches your syntax should be accepted, and go out uncorrupted
(not the same as unmodified).
> > One could use location: like, declaring stuff in /etc/ and
> > /usr/share/doc/ to be text unless proven otherwise, but that's an
> > incomplete hack. Only hashbangs can be considered reliable, but
> > scripts are not where most documentation goes.
> > 
> > Also, should HTML be considered text or not?  Updating http-equiv is not
> > rocket surgery, detecting HTML with fancy extensions can be.
> I think better questions could be: why do you want to regard a file as
> text? For what purpose(s)? For the "all shipped text files in UTF-8"
> rule only?

A shipped config file will have some settings the user may edit and comments
he may read.  Being able to see what's going on is a prerequisite here.

A perl/python/etc script is something our kind of folks often edit and/or

A plain text file ships no encoding information, thus it can't be either
rendered nor edited comfortably if the encoding is different from the system

HTML can include http-equiv which take care of rendering, but editing is
still a problem.  And if you edit it, or, say, fill in some fields from a
database, you risk data loss.  If everything is UTF-8 end-to-end, this risk
goes away.  (I do care about plain text more, though.)
> What about examples whose purpose is to have a file in a charset
> different from UTF-8?

Well, we don't convert those :)

I don't expect a package with a test suite that includes charset stuff to
make such an error by itself, but if there's a need, we could add a syntax
for exclusions.  For example, writing "verbatim" in the charset field.

> > 4a. perl and pod
> > 
> > Considering perl to be text raises one more issue: pod.  By perl's design,
> > pod without a specified encoding is considered to be ISO-8859-1, even if
> > the file contains "use utf8;".  This is surprising, and many authors use
> > UTF-8 like everywhere else, leading to obvious results ("man gdm3" for one
> > example).  Thus, there should be a tool (preferably the one mentioned
> > above) that checks perl files for pod with undeclared encoding, and raises
> > alarm if the file contains any bytes with high bit set.  If a conversion
> > encoding is specified, such a declaration could be added automatically.
> Yes, undeclared encoding when not ASCII should be regarded as a bug.

And if it's declared but not UTF-8, I'd convert it at package build time.


Reply to: