Bug#819888: closed by David Kalnischkies <david@kalnischkies.de> (Re: Bug#819888: libapt-pkg5.0: Files with utf-8 encoding in /etc/apt/preferences.d/ are parsed incorrectly)

On Mon, Apr 4, 2016 at 3:00 PM, Debian Bug Tracking System <owner@bugs.debian.org> wrote:

This is an automatic notification regarding your Bug report
which was filed against the libapt-pkg5.0 package:

#819888: libapt-pkg5.0: Files with utf-8 encoding in /etc/apt/preferences.d/ are parsed incorrectly

It has been closed by David Kalnischkies <david@kalnischkies.de>.

Their explanation is attached below along with your original report.
If this explanation is unsatisfactory and you have not received a
better one in a separate message then please contact David Kalnischkies <david@kalnischkies.de> by
replying to this email.

--
819888: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819888
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

---------- Forwarded message ----------
From: David Kalnischkies <david@kalnischkies.de>
To: Shalom Bhooshi <s.bhooshi@gmail.com>, 819888-done@bugs.debian.org
Cc:
Date: Mon, 4 Apr 2016 14:58:06 +0200
Subject: Re: Bug#819888: libapt-pkg5.0: Files with utf-8 encoding in /etc/apt/preferences.d/ are parsed incorrectly
Control: tags -1 - l10n + wontfix
Control: severity -1 minor

On Sun, Apr 03, 2016 at 03:51:30PM +0200, Shalom Bhooshi wrote:
> Package: libapt-pkg5.0
> Version: 1.2.9
> Severity: important
> Tags: l10n

The 'bug' has nothing to do with localisation, so dropping the tag.
Severity dropping is explained later along with wontfix & done.

> * What led up to the situation?
>
> apt-get, aptitude, apt-cache, etc all report the following error
> E:Invalid record in the preferences file /etc/apt/preferences.d/my.pref, no Package header

In general, it would have been nice if you had attached the offending
file, so that we have a chance to reproduce this. I use UTF-8 characters
all the time (also in this mail) and can't reproduce this problem with
a "UTF-8 Unicode text" file as identified by 'file'.

> * What exactly did you do (or not do) that was effective (or
> ineffective)?
>
> Create a file in /etc/apt/preferences.d/ using an ansible template that is
> UTF-8 encoded.

I have the strong feeling that this is a file with a BOM [0] in which
case this is a minor 'bug' as you could just as well save the file
without a BOM as there is no point in having it for UTF-8 – and it
causes all sorts of problems…

[0] https://en.wikipedia.org/wiki/Byte_order_mark

(Note to self: 'set bomb' to force vim to write it – how fitting)

> * What outcome did you expect instead?
>
> Ideally, The preferences in the new preferences file should be respected and apt should be
> capable of dealing with files of any file encoding or the error message returned to the
> front-end utilities should indicate a problem with the file or its encoding.

We are not going to support UTF-16 encoding (or higher) [regardless of
its endianness] because that is hard™ and there is no point, so that
will probably remain being an error.

The UTF-8 BOM ignoring is debatable, but while it sounds easy, it is
usually hard in practice and usually not done if there isn't a strong
reason – and that is the problem: There is no strong reason here. The
list of linux tools not supporting BOM is long starting with your $SHELL
already. The error messages they show are equally useless, too (if they
show one – cat e.g. will produce invalid files [as the BOM of the second
file isn't removed]). So we are in perfectly good company.

The only tools I could find who actually support it are tools who have
to deal with files created on a Window machine… the amount of those is
going to be small for the time being as you seem to be the very first
person to report such an issue in 18 years (+ 3 days) of APT.

In the end, these files (preferences, but sources and conf are no
different) aren't general propose text files but configuration files
requiring a certain format to be interpreted correctly – and in our case
the file format doesn't allow BOMs, so instead of leaving this report
linger forever as wontfix (as I doubt anyone will ever touch it as there
are hundreds of more interesting & useful things to work on in apt)
I can go a bit further and close this bug as not-a-bug.

Hi David,

Thanks for the insight - indeed it appears to be the BOM and not the encoding

of the file itself.

I fully appreciate the technicalities and agree with your reasoning to close the bug - no problems there.

The problem is that things left as they still means the current error message does not do much to reflect the problem - and so is a bit of a red-herring and seems to only guide the user further away from spotting the real problem. My otherwise valid file with the right stanzas stumbles on a "no Package header" error - only a well versed user of APT would ignore or read past the current error message to look at wider causes.

I just wonder if we can have apt be a bit more intelligent and make the error a bit more befitting of the problem i.e if the error message said something very simply along the lines of "E: Error parsing preferences file /etc/apt/preferences.d/my.pref" - I might have considered a `cat -e` of the file, spotted the problem and saved myself 2 and a bit hours of being fired up.

If this is possible at all, I think this would save many a user's time - I'm happy to raise another bug if that's what it will take.

Best regards

David Kalnischkies

Many thanks!!

Shalom

---------- Forwarded message ----------
From: Shalom Bhooshi <s.bhooshi@gmail.com>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Cc:
Date: Sun, 03 Apr 2016 15:51:30 +0200
Subject: libapt-pkg5.0: Files with utf-8 encoding in /etc/apt/preferences.d/ are parsed incorrectly
Package: libapt-pkg5.0 Version: 1.2.9 Severity: important Tags: l10n

Dear Maintainer,
* What led up to the situation?
apt-get, aptitude, apt-cache, etc all report the following error
E:Invalid record in the preferences file /etc/apt/preferences.d/my.pref, no Package header
* What exactly did you do (or not do) that was effective (or
  ineffective)?
Create a file in /etc/apt/preferences.d/ using an ansible template that is
UTF-8 encoded.
* What was the outcome of this action?
None of the APT front-end utilities respect the preferences (pinning) set in the new preferences file.
* What outcome did you expect instead?
Ideally, The preferences in the new preferences file should be respected and apt should be
capable of dealing with files of any file encoding or the error message returned to the
front-end utilities should indicate a problem with the file or its encoding.
— System Information: Debian Release: stretch/sid Architecture: amd64 (x86_64)

Kernel: Linux 4.3.0-1-amd64 (SMP w/2 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_GB.UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system)

Versions of packages libapt-pkg5.0 depends on: ii libbz2-1.0 1.0.6-8 ii libc6 2.21-6 ii libgcc1 1:5.3.1-12 ii liblz4-1 0.0~r131-1 ii liblzma5 5.1.1alpha+20120614-2.1 ii libstdc++6 5.3.1-12 ii zlib1g 1:1.2.8.dfsg-2+b1

Versions of packages libapt-pkg5.0 recommends: ii apt 1.2

libapt-pkg5.0 suggests no packages.

— no debconf information