Bug#865713: Declaring a charset of UTF-8 for policy files
On Sat, Jun 24, 2017 at 7:12 PM, Paul Wise <pabs@debian.org> wrote:
> On Sun, Jun 25, 2017 at 8:54 AM, Simon McVittie wrote:
>
>> For what it's worth, I agree that declaring the correct charset in HTTP
>> metadata is a better solution than prepending U+FEFF ZERO WIDTH NO-BREAK SPACE
>> (aka the "byte-order mark") in the file content.
Yes, the BOM was only intended for UTF-16, which could actually have
two different byte orders. Because there is no such thing as "byte
order" with UTF-8, the world wide web has rebranded the UTF-8
three-byte version of U+FEFF as the "UTF-8 signature". The original
intention of The Unicode Consortium was that the sequence would never
be used in a UTF-8 document.
In Firefox, if you press Ctrl+Shift+Q you will get an "Inspector". Loading
https://www.debian.org/doc/packaging-manuals/upgrading-checklist.txt
with the Netowrk tab selected in the Inspector shows multiple tabs.
The "Console" tab gives this message, highlighted in pink [for
dramatic effect]:
"The character encoding of the plain text document was not declared.
The document will render with garbled text in some browser
configurations if the document contains characters from outside the
US-ASCII range. The character encoding of the file needs to be
declared in the transfer protocol or file needs to use a byte order
mark as an encoding signature."
So the browser is encouraging the use of this three-byte UTF-8 version
of U+FEFF, even though it was never supposed to be used in a document.
We live in an imperfect world.
Going to the Network tab, reloading the page, and clicking on "Raw
Headers" shows the following information (i just made the request
again):
Request Headers:
Host: www.debian.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
If-Modified-Since: Sat, 24 Jun 2017 20:17:13 GMT
If-None-Match: "e965-552ba67456626-gzip"
Cache-Control: max-age=0
Response Headers:
Accept-Ranges: bytes
Cache-Control: max-age=86400
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 18592
Content-Type: text/plain
Date: Sun, 25 Jun 2017 02:10:24 GMT
Etag: "e965-552ba67456626-gzip"
Expires: Mon, 26 Jun 2017 02:10:24 GMT
Keep-Alive: timeout=5, max=100
Last-Modified: Sat, 24 Jun 2017 20:17:13 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000
Vary: Accept-Encoding
X-Clacks-Overhead: GNU Terry Pratchett
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-XSS-Protection: 1
referrer-policy: no-referrer
So the Content-Type is "text/plain", which results in the "garbled
characters", to quote the Firefox Console window in the Inspector. As
an aside, the Content-Encoding is "gzip", which is a good thing.
On Sat, Jun 24, 2017 at 7:12 PM, Paul Wise <pabs@debian.org> wrote:
> Forcing every text file to UTF-8 isn't the correct solution either,
> since it breaks text files that are not encoded in UTF-8 (such as old
> dedication texts) and does not work on Debian mirrors that are not
> controlled by us.
If using the UTF-8 signature in a document is too aesthetically
distateful (and I don't disagree), and if setting the HTTP header to
denote a UTF-8 charset is not a universal solution because it will
only have effect on Debian's servers, would a tool that converted such
text files to an HTML document be desirable? Such a hypothetical tool
would insert a meta tag in the header saying <meta charset="UTF-8">.
If that is an acceptable solution, I could put together an awk script
for Debian (if it would get used) that would employ awk's BEGIN and
END sections to wrap a UTF-8 document in HTML tags, enclosing the text
itself in <PRE>...</PRE> tags. That would mean that Debian UTF-8
documents intended for being served on the web would have to run such
a utility and be converted into HTML pages for display.
Three possibilities seem to exist, and I am fine with any one being chosen:
1) Use the UTF-8 signature in UTF-8 text files
2) Set the HTTP headers for charset="UTF-8"
3) Convert UTF-8 text files to HTML documents for web display
Paul Hardy
Reply to: