Bug#99933: second attempt at more comprehensive unicode policy
[ CC's trimmed, since mail to the bug will reach -policy ]
On Mon, 2003-01-06 at 16:07, Jason Gunthorpe wrote:
> Fixing progams that handle terminal input is a different matter IMHO, it's
> something that should be decided on a more case by case basis, and alot of
> cases might be effortless handled just by extending ncurses/slang
A lot of programs don't use curses...
> I think the philosophy should be that everything should be converted to
> UTF-8 after it is read from the terminal. Programs that interface with the
> terminal need to convert.
I generally agree with that.
> Changing programs that handle terminal input is a far smaller scope than
> changing every program that touches argv and every program that does
> terminal input.
If by 'touching argv' you mean 'modifying and creating output based on',
then I hope you agree that we will almost certainly have to make those
programs grok Unicode anyways, as I said before. UTF-8 is a multibyte
encoding, and traversing and manipulating it correctly generally
requires one to use different string functions (although stuff like
strchr(foo, '.') will still work).
> If this route is followed then a huge swath of programs are half correct
> already, their only problem is that they will not be converting utf-8 for
> display. That might be best handled through glibc (again, changing
> *everything* just to get around the lack of utf-8 terminals is insane)
Output is a big problem, I agree. But how exactly do you propose to
modify glibc?
> Well, that's not true. At the shell level everything is tagged. The shell
> knows things returned from readdir are utf-8
No, it doesn't! Even if we force users to run a script which converts
all legacy encodings to UTF-8, people will still have files NFS mounted
readonly on other systems, files that they created using a legacy
program, files on CD-ROM or DVD, etc.
What do you mean anyways that everything on the shell level is tagged?
How is that possible?
What if I do something like this:
touch $(nc www.random.org 80)
> When I mean 'all cases' I mean the cases the come up in a system with only
> UTF-8 names in the filesystem, not one that has mixed encodings already
> in the filesystem, that's hopeless.
But mixed encodings will happen in the real world. It is unavoidable.
There is a lot of legacy data.
> > For the case you named above, I think what should happen is that 'ls'
> > converts all the arguments to UTF-8 for internal processing. For the
> > first argument, UTF-8 validation will fail, so ls will try converting
> > from the locale's charset, which will work. The rest of the arguments
> > will validate as UTF-8, so ls just goes on its way.
>
> Eww, that's gross, it isn't definate that UTF-8 validation will always
> fail for non UTF-8 text, you could easially get lucky and type in a word
> that is valid UTF-8, but needs conversion! That's a terribly subtle UI
> bug.
I agree, it sucks and it's pretty gross. But I don't think there is a
better solution.
> Consider the shell to be a scripting language just like python/java and
> look at how it's handled there - all internal strings are UTF-8, functions
> that read/write to the terminal convert automatically, functions exist to
> convert arbitary text/files.
Yes, but even in Python/Java/C# or whatever, you don't always know the
encoding for sure; what if you're opening up a Debian changelog? By
default the strema will be opened using the user's locale encoding, but
we already mandated that Debian changelogs be UTF-8.
> You have everything needed to make the shell work uniformly in any
> environment, but some cases might require an iconv, but the iconv is
> required for *all* users, not just those with different locale settings. I
> think that's a good goal.
I don't see how you can make iconv just make everything work.
> The trouble is, the shell interfaces with the terminal, so it is the only
> thing in a position to know how to convert characters coming from the
> terimal to UTF-8, nothing else can do this.
As I said, I don't think the shell knows everything, and I think just
modifying the shell will not fix everything, even if it did.
Reply to: