Bug#1124802: xterm: Cannot switch to UTF-8 Encoding without resetting the terminal

[Jorgen Grahn <grahn+src@snipabacken.se>]

Package: xterm
Version: 398-1
Severity: normal
Tags: l10n

Dear Maintainer,

I upgraded to Debian 13 the other day, and afterwards, switching
to/from UTF-8 text in xterm behaves differently.

I made a test vector with both UTF-8 and iso8859-1 in it:

% xxd xterm_test
00000000: 7574 6638 3a20 2fc3 a52f c3a4 2fc3 b62f  utf8: /../../../
00000010: e280 932f 0a6c 6174 696e 313a 202f e52f  .../.latin1: /./
00000020: e42f f62f 2d2f 0a                        ././-/.

Here are some steps which show the issue:

1. % LANG=C.utf8 LC_ALL=C.utf8 xterm
-> the xterm Ctrl-right button menu shows "UTF-8 Encoding" enabled
   and greyed out (not possible to change)
-> the new shell has C.utf8 locale
-> 'cat xterm_test' renders the UTF-8 text properly

2. % LANG=sv_SE LC_ALL=sv_SE xterm
-> the xterm menu shows "UTF-8 Encoding" disabled
   but possible to enable
-> the new shell has sv_SE locale
-> 'cat xterm_test' does not render the UTF-8 text properly;
   renders the latin1 ok instead

3. Select "UTF-8 Encoding" in that menu.
-> 'cat xterm_test' still does not render the UTF-8 text properly;
   renders the latin1 instead

4. Run 'reset' in the shell.
-> 'cat xterm_test' now renders the UTF-8 text properly.

5. Deselect "UTF-8 Encoding" again in the menu.
-> 'cat xterm_test' does not render the UTF-8 text properly.

6. Select "UTF-8 Encoding" again in the menu.
-> 'cat xterm_test' renders the UTF-8 text properly.

So it seems that if you start out with a non-UTF-8 xterm but want to
switch to UTF-8, you need to both select "UTF-8 Encoding" in the menu
and run reset(1) in the terminal. After that, you can switch back and
forth using only the menu.

I'm pretty sure this is a change compared to Debian 12 and earlier.
I.e. step 3 would have shown the UTF-8 text, with no need for step 4.
I will be able to confirm that, but not until Saturday when I have
access to a Debian 12 system.

I should probably also disclose that I do my daily terminal work in
sv_SE, i.e. a non-utf8 locale. I have 35 years' worth of iso8859-1
data which I don't fancy converting. I only switch to a UTF-8 locale
when I ssh to the host where I read my mail. I open a new xterm
dedicated to this, use the xterm menu to select "UTF-8 Encoding" and
"UTF-8 Fonts", and then:

  % LANG=sv_SE.UTF-8 LC_CTYPE=sv_SE.UTF-8 ssh -t aphanes mutt

Aphanes runs OpenBSD, thus the odd locale names. Anyway this should
not be relevant to my problem -- unless you feel starting a non-UTF-8
xterm is unsupported.

BR,
/Jörgen

-- System Information:
Debian Release: 13.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.12.57+deb13-amd64 (SMP w/12 CPU threads; PREEMPT)
Locale: LANG=sv_SE, LC_CTYPE=sv_SE (charmap=ISO-8859-1), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xterm depends on:
ii  libc6           2.41-12
ii  libfontconfig1  2.15.0-2.3
ii  libfreetype6    2.13.3+dfsg-1
ii  libice6         2:1.1.1-1
ii  libtinfo6       6.5+20250216-2
ii  libutempter0    1.2.1-4
ii  libx11-6        2:1.8.12-1
ii  libxaw7         2:1.0.16-1
ii  libxext6        2:1.3.4-1+b3
ii  libxft2         2.3.6-1+b4
ii  libxinerama1    2:1.1.4-3+b4
ii  libxmu6         2:1.1.3-3+b4
ii  libxpm4         1:3.5.17-1+b3
ii  libxt6t64       1:1.2.1-1.2+b2
ii  xbitmaps        1.1.1-2.2

Versions of packages xterm recommends:
ii  luit [luit]  2.0.20240910-1
ii  x11-utils    7.7+7

Versions of packages xterm suggests:
pn  xfonts-cyrillic  <none>

-- no debconf information