On Wed, Dec 17, 2025 at 12:22:21PM +0100, Vincent Lefevre wrote:
> On 2025-12-17 04:32:37 -0500, Thomas Dickey wrote:
> > On Wed, Dec 17, 2025 at 04:50:45AM +0100, Vincent Lefevre wrote:
> > > With xterm 405, the use of the U+FE0F VARIATION SELECTOR-16 (VS16)
> > > character after an emoji can completely corrupt the display with
> > > Mutt. GNU Screen also gets broken with the command below (issues
> > > with the last line of the terminal). I suspect that this is due
> > > to an inconsistency between the xterm behavior and wcwidth(),
> > > which may affect various applications that rely on wcwidth().
> >
> > Without the Emoji width feature (which as I mentioned, I see should be
> > configurable), xterm's wcwidth is a close match for glibc's wcwidth.
> > The few differences which I noticed in testing appear to be problems with
> > glibc.
>
> But if xterm's Emoji width feature is enable, I don't see how this
> can be fixable with wcwidth() since the width is contextual. It is
well, yes - I view the spec as flawed because it changes things after
they are processed. But it's not going away, it appears, so I decided
to work through the implementation and resolve issues.
Since it seemed to be in actuality a seldom-used feature,
I didn't anticipate much feedback. :-(
> the spec that would be broken. And there should be a clean way to
> query the terminal about its behavior.
I agree - there should be. (Using a cursor position test isn't clean).
I'm aware of a few terminals which have a query for the larger scope
of "grapheme clusters", but none that are designed to identify the
nuances of this smaller feature.
> > Checking now, mutt has a wcwidth.c, which is not often used (since it's
> > compile-time), which is just as well because it's tables are very old.
> > It has a wrapper for wcwidth which makes some assumptions about iswprint
> > that make its behavior problematic except with glibc.
>
> There was the same issue with GNU Screen in the past, which has
> now been fixed by using the wcwidth function from the C library,
> following my bug reports:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1027733
> https://savannah.gnu.org/bugs/?63634
>
> But I did not have any issue with Mutt itself over the years.
> Well... Actually it seems that Mutt uses its own implementation
> only if such functions are not present in the C library.
I had in mind the BSDs (or MacOS), whose notion of wcwidth differs from glibc.
> In configure.ac:
>
> if test "$wc_funcs" != yes && test "$wc_funcs" != no; then
> AC_CACHE_CHECK([for wchar_t functions], mutt_cv_wc_funcs,
> mutt_cv_wc_funcs=no
> AC_LINK_IFELSE([AC_LANG_PROGRAM([[
> #define _XOPEN_SOURCE 600
> #include <stddef.h>
> #include <stdlib.h>
> #ifdef HAVE_WCHAR_H
> #include <wchar.h>
> #endif
> #ifdef HAVE_WCTYPE_H
> #include <wctype.h>
> #endif]], [[mbrtowc(0, 0, 0, 0); wctomb(0, 0); wcwidth(0);
> iswprint(0); iswspace(0); towlower(0); towupper(0); iswalnum(0)]])],[mutt_cv_wc_funcs=yes],[]))
> wc_funcs=$mutt_cv_wc_funcs
> fi
>
> if test $wc_funcs = yes; then
> AC_DEFINE(HAVE_WC_FUNCS,1,[ Define if you are using the system's wchar_t functions. ])
> else
> MUTT_LIB_OBJECTS="$MUTT_LIB_OBJECTS utf8.o wcwidth.o"
> fi
>
> So, in practice, under Linux (and most systems?), Mutt uses the
> wcwidth function from the C library.
>
> > > I have not checked wcswidth().
> >
> > nor I - actually I don't believe it is often used.
> > mutt imitates it by repeatedly calling wcwidth, and doesn't account for VS16.
> >
> > Because mutt isn't accounting for VS16, that's an issue for which xterm
> > "should" be configurable, so we can accommodate programs which pass through
> > VS15 and VS16 without accounting for their behavior.
>
> Note that if Mutt accounted for VS16, this would break its display
> on most terminals (including GNOME Terminal, which seems commonly
> used under Linux).
>
> BTW, the behavior of GNOME Terminal has the advantage to avoid
> display breakage: the emoji takes 2 cells, but wcwidth is honored.
> So it seems that it is up to the application to advance the cursor
> to avoid partial overlap.
>
> > (I haven't investigated "neomutt", which may provide improvements, though
> > the "neo" cult appears to rely heavily upon hard-coding).
>
> Thre would still be the question of behavior with the terminals
> like xterm 403- and GNOME Terminal.
>
> > > But there are issues even with simple output. In a 80-column terminal:
> > >
> > > perl -C -e 'print "\x{2642}\x{FE0F}"x60, "\n"'
> >
> > perl's yet another pitfall. In developing #404, I looked into the wcwidth
> > data used in NetBSD/OpenBSD, which reportedly is tied to perl. That ignores
> > the East Asian stuff entirely, and doesn't match glibc very well.
>
> Note that I'm using perl here only for its "x" feature to repeat
> a string. There is exactly the same issue with printf (with the
> sequence repeated manually), which has the same output.
sure - see the next line -
> > For your example, perl's irrelevant though - this is just bits...
> >
> > > I get "♂♂" in the last two columns, which is inconsistent with what
> > > is output before. And in case of scrolling, the spaces are missing
> > > in the second line.
> >
> > xterm's handling fullwidth characters by putting a non-character in the
> > second cell. In handling VS16, I may have overlooked some path for doing
> > that (something to investigate). But the behavior in mutt was consistent
> > with my expectation: an extra "blank" cell.
>
> BTW, there was another issue I hadn't reported here: this extra
> "blank" cell does not have the correct background when it is
> different from the default background.
agreed - I'm working on that.
Fixing these display issues and providing a way to turn the feature off
are all I intend for #406
--
Thomas E. Dickey <dickey@invisible-island.net>
https://invisible-island.net
Attachment:
signature.asc
Description: PGP signature