Bug#712024: Characters go missing when ESC [ K occurs when wrapping a line
Package: xterm
Version: 278-4
(Summary, if I'm guessing right: the terminal emulation code has a bug
related to the ESC [ K control sequence if it occurs when the line is
just about to wrap. And grep uses ESC [ K when changing colors, so the
bug is quite easy to see.)
The command:
perl -e 'print "a"x79 . "bc" . "a"x5 . "\n";' | grep --color=always c
when run on a terminal exactly 80 characters wide should print a line
containing a's and one b, and on the next line a colored (red) c plus
five a's, like this:
rjs@anar:~/t$ perl -e 'print "a"x79 . "bc" . "a"x5 . "\n";' | grep --color=always c
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab
caaaaa
rjs@anar:~/t$
(the above lines were cut-and-pasted from a gnome-terminal window, with
a line break added manually after "b", as displayed on-screen; the "c"
is red)
However, on xterm, the b and c go missing; the following is
cut-and-pasted from an xterm window with a line-break added (that's how
it looks on screen, with all characters the same color):
rjs@anar:~/t$ perl -e 'print "a"x79 . "bc" . "a"x5 . "\n";' | grep --color=always c
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaa
rjs@anar:~/t$
Rxvt breaks in the same way as xterm. But the command works properly in
gnome-terminal and konsole, as well as in tmux, even when tmux is run
inside an xterm. But both rxvt-unicode and screen break in another way:
the b at the end of the line goes missing, but the red c remains (in
screen, apparently regardless of where screen is run). I just found out
that (apparently) the same bug has been reported five years ago for rxvt
as bug #477463 <http://bugs.debian.org/477463>.
Which escape sequences does grep actually produce? In an xterm
(TERM=xterm):
rjs@anar:~/t$ perl -e 'print "a"x79 . "bc" . "a"x5 . "\n";' | grep --color=always c | cat -A
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab^[[01;31m^[[Kc^[[m^[[Kaaaaa$
rjs@anar:~/t$
(the bug remains when adding | cat without -A to the end of the command)
So grep prints extra ESC [ K control sequences before and after the
match. Why? The grep source code has a long comment about it:
,---- Debian wheezy grep-2.12/src/main.c lines 143-201
| /* Select Graphic Rendition (SGR, "\33[...m") strings. */
| /* Also Erase in Line (EL) to Right ("\33[K") by default. */
| /* Why have EL to Right after SGR?
| -- The behavior of line-wrapping when at the bottom of the
| terminal screen and at the end of the current line is often
| such that a new line is introduced, entirely cleared with
| the current background color which may be different from the
| default one (see the boolean back_color_erase terminfo(5)
| capability), thus scrolling the display by one line.
| The end of this new line will stay in this background color
| even after reverting to the default background color with
| "\33[m', unless it is explicitly cleared again with "\33[K"
| (which is the behavior the user would instinctively expect
| from the whole thing). There may be some unavoidable
| background-color flicker at the end of this new line because
| of this (when timing with the monitor's redraw is just right).
| -- The behavior of HT (tab, "\t") is usually the same as that of
| Cursor Forward Tabulation (CHT) with a default parameter
| of 1 ("\33[I"), i.e., it performs pure movement to the next
| tab stop, without any clearing of either content or screen
| attributes (including background color); try
| printf 'asdfqwerzxcv\rASDF\tZXCV\n'
| in a bash(1) shell to demonstrate this. This is not what the
| user would instinctively expect of HT (but is ok for CHT).
| The instinctive behavior would include clearing the terminal
| cells that are skipped over by HT with blank cells in the
| current screen attributes, including background color;
| the boolean dest_tabs_magic_smso terminfo(5) capability
| indicates this saner behavior for HT, but only some rare
| terminals have it (although it also indicates a special
| glitch with standout mode in the Teleray terminal for which
| it was initially introduced). The remedy is to add "\33K"
| after each SGR sequence, be it START (to fix the behavior
| of any HT after that before another SGR) or END (to fix the
| behavior of an HT in default background color that would
| follow a line-wrapping at the bottom of the screen in another
| background color, and to complement doing it after START).
| Piping grep's output through a pager such as less(1) avoids
| any HT problems since the pager performs tab expansion.
|
| Generic disadvantages of this remedy are:
| -- Some very rare terminals might support SGR but not EL (nobody
| will use "grep --color" on a terminal that does not support
| SGR in the first place).
| -- Having these extra control sequences might somewhat complicate
| the task of any program trying to parse "grep --color"
| output in order to extract structuring information from it.
| A specific disadvantage to doing it after SGR START is:
| -- Even more possible background color flicker (when timing
| with the monitor's redraw is just right), even when not at the
| bottom of the screen.
| There are no additional disadvantages specific to doing it after
| SGR END.
|
| It would be impractical for GNU grep to become a full-fledged
| terminal program linked against ncurses or the like, so it will
| not detect terminfo(5) capabilities. */
| static const char *sgr_start = "\33[%sm\33[K";
| static const char *sgr_end = "\33[m\33[K";
`----
This bug can of course also be reproduced without using grep:
perl -e 'print "a"x79 . "b\e[01;31m\e[Kc\e[m\e[K" . "a"x5 . "\n";'
However, removing the ESC [ K control sequence fixes the bug:
rjs@anar:~/t$ perl -e 'print "a"x79 . "b\e[01;31mc\e[m" . "a"x5 . "\n";'
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab
caaaaa
rjs@anar:~/t$
(cut-and-pasted from xterm with the line break added; the c is red)
But the comment quoted above suggests that ESC [ K is necessary for
non-default background colors. (And of course ESC [ K should probably
not erase extra characters if it occurs close to a line break...)
It appears that the bug is not related to color changes; removing them
still breaks xterm:
rjs@anar:~/t$ perl -e 'print "a"x79 . "b\e[Kc\e[K" . "a"x5 . "\n";'
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaa
rjs@anar:~/t$
(cut-and-pasted from xterm with the line break added; note that the b
and c are missing)
Reply to: