Bug#995445: logidee-tools: creates incorrect TeX code, breaks other packages via autopkgtests

[Norbert Preining <norbert@preining.info>]

Package: logidee-tools
Version: 1.2.19
Severity: normal
X-Debbugs-Cc: debian-ci@lists.debian.org, debian-tex-maint@lists.debian.org

Hi all,

recent uploads of TeX Live (texlive-base et al) got stalled due to a
failed autopkgtest of logidee-tools.

I analyze the code generated by logidee-tools, and it turned out to be
rather incorrect (La)TeX code, that just by chance worked.

The failed autopkgtest can be boiled down to the following example (all
the code is as taken from the logidee-tools generated code):
*******************
\documentclass[11pt]{minimal}
\newenvironment{note}[1][]{\nettoie}
\long\def\nettoie#1{\mangeblanc{#1}}
\long\def\mangeblanc#1{%
    \ifcat{#1}{ }\else\ifcat\par{#1}\else\stopmange{#1}\fi\fi
\mangeblanc}
\long\def\stopmange#1#2\mangeblanc {\fi\fi{#1}}
\begin{document}
\begin{note}
AAA é  % NOTE without the "AAA " it breaks ALSO in TL2020!
\end{note}
\end{document}
****************************

This worked up to LaTeX2e <2020-10-01>, but with LaTeX2e <2021-06-01>
it fails, and the autopkgtest now prevents texlive-* from entering
testing.

The code is rather convoluted, so I asked the LaTeX Team about it, and
Frank Mittelbach answered the following:

******************************************************
quite surprising that that works with anything ... to me this looks like a
larger bunch of misunderstandings

>     \ifcat{#1}{ }\else\ifcat\par{#1}\else\stopmange{#1}\fi\fi

I presume the idea here is to test if #1 consists of a blank and if not if it
consists of "\par"  or in plain words, to jump of blanks and newlines at the
beginning of the note env body

but if you look at TeXbook page 209: \ifcat expands until it finds two
unexpandable tokens so if you feed that random input from #1 or \par (which
may be the primitive but also may be an expandable macro in LaTeX) you get
totally random results. Certainly not testing what I think was tried to test.

What it actually does is testing if "{" has the same catcode as the first
token in #1 (after full expansion). As that first token normally doesn't have
the catcode of { you get to the second test.

But try \begin{note} \bgroup

and you make it true after which it falls over on the closing } because the {
one has been swallowed by the test.

The second test is equally wrong

 - \par may get expanded after which you compare catcodes of the first two
tokens from that expansion. If that then returns true by chance you have
swallowed 2 tokens from the exapnasion of \par you will then execute the rest
(which of course is incomplete and not intended by the author of the above
macros) followed by {#1} (whatever that contains)

 - if it is the (unexpandable) primitive  -- as it was most of the time but
not always in older LaTeX -- you compare catcode of a macro (16) to catcode of
{ so false  and you jump over that.

Bottom line

 Aus Falschem folgt Beliebiges
*****************************************************

("Aus Falschem folgt Beliebiges" = "ex falso quotlibet")


So it seems that simply the generated code is incorrect according to the
prime expert in this area, and just happened to work for some cases, but
not others (especially when it comes to UTF8, as we see).

That said, I think what is necessary is:
* fixing of the generated code
* disabling of the autopkgtest for as long as there is no proper fix

Thanks a lot

Norbert

-- System Information:
Debian Release: bookworm/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.13.19+futex+ (SMP w/8 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages logidee-tools depends on:
ii  ghostscript                      9.54.0~dfsg-5
ii  imagemagick                      8:6.9.11.60+dfsg-1.3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3
ii  perl                             5.32.1-6
ii  psutils                          1.17.dfsg-4
ii  texlive-fonts-recommended        2021.20210921-1
ii  texlive-lang-french              2021.20210921-1
ii  texlive-lang-german              2021.20210921-1
ii  texlive-latex-extra              2021.20210921-1
ii  texlive-latex-recommended        2021.20210921-1
ii  texlive-pictures                 2021.20210921-1
ii  texlive-pstricks                 2021.20210921-1
ii  xsltproc                         1.1.34-4

Versions of packages logidee-tools recommends:
ii  libxml2-utils  2.9.12+dfsg-5

logidee-tools suggests no packages.

-- no debconf information