Bug#841294: Overrule maitainer of "global" to package a new upstream version

[Ron <ron@debian.org>]

On Mon, Dec 05, 2016 at 10:13:05PM +0100, Philip Hands wrote:
> Ron <ron@debian.org> writes:
> 
> > I'm not insisting that's what we should do.  But it's certainly an
> > option, and it dodges the bullet of having to say "Sucks to be you"
> > without any notice at all.  And it doesn't take anything away from
> > the people who want "new upstream or bust" for Stretch, because it
> > can still be available to them in backports.
> 
> Perhaps you'd be kind enough to either confirm or correct my perceptions
> of the current situation:

I'll try to be concise, but I might fail at 'brief', since I think it's
important to fairly summarise all sides of this.  Leaving people to
somehow reconcile narrowly polarised, conflicting sound bites isn't very
helpful.  And since I don't have the option to abstain from an opinion,
and aren't starting from an immutable one, I want to be as sure as I can
that _I've_ got the whole picture clear in my head before settling on
what I think is really best too.

If I show how I got there, and you disagree with my rationale, then I
have something that _I_ can rethink to maybe agree with you.  If all
I do is state a conclusion, and handwave the details of how I got there,
then at best we can agree to disagree, and at worst we get more random
trolls throwing mindless insults at me again.


>   Version 6 includes a CGI script that one is expected to install in a
>   manner so hopelessly insecure that we'd not accept it in Debian.

For the version that Punit and Vincent wanted me to let them upload
and that people complained that I nacked, which is where this appeal
to the ctte started from, that's absolutely true.  Not only did it
have the 'chmod 777' interface to enable it, it had little gems in it
like this too:

 open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");

Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
The version Punit published and suggested people should use had all
of that enabled in it - and he did that after I'd warned him there
was trouble that shouldn't just be ignored.


That was the shared state and history when this ctte bug was opened.
In the interests of seeing what new options we had _today_, I then did
an initial review of the most recent upstream state, which nobody else
had yet looked at, to see if anything material had changed which might
improve the situation.  My second post here included details of that,
but the short answer is "it got more complicated".

That one had removed the ability to run it from a secure system CGI
location at all, and changed the way that it's generated yet again.
It also made changes that guarantee everyone will need to fork it
for distro use, because it now hardcodes a fun mix of paths, like
/opt/local/bin for perl and /usr/local for global et al.

It does comment out that line from above with a note:
"To use this code, please uncomment in your own responsibility."
But it also outputs a .htaccess enabling execution in the directory
where the output is generated, whether you want to use it from there
or not (and adds a second CGI, and a bunch of jquery doing AJAX too).

It has a bunch of immediately obvious problems:

 - it still passes the unsanitised $pattern to global, which then
   passes it to popen (which again lets it be interpreted by the
   shell).

 - it won't run with perl's taint mode enabled, because that
   simply kills it warning of unsafe operations on untrusted data.

 - perl's warnings aren't enabled, and it complains of other
   rookie code problems if you do.

 - Enabling 'use strict' makes it scream with pain and completely
   fail to compile and run.


It would need more thorough auditing to confirm that there is(n't) an
exploitable path through that, but given that it ignores even the most
basic advice which perl has strongly recommended since perl4 was new
and shiny, then if there isn't, it would be because of luck more than
obvious care.

I'd not think it was a Good Plan to push this out as a "rush to beat the
freeze" upload of this source without a much more careful look at it,
and some effort spent actually fixing the already obvious deficiencies,
but the size of the truck you might drive through its holes is a bit
different to where we started from.

It might not be utterly unfixable, but you'd have to convince upstream
that security is important, or completely fork it for debian, which
brings us back to exactly where we are - unless we just remove it all.
But that would need Time, whoever does it.

It is what we now have enabled in the package that Wookey uploaded to
experimental.


>   However, it is possible to generate static content ...

Yes.

>   ... that achieves the
>   same aims, but at the cost of approximately doubling the disk usage,
>   and presumably also requiring more time to generate.

No.  This isn't just a time-space tradeoff for equivalent functionality
as Punit claimed.  You lose the ability to actually search the code,
which is the whole point of doing this in/with global in the first
place.

If you don't have that, you can get much better results by using
doxygen to do the formatting, even if you don't have any doxygen
specific markup in your source at all, and it will also give you
client-side searchability (which these days might well be better
than global's is anyway) ...

So the options for global6 really boil down to:

 - Do we try to fork/fix the (new) existing CGI, to make it at
   least vaguely secure, modulo not being able to actually
   install it to a secure system CGI location.

 - Do we try to fork/fix htags to remove that, leaving it with
   just (IMO) relatively useless (compared to other options)
   functionality to create static markup.

 - Do we just remove htags completely, and only ship the tagging
   part of global (for Stretch at least, until someone does fix
   the problems with it).


>   Also, for people that want personal access to htags there is a
>   htags-server command that brings up a dedicated server to run the CGI
>   as the invoking user, by default bound to a localhost port.

htags-server is a shell script that generates python or ruby code
to use the HTTP server functionality they can provide.

I have no idea offhand what the security properties of either of
those is is really like, but AIUI the python version at least
claims to run the CGIs as 'nobody' - but I'm not sure how that
works when run as a user unless something is SUID somewhere ...
or if that's patched to work differently in Debian to what the
python docs say it does.  If that's true it would mean you need
everything to be at least world readable to use this.

It still unconditionally drops a .htaccess file into the output
though, so if you're running a real web server, that may or may
not grant CGI execute access to that directory, depending on
how it's otherwise configured - whether you use htags-server for
this or not.


That script doesn't really change any of the things that we'd be
concerned about in practice.  It just means people who don't have
apache (or another web server) installed can install python or
ruby instead.  Either can limit it to localhost or allow access
on a public IP.


>   Version 6 fixes some bugs that are still present in your version 5
>   package and/or provides features that are missing, but bug reports of
>   sufficient quality to allow you to fix/backport to v5 are lacking.

There appears to be two main issues that we 'know' using v6 'fixes'.

One is whatever it is that the third-party ggtags wrapper needs, which
aiui is what Vincent and Punit are most annoyed about.  But I don't
use emacs, and ggtags isn't even in Debian - and they haven't even
told me what error they see, let alone what operation(s) trigger it.

Vincent just gave me the output of global's short --help, and said
"look, there's new options" - but we don't even know if it's actually
a 'missing' command line option that it fails on (or which one that
might be), or something else entirely.  My hunch is that one would
probably be pretty trivial to fix - either in ggtags or global, or
both - if someone who uses it engages with what I asked originally,
to file a separate bug from the 'new upstream' one, detailing the
actual problem, and is willing to test any proposed fixes even if
they aren't up to actually submitting a patch for that themselves.
I don't mind writing a patch if we know what actually needs patching.

Or if that really did prove to be intractable to backport, we'd have
a real data point for the question we're looking at here too.  But
right now, the odds are good that it's something quite minor, because
not that much has really changed.


The other one, Wookey has now given us a good report for as part of
his recent digging - and at first blush it appears to be 'trivial'
in the sense that it doesn't actually appear to be a bug in global's
code, but rather a hard limit in the version of flex that was used
to generate the scanner.  (I do consider it a bug in global that it
isn't regenerated at build time - upstream doesn't even provide
makefile rules to do it, they generate it 'manually' and bundle
the generated file in the source).  That one is only still open
because it's going to be an intrusive patch, even if it's 'easy',
and he wasn't keen to try regenerating it himself, and by that
point I wasn't sure if it was going to just be wasted effort, if
the consensus was going to be to kill off v5 right now.

I've got a bunch of things I want to get done that I do expect to
be released, but I can fit this one onto that priority list if we
do decide that keeping v5 for Stretch is the right choice.

I already uploaded a fix for the other good actionable report that
Wookey made while he was looking at all this.


> Are there any other regressions that you are aware of in v6?

In terms of the upstream code, the issues with htags are the main
'showstopper' I'm currently aware of.  But I haven't tested the
rest of it anywhere near exhaustively yet either.

In terms of the package currently in experimental, there's a bunch
of issues with it that would need to be fixed if we were going to
want it in Stretch.  Wookey already identified some.  And in the
quick look that I've already had at it so far, there's others
which were immediately obvious.  There's a bunch of upstream stuff
that is simply missing from it, the man page for htags-server, the
icons and javascript and css for htags (which it actually complains
loudly about when you try to use it, and makes me wonder how much
they did actually test that or pay attention to what it reported)
but that becomes moot if we go for the "remove htags completely"
option too.

There's little things like it still having .la files in it, and
static libs for things that are supposedly 'plugins'.  Which aren't
a big deal on their own to fix, but again suggests that if 'obvious'
things like that were missed, there's a good chance there will be
more issues than what I've already seen in a cursory review.



> Your suggestion, as I understand it, is that v6 should hit unstable
> after stretch's release, and that people who are currently complaining
> about bugs/missing-features in v5 (or are overly focused on numbers) can
> then grab v6 out of stretch-backports.

If we're going to say that our patience is up with waiting for sanity
to return to htags (and mine pretty much is), then I think that's the
most orderly way we can drain the swamp.  With the least risk of having
people step on the unexploded ordinance we'll expose, in ways that we
can't easily fix under stable release constraints.

I don't think it's the only option, but it looks like the least painful
overall for everyone that I can so far think of, which maximises our
options in the event of something unexpected (which ironically, are
black swans we probably should expect ;)


> Could you consider the relative merits of instead putting v6 into
> stretch now, and dealing with the people that are currently clinging to
> v5 by pointing out that:
> 
>   0) there are now other alternatives to htags that might suit them better.

I genuinely think that's actually the best advice we/I can give anyone,
regardless of what our "for the short term" choice ends up being.  At
least unless somebody suddenly starts making major improvements to htags
to do things that doxygen can't/doesn't already do on its own.

But I'm at that awkward age, where I'm too old to imagine everyone will
take my advice, but not so cranky to think that anyone who doesn't should
have abuse and scorn heaped upon them before being nuked from orbit ...
So I'm pretty sure someone will still want spacebar heating, whatever we
might say.  The difference is just how much warning we give them that we
are taking it away, as to whether we can feel that we were polite and
considerate enough about how we did that or not.

>   1) htags-server lets them run the CGI for local access.
> 
>   2) htags can generate static content, and thus safely provide general
>      access while avoiding the need for a CGI

I think I already covered those above?  (so I'll go back to trying to
be brief and let you ask if there's something I didn't :)

>   3) If there is anyone that cannot do either for some reason, they can
>      install global v5 from jessie, pin it to avoid upgrades, and report
>      the reasons why they had to do this to the BTS.

That is another option.  Though it seems a bit backward from what would
seem like 'normal advice', and means they will fall out of having
security support before Stretch is EOL.  It's basically equivalent to
saying "just maintain your own local build".  I'm not sure I'd really
want to *suggest* that to people, or get them in the habit of thinking
that it's an ok thing to do when upgrading Debian - if it wasn't an
option they figured out for themselves (and hopefully properly
understood the consequences of).


> Have I missed some vital aspect of this?

Knowing what other people don't know is one of the hardest questions
in the world :)  So I hope you'll ask if there's something I didn't
shed the sort of light you were looking for on.

But you're asking good questions, and exploring the same sort of options
that that I'd been mulling over, so I think we're at least in the same
universe, even if we don't quite see it the same.

Optimising for multiple variables is one of the other hardest things
in the world, so I don't see it as a failure of anybody if we don't
(ever) all completely agree on what would be 'best' here.  I'll be
happy if enough of us have at least thought about it hard enough to
show their answer is based on the actual variables at play here.
The center of gravity of those options should at least be a local
maxima, if not the ideal solution.


> Is there a compelling reason to favour the theoretical users that might
> want to stick with v5 over the actual users that we've been hearing ask
> for v6?

That kind of depends on what we decide to do about htags when we move
to v6 (now or later).

That there are users of htags is definitely not theoretical, there was
a new bug reported against it just yesterday - and unlike some of the
more strident voices in this bug log, it's from an actual user of
global, who I've never spoken to before, and wasn't astroturfed in as
'reinforcement'.

If we tell him, "we fixed your bug, by removing htags", I'm pretty
sure his first thought isn't going to be "thank you!".  If we tell
him, "we fixed your bug, and we removed the insecure CGI, but you
can still generate static markup that you can't search" - I don't
know what his answer will be, because he hasn't told us if he does
use that or not.  Similarly if we rush a 'blind' patch into Stretch
to remove that and accidentally break something it would be hard to
fix because there wasn't enough time spent on testing it all, we're
somewhere on the spectrum of Suboptimal.


> If the TC were to achieve consensus that v6 should be in stretch, will
> it be sufficient for us to inform you of that in order to make it so?
> 
> I gather from what you just wrote that it would be sufficient.

Yeah, if we've got enough people willing to put their neck on the block
to say "I think we're better off with v6-only in Stretch, rather than
both via the magic of -backports, and here's why the people who might
get burned by that, and the problems that can/will create, aren't
important enough to exercise any patience to minimise them", then I'd
be a fool not to jump at the chance to let them take the rap for
whatever backlash might occur :)

What I'd _like_ to see a good consensus on though, is a little more
than that - because I don't think "should we ship v6 in Stretch" is
the right question, or rather a _sufficient_ question.  Because if
the answer to that is "yes" - then we still have the question of
"what should we do with htags in v6", and that's actually the one
where things go all pear-shaped if we get it wrong.

And even if the answer to it is "no", then that question _still_
remains as "what should we do with htags in v6 for stretch+1" ...


So if we instead ask "what should we do with htags for Stretch
(and beyond)", then everything else pretty much follows on
automatically from that.

If we say it's important to keep it as fully functional as it is
now through the Stretch cycle, then really the only practical
option is to stay with v5 for that.  v6 has gutted too much of
that to realistically fix in the time we have (or perhaps ever
if upstream continues to fight that).

If we say it's ok to lose the search function of it, or to just
lose it altogether since the search was what made it useful -
then we have nothing to gain by staying with v5 and should ship
v6, and accept whatever fallout occurs from us doing that without
any notice.


Maybe that's what I somehow failed to communicate earlier when I
kept saying htags was the important question I wanted to see get
addressed.

Because people saying "it's irrelevant, uploading 'something newer'
is overwhelmingly more important" completely misses the point that
uploading something newer unavoidably involves someone answering
that question.

And ignoring the issues around it totally leads to fun paradoxes
like Odyx saying that one of the reasons it's important to have v6
is that it "fixes bugs like #590053" - when what the reporter of
that bug wanted fixed is _exactly_ what we would lose by going to
the htags from v6, and he was one of the people who also spent a
considerable amount of time trying to work with upstream to have
a secure system install of the CGI supported ...

Or like Ian's faster than light "no brainer", where if we'd followed
his advice, and made a snap decision without thinking to just let
Vincent upload what Punit had prepared, because Clearly they were
more capable than Evil Me of making an appropriate decision - then
we'd have had a package in stretch with a trivial remote shell
exploit in it ...

Sam's argument that "we don't care" might be justifiable given the
time that's elapsed and the direction upstream has trended in, but
he hasn't yet given any preference for whether we remove htags
entirely, or just try to amputate the CGI - which only gets us
halfway to _what_ we should upload if we go with v6.

I think Tollef and I are somewhere near to on the same page, in that
we think it would suck to burn existing users on short notice, but
agree we can't prioritise them forever, and are open about exactly
how we proceed from here.

I get the feeling that you are leaning toward preferring v6, but
you're asking good questions, so I am interested in how you weigh
all this up.


But I think if we can get a sense of people's preference for what
to do with htags, then we collapse the waveform of what logically
follows from that - and if we don't have as many opinions as we
have people, a reasonable consensus should be fairly quickly
reachable from there, with a self-evident plan of action.  And
faster than it has while each person has focussed on their own
personal subset of What Is Most Important here.


> If however you are likely to insist on resolutions to override the
> maintainer, or transfer the maintainership, I think you ought to be
> up-front about that in order to avoid any accusations of intentional
> time-wasting later on.

I'm pretty sure that if wasting time was my intention, there'd be
far more fun ways to do it than this!

I know people here love their FAOD clarifications, so it helps you
for us to be crystal clear about that, but it does make me sad that
people do project those sort of accusations onto others.  Especially
onto others who *are* being diligent about making good decisions.

I'm taking the time to give detailed answers to good questions
because that's how good decisions get made.  Anyone who thinks that
'threats' of "over my dead body" end with anything better than
dead bodies, still has a lot to learn about how the world works.

I'm pretty sure we can agree on some reasonable plan without the
need for a vote, or to invoke the constitution to force anything.
And I can guarantee you that on the remote chance that I'm wrong
about that, and do violently disagree with some insane consensus
that emerges - that I won't for a moment stand in the way of
letting someone else be the person whose face that blows up in.
If it comes to that, they'll have earned it!


> If you can keep your answers brief, you'll earn my gratitude.

Sorry :(  I'll have to find some other way to earn that, if proving
that your diligence in learning about this before deciding was well
founded doesn't do it.

  Best,
  Ron