Re: Dependency-based running of collection scripts
Russ Allbery wrote:
> Sorry it took me so long to reply to this. I see that all sorts of things
> went into my to-reply list right before the conference I was running last
> year heated up, and then I never got back to them.
Np. The reply to this thread actually comes to me as a surprise as most of
the things I mentioned are already implemented or changed.
<h9er7b$nse$1@ger.gmane.org> is the most up-to-date post and thread.
> What's making me nervous is that I think the solution is pretty complex
> for what's needed for the dependency management, and I tend towards the
> "minimum possible required code" approach that was popularized by the XP
> methodology.
>
> But that being said, I really don't want to block you from continuing over
> that, particularly since I'm not replying very quickly. And you're of
> course right: you have already tested code that you know works. So I
> withdraw my objection here, although I still have a preference for doing
> the simplest solution that will work.
Maybe once you take a look at the new, cleaner, code you will see why it is
needed (or me see how it is _not_ needed if you provide the necessary
changes to remove it :).
>
> Well, currently we're running check scripts in the main process, so if
> we're running the check script, by necessity we're not waiting. We have
> to serialize the check scripts with the current architecture. So we'd run
> all the check scripts we can, then go back and see if anything finished.
> If nothing finished, then we wait until a child process exits, at which
> point we go do whatever's been freed up.
>
> One really doesn't want to poll children, since it's a busy-wait loop.
> One wants to have the kernel suspend the parent process until a child
> process finishes, which is what wait will do for you.
Ack., already mentioned on the new thread.
>
> Abstracting this work in Lintian::Command sounds good, but it's going to
> be tricky to fit that into the existing interface. Lintian::Command
> currently views the world one command at a time. To provide an API for
> this, we need something that manages multiple running processes and can
> provide an additional layer that you can query for the list of ones that
> finished or to wait for one to finish.
>
> The only point I was making is that, underneath that implementation,
> running these commands should probably use something other than IPC::Run,
> since IPC::Run doesn't give you the PID and therefore means that one can't
> use wait in its most natural form. One has to instead poll after each
> child exits to see which specific child exited, when an interface that
> gave you the PID, such as a normal fork, would let you immediately look up
> which child exited.
Agreed. The interface is what has made me delay those changes, I need to
reconsider the interface of Lintian::Command and probably split it.
> Raphael Geissert writes:
>> Summarising:
[...]
>> Comments?
>
> Sounds great -- please go ahead, and I'm happy to review code. It might
> be easier to do the Lintian::Command refactoring separately and get that
> in before doing the frontend restructuring, since we can use the
> management of multiple processes to clean up other places, such as in some
> of the collect scripts that currently background several things.
Okay.
>
> Incidentally, while you're working on this, if you feel inspired, Checker
> should become Lintian::Checker or something similar, and as much of this
> code from the frontend should move into a library as possible.
This should probably be discussed on the other thread (I can't find it with
a quick subject search, though) where I proposed making all checks inherit
a new Lintian::Check class that would allow splitting the checks code into
automagically-run check_* methods.
> My very
> long-term goal is that I'd like to get the frontend/lintian script down to
> just command-line parsing and library setup, and glueing things together,
> and get all the core work into libraries as much as possible.
Yes, that would be great.
The current state of frontend/lintian doesn't help to make other changes
there.
> I also want
> to start installing Lintian's Perl libraries into the regular Perl search
> path. That way, one could embed Lintian in any other Perl application as
> a module, which opens some interesting use cases.
>
That'd be interesting.
Cheers,
--
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net
Reply to: