We've wanted to wrap the second stage of the installation in a debconf
progress bar for a long time. As of a few months ago, all the
infrastructure in apt and debconf et al is there for this, so we just
need to port the base-config hackery I did for Ubuntu.
While talking about this with Joey this evening (edited IRC log
attached), we agreed that it would be useful to split out a component of
that hackery into a self-contained program that runs an apt-like
command, interprets its status output, and transforms it into a progress
bar. That program could then be used in base-config or tasksel or
wherever it ends up going.
This is a first draft (in the form of a man page) of a design for that
helper program, working name 'apt-debconf-progress'. For reference, it
will replace roughly the following shell code, although to make it
possible to implement a missing feature (media-change requests) within
the limits of my sanity it will probably be written in Perl instead. I
don't expect it to be an especially complicated program. It will
probably live in the d-i repository, partly on a "why not?" basis and
partly to take advantage of the existing translation infrastructure for
the couple of translated debconf templates it'll need.
# Wrap a subprocess such that it will use the debconf passthrough
# frontend.
passthrough () {
# The passthrough frontend doesn't yet know how to fetch its
# priority from the UI agent. Work around this.
db_get debconf/priority
priority="$RET"
DEBIAN_HAS_FRONTEND= DEBCONF_REDIR= \
DEBIAN_FRONTEND=passthrough DEBIAN_PRIORITY="$priority" \
DEBCONF_READFD=5 DEBCONF_WRITEFD=6 \
"$@" 5<&0 6>&3 3>&- </dev/null
}
# Wrap package installation in a progress bar. The nominated
# subprocess should output apt install-status information to fd 4,
# and should ensure to keep fds 5 and 6 open when invoking debconf
# (directly or indirectly), as those fds will be used for the
# debconf passthrough protocol.
progress () {
MIN="$1"
MAX="$2"
shift 2
(if passthrough "$@" 4>&1 >&2; then
:
else
# Propagate the exit status to the last process in
# the pipeline so that our caller can see it.
echo "_error:$?"
exit "$?"
fi | while IFS=: read -r status pkg percent description; do
# Crude waypointing. 15% was chosen to match
# base-installer, but could benefit from timing
# tests under various bandwidth conditions.
case $status in
dlstatus)
min=0
len=15
;;
pmstatus)
min=15
len=85
;;
_error)
exit "$pkg"
;;
*) continue ;;
esac
percent="${percent%%.*}"
percent="$(($percent * $len / 100 + $min))"
position="$(($percent * ($MAX - $MIN) / 100 + $MIN))"
db_progress SET "$position" <&4
db_subst base-config/progress/info DESCRIPTION \
"$description" <&4
db_progress INFO base-config/progress/info <&4
done) 4<&0
}
Cheers,
--
Colin Watson [cjwatson@debian.org]
* joeyh looks at the most recent version of kamion's pkgsel progress patch <Kamion> I still haven't put in the several solid hours of thinking it would take me to figure out how to port that patch to tasksel :-/ <joeyh> so, I'm trying to figure out where in d-i this code would best go. If we really do manage to get rid of base-config, leaving it there would be wrong. Maybe tasksel? <joeyh> except of course ubuntu doesn't use tasksel for some reason even though it should be a flexible enough format <Kamion> I wanted to use tasksel, but lost the argument; at the time tasksel wasn't as good <joeyh> s/format/framework/ <joeyh> hmm, I don't see any large problems to integrating it to tasksel? <Kamion> it might be worth porting Ubuntu over to use tasksel now, but as far as resource assignments go it qualifies as busy-work <Kamion> since what we have works ... <joeyh> sure <Kamion> it's just (a) figuring out exactly where the file descriptor and environment variable handling ought to go, (b) arranging for it not to be a flag-day where you have to have both new base-config and new tasksel simultaneously <Kamion> oh and also that patch doesn't handle apt media-check requests yet, and doing so in shell is actually really hard <Kamion> you end up trying to construct a circular pipe <joeyh> Kamion: seems to me that the current pkgsel script could be generalised to something that tasksel runs and passes a aptitude command line to and then it just does the debconf progress bar stuff in that script <Kamion> because you're reading from one of apt's output file descriptors and then in the handler for that you find yourself wanting to stick a newline into apt's stdin <joeyh> well, I'd probably rewrite it in perl too <Kamion> joeyh: oh, invert it? that would suck less ... <Kamion> it would be nice to have the possibility of tasksel-udeb ... <Kamion> the file descriptor handling would be much easier in either C or Perl than in shell, much though I love shell <Kamion> hmm, I suppose tasksel-udeb could ask the questions and then chroot to run pkgsel, which could then use perl <joeyh> yeah, tasksel udeb is hard because of 1. perl and 2. not having Packages info early <Kamion> could we generalise the retriever a bit for that? <joeyh> not much point to the udeb unless it can run before partman, imho <Kamion> agreed <Kamion> or before base-installer, anyway <joeyh> would be hard, there's a reason apt-setup runs late <Kamion> sure, it needs to run apt-get ... don't we already have most of the necessary logic in *-retriever though? <joeyh> well, maybe. I was pondering earlier about something that maybe just used the Release file to work out task info <Kamion> if we could run bits of apt-setup earlier to say what we want to have in sources.list, (then with tasksel-udeb have the retrievers look at that and fetch what they can), then apt-setup-verify after base-installer, it seems that that would work <Kamion> unless there are generators that really *have* to have the results of a previous apt-setup-verify <Kamion> I know security uses those results at the moment <joeyh> I did some changes to 50mirror to make it use verification to make sure that transient network problems don't result in a sources.list that doesn't have any useful sources <Kamion> so what about the other things pkgsel does? can it just do those before running the thing tasksel passes to it? <Kamion> having pkgsel control the progress bar is sort of useful because there are several pieces of it that need space allocated separately <joeyh> if you mean popcon etc, I'd be removing that <joeyh> unless hmm.. unless we want them progress barrted <Kamion> mdetect/read-edid, scrollkeeper-update, in the case of Ubuntu language packs, etc. <Kamion> I progress-barred everything, it's kind of nice I think <Kamion> one big progress bar for the second stage <Kamion> well, pkgsel anyway, but that's nearly all that's left of it :) <joeyh> if we keep that in, it wouldn't belong in tasksel <Kamion> so that's why I was thinking of running tasksel inside the progress() function or similar <joeyh> tasksel is now smart enough to run under debconf and limit itself to debconf protocol, except or the progress fd stuff <joeyh> but that would still leave two streams coming out of tsksel that would have to be combined <Kamion> for pkgsel-progress it would need to avoid closing fds 5 and 6 <Kamion> which probably involves a little horribleness in perl, like what debconf itself has to do <joeyh> how did you manage that with aptitude? <Kamion> same way as I manage everything else with aptitude, I got mvo to add a little tweak ... <Kamion> hang on 'til I find the reference <Kamion> I think it was in apt actually <joeyh> APT_OPTS='-o APT::Keep-Fds::=5 -o APT::Keep-Fds::=6' <Kamion> that's the one <joeyh> I'd wondered wtf that option was for <Kamion> # The passthrough frontend doesn't yet know how to fetch its <Kamion> # priority from the UI agent. Work around this. <Kamion> hmm, I should probably get round to fixing that in passthrough proper ... <joeyh> actually, tasksel invokes debconf via a shell scrpt, so I dunno if I could do that <Kamion> I was hoping all that could go away <Kamion> notice how my pkgsel does practically everything inside a confmodule, apart from a couple of debconf-copydbs at the top <Kamion> if you're progress-bar/passthroughing everything then it's safe to do all the package installations inside debconf <Kamion> rewriting progress() in Perl would probably be good enough to let us do media change requests, without having to rewrite all of pkgsel (most of which is fine in shell) <joeyh> hmm, you know, I'm still leaning toward preferring inverting it, perhaps with the caveat that there might need to be a way to make tasksel bahave like run-debootstrap and not bring up the progress bar itself <Kamion> I'm not entirely sure I understand how the result would look, although I'm not as familiar with tasksel as I ought to be <Kamion> how about a simple declarative kind of thing at the top that just does progress bar segment lengths and what to run in each bit, and the biggest bit would be running tasksel? <joeyh> well, tasksel would run and emit only debconf protocol output and would run this pkgsel derived thing that handled converting the apt progress <Kamion> which would then call back to something that did the aptitude progress logic <joeyh> really there's so reason we couldn't use the same thing outside tasksel.. might be nice to use it in base-installer for that long kernel package download <Kamion> personally I like the idea of splitting off the progress conversion logic into a separate script so that it can be used in multiple places, rather than having it be part of the thing which installs packages tasksel wants <joeyh> right, it would just step the bar from x to y <Kamion> for instance there's the X-preconfiguration stuff at the moment, and in Ubuntu I deliberately install language support stuff in a separate apt run so that, if language support fails, at least you get a working desktop install <Kamion> (apt being too willing to just give up and do nothing if you tell it to do lots of stuff at once) <joeyh> I haven't read that part of the code in detail, but don't see why not <joeyh> yeah, apt sucks, in $WORK we have a script that runs apt ~20 times for this reason ;-) <joeyh> Kamion: so, how's this for an interface: apt-debconf-progress start-waypoint end-waypoint command [args ...] <joeyh> command is required to take apt options, so apt or aptitude <Kamion> is exactly the same as the current progress function's interface, so fine by me :) <Kamion> well, not quite <Kamion> if it's required to take apt options, then apt-debconf-progress can give it the status-fd/keep-fds options itself, which is better <Kamion> then the same fd requirements, must avoid interfering with fds 4/5/6 <Kamion> apt-debconf-progress should be a debconf confmodule, right? <joeyh> hrm, probably <Kamion> then it needs a couple of templates - base-config/* or somewhere else? <joeyh> so you can $ apt-debconf-progress apt-get install myblah # look how pretty <joeyh> cool <joeyh> please let's not add templates to base-config at this point <Kamion> it needs to get at base-config/progress/info at the moment to fiddle with descriptions <Kamion> ok <joeyh> it's so anemic it can't stand up much longer <Kamion> you want this a separate package, then? <joeyh> perhaps "gutted" is a better word <joeyh> leaning that way. It's not very related to tasksel <Kamion> if it's *completely* standalone like that, it can't do waypoints in the same way ... <joeyh> or put it in apt :-P <Kamion> how about apt-debconf-progress --waypoints 5-95 command [args ...] <joeyh> I guess the waypoints would better be --start --end type things <joeyh> yeah <Kamion> ok, same idea <Kamion> then if you omit them it does PROGRESS START and STOP itself <joeyh> right. <Kamion> hmm, this is interesting if you want to have a multi-segment progress bar <joeyh> hmm, this will allow me to invert tasksel so it doesn't use that damned shell script for debconf <joeyh> I think <Kamion> you'd want to have apt-debconf-progress --start-bar/--end-bar so that it could control its own templates rather than callers having to know their names <Kamion> right, that's exactly the idea <Kamion> shall I knock up something in perl tomorrow then? <joeyh> that would be terrific <Kamion> ok for d-i svn? <joeyh> oh, I suppose <Kamion> might as well use the translation infrastructure * Kamion watches bubulle run away screaming at the idea of even more translations <joeyh> yeah <joeyh> eh, I thought bubulle was the one rocking to himself in the corner over there <Kamion> where would we put the specialised progress templates for things like the scrollkeeper-update run? <joeyh> you wouldn't want to use this for scrollkeeper-update would you? <Kamion> I do at the moment - it takes a while <Kamion> "Registering documentation, please wait..." <Kamion> you wouldn't want to run it inside apt-debconf-progress, indeed <joeyh> oh, I had missed that went through the apt interface <Kamion> but it seems to belong at the end of the same progress bar <joeyh> yes, exactly <Kamion> I think that one might have to go in base-config for now as long as it's calling scrollkeeper-update <joeyh> right, but it would use the same progress bar and do that just before taking it down <Kamion> yep * Kamion decides writing the man page first is the One True Way to avoid rewriting this thing over and over again <joeyh> yes <joeyh> you could even post it to -boot tonight, my brain should de-fry after dinner
.Dd November 29, 2005 .Os Debian .ds volume-operating-system Debian .Dt APT\-DEBCONF\-PROGRESS 8 .Sh NAME .Nm apt\-debconf\-progress .Nd install packages within a debconf progress bar .Sh SYNOPSIS .Nm Ar command Op Ar args ... .Nm .Fl Fl start .Nm .Fl Fl from Ar waypoint .Fl Fl to Ar waypoint .Ar command Op Ar args ... .Nm .Fl Fl end .Sh DESCRIPTION .Nm installs packages within a .Ic debconf progress bar. The given .Ar command should be any command-line .Ic apt frontend; specifically, it must send progress information to the file descriptor selected by the .Li APT::Status\-Fd configuration option, and must keep the file descriptors nominated by the .Li APT::Keep\-Fds configuration option open when invoking .Ic debconf (directly or indirectly), as those file descriptors will be used for the .Ic debconf passthrough protocol. .Pp The .Fl Fl start , .Fl Fl end , .Fl Fl from , and .Fl Fl to options may be used to create a progress bar with multiple segments for different stages of installation, provided that the caller is a .Ic debconf confmodule. Use .Fl Fl start to start up the progress bar, .Fl Fl from and .Fl Fl to to install packages with their progress bar bounded by the given beginning and ending .Dq waypoints , and .Fl Fl stop to stop the progress bar when installation is complete. The caller may interact with the progress bar itself using the .Ic debconf protocol if it so desires. .Sh EXAMPLES Install the GNOME desktop within a progress bar: .Pp .Dl $ apt\-debconf\-progress apt\-get install gnome .Pp Install the GNOME, KDE, and XFCE desktops within a single progress bar, allocating 45% of the space for each of GNOME and KDE and the remaining 10% for XFCE: .Pp .Bd -literal -offset indent \&. /usr/share/debconf/confmodule apt\-debconf\-progress \-\-start apt\-debconf\-progress \-\-from 0 \-\-to 45 apt\-get install gnome apt\-debconf\-progress \-\-from 45 \-\-to 90 apt\-get install kde apt\-debconf\-progress \-\-from 90 \-\-to 100 apt\-get install xfce4 apt\-debconf\-progress \-\-end .Ed .Sh AUTHORS .An "Colin Watson" Aq cjwatson@debian.org .An "Joey Hess" Aq joeyh@debian.org
Attachment:
signature.asc
Description: Digital signature