Re: Bug#50284: a better apt configuration method

[Joey Hess <joeyh@debian.org>]

I've attached current version of the program and templates file.

Jason Gunthorpe wrote:
> > I disagree. I don't think people in australia are best served by downloading
> > from http.us.debian.org.
> 
> The internet is not geographically routed.

I'm well aware of how network routing works, thanks.

> You cannot use Geography to
> determine the best mirror for a location. In general many of the non-us
> sites don't have large local peering arragements like you see in the US -
> for instance until recently in Australia two of the big networks did not
> locally peer, if you wanted to get to one from the other you needed to go
> to the US and back again! 
> 
> Heck, even in Canada it used to be like that 4 years ago, to go down the
> street my packets went through Vancouver and Toronto!
> 
> In the majority of cases blindly picking a site by country will be worse
> than using the default US site, particularly on a modem. You really need
> to use something like netselect to make this feature truely usefull.

Unfortunatly, netselect isn't always going to be available. I'd love to use
it.

Grouping the mirror list by country is still useful despite what you've said.
If you're in Canada and know the US is going to be faster, then you select
the US off the list. If you're in the US and know connectivity to australia
sucks, you know the one item _not_ to choose is Austrialia. In either case,
you've made a more informed decision than picking one item at random off a
long list of choices. 

I did make the US the default of course.

It's worth noting that the debian mirrors web page
(http://www.debian.org/misc/README.mirrors) is ordered by country, and
indeed I think that's why you include the country data in Mirrors.masterlist
in the first place, right?

I can tune the description to be something like:

Description: Select a country 
 Select the country the mirror should be in. The goal is to find a mirror that
 is close to you on the network, and be aware that near countries, or even
 your own, may not be the best choice.
 . 
 If you want to enter mirror information manually, select the last item on  
 the list.

> Because apt-get won't work? If apt-cdrom doesn't run correctly with out
> any options then apt-get has no hope of having normal operation. APT has
> to be able to mount discs on its own!

I'll talk to the boot-floppies folks and make sure new cd installs come with
a fstab line for /cdrom. Unless I'm missing something, this will make
apt-get work.

The reason I'm using --no-mount is because I don't want apt-cdrom to prompt
for the cdrom to be inserted, since my own prompts already ask that that be
done, and so the autoprobing can happen without user interaction.

> > > The way you do this is an extremely bad scheme.. If a pre-existing entry
> > > is down then all new entries you try to add will mysteriosly fail!
> 
> > That's true. Hardly mysteriosly though, since you get the error message.
> 
> Since you can't seem to remove sites with your gui this kinda makes it
> impossible to add new sites once one is dead.

True, removal would be a nice thing to have. I'd be happy to add it in the
future.

> > Wasn't aware of it. I suppose I can use it to get the location of apt.conf,
> > that's the only useful information I see it providing for this program.
> 
> You need to use it to get the location of sources.list, the 'apt-get'
> binary and the 'apt-cdrom' binary at least.

Ah yes I meant sources.list and I'm querying apt-config for that now.

Apt-cdrom dump doesn't seem to output a location of apt-get or apt-cdrom,
and besides, I don't care where they are as long as they're in the path.

> > > Your mime scheme didn't preserve filenames so I called one aptconf and the
> > > other aptconf.templates and threw them in /tmp for testing..
> 
> > As I said (I think), you need to have debconf 0.2.33 or above.
> 
> I'm using 0.2.34

Oh, I read too fast. The filenames (which mutt *does* preserve in its mime
messages, I wonder if the BTS screwed it up?) were setup and
setup.templates.

> > > It runs, but
> > > I think it screwed up around 5 times just while I was futzing around, I
> > > never did get it to load the mirror list [yes I changed the location in
> > > the script] and it insisted in writing invalid urls! 
> > 
> > Can you provide details?
>  
> Not loading the mirror list was a big one

That is truely strange. Where did you put it, and how did you modifiy the
script and did it give error messages to stderr about it?

> it tried to mount the cd rom a
> few times even though I never selected cdrom

Yes, it probes for cdrom on startup -- the boot-floppies people want that to
smooth the setup process for people who have just installed from cdrom. That
should be turned off when it's being run from dpkg (done).

> it actually didn't seem to
> notice when APT failed to update a site since it didn't give any message
> just went back to the main screen

if ! apt-get update 2>$tempfile; then
	[ show tempfile ]

I don't see how this could fail to notices apt's return code. Do you? Are
you sure apt failed?

> you know the sizes of some of your
> dialogs randomly change sometimes when you visit?

Randomly? Debconf sizes the dialog to the text that is inside it.

> If you hit 'cancel' in
> some of the dialogs it exits instead of logically returning to the start,

Which is logical is debatable..

> probably a few more I forget. 

Well you said it "insisted in writing invalid urls", I still don't have
details of that.

-- 
see shy jo

#!/bin/sh -e
# Apt configurator.
# GPL 199 by Joey Hess

# Options passed from dselect
VARDIR=$1
GROUP=$2
METHOD=$3

# Pass something as parameter 4 to enable cd autoprobing on startup.
CDPROBE=$4

# Where is the mirrors master list?
#MIRRORLIST=/home/joey/debian/webwml/english/mirror/Mirrors.masterlist
MIRRORLIST=$VARDIR/methods/$GROUP/Mirrors.masterlist

. /usr/share/debconf/confmodule

# Set APTETC to the directory apt's config files are in.
eval `apt-config shell APTETC Dir::Etc`

# Pass in a URI type; this function returns a list of countries
# that have mirrors in them that match the type.
country_list () {
	perl -ne '
		BEGIN { $/="\n\n"; $type=shift }
		if (/Archive-$type:/) {
			($c)=/Country: (.*?)\s*\n/;
			$countries{$c}=1 if $c;
		}
		END { print join(", ", sort(keys %countries)) }
	' $1 $MIRRORLIST
}

# Pass in a URI type and a country; this function returns a 
# list of mirrors of the correct type in the country.
# The list is ordered with push mirrors at the top.
# Other orderings can be added here.
mirror_list () {
	perl -ne '
		BEGIN { $/="\n\n"; $type=shift; $country=shift }
		if (/Archive-$type:/ && /Country: $country\s*\n/) {
			($s)=/Site: (.*?)\n/;
			($t)=/Type: (.*?)\n/;
			$rating=0;
			$rating=1 if $t=~/push/i;
			$rating=2 if $t=~/push-primary/i;
			$mirrors{$s}=$rating;
		}
		END { print join(", ",
			sort { $mirrors{$b} <=> $mirrors{$a} }
			(keys %mirrors)) }
	' $1 "$2" $MIRRORLIST
}

# Pass in the URI type and the hostname of a mirror; this returns the
# directory the debian mirror is located in on that host.
mirror_dir () {
	perl -ne '
		BEGIN { $/="\n\n"; $type=shift; $mirror=shift }
		if (/Archive-$type: (.*)\n/) {
			print $1;
			exit;
		}
	' $1 $2 $MIRRORLIST
}

# After mounting a cd, call this function to scan
# it with apt-cdrom, which also adds it to the sources.list.
scan_cd () {
	clear
	echo "Scanning CD, this will take a minute."
	if apt-cdrom add --no-mount; then
		return 0
	else
		# Apt didn't like the cdrom for some reason.
		db_fset aptconf/cd/bad isdefault true
		db_input critical aptconf/cd/bad || true
		db_go
		return 1
	fi
}

# This function should be called only after one cd has been successfully
# scanned. It prompts the user if there are more cd's to scan and
# scans them. Pass in the device that is known to be the cd drive.
handle_rest_cds () {
	LOOP=1
	while [ "$LOOP" ]; do
		# Make sure the cd is unmounted, we may be prompting
		# them to change cd's.
		umount /cdrom 2>/dev/null || true

		db_set aptconf/cd/another false
		db_fset aptconf/cd/another isdefault true
		db_input medium aptconf/cd/another || true
		db_go
		db_get aptconf/cd/another
		if [ "$RET" = true ]; then
			while ! mount $1 /cdrom -o ro -t iso9660
			do
				db_fset aptconf/cd/dev isdefault true
				db_input critical aptconf/cd/dev || true
				db_go
				db_get aptconf/cd/dev
				CDDEV="$RET"
			done
			scan_cd
		else
			LOOP=''
		fi
	done
}

# This function will ask the user if they want to add another
# apt source. If so, it returns true.
add_another () {
	# Now ask them if they want to add another entry, 
	# and if so, restart.
	db_set aptconf/another false
	db_fset aptconf/another isdefault true
	db_input medium aptconf/another || true
	db_go
	db_get aptconf/another
	if [ "$RET" = true ]; then
		# Before looping, make a new backup.
		cp -f $APTETC/sources.list $APTETC/sources.list.bak
		return 0
	fi
	return 1
}

# Make sure the file exists, and back it up.
touch $APTETC/sources.list
cp -f $APTETC/sources.list $APTETC/sources.list.bak

# Establish the preliminaries.
db_version 2.0
#db_capb 'backup'
db_title 'Apt Configuration'

MAINLOOP=1

# First, try to do cdrom autodetection. This is so in the most common case,
# you don't have to configure anything at all. Note that /dev/cdrom is made
# by the install process if you used the cd to install.

# If /dev/cdrom exists, use it.
if [ -e /dev/cdrom ]; then
	CDDEV=/dev/cdrom
	db_set aptconf/cd/dev "$CDDEV"
fi
# TODO: other probing here.

if [ "$CDPROBE" -a "$CDDEV" ]; then
	umount /cdrom 2>/dev/null || true
	# Try mounting the detected cd rom.
	if [ "$CDDEV" ] && mount $CDDEV /cdrom -o ro -t iso9660 && scan_cd; then
		handle_rest_cds $CDDEV
		MAINLOOP=''
	else
		# Unable to mount it. Make sure that cdrom is the default, and
		# just go on to ask them where they want apt to install from.
		db_set aptconf/uri_type "cdrom"
		db_subst aptconf/uri_type note "You probably used a CD to install the Debian base system, but it is not currently in the drive. You should probably just insert it and select \"cdrom\"."
	fi
fi

while [ "$MAINLOOP" ] || add_another; do
	MAINLOOP=''

	# Ask what source apt should install from.
	db_fset aptconf/uri_type isdefault true
	db_input critical aptconf/uri_type || true
	db_go
	db_get aptconf/uri_type
	URI="$RET"
	# Clear any note that is on the uri_type, now that they've seen it.
	db_subst aptconf/uri_type note ""

	# If they chose to use CD, there is little point in asking
	# these questions, since the CD's they insert will answer them for us.
	# Same goes if they are entering manually.
	if [ "$URI" != "cdrom" -a "$URI" != "manual entry" ]; then
		db_beginblock
		db_fset aptconf/distribution isdefault true
		db_input low aptconf/distribution || true
		db_fset aptconf/non-free isdefault true
		db_input medium aptconf/non-free || true
		db_endblock
		db_go

		# If they choose to use non-free, ask about contrib.
		# Doesn't seem to make much sense to even ask, otherwise.
		db_get aptconf/non-free
		if [ "$RET" = true ]; then
			db_fset aptconf/contrib isdefault true
			db_input medium aptconf/contrib || true
			db_go
		fi
	fi

	case "$URI" in
	ftp|http)
		# Ask them which country they're in.
		db_subst aptconf/country countries "`country_list $URI`"
		db_fset aptconf/country isdefault true
		db_input critical aptconf/country || true
		db_go
		
		db_get aptconf/country
		if [ "$RET" != "enter information manually" ]; then
			# Now prompt with the mirrors in the selected country.
			db_subst aptconf/mirror mirrors "`mirror_list $URI \"$RET\"`"
			db_fset aptconf/mirror isdefault true
			db_input critical aptconf/mirror || true
			db_go
			
			# Now shove the data about the mirror into some other
			# questions.
			db_get aptconf/mirror
			MIRROR="$RET"
			db_set aptconf/hostname $MIRROR
			db_set aptconf/directory "`mirror_dir $URI $MIRROR`"
		else
			# They elected to enter info manually.
			# Prompt for hostname and directory the mirror is in.
			db_beginblock
			db_fset aptconf/hostname isdefault true
			db_input critical aptconf/hostname || true
			db_fset aptconf/directory isdefault true
			db_input critical aptconf/directory || true
			db_endblock
			db_go
		fi
	;;
	cdrom)
		# We've already probed earlier to try to figure out the
		# cd device.
		LOOP=1
		while [ "$LOOP" ]; do
			# Make sure the cd is unmounted, we may be prompting
			# them to change cd's.
			umount /cdrom 2>/dev/null || true
		
			# Prompt for the cd device if it wasn't autodetected.
			# TODO: We could give them a list of likely devices..
			#       This is unfriendly right now.
			if [ -z "$CDDEV" ]; then
				db_input critical aptconf/cd/dev || true
				db_go
				db_get aptconf/cd/dev
				CDDEV="$RET"
			fi
			
			# Now try to mount the cdrom. If this fails, we loop,
			# and prompt again.
			if mount $CDDEV /cdrom -o ro -t iso9660; then
				scan_cd && LOOP=""
			else
				db_fset aptconf/cd/dev isdefault true
				CDDEV=""
			fi
		done
		
		handle_rest_cds $CDDEV
	;;
	
	filesystem|nfs)
		# nfs and filesystem are the same, really -- they need to
		# have a mirror already mounted somewhere.
		URI=file
		LOOP=1
		while [ "$LOOP" ]; do
			db_fset aptconf/directory isdefault true
			db_input critical aptconf/directory || true
			db_go
			db_get aptconf/directory
			
			# A very simple mirror sanity check.
			if [ ! -d "$RET/dists" ]; then
				db_fset aptconf/baddir isdefault true
				db_input critical aptconf/baddir || true
				db_go
			else
				LOOP=''
			fi
		done
	;;
	"manual entry")
		# Prompt them for the line to add.
		db_fset aptconf/wholeline isdefault true 
		db_input critical aptconf/wholeline || true
		db_go
	esac

	# Generate the sources.list line if we haven't already.
	if [ "$URI" != cdrom ]; then
		if [ "$URI" != "manual entry" ]; then
			db_get aptconf/hostname
			HOST="$RET"
			db_get aptconf/directory
			DIR="$RET"
			db_get aptconf/distribution
			DIST="$RET"
			db_get aptconf/non-free
			if [ "$RET" = true ]; then
				NONFREE="non-free"
			fi
			db_get aptconf/contrib
			if [ "$RET" = true ]; then
				CONTRIB="contrib"
			fi
		
			if [ "$URI" = ftp -o "$URI" = http ]; then
				SEP=//
			fi
	
			echo "deb $URI:$SEP$HOST$DIR $DIST main $NONFREE $CONTRIB" >> $APTETC/sources.list
			echo "deb-src $URI:$SEP$HOST$DIR $DIST main $NONFREE $CONTRIB" >> $APTETC/sources.list
		else
			db_get aptconf/wholeline
			echo $RET >> $APTETC/sources.list
			echo 
		fi

		# As a final sanity check, run apt-get update, and catch the
		# return code and errors. This is unnecessary (I think)
		# if we just added a cdrom, since apt-cdrom does enough checks.
		tempfile=`tempfile`
		clear
		echo "Testing apt sources ..."
		if ! apt-get update 2>$tempfile; then
			# Show the user the error message, remove the bad
			# entry, and loop.
			db_subst aptconf/badsource apt_error `tr '\n' ' ' < $tempfile`
			db_fset aptconf/badsource isdefault true
			db_input critical aptconf/badsource || true
			db_go
			rm -f $tempfile
			cp $APTETC/sources.list.bak $APTETC/sources.list
			MAINLOOP=1
		fi
	fi
done

Template: aptconf/uri_type
Type: select
Choices: cdrom, http, ftp, nfs, filesystem, manual entry
Default: cdrom
Description: Choose what method apt will use to access the Debian archive
 Apt can access the Debian archive in a variety of ways. Choose the access
 method apt should use. For example if you have a Debian cd, select "cdrom",
 while if you plan to install via a Debian mirror, choose "ftp" or "http".
 .
 ${note}

Template: aptconf/country
Type: select
Choices: ${countries}, enter information manually
Default: US
Description: Select a country
 Select the country the mirror should be in. The goal is to find a mirror that
 is close to you on the network, and be aware that near countries, or even 
 your own, may not be the best choice.  
 .   
 If you want to enter mirror information manually, select the last item on
 the list.

Template: aptconf/mirror
Type: select
Choices: ${mirrors}
Description: Choose the Debian mirror to use
 Select the Debian mirror apt should use. You should select a mirror that
 is close to you on the net.

Template: aptconf/hostname
Type: string
Description: Enter mirror hostname
 Enter the hostname of the Debian mirror you want to use.

Template: aptconf/directory
Type: string
Default: /debian
Description: Enter mirror directory
 Enter the directory the Debian mirror is located in.

Template: aptconf/baddir
Type: note
Description: Mirror not found.
 The directory you entered does not contain a subdirectory named "dists",
 so it doesn't seem to be a Debian mirror.

Template: aptconf/cd/dev
Type: string
Description: Enter CD ROM device file
 This program was unable to auto-detect a CD ROM drive, or there is no CD
 in the drive. Enter the device file to use to access your CD ROM, and place
 a Debian CD in the drive.

Template: aptconf/cd/bad
Type: note
Description: Bad CD
 Your CD was detected, but it does not seem to be a Debian CD.

Template: aptconf/cd/another
Type: boolean
Default: false
Description: Scan another CD?
 If you have another Debian CD (for example, the second in a two CD set),
 you should scan it now.

Template: aptconf/non-free
Type: boolean
Default: false
Description: Use non-free software?
 Debian makes available some non-free software. though this software is not
 part of Debian proper, it can be installed with apt. This software has
 limitations on its distribution, modification, and/or use. Do you wish to
 use this software?

Template: aptconf/contrib
Type: boolean
Default: false
Description: Use contrib software?
 Debian includes some software that, while free, does not work without other,
 non-free software. Do you wish to use this software?

Template: aptconf/distribution
Type: select
Choices: stable, unstable
Default: stable
Description: Select Debian distribution to use
 Debian comes in two flavors. Stable is well-tested and rarely changes.
 Unstable is untested and frequently changing. Which of the two do you wish
 to install?

Template: aptconf/badsource
Type: note
Description: Failed to access the Debian archive
 When I tried to access the debian archive using the information you
 provided, apt gave the following error. I will run through the
 questions again, try to correct the error.
 .
 ${apt_error}

Template: aptconf/another
Type: boolean
Default: false
Description: Add another apt source?
 Apt can optionally install from multiple Debian archives. You have just
 successfully added a source to apt, and if you like, you can add another
 one.

Template: aptconf/wholeline
Type: string
Description: Manually enter the apt source line
 Manually enter here the line you wish to add to apt's sources list.
 If you received a line from someone else, you should just be able to paste
 it in here.
 .
 See the sources.list(5) man page for details. The format looks something
 like:
 .
 deb uri distribution [component1] [componenent2] [...]