[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Progress on reprepro frontend

On Sunday 21 June 2009 03:33:33 Goswin von Brederlow wrote:
> > Overall, I think that reprepro does a good job of maintaining a local
> > repository, and we shouldn't reimplement what it does.  Reprepro also
> > seems flexible enough to implement most of the backend with simple
> > commands and options.  I've never tried to implement a new apt-method
> > before, so I think that would take a bit more research from me.
> I totally agree that reprepro as the cache/storage backend would be
> great use of existing software.
> The problem I have with it being an apt method is that the apt method
> runs on a different host than the reprepro. That would require ssh
> logins from all participating clients or something to alter the
> reprepro filter.
> MfG
>         Goswin

I've started on writing the reprepro frontend part of the program.  The 
frontend isn't really complete, but I think it's a pretty good start.

I decided to make a system user "repserve" that will control reprepro.  This 
makes it easier to generate and use a gpg key with an unencrypted private 
key.  Since the key should only be readable by the repserve user, and since 
the application is designed to make personal, partial mirrors, I think that 
this strategy should be sufficient for how it will be mainly used.  The 
repserve user's home is at /var/lib/repserve

Running "repserve intialize" should create the gpg key, export the key to 
/var/www/repserve/archive.gpg, and create the initial repserve.conf file.  I 
don't know what to do about gathering entropy so that the key can be made 
more automatically.

Instead of directly configuring reprepro,  I've put the main configuration in 
a python config file (read by ConfigParser).  I've made code that parses this 
config file (~/repserve.conf), and generates the reprepro configuration from 
this.  This should make it easier to generate some of the more commonly used 
configurations for reprepro.

I've made it so that the configuration can be filled from the contents of a 
sources.list file.  Every 'deb' line uses dpkg --print-architecture to 
determine the arch to use.  The deb-src lines use the "source" arch in 
reprepro.  There is an --arch option that will let you specify the arch to 
use for the 'deb' entries, so it can be used like so:

repserve addsources /etc/apt/sources.list
repserve --arch=i386 addsources /etc/apt/sources.list

Sources can also be fed from stdin:
cat /etc/apt/sources.list | repserve --arch=c64 addsources


cat /etc/apt/sources.list | ssh repserve@mirror repserve addsources

I have made a simple function that tries to guess the name of the repository 
from the method.  So a method like http://security.debian.org/ gets the 
name "security", etc.  This function doesn't work all that well yet, at it 
doesn't try to look at the names of the official mirrors to figure out if 
it's a debian mirror, and use the name "debian" for those.  If a name isn't 
guessed, it tries to use "repos1", "repos2", etc.  Adding additional sources 
will use the same repository name for each method already contained in the 
config file, so once you set a name, it should stay set.  Changing the 
upstream mirror in the sources.list may cause an extra repository to be made 
if the mirror isn't identified in the guessing function.

For each source in the sources.list, the Release file is retrieved and parsed.  
From the Release file the extra options such as Origin, Version, etc. are 
used.  This make a better reprepro configuration without having to manually 
fill out all those fields.  The release files are currently being retrieved 
using urllib2, but should be using python-apt.  I haven't had time to mess 
with this yet, as I wanted to get other parts working first.

The reprepro configuration isn't created automatically after adding sources, 
in case some of those repository names need to be changed.  The reprepro 
configuration is created by:

repserve reconfigure

Each unique url in the sources list defines a separate repository.  Each 
section in the repserve.conf file corresponds to a repository and the dist 
(codename).  Each repository is split between the basedir and outdir, which 
makes it easier to use the outdirs in apache, or maybe ftpd (the default 
outdir parent is /var/www/repserve).  The basedirs are located 
in /var/lib/repserve/repos-db/ .

I have started making the bare minimum code to help manage filterlists.  Since 
it hasn't really been decided how those lists are to be generated, and which 
repository is going to use which filterlist, I'm somewhat stuck here.  I've 
tried to keep things flexible, so once something is decided, it should be 
relatively easy to implement.

Reprepro isn't really being used yet.  Only the configuration is being managed 
so far, which has been the most difficult part.  There is some code that 
handles running reprepro, but it hasn't really been used yet.  Only update 
and export are handled now, but it shouldn't be too difficult get this part 
going.  I have been more concerned with getting reprepro configured in a way 
that makes it easy to use as a backend with a simple frontend configuration.

I'm not really happy with the name "repserve", but I picked it out of the air, 
because I needed to start with something.  I would like to use another name, 
but I can't think of one that will work.  I'm open to suggestion here.  I'm 
also open to suggestion concerning anything that I've written above, although 
some suggestions should be accompanied by a patch or example.  I would really 
like to gather suggestions on how to name the repositories.  I think that my 
guessing function is a good start, but it could use a lot of improvement.  
When the guessing gets good enough, the function could raise a warning when a 
name couldn't be guessed, so the user can then edit the repserve.conf to fix 
the problem.

All the code is here:

I would like to move the code to it's own project space, but I need to name it 
something before that happens.

I have been playing around with germinate a bit, in case we want to make a 
short list of manually selected packages, and use germinate to resolve the 
dependencies and create the filterlists.  I don't expect this part to be 
working properly anytime soon.

Joseph Rawson

Attachment: signature.asc
Description: This is a digitally signed message part.

Reply to: