Re: partial Debian mirroring - Was: dpkg rewrite in Java
"Tobias Hertkorn" <t.hertkorn@gmx.de> writes:
> Goswin von Brederlow wrote:
>> On Sun, 25 Apr 2004, Tobias Hertkorn wrote:
>>
>>> Hi community!
>>>
>>> I am currently starting to rewrite parts of the dpkg functionality
>>> in Java (For startes the dpkg --info part and a package list
>>> parser). I need this for my senior programming project
>>> apt-got, which is a partial mirror application that focuses
>>> as a subproject on Debian archives.
>>
>> What kind of partial mirroring?
>>
>> I ask because hopefully any partial mirroring should be
>> possible with debmirror2 and it would be wastefull to
>> implement yet another mirroring script.
>
> Hi Goswin,
>
> apt-got[1] is the first subproject of a modular mirroring application
> mainly written in Java (The apache module of the engine is written in
> C). It does not focus on Debian archives, but it was reasonable to
> start with a Debian repository module - because that is the part I
> need most for my home and the Universitaet Bayreuth. Modules for
> mirroring other kinds of repositories (e.g. ftp://www.kernel.org/pub/)
> are planned.
>
> I did some research before I started this project (Jan 2004) and I
> came across debmirror. But debmirror did, as far as I could tell,
> waste more overall bandwidth than save for my particular setup (10
> machines - client and server setups - different archs - mix of stable
> and testing, and some pinned unstable). This setup calls for a proxy
> solution, but all proxy scripts out there had some (major) flaws that
> I set out to solve in apt-got.
So its more an apt-proxy rewrite than debmirror. Thats quite a
different thing so I don't see much duplication with debmirror. But
you might want to talk to the apt-proxy and similar maintainer and
maybe unify your ideas into a signle deb.
> Apt-got will - in the first instance - be a repository proxy. That
> means, it seems as if all packages are available in the local
> repository, but packages are loaded from the remote repository on
> client request only. (And then locally stored to further increase
> delivery speed on later requests). But configurable pseudo-AI
> algorithms are planned (eg prefetch the newest version of packages
> that are frequently requested). My goal is to put as little stress on
> the remote repository as possible. That means for example using
> project/trace/ timestamps to predict possible updates, etc.
I use the Release file. The trace just says the mirror script did
run. The Release file says that and what has changed.
> I did not know about debmirror2 up to now. Will it be written in perl
Its written in ocaml. Its more memory efficient than perl, I hate perl
and I know a loot more ocaml than perl. It was also trivial to add a
caching db to debmirror in ocaml. No more lengthy/noisy `find /mirror`
at every run.
> as debmirror is now? Will it support (optinal) acting as proxy - that
Its core design is still to mirror and not proxy. But I'm planning a
nice interface to add filters to what is mirrored. One filter I plan
would monitor installed packages and only mirror those. You can then
put the local mirror and a full mirror in apt/sources.list and
debmirror starts mirroring everything you install and stops mirroring
things you remove.
Maybe I could add a proxy server that stores new files in the mirror
archive and adds them to debmirror in a filter. But thats for the future.
> means, populating it on the fly? How far along is it?
I have debmirror2 running at home but it only knows abour rsync for
now. Also one can't easily change the configuration without rebuilding
the database (which means md5suming all files). Once those two
problems are done I will replace debmirror with it.
> As a sidenote: apt-got is my Computer Science Capstone[2] - so I
> better have a reasonably stable and functional module by June 5. :-D
> After that date I am allowed to and want to accept co-developers. This
> will be fairly easy, as the project is already sourceforge based[3].
>
> Greets,
> T
>
> [1] http://hacktor.fs.uni-bayreuth.de/apt-got/
> [2] http://www.cs.ucsb.edu/~holl/CS189B/
> [3] http://sourceforge.net/projects/apt-got/
MfG
Goswin
Reply to: