Re: RFC: implementation of package pools
James Troup wrote:
> For the last month and a half, I've been working on re-implementing
> dinstall and switching to package pools. Here are the details.
> Comments would be appreciated, _but_ I'm really not looking for
> "Wouldn't it be nice if" or "this <small detail> is broken" type
> comments. One of the points of re-implementing dinstall is so that I
> can work on it, fix bugs etc., but that's for the future, my concern
> right now is with nothing other than getting it up and running in
> place of the current system, as soon as possible.
Okay, I've just done one of the things you were not looking for. I'll
try to provide the type of comments you've been looking for now :)
> Mirrors:
>
> Obviously some pretty big changes :->. People wanting to mirror
> single architectures will obviously have to do some excluding. This
> is trivial on just about any mirror program/script I've even seen,
> so I don't see this as a problem. If someone enterprising person
> wants to make a nice easy HOWTO document, that would be lovely. We
> lose the ability to mirror a single distribution but it's
> questionable whether this was trivially possible in the first place.
> I don't see a way round this that still implements true package
> pools and I think it's an acceptable trade off for the benefits that
> package pools gives us.
>
In order to avoid any further misunderstanding: This *is* an implementation
of package pools. But you don't see a way around this.
Can I suggest one? A custom debian mirror tool for archives, or/and
a system that implements symlink-farms. I believe this to have
been suggested at least once (I did ;)
> The package pool will be in
>
> /org/<hostname>.debian.org/ftp/pool/
>
why /ftp directory? sure it's not hardcoded? where do you keep
rw arch independent [right? ;)] data according to FHS? according
to GNU coding standards, that's $(prefix)/com but I guess we
don't have that in Debian. [sorry if i'm missing some very basic
point here.]
> The pool is split by <component>/<first letter of source name>/<source name>/.
> All source and binaries for all distributions go into the same
> pool directory. e.g.:
>
> [..]/pool/main/b/bzip2/
> [..]/pool/main/b/bzip2/bzip2_1.0.1-2.diff.gz
> [..]/pool/main/b/bzip2/bzip2_1.0.1-2.dsc
> [..]/pool/main/b/bzip2/bzip2_1.0.1-2_alpha.deb
> [..]/pool/main/b/bzip2/bzip2_1.0.1-2_i386.deb
> [..]/pool/main/b/bzip2/libbz2_1.0.1-2_alpha.deb
> [..]/pool/main/b/bzip2/libbz2_1.0.1-2_i386.deb
> [..]/pool/main/b/bzip2/libbz2-dev_1.0.1-2_alpha.deb
> [..]/pool/main/b/bzip2/libbz2-dev_1.0.1-2_i386.deb
> [...]
>
> An exception to this rule is library packages which "lib" packages
> which are split into <component>/<first 4 letters of source name>/<source name>. e.g:
>
> [..]/pool/main/libm/libmng/
> [...]
>
Here's a justification of this: (from an up-to-date packages list)
orion:packages$ for x in a b c d e f g h i j k l m n o p q r s t u v w x y z;
do echo $x ` grep -E ^\$x.* packages | wc -l` ; done
a 203
b 152
c 251
d 239
e 139
f 166
g 401
h 76
i 156
j 61
k 224
l 1206
m 222
n 177
o 63
p 385
q 37
r 118
s 306
t 261
u 57
v 60
w 138
x 394
y 23
z 43
orion:packages$
orion:packages$ for x in a b c d e f g h i j k l m n o p q r s t u v w x y z; do echo lib$x ` grep -E ^lib\$x.* packages | wc -l` ; done
liba 58
libb 7
libc 65
libd 36
libe 23
libf 36
libg 170
libh 16
libi 18
libj 12
libk 4
libl 23
libm 51
libn 30
libo 23
libp 65
libq 8
libr 94
libs 76
libt 44
libu 19
libv 22
libw 36
libx 39
liby 2
libz 4
(total 981)
[i don't know bash coding well..;) ]
the distribution is uneven showing that the first letter rule
is not really a good hashing function for software package names.
however, this is not what matters here as this directory layout
is devised because of performance reasons (of the ext2fs's inability
to deal with large directories) and it will not matter for a long time ;)
[in support of jason gunthorpe's reply to me]
Congratulations on this great work!
Keep hacking,
--
Eray (exa) Ozkural
Comp. Sci. Dept., Bilkent University, Ankara
e-mail: erayo@cs.bilkent.edu.tr
www: http://www.cs.bilkent.edu.tr/~erayo
Reply to: