[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Debian Mirror



> 
> I am not very familiar with the Debian distribution, and would like
> suggestions on what to include in the Debian mirror on ftp.tux.org that
> would fit within 2 GB maximum.
> 
> The current exclude_patt for mirror is rather ugly and over-micromanaged, I
> suspect it is trying to do too much in trying to keep parts of hamm and bo.
> Perhaps it would be better to concentrate just one of them, and leave
> only key parts of the other?
> 
> exclude_patt=(^|/)(\.in\..*\.$|Incoming|doc|indices|non-free|tools|dists|
> contrib|project|hamm/non-free|hamm/contrib|hamm/hamm/disks-.*|
> lost\+found/|.*/lost\+found/|hamm/lost\+found)
> 
> 
> David
> Niemi@tux.org           703-810-5538          Reston, Virginia, USA
>     "Down that path lies madness.  On the other hand, the road to
>      hell is paved with melting snowballs."  --  Larry Wall, 1992
> 
> 

The following is a chunk from my mirror.defaults.
You will find at the end of "package=debian" my "exclude_patt" for debian.
This "exclude_pattern" is my response to David Niemi's email.

After that "exlude_patt", I include every line I use for debian-non-us.
Throughout, I include additional information here for the typical home modem 
user.
My home mirror initially takes about 4 full days through with a 33.6 modem, 
though my chatscript (with "AT \\V4 ...") indicates through the log files that 
my ISP gives me 24000 bps service.
With a full mirror, my modem contacts my ISP from 3 minutes to 5 hours nightly.
My home mirror uses 1010MB of disk, about the same as a similar "exclude_patt" 
uses for RedHat, since both currently have two active versions.

I include here more lines than just "exclude_patt" to show some un-needed but 
useful variables.
I also include the "debian-non-us" lines because a US mirror site probably can 
not legally mirror it, so you may want to mirror it yourself at home.

Of course, after individuals (ie, not official mirror sites for Debian) get a 
rudimentary Linux working, they should never use a Linux CD again.
Like RedHat, the Debian CD excludes even from its US site the directories
	debian/non-free
	debian/contrib
And the US sites certainly exclude the non-US Debian directories found in 
other countries.

Before David Niemi considers anything like an official mirror of debian, 
consider the failure of what I considered the three best mirror sites for 
Debian:
	tsx-11.mit.edu
	uiuc.edu
	crosslink.net
I saw them sequentially fail over several months, initially providing perfect 
mirrors, but eventually failing to properly mirror debian.
They generally failed when debian added a new version, and hence doubled the 
required disk space for debian.
This caused me agony, since in each case I initially presumed that my mirror 
program had failed.
Each of these sites gave no indication for months that their mirrors of debian 
had become aberrant.
I give a few more details about these sites below and I suggest that NO ONE 
USE A SECONDARY MIRROR.
David Niemi might give a lot of information at his site about mirroring the  
debian distribution in our homes or in our offices, but considering the 
failure of other debian mirrors and the trouble caused to those who relied on 
those mirrors, I SUGGEST HE NOT MIRROR DEBIAN.
Perhaps the trouble of previous debian mirrors could be aleviated with 4GB of 
disk space.
I wonder how big the whole debian site is, since it took out both MIT and UIUC.
Despite problems of bandwidth, I SUGGEST THAT THE FINAL USERS THEMSELVES
MIRROR ftp.debian.org DIRECTLY!
	
	Each of the following three sites were once in debian's README.mirrors file.
	# 1. tsx-11.mit.edu        
	#    I mirrored off this site in October and November of 1996. 
	#    I believe when the Linux kernel changed and debian started the rex
	#    distribution that this site began to keep old debian directories.
	#    This lasted for months, ftp.debian.org eventually commented about
	#    it in its README.mirrors, and tsx-11.mit.edu finally removed all
	#    traces of a debian mirror.
	# 2. uiarchive.cso.uiuc.edu
	#    This is one of the most serious sites in the world and they
	#    couldn't handle debian. 
	#    I mirrored this site from around November, 1996,
	#    until August, 1997. 
	#    My first hint of problems came on November 13, 1996 when it failed
	#    to mirror the then new version, "bo".
	#    In August 1997,  after 6 months of mirroring debian through
	#    uiarchive.cso.uiuc.edu, my PC began to get the whole mirror again.
	#    Checking dates, I found that uiuc changed archive dates.
	#    In response to my email to the UIUC administrator, I was told that
	#    due to disk space problems UIUC may drop the debian mirror.  
	#    They eventually dropped the debian mirror.
	# 3. debian.crosslink.net
	#    This is a local site. 
	#    These guys came to a Lugman meeting this past year.
	#    They seem to play debian in a big way, being a Debian sponsor and
	#    mirroring http://www.debian.org at http://debian.crosslink.net.
	#    I mirrored off debian.crosslink.net from around August 1997
	#    to November 1997.
	#    In October, I noticed this site failed to mirror several
	#    random files.
	#    Some files were not mirrored, while other files were
	#    month old copies. 
	#    For example, they had a month old copy of
	#    libc5-libc6-Mini-HOWTO.txt;  and as I left them,
	#    for several weeks they had no copy of what 
	#    many new packages depended upon, libreadlineg2_2.1-7.deb.
	#    debian.crosslink.net is such a big debian player that it has 
	#    probably corrected these problems, but I saw it in disarray for
	#    so many weeks at such fundamental levels that I probably will never
	#    use this site as a debian mirror again.





package=debian
        # The following stores the remote listings in a file rather than just 
memory.
        # From this, I can see files that might be downloaded from the remote 
directory.
        # If communications are slow, I can also temporarily alter some
	# parameters and need not recreate the file.
	# I also like to see when the file gets larger than 0, so I know my mirroring 
has started.
        store_remote_listing=/mirror/mirrors/store_remote_listings/listings_deb
ian

        # Changing sites sometimes causes a time mismatch, so correct files 
are gotten again.
        # If mirror is getting the whole distribution again 
	# (as when I reset my clock badly or switch ISP's), run
        #     ***mirror -T   -p debian***

        #
	#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	#I CURRENTLY MIRROR FROM THE FOLLOWING DEBIAN SITE:
	# This site has four equivalent directory paths for debian:
	# /debian				#links to /net/ac121/linux/distributions/debian ---responds: 
"Cannot get remote directory details (/debian )"
	# /pub/debian				#links to /net/ac121/linux/distributions/debian ---responds: 
"Cannot get remote directory details (/pub/debian )"
	# /pub/linux/distributions/debian	#This is not a link, but I suspect this is 
a mount of /net/ac121/linux/distributions/debian   ---this "remote_dir" 
connects properly!!!!
	# /net/ac121/linux/distributions/debian.
	site=ftp.debian.org
	remote_dir=/pub/linux/distributions/debian
        #
	# Warning: for lack of disk space and to save mirror time, I no longer get 
any instances of Contents-i386 (14MB file),
	# but I do get Contents-i386.gz from the mirror site (already compressed).
	# In the following "exclude_patt", I exclude "*.dvi" and "*.sgml" 
	# because I personally use only "*.ps" or "*.html" or "*.txt" files.
	# Getting all possible documentation versions quickly adds disk space.
	# I also exclude all non-intel versions: m68k, alpha, powerpc, sparc.
	# The entries for "buzz" and "rex" prevent mirroring old distributions by 
these names.
 
# MY ANSWER TO DAVID NIEMI:       exclude_patt=(^|/)(\.mirror$|core$|\.cap|\.in
\..*\.$|MIRROR\.LOG|#.*#|\.FSP|\.cache|\.zipped|\.notar|\.message|lost\+found/|
Network Trash Folder|dvi/|.*\.dvi||.*\.sgml|JE|JF|.*A4.ps|.*-sparc|debian-alpha
|binary-alpha|binary-powerpc|disks-alpha|disks-powerpc|.*-m68k|.*_m68k|.*-68k-|
msdos-i386|source|Incoming|private|buzz/|rex-updates|rex/|rex-fixed/|Contents-i
386$)
	#
        # The following xbooks* has size 14 MB and contains only X-windows 
arcane documentation.
	# Those mirroring over a modem probably do not want to mirror this file.
	# You might include it in the above "exclude_patt".
	local_ignore=(^|/)xbooks.*



###############################################################################
####
package=debian-non-us
        store_remote_listing=/mirror/mirrors/store_remote_listings/listings_deb
ian-non-us
        #
        comment=The Debian non-US distribution. 
	comment=This site also has the debian Incoming, though I don't mirror it.
	site=nonus.debian.org
	remote_dir=/pub/debian-non-US
        #
        local_dir=/mirror/mirrors/debian-non-US
        do_deletes=true
        make_bad_symlinks=true
        mode_copy=true
        compress_conv_patt=
        compress_conv_expr=
        compress_patt=
	#max_delete_files=0%   #DELETE THIS LINE LATER --dupe of defaults
	#max_delete_dirs=0%    #DELETE THIS LINE LATER --dupe of defaults
        exclude_patt=(^|/)(\.mirror$|core$|\.cap|\.in\..*\.$|MIRROR\.LOG|#.*#|\
.FSP|\.cache|\.zipped|\.notar|\.message|lost\+found/|Network Trash 
Folder|.*-sparc|debian-alpha|binary-alpha|binary-powerpc|disks-alpha|disks-powe
rpc|.*-m68k|.*_m68k|.*-68k-|msdos-i386|source)



-- 
Jim Burt, NJ9L,		Fairfax, Virginia, USA
jameson@mnsinc.com	http://www.mnsinc.com/jameson
jameson@pressroom.com

"If merely 'feeling good' could decide, drunkenness would be the supremely 
valid human experience."      --William James, Varieties of Religious 
Experience



--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-user-request@lists.debian.org . 
Trouble?  e-mail to templin@bucknell.edu .


Reply to: