Re: portable mirror?
On Thu, 2002-02-07 at 18:19, Paulo Jose da Silva e Silva wrote:
> Hello,
>
> I didn't know about apt-move or apt-mirror, so I developped a solution to
> this problem myself, let me share it :-)
There's a program out there called aptcached (you'll have to google
for it) that works _very_ well for me. It sets up a dedicated proxy,
so most of what you do is transparent. It does use 2x as much disk
space, but "apt-get clean" could solve that,
I'll try yours, though, too, since I like python.
>
> The idea is very simple. If you do an "apt-get update" in your home machine
> (the one with slow connection) then you can ask apt to give you the list of
> the packages that would be upgraded. Well, I decided to write a small python
> script that read the list of packages to be upgraded and generate small
> scripts to download them in another machine with a good internet connection.
> Finally copy the downloaded files to the original machine and finish the
> "apt-get upgrade". My script can download the packages to multiple disks too,
> but you need disks that are big enough to fit the biggest deb (that may be
> 10MB or bigger), like a zip.
>
> Well, it helped me a lot before I got a good connection. Take a look and see
> if it is worthy. Precise instructions can be found on the top of the script
> attached.
>
> Paulo
>
> PS: You will net wget installed.
>
> PS2: This was my first python script, so don't look for good programming
> style.
>
> PS3: Any problem, please contact me.
>
> --
> Paulo José da Silva e Silva
> Professor Assistente do Dep. de Ciência da Computação
> (Assistant Professor of the Computer Science Dept.)
> Universidade de São Paulo - Brazil
>
> e-mail: rsilva@ime.usp.br Web: http://www.ime.usp.br/~rsilva
>
> Teoria é o que não entendemos o (Theory is something we don't)
> suficiente para chamar de prática. (understand well enough to call practice)
> ----
>
> #!/usr/bin/env python
> #
> # Simple script to create scripts to download a debian upgrade in zip disks
> # or any "big enough" disks.
> #
>
> # Author: Paulo Silva (rsilva@ime.usp.br).
> #
> # This script should be useful if you want to upgrade a Debian box which
> # has a slow connection to the internet, but you have access to a faster one.
> # The idea is to use the slow connection to update your package list and then
> # use the fast connection to download the deb files for the upgrade. The
> # files should be transported to the machine with slow connection with a
> # zip disk, or any big enough media. You'll need python installed (of course).
> #
> # The use is the following:
> # 1) Use the slow connection to update the packages list from the net. You
> # do this by changing your /etc/apt/source.list file to point to a
> # newer Debian version (like potato right now) and doing an
> #
> # apt-get update
> #
> # from the prompt.
> #
> # 2) Generate the scripts that will download the file from the machine with
> # fast connection. To achieve this you must do:
> # a) Generate a list of the files to download:
> # Just type:
> #
> # apt-get -qq --print-uris dist-upgrade > uris_file
> #
> # b) Move the uris file to the apt-download-offline directory and type:
> #
> # python apt-download-offline.py uris_file disk_size[bkm]
> #
> # For example: 96 megabytes zip disks and using wget you should type:
> #
> # python apt-download-offline.py uris_file 96m
> #
> #
> # 3) Move the download scripts (apt-download-disk?.py) and run them
> # in _different directories_ in the fast connected machine.
> # You should make them executable or type something like:
> #
> # python apt-download-disk?.py
> #
> # The files will be downloaded in the current directory.
> #
> # 4) Take the downloaded deb files with the zip disks, copy them to the
> # /var/cache/apt/archives directory in the slow connected machine and
> # type
> #
> # apt-get dist-upgrade
> #
> # The upgrade should take place and no downloads needed.
> #
> # Enjoy!
> #
> # LIMITATIONS:
> # The disks must be big enough to fit the biggest deb file
> # that must be downloaded. zip should do.
> # The number of disks used is not optimal. I believe that it is possible
> # to prove that with my solution you may use up to 50% more disks than
> # the optimal set. This is good enough for me :-)
> import sys, string, re, commands
>
> # Open a file for reading, aborting on exeptions.
> def openfile(name):
> try:
> handle = open(name, 'r')
> except:
> print 'Problems to open file %s!' % name
> sys.exit(1)
> return handle
>
> # Deal with command line.
> def getarguments(argv):
> if len(argv) != 3:
> print 'Use: apt-download-offiline.py <uris_file> <disk_size[bkm]>'
> sys.exit(1)
> # Opens input file.
> input = openfile(argv[1])
>
> # Gets disksize.
> # Verifies if disk size is in the correct format.
> disksize = re.match('[0-9]+[bkm]?$', argv[2])
> if not disksize:
> print 'Wrong disksize format!'
> sys.exit(1)
> disksize = disksize.group(0)
> # Sets the disksize taking into accound the correct multiplier.
> mults = {'b': 1, 'k': 1024, 'm': 1024*1024}
> if disksize[-1] in mults.keys():
> disksize = int(disksize[:-1])*mults[disksize[-1]]
> else:
> disksize = int(disksize)
>
> # Return parameters.
> return input, disksize
>
> # Save the script to download a disk.
> def printdisk(disk, disknumber, size):
> print 'Disk %d size: %d kbytes' % (disknumber, size/1024)
> try:
> output = open('apt-download-disk%d.py' % disknumber, 'w')
> except:
> print 'Problems to create file apt-download-disk%d.py!' % disknumber
> sys.exit(1)
>
> # Write the head of the python script.
> output.write("""#!/usr/bin/env python
> # Disk %d
> # Automatically generated by apt-download-offline.py
> #
> from string import *
> from commands import *
> import os
> """ % disknumber)
> output.write("""
> # Function to download a file.
> def download(url, file, md5sum):
> while 1:
> os.system("wget -c %s" % url)
> original = split(url, '/')[-1]
> os.rename(original, file)
> trymd5 = getoutput("md5sum %s" % file)
> trymd5 = split(trymd5)[0]
> if trymd5 == md5sum:
> break
> else:
> os.remove(file)
>
> # Download all the required files.
> """)
> for deb in disk:
> output.write("download(%s, '%s\', '%s')\n" % (deb[1], deb[2], deb[3]))
>
> # Read URI file
> def readuri(input, disksize):
> urilist = []
> totalsize = 0
> for line in input.readlines():
> uri, package, size, md5 = string.split(line)
> size = int(size)
> # Asserts that there isn't any package too big to fit a disk.
> if (size > disksize):
> print 'There is a package bigger than the disk!\n'
> sys.exit(1)
> totalsize = totalsize + size
> urilist.append([size, uri, package, md5])
> print 'Total download size: %d kbytes' % (totalsize/1024)
>
> return urilist
>
> ###############################################################################
> #
> # Main script. Creates the script to download disks.
> #
>
> # Gets command line arguments.
> input, disksize = getarguments(sys.argv)
>
> # Reads the uri file.
> urilist = readuri(input, disksize)
> urilist.sort()
>
> # Starts first disk.
> disknumber = 1
> disk = []
> slackness = disksize
>
> # Scans the list to create the disks.
> while urilist:
> # Scans for a package that is small enough to fit the disk.
> for i in urilist:
> if i[0] <= slackness: break
>
> # Appends the package if it fits the current disk.
> if i[0] <= slackness:
> disk.append(i)
> slackness = slackness - i[0]
> urilist.remove(i)
> # Starts a new disk if the package doesn't fit.
> else:
> # prints the script for this disk.
> printdisk(disk, disknumber, disksize-slackness)
> # Start a new disk with actual package (note that we already know
> # that there is no package to big to fit a disk).
> disknumber = disknumber + 1
> slackness = disksize
> disk = []
>
> # Prints the script for the last disk.
> printdisk(disk, disknumber, disksize - slackness)
>
--
+------------------------------------------------------------+
| Ron Johnson, Jr. Home: ron.l.johnson@cox.net |
| Jefferson, LA USA http://ronandheather.dhs.org:81 |
| |
| 484,246 sq mi are needed for 6 billion people to live, 4 !
! persons per lot, in lots that are 60'x150'. |
! That is ~ California, Texas and Missouri. !
! Alternatively, France, Spain and The United Kingdom. |
+------------------------------------------------------------+
Reply to: