Re: split descriptions Re: PROPOSAL to sarge+1 - Split main in sub-repositories
Mike Fedyk <mfedyk@matchmail.com> writes:
> Goswin von Brederlow wrote:
>
>>Mike Fedyk <mfedyk@matchmail.com> writes:
>>
>>
>>
>>>Andrea Mennucc wrote:
>>>
>>>
>>>
>>>>hi
>>>>
>>>>first of all I redid the computazion using gzip
>>>>
>>>> cp /var/lib/apt/lists/*sarge_main_binary-i386_Packages /tmp/main_all
>>>> cd /tmp
>>>> egrep '^ |^Description|^Packa' main_all > main_descr
>>>> egrep -v '^ |^Description' main_all > main_data
>>>> ls -s main_*
>>>> gzip -v main_*
>>>> ls -s main_*
>>>>
>>>>result: 3044 main_all.gz 1220 main_data.gz 1672 main_descr.gz
>>>>
>>>>as you see we are saving 1824kb here (each time we do not download
>>>>the descriptions), or 152 kb, (downloading both)
>>>>
>>>>
>>>Andrea, have you not heard of the (I forget who said it first)
>>>proposal to have a daily and weekly updated packages file?
>>>
>>>The daily files only contain the packages that have changed the
>>>previous day up until the point the packages file was generated.
>>>
>>>The weekly only contain the packages that have changed the previous
>>>week up until the point the packages file was generated.
>>>
>>>You could have several weekly files, and even a monthly, but that's
>>>debatable.
>>>
>>>I'd say this would save you much more bandwitdh than splitting out the
>>>descriptoin files, and it doesn't change the format, only includes
>>>fewer pakcages in the daily and weekly files. Of course splitting the
>>>descriptoins into seperate files can be combined with this, but the
>>>need to do so becomes much less pressing.
>>>
>>>What do you guys think?
>>>
>>>
>>
>>And an apt sources.list like this? [assuming no new apt syntax]
>>
>>deb http://ftp.debian.org/debian sid main/weekly
>>deb http://ftp.debian.org/debian sid main/weekly+1
>>deb http://ftp.debian.org/debian sid main/weekly+2
>>deb http://ftp.debian.org/debian sid main/weekly+3
>>deb http://ftp.debian.org/debian sid main/weekly+4
>>deb http://ftp.debian.org/debian sid main/weekly+5
>>deb http://ftp.debian.org/debian sid main/weekly+6
>>
>>The idea of incremental packages files addresses the problem of
>>downloading the same data over and over partly. You still download the
>>same data over and over once a week. Adding a monthly file (+4 updates
>>for each week) would reduce this further.
>>
>>What it doesn't address is client diskspace, memory usage and cpu
>>time.
>>
> I didn't think it would require a change in the sources.list file at
> all. Just check for a "Packages.daily.gz" file or some such logic and
> use it if it exists. That way, repositories can start using it as
> soon as it is in apt, and apt will start using it once the client
> includes that functionality.
That means only sidusersbenefit from it. Using a directory structure
means even woody users can use it if they wish.
That doesn't mean future apt could not use the directories
automatically with the existing short sources.list entries. Using the
path just allows unchanged apts to use it too.
Another advantage is that you could setup a mirror for this weeks
updates with the existing partial mirror scripts.
MfG
Goswin
Reply to: