Re: Announcing metalink for Knoppix 7.0.5
Hello Klaus,
First of all my personal thanks to your awesome work on Knoppix and I have been using it as my desktop for a few years now essentially as a read only OS + my customizations.best of both worlds :-)
Pls see my comments below.
Hello Bhavesh,
On Tue, Jun 04, 2013 at 12:40:06PM -0400, Bhavesh Patel wrote:
> Lots of linux distros and other popular packages have created metalinks for the distribution of their software in a way that failover and redundancy and being able to download files from multiple mirros is supported via simple xml file.
> You can take a look at all the distros and packages using metalink at this website http://metalinker.org/
>
> Looking at the popularity of the metalinks , IETF has approved and it is now a standard.
>
> So I took the opportunity of converting all the mirror information(http/ftp) of Knoppix for version 7.0.5 and have created metalinks for it in both ver3 (extension .metalink) and version 4(approved by IETF as a standard) with file name extension as meta4.
> With these metafiles you are limited to only your download bandwidth of your ISP as you can parallely download from multiple mirrors and even if some fails it moves to the other in the list.
>
> Pls be sure to read the documenation of your download manager to ensure they support Metalinks and also pls be sure to understand the metalinks which I have published have all the flavours of ISO images(English DVD/CD,German CD/DVD) so pls pay attention when you download and only select the flavor you are interested in.
>
> I have tested that each and every mirror in my metalink file is working at this time.There were lot of dead links on the original mirror webpage on Knoppix site.
I must admit that I have not heared of metalinks up to now.
But first, thanks for your work! Maintaining a list of mirrors becomes
indeed difficult nowadays where web space providers seek to gain
popularity by offering a mirror location for free, then after a year
they abandon or neglect it, it's never updated and finally removed
without me even noticing, so I have a lot of dead links in the mirror
list. I would have to spend a whole day every now and then, updating the
mirrors table and checking all mirrors, which I simply don't have enough
time to do regularly.
BP : Yes Klaus I completely understand the work it takes to keep it updated.
BP : My plan was to upload the metalink file on alioth.debian.org(I was hoping someone can give me permissions for it) or github.com or somewhere(if people liked downloading via metalinks) like sourceforge.net so that anyone can update the metalink file(after my review and running tools to ensure that each and every link for mirror is working fine.) as it is purely an xml file.
a reference be maintained to the latest copy of it which has the latest mirrors.
But here is the catch:
1. How does metalink save me the work of adding and removing mirrors?
Someone has to do this, and react if new mirrors are offered or removed.
I'm not through with the metalink documentation yet, so maybe you have a
quick solution for this that has a nice form, rather than me editing
huge HTML tables each and every time.
BP : If we need a html page then we have 3 options
1>we need to host it on a github.com or sourceforge.net kind of a place where anyone(approved by you) can edit it.The volunteers will do their due deligence in making sure all links are working on a regular basis.
2>Continue hand editing it but give someone else edit options as volunteers.I will be more then happy to help with that effort as well.
3>Look for a script or write a script which converts metalink to an html page, the format might be different then what we have today.
2. Will I still be able to put my "Accept tht GPL and please don't sue
me if you break your computer" license first before the actual download link appears?
BP : Metalink is a purely xml file so currently it does not have anyway for end user interaction.I could be wrong but this is not possible with Bittorrent downloads as well today.
BP : One idea could be to ask people to accept the license before they are provided with metalink to download.The other option could be that on startup , show the GPL page when the browser launches on automatic startup.
3. Can people without special plugins or browsers still access the
mirror list, or a specific mirror? Is it possible without Javascript, or
from the text console? Elinks? Accessibility? Would I still have to
maintain the HTML table for people without metalink support in their
download software?
BP : Metalink is only supported by download managers.Out of the box most popular browsers do not support metalinks(even though IETF has approved it as an international standard).
BP : Metalink should be considered specialized download manager for advanced users just like Bittorrent.The advantage metalink provides is that even if there are broken mirrors , then it skips to the next mirror in the metalink file automatically and also does "segmented parallel downloads" from multiple mirrors in a given/chosen country so that the load is spread evenly across all mirrors and the user does not have to worry about broken mirrors.Also the other advantage of metalinks is it supports http/ftp/rsync/bittorrent/edonkey/gnutella/magnet links .So one file can handle all combinations of downloads.
> Pls let me know if there are any bugs etc.Also let me know if you find this useful then I can probably make metalinks to speed up the downloads and spread the load across various mirrors with the upcoming Knoppix 7.1.0 release as well.
(Well, looks like the new version will rather be a 7.2, many bugfixes
and new features compared to the CeBIT edition.)
Is there a PHP sample code that will turn the metalink file into a HTML
table again? This way, I could at least save the work of updating the
mirror page manually.
BP : I do not know of any script which will do this.
> Pls note since the Torrent for Knoppix has multiple files in it we cannot add it as a metafile.Currently if you want parallel downloads with HTTP/FTP/BIttorrent simultaneously then the torrent should only have one file which is the ISO file which is on the http/ftp sites.
Since there are regularly different derivates of the same version of
Knoppix (DVD+CD * English+German * ADRIANE+Standard), offering a single
ISO file is difficult. Plus, I have to provide signed checksums as well,
so users can always verify checksums for files they downloaded earlier,
even when the old version was already superseded by a new one.
BP : Yes I understand. My comment was related to the torrent file content only.
What I meant is that today we download KNOPPIX_V7.0.5DVD-2012-12-21-EN.iso alone.
But the corresponding torrent for example http://torrent.unix-ag.uni-kl.de/torrents/KNOPPIX_V7.0.5DVD-2012-12-21-EN.torrent consists of other files as well as noted below
KNOPPIX_V7.0.5DVD-2012-12-21-EN.iso
File:KNOPPIX_V7.0.5DVD-2012-12-21-EN.iso.md5
File:KNOPPIX_V7.0.5DVD-2012-12-21-EN.iso.md5.asc
File:KNOPPIX_V7.0.5DVD-2012-12-21-EN.iso.sha1
File:KNOPPIX_V7.0.5DVD-2012-12-21-EN.iso.sha1.asc
File:dpkg-l-dvd-705.txt
File:knoppix-cheatcodes.txt
BP : Now the problem is that the torrent and the file we are downloading has different content and metalink today cannot handle it.But in future if either the torrent is synchronized , or the metalink clients support file across multiple file torrent then users will be able to download simultaneously from http/ftp and bittorrent.
> For detailed instructions(including actual metalink files) and an example using aria2 , please look at my blog post http://bhavesh.freeshell.org/blog/worklog/456321123.html
For me as a beginner user of metalink, it would not be transparent that
opening the .metalink file with firefox just results in the same
question being asked again and again. Apparently, a plugin is needed.
BP : Best is to use a plugin for firefox like downthemall or flashgot which understands metalinks or hands off the control to metalink download managers like aria2 etc.Until browsers support metalink out of the box a plugin will be needed.
BP : Overall the advantage is that it uses existing http/ftp mirrors and does not overload any mirrors and automatically moves to mirrors which are down.
BP : My hope is that between using bittorrent , metalinks etc 80-90% downloads of Knoppix should be covered.The rest 10% could continue to use the html mirror page through their browser(not the most efficient way for Knoppix mirror hosters).
BP : Also as you know bittorrent is not allowed in most companies/organizations.But with metalink since the downloads happen via http/ftp people can download it while they are in their companies/organization and utilize the maximum possible download speeds versus a regular browser download from a single mirror.
Regards
-Klaus
-----Original Message-----
From: "Klaus Knopper" [debian-knoppix@knopper.net]
Date: 06/04/2013 02:38 PM
To: "Bhavesh Patel" <bhaveshp@excite.com>
CC: "" <debian-knoppix@lists.debian.org>
Subject: Re: Announcing metalink for Knoppix 7.0.5
Hello Bhavesh,
On Tue, Jun 04, 2013 at 12:40:06PM -0400, Bhavesh Patel wrote:
> Lots of linux distros and other popular packages have created metalinks for the distribution of their software in a way that failover and redundancy and being able to download files from multiple mirros is supported via simple xml file.
> You can take a look at all the distros and packages using metalink at this website http://metalinker.org/
>
> Looking at the popularity of the metalinks , IETF has approved and it is now a standard.
>
> So I took the opportunity of converting all the mirror information(http/ftp) of Knoppix for version 7.0.5 and have created metalinks for it in both ver3 (extension .metalink) and version 4(approved by IETF as a standard) with file name extension as meta4.
> With these metafiles you are limited to only your download bandwidth of your ISP as you can parallely download from multiple mirrors and even if some fails it moves to the other in the list.
>
> Pls be sure to read the documenation of your download manager to ensure they support Metalinks and also pls be sure to understand the metalinks which I have published have all the flavours of ISO images(English DVD/CD,German CD/DVD) so pls pay attention when you download and only select the flavor you are interested in.
>
> I have tested that each and every mirror in my metalink file is working at this time.There were lot of dead links on the original mirror webpage on Knoppix site.
I must admit that I have not heared of metalinks up to now.
But first, thanks for your work! Maintaining a list of mirrors becomes
indeed difficult nowadays where web space providers seek to gain
popularity by offering a mirror location for free, then after a year
they abandon or neglect it, it's never updated and finally removed
without me even noticing, so I have a lot of dead links in the mirror
list. I would have to spend a whole day every now and then, updating the
mirrors table and checking all mirrors, which I simply don't have enough
time to do regularly.
But here is the catch:
1. How does metalink save me the work of adding and removing mirrors?
Someone has to do this, and react if new mirrors are offered or removed.
I'm not through with the metalink documentation yet, so maybe you have a
quick solution for this that has a nice form, rather than me editing
huge HTML tables each and every time.
2. Will I still be able to put my "Accept tht GPL and please don't sue
me if you break your computer" license first before the actual download link appears?
3. Can people without special plugins or browsers still access the
mirror list, or a specific mirror? Is it possible without Javascript, or
from the text console? Elinks? Accessibility? Would I still have to
maintain the HTML table for people without metalink support in their
download software?
> Pls let me know if there are any bugs etc.Also let me know if you find this useful then I can probably make metalinks to speed up the downloads and spread the load across various mirrors with the upcoming Knoppix 7.1.0 release as well.
(Well, looks like the new version will rather be a 7.2, many bugfixes
and new features compared to the CeBIT edition.)
Is there a PHP sample code that will turn the metalink file into a HTML
table again? This way, I could at least save the work of updating the
mirror page manually.
> Pls note since the Torrent for Knoppix has multiple files in it we cannot add it as a metafile.Currently if you want parallel downloads with HTTP/FTP/BIttorrent simultaneously then the torrent should only have one file which is the ISO file which is on the http/ftp sites.
Since there are regularly different derivates of the same version of
Knoppix (DVD+CD * English+German * ADRIANE+Standard), offering a single
ISO file is difficult. Plus, I have to provide signed checksums as well,
so users can always verify checksums for files they downloaded earlier,
even when the old version was already superseded by a new one.
> For detailed instructions(including actual metalink files) and an example using aria2 , please look at my blog post http://bhavesh.freeshell.org/blog/worklog/456321123.html
For me as a beginner user of metalink, it would not be transparent that
opening the .metalink file with firefox just results in the same
question being asked again and again. Apparently, a plugin is needed.
Regards
-Klaus
--
To UNSUBSCRIBE, email to debian-knoppix-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: [🔎] 20130604183856.GW7888@knopper.net">[🔎] 20130604183856.GW7888@knopper.net">http://lists.debian.org/[🔎] 20130604183856.GW7888@knopper.net
Reply to: