On Fri, Nov 05, 2021 at 05:31:49PM +0100, David Kalnischkies wrote:
> On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote:
> > On 03-11-2021 16:45, Julian Andres Klode wrote:
> > > There is some software "parsing" sources.list on its own, most of that
> > > is better served by `apt-get indextargets` (and for downloading stuff
> > > based on the urls, `apt-helper download-file`, such that it respects
> > > proxies and supports all transports users may use in sources.list)
> >
> > Like autopkgtest. When I was working on it to support Debian's migration
> > testing, I looked at python-apt and because that didn't support it,
> > stopped thinking. With indextargets and download-file I guess we could
> > work on it again. When were those introduced? Ubuntu needs it on old
> > releases so before autopkgtest can change it, we'd need support for a while.
>
> `apt-get indextargets` is from 2015 and a part of the acquire-additional-
> files feature used mainly by apt-file and appstream to have apt download
> files it isn't using itself, so those tools don't have to implement it.
>
> The job of indextargets is it mostly to give access to metadata (and
> crucially filenames) for those previously configured and hopefully now
> downloaded files. apt-file e.g. asks for the Contents files in this way
> to avoid exposing file naming logic and location to other tools.
>
> So, for the filenames of all (downloaded) Packages files:
> apt-get indextargets --format '$(FILENAME)' 'Identifier: Packages'
> (the default output is deb822 stanzas you could grep with more powerful
> tools than the simple inbuilt line-based filter)
>
> Note that you either have to implement opening compressed files yourself
> or use `/usr/lib/apt/apt-helper cat-file`.
>
>
> That was historically the most common reason to fiddle with sources.list
> parsing, hence Julian referring to it, but this seems not what autpkgtest
> is aiming for. On a casual look (well, grep) I see only:
>
> * lib/adt_testbed.py apt-pocket codepath seems to want to construct new
> sources.list entries based on existing ones. That should be possible
> with some indextargets busywork in general, but I am not completely
> sure what is going on here and the gymnastics should be similar to…
>
> * setup-commands/setup-testbed tries to find the mirror (and release)
> used for target distribution based on your current system. I am
> a bit surprised that works actually…
>
> Anyway, the later could perhaps be implement with:
> apt-get indextargets --format '$(ORIGIN)|$(REPO_URI)' | sort -u | \
> grep -i -e "^$(. /etc/os-release; echo "$ID")" | cut -d'|' -f 2
>
> (That is one line only for posterity – as you see, I am trying to fix
> the too general search by checking against Origin as defined by the
> Release file of a repository, but that would need work still to
> eliminate same-origin-but-different-repo cases)
>
>
> Parsing of the sources files is not really indextargets job through, so
> it might not always work for that task: It e.g. doesn't work if the data
> files are not on disk which might or might not be okay for you (there is
> 'guess' mode, but that of course has no metadata extracted from the
> Release files – the Origin I was using above).
>
> The apt family doesn't really have a publicly exposed way of reasoning
> about sources.list (or .sources) files and I am not quite sure it really
> can as subtil differences between repositories make it hard to give them
> all a common interface which makes sense. (I will probably be proven
> wrong by Julian though.)
>
>
> Like, for example, if stable is in the sources, make sure there
> is also updates and security there and/or add them. What for Debian are
> three distinct repositories might for others very well be components.
>
> Assuming you even know which line refers to Debian: I was using Origin
> above for this task as we can't really guess based on the URI. And even
> then… that logic above finds the tor+mirror+file source I am using,
> that won't work for autopkgtest, but I am special and this just
> a default fallback, so I might be thinking way too much about it…
We could render what we passed as a deb822 format in apt{,-get}
sources list (remind me to add subcommand support to the parser).
Possibly with some normalization in there. Presumably if you have
a paragraph "Suites: foo bar Components: x y" you end up with 2
paragraphs in the output, as IIRC we group our sources entries
by Release file.
I think that provides a reasonable top-level interface. You could
of course then also normalize this further such that all fields only
have one value, and give them all IDs and then you can apt source
remove <id> or stuff, idk.
Will have to see how I implement the deb822 parsing and editing
on the python-apt side.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Attachment:
signature.asc
Description: PGP signature