Re: Proposed scheme for emdeb source structure
On Thu, Nov 09, 2006 at 09:18:53PM +0000, Neil Williams wrote:
> On 09/11/06 16:35:23, Julian Gilbey wrote:
> >We start with developers.
> >
> >Default directory structure
> >---------------------------
> >
> ><emdebian working directory>/
> > emdeb-patches/ -- this holds the SVN repository of packages'
> >patches
> > package1/
> > .svn/ -- SVN metadata
> > ALTERNATIVE 1:
> > package1.em.diff -- the emdeb diff file for package1,
> > representing the unified diff between the
> > Debian version of the package and the
> >emdebian one
> > ALTERNATIVE 2:
> > file1.diff -- every file which has a diff is stored
> >separately.
> > debian/ They are all made by appending ".diff"
> > rules.diff (except for debian/changelog.emdeb) - and
> >obvious
> > changelog.emdeb special cases if there's a file which
> >ends
> > ".diff" already present in the package
>
> I'd vote for alternative2 - I know there are tools for editing diff
> files but it's a lot easier with separate diff files in SVN because SVN
> is so helpful at showing you which files have changed - as well as how.
>
> > This way, someone wanting to download the SVN emdeb repository for
> >a particular package to work on it doesn't have to download the
> >whole repository.
>
> Where will the svn trunk/ be located? Will each package need it's own
> trunk? Using svn-buildpackage and it's tags may be easier to understand
> if the tags are package specific. I can't see a need for emdebian
> branches of individual packages but tags may need to be package
> specific.
Sorry, I wasn't clear: this is the structure on the developer's
machine.
The SVN repository layout is something like:
embed-patches/
package1/
trunk/
tags/
package2/
trunk/
tags/
...
with branches/ added where necessary. Clearly, tags (corresponding to
package version numbers when released) need to be package-specific.
> > <work-area>/
> > package1/
>
> i.e. foo/
>
> > build-area/ -- for building the packages, as in
> >svn-buildpackage
> > tarballs/ -- contains .orig.tar.gz and Debian .diff.gz
> >files
> > package1/ -- working source tree with emdebian patches
> >applied
>
> So that would be foo-1.2.3/ rather than simply foo/ ? - as you would
> get in Debian by doing:
> mkdir package
> cd package
> apt-get source package
On my machine, I have this, for example:
burnside:~/debian/mftrace $ ls
build-area/ mftrace/ tarballs/
burnside:~/debian/mftrace $
Here, mftrace is my working checked-out SVN tree. The package version
number changes regularly, and there's no need to stick it on the end
of the directory name. There's no harm in doing so, either.
> svn-move can cope with the name changes. Which is simpler: ensuring
> existing tools do not complain if the directory name does not match the
> version being built, or hooking 'svn move' into whatever script we use
> to update to the next debian version. After all, emdebian will be
> changing version numbers more frequently than Debian because we will
> have our own updates in-between Debian uploads which, in turn, are
> in-between source releases.
The latter is mostly irrelevant, as directory name-version usually
uses the upstream version number. And as long as the tools basically
don't care what the working directory is called, there shouldn't be a
problem. (The devscripts tools, when they check the directory name,
check for something like PACKAGE(-.*)? )
> >Workflow/tasks
> >--------------
> >
> >We work with the package called package1 throughout the following.
> >
> >(1) Setting up a package work area
> >
> > Script which:
> > - creates the directories package1/{build-area,tarballs,package1}
> > - runs apt-get source to get the Debian sources (this may be from
> > upstream Debian for a package new to emdebian, or from the
> > emdebian repository for an existing emdebian package); puts
> > these in tarballs, and runs dpkg-source -x to put the correct
> > content in package1/, and leaves a copy of the Debian tree in
> > either tarballs/ or build-area/
> > - runs svn checkout to download the emdebian patch if it exists,
>
> if not, runs emlocale and/or other emdebian-tools that prepare
> a Debian package as a prototype emdebian package, ready for the
> maintainer to edit and tweak. Hopefully, these scripts will improve to
> the point where at least some packages don't need editing.
I think this should be a separate script to be run intentionally
rather than automatically?
> > and then applies it to the working package1/ (moving
> > debian/changelog to debian/changelog.Debian and the emdeb
> > changelog to debian/changelog)
>
> Ah, the tools would be better run here, after the changelog is prepared.
>
> > Then:
> > - checks OUT the current SVN version
> > - if this is successful, then applies the current set of patches
> > to the working tree, otherwise warn the user, highlight the
> > problems and ask them to fix them before going further
>
> Won't svn do this for us anyway, via conflicts and svn resolved?
Careful: there's the SVN patches tree (stored in emdeb-patches/...)
and there's the working full source tree. The emdeb patches,
downloaded by SVN, then need applying to the working source tree. If
we want SVN to do the whole thing, we need to store everything in SVN
(including the upstream source package).
> >(5) Upgrading to a new Debian/upstream release
> >
> > A script which is essentially a modification of the core of
> > uupdate:
> >
> > - ensures that the working tree is up-to-date vis-a-vis SVN -
> > warns if not
> > Alternative 1:
> > - replaces the entire working tree with the new Debian version
> >and
> > then tries to apply the emdebian patches to this new tree
> > Alternative 2:
> > - creates a patch from old Debian version to new Debian version
> > and tries to apply this to the working directory
> >
> > Alternative 2 makes the huge assumption that binary files will
> >not
> > change, otherwise diff will complain. And this is almost
> > certainly not likely to be the case. However, reading the diff
> >is
> > useful for humans to check whether anything is likely to need
> > modifying emdeb-wise. Maybe do alternative 1 and also provide
> >the
> > user with the output of debdiff?
>
> I like that option - alternative3. :-)
>
> >
> >I think that covers pretty much everything.
> >
> >Note, incidentally, that this also means that Debian native packages
> >will produce Emdebian native packages (only a .tar.gz) and Debian
> >non-native packages will produce Emdebian non-native packages
> >(.orig.tar.gz and .diff.gz).
>
> Hmm. Native packages might be a problem. We've changed the Debian
> native package and it could well appear that we have "co-opted" the
> Debian native package as our own because there is no diff.gz.
>
> i.e. a developer cannot easily now get from a Debian native source
> tarball to the emdebian package just by downloading patches - unless we
> retain and maintain a diff in SVN?
Of course we maintain a diff in SVN, just as with every package. The
only difference between the Debian native and non-native is a question
of how we package it. For a Debian native package, there is no
.orig.tar.gz, so if we were to create a .diff.gz, we'd also need to
create an .orig.tar.gz. Doesn't seem worth the effort.
> It kind-of breaks the idea of Debian being upstream of emdebian.
>
> What about packages that actually were written just for emdebian?
> There'd be no way to tell. Is that a problem? (Are there likely to be
> any that are visible to the user?)
Interesting question. Call the package foo-emdeb. I don't know.
Does it matter if there are?
> Maybe we should use this version convention for Debian natives:
> foo_1.2_arm.deb => foo_1.2-1emN_arm.deb
> or
> foo_1.2_arm.deb => foo_1.2-emN_arm.deb
>
> i.e. *generate* a non-native version string (by adding the hyphen and
> possibly an additional '1' artefact) onto all Debian native packages to
> deliberately create a .diff.gz in emdebian.
And then have to create an .orig.tar.gz as well? You could do that
just by renaming, but then apt-get source might have issues...
> >Users
> >-----
> >
> >Well, that's easy: they just download the .orig.tar.gz and .diff.gz
> >(or .tar.gz) and build it just like an ordinary package.
>
> Native packages are a concern because of the disjoint between the
> .tar.gz and the emdebian package.
See above.
> >Needs comments please (good and bad) before I start thinking about
> >coding any of this!
> >
> > Julian
>
> I like it - the layout is clear and simple. Just one further thought.
> We may need to use an extra layer in the package directory structure
> along the lines of the debian pool:
>
> pool/main/f/foo/
> rather than
> poo/main/foo
>
> This will limit the number of directories listed in the top level
> folders to a more usable 26 instead of thousands!
>
> emdeb-patches/f/foo/
>
> instead of
> emdeb-patches/foo
>
> (as well as making it easier to eventually migrate URL's to debian
> locations.)
Yes, that seems reasonable.
Julian
Reply to: