Re: Splitting the security-tracker repo [was Re: [PATCH 00/12] Plannings for secure-testing repository migration to git]
Hi Guido,
On Thu, Dec 28, 2017 at 06:59:11PM +0100, Guido Günther wrote:
> Hi,
> On Thu, Dec 28, 2017 at 12:33:27PM +0100, Guido Günther wrote:
> > Hi,
> > On Wed, Dec 27, 2017 at 11:32:53PM +0100, Salvatore Bonaccorso wrote:
> > [..snip..]
> > > have that already :). But I'm unahappy with my current versions, thus
> > > I have not sent those yet. Basically: I'm struggling with a proper
> > > replacement of such constructs:
> > >
> > > cd /srv/security-tracker.debian.org/website/secure-testing/data && svn update && ../bin/update && svn commit -qm "automatic update" CVE && cd CVE && ../../bin/bts-update list
> >
> > cd /srv/security-tracker.debian.org/website/security-tracker/data && git pull origin master && ../bin/update && git commit -qm "automatic update" CVE && git push origin master && cd CVE && ../../bin/bts-update list
> >
> > git commit might fail if there are no changes so we need to either guard
> > for that or use --allow-empty. git push might fail as well if there were
> > pushes in between. Since there are several commands that
> >
> > pull, modify, commit push
> >
> > should we move this to a script that
> >
> > pulls, invokes the modification command, commits when there are
> > changes, pushes. If that fails merges again && pushes again.
> >
> > That won't save you from merge conflicts but that's the same with SVN a
> > assume?
>
> Salvatore pointed out to me that it would be good to not mix tracker
> code updates with data updates so that the above commands that pull and
> push data to/from git would only operate on the data part but not on the
> tracker itself. One possible solution would be to split the current
> secure-testing repository into two:
>
> security-tracker: containing the data that is the data/ packages/ org/
> doc/ and conf/ directories.
> security-tracker-bin: containing the security tracker service and other
> scripts and python modules: check-external, bin, tools, lib,
> templates, website
>
> security-tracker-bin could then become a submodule of security
> tracker. To ease migration we would create symlinks so the directory
> layout stays the same. The top level of the secure-testing repo would
> look something like this:
>
> -rw-r--r-- 1 agx agx 3449 Dez 28 11:56 TODO.gitmigration
> drwxr-sr-x 9 agx agx 4096 Dez 28 11:56 data
> drwxr-sr-x 2 agx agx 4096 Dez 28 11:56 packages
> drwxr-sr-x 2 agx agx 4096 Dez 28 11:56 org
> drwxr-sr-x 5 agx agx 4096 Dez 28 11:56 doc
> -rw-r--r-- 1 agx agx 8450 Dez 28 12:01 Makefile
> drwxr-sr-x 2 agx agx 4096 Dez 28 13:30 conf
> -rw-r--r-- 1 agx agx 179 Dez 28 17:20 .gitmodules
> lrwxrwxrwx 1 agx agx 35 Dez 28 17:20 check-external -> security-tracker-bin/check-external
> lrwxrwxrwx 1 agx agx 24 Dez 28 17:20 bin -> security-tracker-bin/bin
> lrwxrwxrwx 1 agx agx 26 Dez 28 17:20 tools -> security-tracker-bin/tools
> lrwxrwxrwx 1 agx agx 30 Dez 28 17:20 templates -> security-tracker-bin/templates
> lrwxrwxrwx 1 agx agx 27 Dez 28 17:20 static -> security-tracker-bin/static
> lrwxrwxrwx 1 agx agx 24 Dez 28 17:20 lib -> security-tracker-bin/lib
> lrwxrwxrwx 1 agx agx 28 Dez 28 17:20 website -> security-tracker-bin/website
> drwxr-sr-x 10 agx agx 4096 Dez 28 17:20 .
> drwxr-sr-x 9 agx agx 4096 Dez 28 17:25 security-tracker-bin
> drwxr-sr-x 2 agx agx 4096 Dez 28 17:26 stamps
> drwxr-sr-x 9 agx agx 4096 Dez 28 17:26 .git
> drwxr-sr-x 4 agx agx 4096 Dez 28 18:17 ..
>
> I have scripts¹ that produce such a layout and pushed temporary repos
> here
That looks good to me, can you push your conversion scripts to
tools/git-migration?
> https://salsa.debian.org/agx/security-tracker-with-submodule
> https://salsa.debian.org/agx/security-tracker-bin
>
> So if you want to see how it looks to a
>
> git clone --recursive https://salsa.debian.org/agx/security-tracker-with-submodule
>
> With some follow up work we can get rid of most of the symlinks but
> this could happen after the git migration. So soriano would only see an
> update of security-tracker-bin code if we bump the submodule revision
> (end even then only if we do a "git submodule update").
Indeed, if we want that we should do so only after migration.
>
> Does this make sense or does it introduce too much complexity? Is the
> above split about right or should some of the directories move between
> security-tracker and security-tracker-bin (e.g. website or static).
This looks good to me (still testing a bit right now), but I guess we
should to that change during the migration because of the above
mentioned problem where we only want changed to data/ autmoatically
updated.
With that i think we have most all together to do the switch, any
comments from others? Objections? Concerns? Or somewhing we have
overseen?
I will write up a migration plan/checklist, and would be happy to work
together with Guido further on doing the switch.
Regards,
Salvatore
Reply to: