RFC: rmadison query in review-update-needed script

[Roberto C. Sánchez <roberto@debian.org>]

Hello all,

Some days ago I claimed tigervnc in dla-needed.txt and began working on
it.  Only, after about an hour it dawned on me that tigervnc was not
present in jessie.  I went about trying to determine the best way to
ensure that only packages actually in the appropriate suite
(oldoldstable for now for jessie) end up in dla-needed.txt.

After asking on debian-devel regarding API accessibility of
distro-tracker and reviewing the limited options, I came up with a minor
modification to the review-update-needed script that queries rmadison
for information regarding the existence of the packages in
dla-needed.txt.

I have done my best to seamlessly integrate with the existing
capabilities of the script and to integrate in such a way that, for
example, the new capability could be easily leveraged by those working
front-desk or even automated by cron to quickly alert of the presence of
packages in dla-needed.txt that are not present in jessie.  The
implementation should also work for dsa-needed.txt if that might be
useful to those who maintain that file.

Please have a look at the attached patch and let me know what you think.
If the consensus is that this capability would be useful, I will go
ahead and commit/push the change to the security-tracker repo.

Here is some example output:

roberto@build01:~/src/security-tracker.git (master *=)$ ./bin/review-update-needed --lts --exist-in oldoldstable | tail

Package: iperf3
Claimed-By: Thorsten Alteholz
Claimed-Date: 2020-01-26 21:47
Last-Update: 2020-01-27 11:16

Package: tigervnc
Missing-From: oldoldstable
Unclaimed-Since: 2020-01-27 15:53


Regards,

-Roberto

-- 
Roberto C. Sánchez

diff --git a/bin/review-update-needed b/bin/review-update-needed
index b931221b71..49f64bfbf5 100755
--- a/bin/review-update-needed
+++ b/bin/review-update-needed
@@ -47,6 +47,8 @@ else:
                         help='Automatically unclaim entries older than N seconds (default: %(default)s)')
 parser.add_argument('--exclude', nargs='+', metavar='PACKAGE', default=[],
                     help='completely ignore packages specified PACKAGE')
+parser.add_argument('--exist-in', nargs='+', metavar='SUITE', default=[],
+                    help='query rmadison for existence in SUITE')
 args = parser.parse_args()
 if args.verbose and args.quiet:
     args.error("--verbose and --quiet contradiction")
@@ -123,11 +125,38 @@ if retcode != 0:
 
 all_entries.sort(key=lambda x: x[args.sort_by])
 
+if args.exist_in:
+    process = subprocess.Popen(["rmadison", "-u", "debian", "-a", "source",
+                                "-s", ",".join(args.exist_in),
+                                " ".join([entry['pkg'] for entry in all_entries])],
+                                stdout=subprocess.PIPE)
+
+    rmadison = []
+    for line in process.stdout:
+        pkg, ver, suite, arch = [f.strip() for f in line.decode('utf-8').split('|')]
+        rmadison.append((pkg, suite))
+    for entry in all_entries:
+        in_suites = set()
+        for suite in args.exist_in:
+            if (entry['pkg'], suite) in rmadison:
+                in_suites.add(suite)
+        missing_from = set(args.exist_in).difference(in_suites)
+        if len(missing_from) > 0:
+            entry.update({'missing-from': ",".join(missing_from)})
+        else:
+            entry.update({'missing-from': None})
+
+    retcode = process.wait()
+    if retcode != 0:
+        sys.stderr.write("WARNING: rmadison returned error code {}\n".format(retcode))
+
 unclaim_pkgs = []
 for entry in all_entries:
     if args.skip_unclaimed and not entry['claimed-by']:
         continue
     args.quiet or print("Package: {}".format(entry['pkg']))
+    if entry['missing-from']:
+        args.quiet or print("Missing-From: {}".format(entry['missing-from']))
     if entry['claimed-by']:
         args.quiet or print("Claimed-By: {}".format(entry['claimed-by']))
         args.quiet or print("Claimed-Date: {}".format(format_date(entry['claimed-date'])))