--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: britney: patch to speed up loop performance
- From: Robert Bruce Park <robert.park@canonical.com>
- Date: Wed, 21 Sep 2016 12:00:47 -0700
- Message-id: <147448444712.30468.11333945321908444708.reportbug@rouge>
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: britney, patch
Hi,
In Britney.write_excuses, there's a number of for loops doing slow operations redundantly. I've created a small patch that optimizes those loops with a good performance improvement. Here's some python profiling that proves the concept:
http://paste.ubuntu.com/23212621/
-- System Information:
Debian Release: stretch/sid
APT prefers xenial-updates
APT policy: (500, 'xenial-updates'), (500, 'xenial-security'), (500, 'xenial'), (100, 'xenial-backports')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.4.0-36-generic (SMP w/8 CPU cores)
Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
>From daf514d526d47646bb15f5b0a91cd9b8abe69820 Mon Sep 17 00:00:00 2001
From: Robert Bruce Park <robert.park@canonical.com>
Date: Tue, 20 Sep 2016 14:03:02 -0700
Subject: [PATCH] Python loop performance enhancements.
---
britney.py | 30 +++++++++++++++++-------------
1 file changed, 17 insertions(+), 13 deletions(-)
diff --git a/britney.py b/britney.py
index b28c62f..79ad49c 100755
--- a/britney.py
+++ b/britney.py
@@ -1751,59 +1751,63 @@ class Britney(object):
should_upgrade_srcarch = self.should_upgrade_srcarch
should_upgrade_src = self.should_upgrade_src
+ unstable = sources['unstable']
+ testing = sources['testing']
+
# this list will contain the packages which are valid candidates;
# if a package is going to be removed, it will have a "-" prefix
upgrade_me = []
+ upgrade_me_append = upgrade_me.append # Every . in a loop slows it down
excuses = self.excuses = {}
# for every source package in testing, check if it should be removed
- for pkg in sources['testing']:
+ for pkg in testing:
if should_remove_source(pkg):
- upgrade_me.append("-" + pkg)
+ upgrade_me_append("-" + pkg)
# for every source package in unstable check if it should be upgraded
- for pkg in sources['unstable']:
- if sources['unstable'][pkg][FAKESRC]: continue
+ for pkg in unstable:
+ if unstable[pkg][FAKESRC]: continue
# if the source package is already present in testing,
# check if it should be upgraded for every binary package
- if pkg in sources['testing'] and not sources['testing'][pkg][FAKESRC]:
+ if pkg in testing and not testing[pkg][FAKESRC]:
for arch in architectures:
if should_upgrade_srcarch(pkg, arch, 'unstable'):
- upgrade_me.append("%s/%s" % (pkg, arch))
+ upgrade_me_append("%s/%s" % (pkg, arch))
# check if the source package should be upgraded
if should_upgrade_src(pkg, 'unstable'):
- upgrade_me.append(pkg)
+ upgrade_me_append(pkg)
# for every source package in *-proposed-updates, check if it should be upgraded
for suite in ['pu', 'tpu']:
for pkg in sources[suite]:
# if the source package is already present in testing,
# check if it should be upgraded for every binary package
- if pkg in sources['testing']:
+ if pkg in testing:
for arch in architectures:
if should_upgrade_srcarch(pkg, arch, suite):
- upgrade_me.append("%s/%s_%s" % (pkg, arch, suite))
+ upgrade_me_append("%s/%s_%s" % (pkg, arch, suite))
# check if the source package should be upgraded
if should_upgrade_src(pkg, suite):
- upgrade_me.append("%s_%s" % (pkg, suite))
+ upgrade_me_append("%s_%s" % (pkg, suite))
# process the `remove' hints, if the given package is not yet in upgrade_me
for hint in self.hints['remove']:
src = hint.package
if src in upgrade_me: continue
if ("-"+src) in upgrade_me: continue
- if src not in sources['testing']: continue
+ if src not in testing: continue
# check if the version specified in the hint is the same as the considered package
- tsrcv = sources['testing'][src][VERSION]
+ tsrcv = testing[src][VERSION]
if tsrcv != hint.version:
continue
# add the removal of the package to upgrade_me and build a new excuse
- upgrade_me.append("-%s" % (src))
+ upgrade_me_append("-%s" % (src))
excuse = Excuse("-%s" % (src))
excuse.set_vers(tsrcv, None)
excuse.addhtml("Removal request by %s" % (hint.user))
--
2.7.4
--- End Message ---
--- Begin Message ---
- To: 838514-done@bugs.debian.org
- Subject: Re: ITM: Britney2 branch britney-patch-bundle-2016-09 into master
- From: Niels Thykier <niels@thykier.net>
- Date: Thu, 27 Oct 2016 06:04:00 +0000
- Message-id: <bdb38b5a-1c5d-82ee-1d4b-4d63cd5b2371@thykier.net>
- In-reply-to: <b87c071b-7ecf-7283-43de-ea628de34cf8@thykier.net>
- References: <b87c071b-7ecf-7283-43de-ea628de34cf8@thykier.net> <b87c071b-7ecf-7283-43de-ea628de34cf8@thykier.net>
On Sun, 25 Sep 2016 16:47:00 +0000 Niels Thykier <niels@thykier.net> wrote:
> Hi,
>
> I intend to merge the branch [britney-patch-bundle-2016-09] consisting
> of 22 patches into master no later than Monday the 3th of October. It
> has been tested with [no regressions].
> * CC: People known to run Britney
> * BCC: contributors + the debian bug
>
> [...]
> Bugs fixed/Merges:
> * #838514 (patch to speed up loop performance)
> * GH#12 (Handle missing arches in p-u gracefully)
> * GH#13 (Use better exception handling)
>
> [...]
>
> Robert Bruce Park (1):
> Python loop performance enhancements.
>
>
> Thanks,
This was merged a while back, closing the bug.
Thanks,
~Niels
--- End Message ---