Re: jenkins vs. debile [was: Re: Bits from the Release]
Hi!
2013/9/27 Michael Tautschnig <mt@debian.org>:
>> [...]
>
> Would you/Matthias be willing to share the feedback on jenkins more widely? I do
> presently run a system with ~60,000 jobs, and, yes, this is probably somewhat
> beyond what jenkins was meant to scale to. Yet jenkins does have the practical
> advantage of being widely used, with lots of plug-ins, and being well
> documented. Not all of this seems to be the case for debile at present...
>
>> I don't doubt that jenkins.d.n can be leveraged for the time being,
>> giving the low amount of autopkgtests currently available. But you might
>> want to check with Matthias or similar experiences before committing to
>> using Jenkins for this.
>>
> [...]
>
Additionally to the points Paul already mentioned: The lots of plugins
stuff does not really matter for the Debian use-case, because Jenkins
was built for an entirely different purpose.
For example: In Debian, we need at least 5 states a build can be in:
dependency-wait, building, uploaded, installed and failed. Jenkins
only provides built and failed, and adding more states is lots of
work.
Also, packages function differently compared to Jenkins jobs: You can
e.g. build different versions of a single package, so foobar is build
in versions 1.0 and 1.2 for different target suites. This can not be
easily reflected in Jenkins.
We worked around all of these issues in Tanglu. For example, we have a
depwait state by taking the control over builds from Jenkins to an
external application and adding dependency-wait information to the job
descriptions. We can then find stuff in depwait using a RegEx.
To solve the multiple-versions issue, we append the version number to
the jobname, and jobs get renamed if necessary by the
Jenkins-controlling tool. Because we don't have uploaded/installed
states, the Jenkins-helper is patrolling the jobs to guess these
states and schedule rebuilds.
You can see the stuff in action here: http://buildd.tanglu.org/
(currently, some build slaves are broken, which is really sad and
needs to be fixed soon)
So, I really think that Debile is a sane approach with less hackish
solutions. I will start to work on it too, as soon as I find time, and
most likely migrate Tanglu to it, so we do no longer rely on Jenkins
for that.
On the pro side, of course, Jenkins has cool plugins to analyze job
output, and it has live buildlogs - but having it build the Debian
archive is hackish. Also, Jenkins gets incredibly slow... The machine
powering our build-master and archive is very powerful (Octocore, I
think at least 16GB of RAM), and Jenkins performance is still bad.
Maybe because it is using one large XML file to store job data.
Cheers,
Matthias
--
I welcome VSRE emails. See http://vsre.info/
Reply to: