Re: How to package Nuxeo DM, a Java EE application, in Debian
On Feb 6, 2011, at 10:15 PM, Niels Thykier wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 2011-02-06 20:24, Stefane Fermigier wrote:
>> We are fully aware that our packages are not built in a way similar to the way a Linux package is usually built (i.e.: ./configure ; make ; make install). But we believe that:
>>
>> 1. We don't have another reasonable choice for how to build these packages.
>>
>> 2. The issues (and discrepencies with the packaging guidelines for Ubuntu or Debian) are not specific to our project, but common to every project that uses Maven (which seems to be the most popular build tool for Java projects these days) as its main build tool, and more generally to every large-scale Java EE application (such as: XWiki, OpenBravo, Compiere, Open-Xchange, OBM...).
>>
>
> Actually, most Java packages do not use the ./configure && make && make
> install approach. Most I have seen tend to use ant or maven; ant is
> rather well supported (assuming the build file is written sanely) and I
> believe we got some very good support for maven2 as well. :)
Sorry if I made myself unclear: we DO use maven for building Nuxeo. Maven 2.2.1 to be precise (with lot of sensitivity upon the exact maven version, actually).
>> Here are the main objection that have been raised (by some Ubuntu guys) about the way we are making our packages:
>>
>> 1. "It looks like they're bundling their own Tomcat. We haven't allowed this in the past. Ask that they use our version"
>>
>> 2. "They bundle a TON of JARs, many of which we provide. We may be able to work with this, but ideally you will want to use our jars where possible."
>>
>
> I have to admit, these objections applies to Debian too. One of the
> issues with embedding other libraries/applications into another
> application is that it makes it harder to for us to fix security issues.
If there is a security issue in Application A, then it's Application A vendor to provide a fix for this issue, even if this issue comes from one of the jars embedded in it.
>> a. We're not using a "stock" Tomcat distribution, but one "patched" by adding a few jars in the "lib" directory, which means that other applications which would like to use the same tomcat instance could end-up with unexpected behaviour.
>>
>
> Could these changes possible by ported to the standard Tomcat (that may
> require you to re-license your changes under Apache-2.0, which the
> Tomcat upstream uses as I recall)?
Actually we don't change the Tomcat source code, but we add several more JARs to the Tomcat lib, for instance to add transaction control, OSGi support, H2 database with fulltext indexing support, etc.
These changes must be (for some reason that I trust my developers / CTO to be right about) done in a global way and change the tomcat behavior, at the risk of breaking the other webapps (that are usually just a WAR) that would be installed in the same Tomcat instance.
>> b. We're not using any version of Tomcat, but the one that has been proven (by our test suite and manual QA process) to work properly. While it's probable that other versions of Tomcat could also work, we have no proof of it will unless we base our own "standard" distribution on the exact Tomcat version that's shipped with Ubuntu.
>>
>
> A lot of the Ubuntu contributors for Java also contribute to the Debian
> Java team, which means that Ubuntu usually ships (about) the same
> version as Debian has in the archive when Ubuntu starts its freeze. The
> thing is that Debian usually do not release anyway near the same date as
> Ubuntu since we have very different release processes.
Which means to go this route we would have to support several different Tomcat version, which we haven't the resources to do.
>
>>
>> [...]
>>
>> We understand that this looks counter-intuitive to the Linux / C / C++ developers, but our experience with open source Java development is that you have to be very careful about changing the version of your librairies.
>>
>
> We have had some similar experiences with this. In fact there was a
> proposal at DebConf10 to assist us with catching some of these issues by
> introducing something like a "SONAME" for Java Libraries.
> In fact I hope we can go beyond that and actually implement something
> similar to the "symbols" files we have for C/C++ libraries.
>
> In case you are not familiar with the symbols files; we have files that
> lists all the public symbols of shared libraries. They can both be used
> to calculate the minimum dependency version based on which symbols an
> application uses and also catch some API/ABI breakages.
IIUC, we don't trust this kind of automated dependency computation. The only dependencies we trust are the one we have validated through unit testing + integration testing + selenium testing + manual testing + community feedback on release candidates.
>
>> [...]
>>
>> I understand that these issues put us at odds with the Debian Java policies, and I understand mostly why these policies are in place coming myself from a Linux background (as a Slackware / Red Hat / Mandriva / Debian and Ubuntu user, and having made a few dozens of RPM packages in the past and a couple .deb).
>>
>> But I'm also convinced that the problems I'm mentioning are not specific to Nuxeo, but actually common to most large server-side Java (EE) applications.
>>
>
> Indeed; in fact any large application either with a company or a
> foundation behind it usually have this kind of issue.
The fact that a company or foundation is behind an application seems an orthogonal concern to me.
The problem lies with large (and complex) applications, which tend to be developed by companies or foundations.
>> I don't know how many (if any) such applications have been already packaged into Debian. The only one I know is openbravo-erp from the Ubuntu partner repository (http://archive.canonical.com/ubuntu/pool/partner/o/openbravo-erp/), which already embeds 92 jars:
>>
>> darkstar% dpkg --contents openbravo-erp_2.50MP-25EU1-1maverick1_all.deb > /tmp/openbravo-erp.contents
>> darkstar% egrep "\.jar$" /tmp/openbravo-erp.contents| wc -w
>> 92
>>
>
> Btw, I am not sure if that method works as intended if the package
> merely symlinks to other jar files that are available in other packages
> (not intended to undermine your example, which I have not checked - just
> a heads up).
Only one of them is:
darkstar# dpkg --contents openbravo.deb > /tmp/openbravo-erp.contents darkstar# egrep "\.jar$" /tmp/openbravo-erp.contents
-rw-r--r-- root/root 250643 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-core/lib/openbravo-core.jar
-rw-r--r-- root/root 6608 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/dbmanager.jar
-rw-r--r-- root/root 65261 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/jakarta-oro-2.0.8.jar
-rw-r--r-- root/root 207723 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/commons-lang-2.1.jar
-rw-r--r-- root/root 324441 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/postgresql-jdbc3-8.2.jar
-rw-r--r-- root/root 718907 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/dbsourcemanager.jar
-rw-r--r-- root/root 26514 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/stax-api-1.0.1.jar
-rw-r--r-- root/root 486522 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/dom4j-1.4.jar
-rw-r--r-- root/root 474464 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/wstx-asl-3.0.2.jar
-rw-r--r-- root/root 242227 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/database/lib/commons-betwixt-0.8.jar
-rw-r--r-- root/root 6608 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-db/build/lib/dbmanager.jar
-rw-r--r-- root/root 422533 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-wad/lib/openbravo-wad.jar
-rw-r--r-- root/root 23328 2011-01-08 01:42 ./usr/share/openbravo-erp/src/src-trl/lib/openbravo-trl.jar
-rw-r--r-- root/root 443432 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/antlr-2.7.6.jar
-rw-r--r-- root/root 92015 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/antlr-runtime-3.0.jar
-rw-r--r-- root/root 11816 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ant-launcher.jar
-rw-r--r-- root/root 307860 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jcommon-1.0.13.jar
-rw-r--r-- root/root 42492 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-pool.jar
-rw-r--r-- root/root 1599570 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/axis.jar
-rw-r--r-- root/root 216638 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.eclipse.emf.ecore.xmi_2.4.0.v200806091234.jar
-rw-r--r-- root/root 352668 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/log4j-1.2.8.jar
-rw-r--r-- root/root 45124 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/xmlrpc-client-3.1.jar
-rw-r--r-- root/root 188671 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-beanutils-1.7.jar
-rw-r--r-- root/root 456234 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.openarchitectureware.core.expressions_4.3.1.20080910-1400PRD.jar
-rw-r--r-- root/root 2111580 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/batik.jar
-rw-r--r-- root/root 3555707 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jdtcore-3.1.0.jar
-rw-r--r-- root/root 34875 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ws-commons-util-1.0.1.jar
-rw-r--r-- root/root 187370 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.eclipse.emf.common_2.4.0.v200806091234.jar
-rw-r--r-- root/root 109043 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-io-1.4.jar
-rw-r--r-- root/root 356519 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/mail.jar
-rw-r--r-- root/root 1311979 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jfreechart-1.0.10.jar
-rw-r--r-- root/root 1067396 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/iText-2.1.3.jar
-rw-r--r-- root/root 248383 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/tika-core-0.6.jar
-rw-r--r-- root/root 1223877 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/xercesImpl.jar
-rw-r--r-- root/root 104038 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/xmlrpc-common-3.1.jar
-rw-r--r-- root/root 411090 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/xstream-1.3.jar
-rw-r--r-- root/root 97704 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/servlet-api.jar
-rw-r--r-- root/root 313898 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/dom4j-1.6.1.jar
-rw-r--r-- root/root 46725 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-codec-1.3.jar
-rw-r--r-- root/root 27256 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-fileupload.jar
-rw-r--r-- root/root 56702 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jettison-1.0.1.jar
-rw-r--r-- root/root 559366 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-collections.jar
-rw-r--r-- root/root 29836 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.apache.commons.cli_1.0.0.jar
-rw-r--r-- root/root 94649 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.eclipse.equinox.common_3.4.0.v20080421-2006.jar
-rw-r--r-- root/root 660390 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jxl-2.6.jar
-rw-r--r-- root/root 244646 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.eclipse.text_3.4.0.v20080605-1800.jar
-rw-r--r-- root/root 26708 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/catalina-ant.jar
-rw-r--r-- root/root 3762982 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/hibernate3.jar
-rw-r--r-- root/root 3056 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ant-apache-log4j.jar
-rw-r--r-- root/root 37064 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/lam-client.jar
-rw-r--r-- root/root 1465682 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/fop.jar
-rw-r--r-- root/root 58880 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.openarchitectureware.core.emftools_4.3.1.20080910-1400PRD.jar
-rw-r--r-- root/root 52915 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-logging-1.1.jar
-rw-r--r-- root/root 31191 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jaxrpc.jar
-rw-r--r-- root/root 446063 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/quartz-1.6.2.jar
-rw-r--r-- root/root 143602 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-digester-1.8.jar
-rw-r--r-- root/root 208048 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ehcache-1.2.3.jar
-rw-r--r-- root/root 160862 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.openarchitectureware.core.workflow_4.3.1.20080910-1400PRD.jar
-rw-r--r-- root/root 324238 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/cglib-nodep-2.1_3.jar
-rw-r--r-- root/root 2082142 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jasperreports-3.0.1.jar
-rw-r--r-- root/root 324441 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/postgresql-jdbc3-8.2.jar
-rw-r--r-- root/root 158555 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/barcode4j-fop-ext-0.20.5-complete.jar
-rw-r--r-- root/root 126771 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/wsdl4j-1.5.1.jar
-rw-r--r-- root/root 73425 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/avalon-framework-4.1.5.jar
-rw-r--r-- root/root 105672 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/nekohtml.jar
-rw-r--r-- root/root 8812 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jta.jar
-rw-r--r-- root/root 436697 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ant-nodeps.jar
-rw-r--r-- root/root 290823 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.openarchitectureware.core.xpand2_4.3.1.20080910-1400PRD.jar
-rw-r--r-- root/root 64731 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/hybridlabs-beautifier-1.1.9.jar
-rw-r--r-- root/root 960134 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/jalopy-1.5-rc3p1.jar
-rw-r--r-- root/root 1034902 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/org.eclipse.emf.ecore_2.4.0.v200806091234.jar
-rw-r--r-- root/root 55932 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/activation.jar
-rw-r--r-- root/root 1323005 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ant-1.7.1.jar
-rw-r--r-- root/root 107631 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-dbcp.jar
-rw-r--r-- root/root 194354 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/xml-apis.jar
-rw-r--r-- root/root 1988051 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/ojdbc6.jar
-rw-r--r-- root/root 32784 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/axis-ant.jar
-rw-r--r-- root/root 250643 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/openbravo-core.jar
-rw-r--r-- root/root 5893 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/renderFoRmi.jar
-rw-r--r-- root/root 18979 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/saaj.jar
-rw-r--r-- root/root 71442 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/runtime/commons-discovery-0.2.jar
-rw-r--r-- root/root 361757 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/google-collections.jar
-rw-r--r-- root/root 871260 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/js.jar
-rw-r--r-- root/root 109043 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/commons-io-1.4.jar
-rw-r--r-- root/root 74204 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/smartsprites-0.2.1-alpha.jar
-rw-r--r-- root/root 812720 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/ob-rhino-1.6R7.jar
-rw-r--r-- root/root 850797 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/yuicompressor-2.4.2.jar
-rw-r--r-- root/root 261809 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/commons-lang-2.4.jar
-rw-r--r-- root/root 338488 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/commons-math-1.2.jar
-rw-r--r-- root/root 6662 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/YUIAnt.jar
-rw-r--r-- root/root 116799 2011-01-08 01:42 ./usr/share/openbravo-erp/src/lib/build/junit.jar
lrwxrwxrwx root/root 0 2011-01-08 01:42 ./usr/share/tomcat6/lib/tools.jar -> ../../../lib/jvm/java-6-sun/lib/tools.jar
>
>> This is less than us (more than 250 third-party jars + 187 of our owns), but this is just because their application is less complex than ours, not because they are packaging it differently.
>>
>
> I am not familiar with the Ubuntu partner repository and requirements
> for using that, so if openbravo is from the partner repository I have
> very little means to comment on the sanity of that package.
Indeed, do you have examples of large scale (i.e. > 50 jars, > 50 MB) Java EE application that are already present in Debian, so we can look how the problems have been solved in their case ?
S.
--
Stefane Fermigier, Founder and Chairman, Nuxeo
Open Source, Java EE based, Enterprise Content Management (ECM)
http://www.nuxeo.com/ - +33 1 40 33 79 87 - http://twitter.com/sfermigier
Join the Nuxeo Group on LinkedIn: http://linkedin.com/groups?gid=43314
New Nuxeo release: http://nuxeo.com/dm54
"There's no such thing as can't. You always have a choice."
Reply to: