[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Third party code inside Ugene

Hi Andreas,
I see you still have UGENE_USE_BUNDLED_ZLIB=1 in your qmake file.  Could you try to set it to 0 (UGENE_USE_BUNDLED_ZLIB=0) and build again? 

Thank you,
Yuliya.


On 02/01/2019 06:10 PM, Andreas Tille wrote:

Hi again,

I took the time and re-added

     src/libs_3rdparty/zlib/zlib.pr[io]

and tried to set the variable below.  Unfortunately it does not work with
removed zlib code:

...
cd src/libs_3rdparty/zlib/ && ( test -e Makefile || /usr/lib/qt5/bin/qmake -o Makefile /build/ugene-1.31.1+dfsg1/src/libs_3rdparty/zlib/zlib.pro 'QMAKE_CFLAGS_RELEASE=-g -O2 -fdebug-prefix-map=/build/ugene-1.31.1+dfsg1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' 'QMAKE_CFLAGS_DEBUG=-g -O2 -fdebug-prefix-map=/build/ugene-1.31.1+dfsg1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' 'QMAKE_CXXFLAGS_RELEASE=-g -O2 -fdebug-prefix-map=/build/ugene-1.31.1+dfsg1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' 'QMAKE_CXXFLAGS_DEBUG=-g -O2 -fdebug-prefix-map=/build/ugene-1.31.1+dfsg1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' 'QMAKE_LFLAGS_RELEASE=-Wl,-z,relro -Wl,-z,now' 'QMAKE_LFLAGS_DEBUG=-Wl,-z,relro -Wl,-z,now' QMAKE_STRIP=: PREFIX=/usr QMAKE_CFLAGS_ISYSTEM= QMAKE_CXXFLAGS_ISYSTEM= UGENE_WITHOUT_NON_FREE=1 UGENE_LRELEASE=lrelease-qt5 UGENE_LUPDATE=lupdate-qt5 UGENE_USE_BUNDLED_ZLIB=1 ) && make -f Makefile
WARNING: Failure to find: src/adler32.c
WARNING: Failure to find: src/compress.c
WARNING: Failure to find: src/crc32.c
WARNING: Failure to find: src/deflate.c
WARNING: Failure to find: src/gzio.c
WARNING: Failure to find: src/infback.c
WARNING: Failure to find: src/inffast.c
WARNING: Failure to find: src/inflate.c
WARNING: Failure to find: src/inftrees.c
WARNING: Failure to find: src/minigzip.c
WARNING: Failure to find: src/trees.c
WARNING: Failure to find: src/uncompr.c
WARNING: Failure to find: src/zutil.c
WARNING: Failure to find: src/crc32.h
WARNING: Failure to find: src/deflate.h
WARNING: Failure to find: src/inffast.h
WARNING: Failure to find: src/inffixed.h
WARNING: Failure to find: src/inflate.h
WARNING: Failure to find: src/inftrees.h
WARNING: Failure to find: src/trees.h
WARNING: Failure to find: src/zconf.h
WARNING: Failure to find: src/zlib.h
WARNING: Failure to find: src/zutil.h
WARNING: Failure to find: src/adler32.c
WARNING: Failure to find: src/compress.c
WARNING: Failure to find: src/crc32.c
WARNING: Failure to find: src/deflate.c
WARNING: Failure to find: src/gzio.c
WARNING: Failure to find: src/infback.c
WARNING: Failure to find: src/inffast.c
WARNING: Failure to find: src/inflate.c
WARNING: Failure to find: src/inftrees.c
WARNING: Failure to find: src/minigzip.c
WARNING: Failure to find: src/trees.c
WARNING: Failure to find: src/uncompr.c
WARNING: Failure to find: src/zutil.c
WARNING: Failure to find: src/crc32.h
WARNING: Failure to find: src/deflate.h
WARNING: Failure to find: src/inffast.h
WARNING: Failure to find: src/inffixed.h
WARNING: Failure to find: src/inflate.h
WARNING: Failure to find: src/inftrees.h
WARNING: Failure to find: src/trees.h
WARNING: Failure to find: src/zconf.h
WARNING: Failure to find: src/zlib.h
WARNING: Failure to find: src/zutil.h
make[2]: Entering directory '/build/ugene-1.31.1+dfsg1/src/libs_3rdparty/zlib'
make -f Makefile.Release
make[3]: Entering directory '/build/ugene-1.31.1+dfsg1/src/libs_3rdparty/zlib'
make[3]: *** No rule to make target 'src/adler32.c', needed by '_tmp/obj/release/adler32.o'.  Stop.
make[3]: Leaving directory '/build/ugene-1.31.1+dfsg1/src/libs_3rdparty/zlib'
make[2]: *** [Makefile:40: release] Error 2
...

Kind regards

        Andreas.


On Fri, Feb 01, 2019 at 08:15:29AM +0700, Yuliya Algaer wrote:

Hi Andreas,

Lets start with zlib first: it is easier that way we do not use any custom
flags, renaming and even have initial support to use system library first.

Check this code from ugene_globals.pri:

# By default, UGENE uses bundled zlib on Windows (libs_3rdparty/zlib) and OS
version on Linux.
# To use bundled version on any platform set UGENE_USE_BUNDLED_ZLIB = 1

defineTest( use_bundled_zlib ) {
     contains( UGENE_USE_BUNDLED_ZLIB, 1 ) : return (true)
     contains( UGENE_USE_BUNDLED_ZLIB, 0 ) : return (false)
     win32 {
         return (true)
     }
     return (false)
}


This way you can switch to the system library: bundled sources will not be
used at all.

So the next steps for Debian packaging could be:

1) Switch from CMake to qmake build. This is how we build UGENE for
production.

2) Try to build it with no bundled zlib. If it works you can completely
remove zlib folder from UGENE sources for Debian and add zlib-dev as the
required dependency.


About Phylip: this is plugin, so it can easily removed from the build &
source code. While most of the 3rd party tools are run from UGENE as
external processes, some of them, and Phylip is the one of them, are
integrated on the source level. Examples of such plugins are: Phylip,
Muscle, HMMER2, KAlign. Unfortunately we can't replace them with pre-built
originals because we integrated them on the source level (modified them).
The good news is that there are only 4-5 of such plugins, while 90% (like
BLAST, HMMER3...) are run as external programs.


Please let me know if your experiment with zlib is successful. If yes, we
can try the same with SQLite.


Kind regards,

Yuliya



On 01/31/2019 02:08 PM, Andreas Tille wrote:

Hi Yuliya,

thanks again for the very productive discussion.

On Thu, Jan 31, 2019 at 08:58:47AM +0700, Yuliya Algaer wrote:

-lugenedbd and -lugenedb: what is the difference between these two?

The first one is debug build of the library. The second one is release. All
libraries in UGENE are built this way.

OK, that's very helpful.  In Debian by default code will be build
including debug information and distributed in a separate package.


I would recommend to check for the existence of zlib on the system that
builds Ugene and if it exists use the zlib from the system.

We can add this check into the build script. But in either case we have to
have zlib sources in the bundle UGENE to use it as a fallback.

Yes, for sure you can do this.  For the packaging it is just extremely
helpful that we can link to the system libraries without needing to
worry about copies in your code.  This saves us from creating patches
and adapting these for every new upstream version.  Finally this kind of
friction is the reason why we are lagging behind your latest version.


Another alternative here is to re-check if some compatible form of zlib is
available via Qt library. I will check it by myself. Thank you for the hint.

That's another option but I know to less about Qt to comment on this.

In case the system has an up to date library there is no need to make use
of private copies.

Same issue as with zlib here. If there is no library on the system we have
to use the one from our sources or ask a user to provide one. UGENE builds
on many platforms. On Windows, for example, there are no SQLite by default
at all.

That's fine in the same way.  We have no problem if the code copy exists
since we can even strip it from the source archive without worrying
whether this might influence the functionality.


I'm not sure whether phylip is the only example

Phylip is a plugin and can easily be excluded from the build.

That's interesting.  How can we make sure that the Debian packaged
phylip is used as plugin?  We really want to provide the full
functionality but I think to do so we have to understand this plugin
system better.


The only real
UGENE dependencies are QT5, zlib & SQLite. If you do not mind that we carry
the last 2 in our sources, but automatically detect and use system ones on
Debian (since we know they are compatible) this task is not very complex and
I will be glad to help you here.

This would be really great and we would appreciate this very much since
it would drastically reduce our work.

By using cmake features you can easily check whether a user has these
libraries installed

We have only unofficial experimental build with CMake and have no active
plans on it. So we have to use qmake here.

Fine with me - whatever you might implement.  While the Debian packaging
is currently using your CMake files this is probably not a good idea if
it is only experimental.  We can adapt to whatever build system you are
using (even if my personal observation from lots of software is that
several projects moved from qmake to cmake.  I can not compare these
systems since I only know these as "consumer" but those upstreams who
switched seem to have their reasons.

Thank you,
Yuliya.

Thanks a lot to you

        Andreas.
Reply to: