Re: [MoM] FAST progress (was Re: [MoM] fast: Add further dependencies to enable chroot / cowbuilder to build)
Hello Shayan,
On Sat, Aug 31, 2019 at 05:30:06PM +0100, Shayan Doust wrote:
> Hello Andreas,
>
> Will extend on your suggestion for the binary examples.
>
> Firstly, I reverted to not including the fonts directory (this is
> redundant and extra license) in doc and instead using the ubuntu family
> fonts package. I will repackage to not include the doc/fonts. The patch
> will instead look in /usr/share/fonts. This means that now, the test
> suite should run (I see no more of that error / assertion you got with
> regards to fonts and graphics but the test suite itself needs testing).
>
> I will not add the sources to the binary because even though these are
> examples, the full tutorial and walkthrough is available on the wiki (I
> will write some documentation guiding the user with using the example
> binaries and referring to the wiki). Additionally, the binaries
> themselves (especially the segmentations and visualisations) can be used
> for production as they rely on a dataset as input (presumably these data
> can be created by the user themselves so they do not have to strictly
> use the example data). These binaries are more or less multi-purpose.
Those binaries that have a purpose as user utilities should probably go
to /usr/bin and adding a manpage (perhaps by using createmanpages[1]).
Kind regards
Andreas.
[1] https://salsa.debian.org/med-team/community/helper-scripts/blob/master/createmanpages
> Thanks & kind regards,
> Shayan Doust
>
> On 30/08/2019 20:46, Shayan Doust wrote:
> > Hello Andreas,
> >
> > Need a suggestion for a directory because something is slightly unclear.
> > I am using the contents in doc/fonts and doc/images and copying these
> > over to /usr/share/doc/FAST/assets. FAST is designed to dynamically look
> > for this via <current library (*.so) location>/../share/doc/FAST. This
> > is dynamic because these assets are needed both at compilation time, and
> > during runtime and the build environment will not have these installed
> > statically in /usr/share/doc/FAST/assets.
> >
> > The only time I see these assets ever coming to use are the examples and
> > the test suite itself (FAST likes to set and display its own font from
> > the font directory and images from the image directory). In a sense,
> > this is needed for the functionality however on the other hand, these
> > assets seem to be useful from the example code on the wiki and could
> > maybe still be classed as documentation in itself.
> >
> > Should I keep using /usr/share/doc/FAST/assets or should I instead use
> > /usr/lib/FAST in cases like these.
> >
> > Thanks for your time,
> > Shayan Doust
> >
> > On 30/08/2019 17:26, Shayan Doust wrote:
> >> Hello Andreas,
> >>
> >>> Unable to open directory /usr/../doc//fonts
> >>
> >> That's very annoying. The docs file is expected to be installed in /usr
> >> and contains "FAST" image branding and some other assets that are used
> >> in the program. I will see if FAST allows for dynamic pathing and where
> >> it actually wants me to install the doc folder. The reason I did not
> >> install the doc folder under /usr/share and excluded it for now is
> >> because it serves no purpose to the end user. So now I will look into it
> >> and if not, introduce another patch to cater for the doc directory to be
> >> under a different location.
> >>
> >> SIGSEVG probably because it expects these "doc" assets. Luckily there is
> >> no early assertion of CL which means the CL patch probably worked. I
> >> will try offloading this onto my build server, and manually testing the
> >> test suite on another spare laptop and hopefully opencl is supported.
> >>
> >>> A general remark: The package libfast-examples contains compiled
> >>> executables in /usr/lib/fast. Users usually expect examples in
> >>>
> >>> /usr/share/doc/PKGNAME/examples
> >>>
> >>> (that's where dh_installexamples puts files). Usually these are not
> >>> binaries. It depends a bit what we want to teach users by these
> >>> examples. If it is how to develop with libfast-dev than it would be
> >>> better to provide the source code files including a Makefile to compile
> >>> these examples (and may be change something and try what's happening).
> >>>
> >>> If something else should be demonstrated and these executables might
> >>> have some extra value I'd consider it necessary to add some information
> >>> under /usr/share/doc/libfast-examples/examples or similar to inform
> >>> users where they can find those files and what to do.
> >>
> >> Ahh I see. I am still fairly uninformed as to what these examples
> >> represent (until I rectify this opencl issue). I thought these binaries
> >> would be nice because they represent the examples on the wiki[1].
> >>
> >>> If it is how to develop with libfast-dev than it would be
> >>> better to provide the source code files including a Makefile to
> >>> compile these examples (and may be change something and try what's
> >>> happening).
> >>
> >> That could work, although that would mean writing the examples and
> >> writing my own makefile which could work as this is not readily
> >> available, however I think a simpler option would be the second point
> >> you suggested. Maybe adding a documentation / some file under
> >> /usr/share/doc/libfast-examples/examples to both explain the purpose,
> >> refer to the wiki and the new example location. I will play with these
> >> ideas once I figure out this opencl.
> >>
> >> Kind regards & thanks for your help,
> >> Shayan Doust
> >>
> >> [1]: https://github.com/smistad/FAST/wiki/Examples
> >>
> >> On 30/08/2019 17:05, Andreas Tille wrote:
> >>> Hi Shayan,
> >>>
> >>> please git pull for two commits (library name to make sure fast-examples
> >>> can be installed and Standards-Version).
> >>>
> >>> On Thu, Aug 29, 2019 at 01:22:51AM +0100, Shayan Doust wrote:
> >>>> Hello Andreas,
> >>>>
> >>>>> In libhmsbeagle I had to deal with this in build time tests. May be
> >>>>> you check the Build-Depends for some additional inspiration. No idea
> >>>>> whether this might help.
> >>>>
> >>>> Many thanks for your reply. This has made some slight difference. A few
> >>>> things to note.
> >>>>
> >>>> Running clinfo returns my CPU as a platform, however "what():
> >>>> clGetPlatformIDs" is still thrown. Running the test binary, I see some
> >>>> more information. I am not sure the significance of these as opposed to
> >>>> the separate example binaries, however here is a short snippet:
> >>>>
> >>>> ...
> >>>>
> >>>> -------------------------------------------------------------------------------
> >>>> /home/shayandoust/Desktop/FAST/fast/source/FAST/Tests/Benchmarks.cpp:106
> >>>> ...............................................................................
> >>>>
> >>>> /home/shayandoust/Desktop/FAST/fast/source/FAST/Tests/Benchmarks.cpp:106: FAILED:
> >>>> due to unexpected exception with message:
> >>>> There was an error while getting OpenCL devices: clGetDeviceIDs
> >>>>
> >>>> INFO: QApp already exists..
> >>>> INFO: Device manager initialize..
> >>>> INFO: Initial GLX context 0x5574dc453ae0
> >>>> INFO: Found 1 OpenCL platforms.
> >>>> INFO: 1 platforms selected for inspection.
> >>>> INFO: Platform 0: Mesa
> >>>> INFO: This platform has
> >>>> -------------------------------------------------------------------------------
> >>>> Pipeline C
> >>>> -------------------------------------------------------------------------------
> >>>> /home/shayandoust/Desktop/FAST/fast/source/FAST/Tests/Benchmarks.cpp:167
> >>>> ...............................................................................
> >>>>
> >>>> /home/shayandoust/Desktop/FAST/fast/source/FAST/Tests/Benchmarks.cpp:167: FAILED:
> >>>> due to unexpected exception with message:
> >>>> There was an error while getting OpenCL devices: clGetDeviceIDs
> >>>>
> >>>> ===============================================================================
> >>>> test cases: 221 | 32 passed | 189 failed
> >>>> assertions: 385 | 196 passed | 189 failed
> >>>
> >>> I get:
> >>>
> >>> $ sh run-unit-test
> >>> Unpacking test data
> >>> Invoking testFAST
> >>> INFO: Creating new QApp
> >>> WARNING: Unable to open the configuration file /usr/fast_configuration.txt. Using defaults instead.
> >>>
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> testFAST is a Catch v1.5.9 host application.
> >>> Run with -? for options
> >>>
> >>> -------------------------------------------------------------------------------
> >>> Airway segmentation
> >>> -------------------------------------------------------------------------------
> >>> /build/fast-3.0.0~rc3+ds/source/FAST/Algorithms/AirwaySegmentation/airwaySegmentationTests.cpp:49
> >>> ...............................................................................
> >>>
> >>> /build/fast-3.0.0~rc3+ds/source/FAST/Algorithms/AirwaySegmentation/airwaySegmentationTests.cpp:49: FAILED:
> >>> due to unexpected exception with message:
> >>> Unable to open directory /usr/../doc//fonts
> >>>
> >>> INFO: QApp already exists..
> >>> INFO: Creating new GL context for computation thread
> >>> WARNING: Your system uses comma as decimal point.
> >>> WARNING: This will now be changed to dot to avoid any comma related bugs.
> >>> INFO: Device manager initialize..
> >>> INFO: Initial GLX context 0x563a2ac8d8c0
> >>> INFO: Found 1 OpenCL platforms.
> >>> INFO: 1 platforms selected for inspection.
> >>> INFO: Platform 0: Intel
> >>> INFO: This platform has 0 available devices in total
> >>> INFO: Looking for GPU devices only.
> >>> INFO: 1 selected.
> >>> INFO: Inspecting device 0 with the name Intel(R) HD Graphics Haswell Ultrabook GT2 Mobile
> >>> INFO: Initial GLX context 0x563a2ac8d8c0
> >>> -------------------------------------------------------------------------------
> >>> Block matching 2D
> >>> -------------------------------------------------------------------------------
> >>> /build/fast-3.0.0~rc3+ds/source/FAST/Algorithms/BlockMatching/Tests.cpp:10
> >>> ...............................................................................
> >>>
> >>> /build/fast-3.0.0~rc3+ds/source/FAST/Algorithms/BlockMatching/Tests.cpp:10: FAILED:
> >>> due to a fatal error condition:
> >>> SIGSEGV - Segmentation violation signal
> >>>
> >>> ===============================================================================
> >>> test cases: 2 | 2 failed
> >>> assertions: 2 | 2 failed
> >>>
> >>>> spent a few hours reading the source and attempting to search up the
> >>>> error message, although I'm trying not to put any additional load on
> >>>> you. If you do lack time, I could always ask in mentors.
> >>>
> >>> The missing font is a bit suspicious but I'm absolutely uninformed
> >>> about all this graphics stuff to be more helpful here.
> >>>
> >>>
> >>> A general remark: The package libfast-examples contains compiled
> >>> executables in /usr/lib/fast. Users usually expect examples in
> >>>
> >>> /usr/share/doc/PKGNAME/examples
> >>>
> >>> (that's where dh_installexamples puts files). Usually these are not
> >>> binaries. It depends a bit what we want to teach users by these
> >>> examples. If it is how to develop with libfast-dev than it would be
> >>> better to provide the source code files including a Makefile to compile
> >>> these examples (and may be change something and try what's happening).
> >>>
> >>> If something else should be demonstrated and these executables might
> >>> have some extra value I'd consider it necessary to add some information
> >>> under /usr/share/doc/libfast-examples/examples or similar to inform
> >>> users where they can find those files and what to do.
> >>>
> >>> Kind regards
> >>>
> >>> Andreas.
> >>>
> >>>> Many thanks,
> >>>> Shayan Doust
> >>>>
> >>>> On 28/08/2019 20:04, Andreas Tille wrote:
> >>>>> On Wed, Aug 28, 2019 at 06:18:59PM +0100, Shayan Doust wrote:
> >>>>>> Just as I anticipated, it looks like it will not work as I use
> >>>>>> virtualbox and debian is essentially virtualised. So not only can I not
> >>>>>> passthrough the host GPU, I am unable to use any drivers which means
> >>>>>> that the error I get even when running any of the fast-examples binaries
> >>>>>> (apart from OpenIGTLinkServer) is "what(): clGetPlatformIDs", which is
> >>>>>> verified by running "clinfo" which returns 0 as the number of platforms
> >>>>>> - it seems like I have hit a dead end unless there is some work around
> >>>>>> which I have my doubts in.
> >>>>>
> >>>>> In libhmsbeagle I had to deal with this in build time tests. May be
> >>>>> you check the Build-Depends for some additional inspiration. No idea
> >>>>> whether this might help.
> >>>>>
> >>>>> Kind regards
> >>>>>
> >>>>> Andreas (offline for the next 20 hours)
> >>>>>
> >>>>>> Kind regards,
> >>>>>> Shayan Doust
> >>>>>>
> >>>>>> On 28/08/2019 16:33, Shayan Doust wrote:
> >>>>>>> Hello Andreas,
> >>>>>>>
> >>>>>>>> Ahhh, that really saves time. I had an extremely busy weekend -
> >>>>>>>> otherwise I would have answered before.
> >>>>>>>
> >>>>>>> No problem. At least I prevented you from answering an old question :).
> >>>>>>>
> >>>>>>>> The symbols file changes are really strange. I was running again in a
> >>>>>>>> diff. I have not commited my changes yet but for the moment I have
> >>>>>>>> deactivated the symbols file since it seems we are playing diff - patch
> >>>>>>>> ping-pong for no good reason.
> >>>>>>>
> >>>>>>> I haven't seen this happen with other libraries so it is strange.
> >>>>>>> Perhaps our environments are different in some way? It still shouldn't
> >>>>>>> matter though as the library itself is one version and not changing.
> >>>>>>>
> >>>>>>>> I admit I have never seen this. D-shlibs enforces
> >>>>>>>> (= ${binary:Version})
> >>>>>>>> and this should be there. Your. May be your change was needed once
> >>>>>>>> the package was arch: all ?
> >>>>>>>
> >>>>>>> You're correct. I changed the depends before I realised the arch was
> >>>>>>> incorrect, and as usual I forgot to backtrack and undo my change to
> >>>>>>> depends. I've just pushed this change.
> >>>>>>>
> >>>>>>>> Testing with opengl is always a nuisance. Even with emulation this
> >>>>>>>> tends to fail unfortunately. But you might like to try and we'll see
> >>>>>>>> how it might turn out.
> >>>>>>>
> >>>>>>> I'll let you know how this goes. This is currently taking the most time
> >>>>>>> for the packaging as opengl seems to be a real pain from past
> >>>>>>> experience. Probably the VM factor doesn't help it as maybe there is
> >>>>>>> some more abstraction. Assuming autopkgtest is also ran on debian
> >>>>>>> servers, I wonder if this issue is still present.
> >>>>>>>
> >>>>>>> Kind regards,
> >>>>>>> Shayan Doust
> >>>>>>>
> >>>>>>> On 28/08/2019 12:33, Andreas Tille wrote:
> >>>>>>>> Hi Shayan,
> >>>>>>>>
> >>>>>>>> On Tue, Aug 27, 2019 at 03:40:04AM +0100, Shayan Doust wrote:
> >>>>>>>>> Hello Andreas,
> >>>>>>>>>
> >>>>>>>>> Just to update you on what has been going on. Please disregard my
> >>>>>>>>> previous email to save you time from writing regarding something redundant.
> >>>>>>>>
> >>>>>>>> Ahhh, that really saves time. I had an extremely busy weekend -
> >>>>>>>> otherwise I would have answered before.
> >>>>>>>>
> >>>>>>>>> I have added some more overrides and updated the symbol file again to
> >>>>>>>>> which the package is now being built successfully :).
> >>>>>>>>
> >>>>>>>> The symbols file changes are really strange. I was running again in a
> >>>>>>>> diff. I have not commited my changes yet but for the moment I have
> >>>>>>>> deactivated the symbols file since it seems we are playing diff - patch
> >>>>>>>> ping-pong for no good reason.
> >>>>>>>>
> >>>>>>>>> A few errors persist via Lintian when I first built. The Lintian errors
> >>>>>>>>> are "arch-independent-package-contains-binary-or-object" referencing
> >>>>>>>>> usr/lib/x86_64-linux-gnu/libFAST.a and
> >>>>>>>>> "triplet-dir-and-architecture-mismatch" referencing
> >>>>>>>>> usr/lib/x86_64-linux-gnu.
> >>>>>>>>>
> >>>>>>>>> For arch-independent-package-contains-binary-or-object: I have changed
> >>>>>>>>> the architecture of libfast-dev from "all" to "any".
> >>>>>>>>
> >>>>>>>> That's definitely correct - architecture should be any.
> >>>>>>>>
> >>>>>>>>> non-binmutable-all-depends-any: For the depends field, I have used
> >>>>>>>>> "libfast0 (>= ${source:Version}), libfast0 (<< ${source:Version}.1~)"
> >>>>>>>>> instead of "libfast0 (= ${binary:Version})". Please advise if this
> >>>>>>>>> should be something else.
> >>>>>>>>
> >>>>>>>> I admit I have never seen this. D-shlibs enforces
> >>>>>>>> (= ${binary:Version})
> >>>>>>>> and this should be there. Your. May be your change was needed once
> >>>>>>>> the package was arch: all ?
> >>>>>>>>
> >>>>>>>>> I have modified the --movedev option of d-shlibmove as we do not need
> >>>>>>>>> the CL include directory as this will clash with OpenCL's headers, and
> >>>>>>>>> the patch I put erradicates the need for this additional FAST's CL
> >>>>>>>>> directory anyways.
> >>>>>>>>
> >>>>>>>> Sounds sensible even if I can not judge about this since I do not
> >>>>>>>> know the internals of this program.
> >>>>>>>>
> >>>>>>>>> I will try run autopkgtest to make sure the patch has
> >>>>>>>>> not missed anything out although I am personally having opengl driver
> >>>>>>>>> issues that is stopping me from doing this for now.
> >>>>>>>>
> >>>>>>>> Testing with opengl is always a nuisance. Even with emulation this
> >>>>>>>> tends to fail unfortunately. But you might like to try and we'll see
> >>>>>>>> how it might turn out.
> >>>>>>>>
> >>>>>>>>> I've written this email in time order while I have been applying some
> >>>>>>>>> commits, so as of right now I do not see anymore Lintian errors or
> >>>>>>>>> warnings. In the meantime, I've also spent the time going through any
> >>>>>>>>> devel documentation I could find with regards to policy and guides.
> >>>>>>>>
> >>>>>>>> I've started a build at home and will send further comments once I'm
> >>>>>>>> home again.
> >>>>>>>>
> >>>>>>>> Kind regards
> >>>>>>>>
> >>>>>>>> Andreas.
> >>>>>>>>
> >>>>>>>>> On 24/08/2019 18:32, Shayan Doust wrote:
> >>>>>>>>>> Hello Andreas,
> >>>>>>>>>>
> >>>>>>>>>> Just to extend the previous email I sent.
> >>>>>>>>>>
> >>>>>>>>>> I have corrupted my VM which means I cannot get the exact error message
> >>>>>>>>>> however off the top of my head, it seems like d-shlib is failing.
> >>>>>>>>>>
> >>>>>>>>>> The error message consists of many lines stating "There is no package
> >>>>>>>>>> matching [<some qt libraries>] and noone provides it, plese report but
> >>>>>>>>>> to d-shlibs maintainer".
> >>>>>>>>>>
> >>>>>>>>>> I am sure you can replicate this. Does this look like a reporable issue
> >>>>>>>>>> or is this related to perhaps some flags you set for d-shlibs?
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Shayan Doust
> >>>>>>>>>>
> >>>>>>>>>> On 23/08/2019 21:28, Shayan Doust wrote:
> >>>>>>>>>>> Hello Andreas,
> >>>>>>>>>>>
> >>>>>>>>>>>> Something which is really strange is that dh_makeshlibs created a diff
> >>>>>>>>>>>> for the symbols file. I applied that diff and started a rebuild - no
> >>>>>>>>>>>> idea why this is happening.
> >>>>>>>>>>>
> >>>>>>>>>>> I've seen this happen a few times and I've ended up rewriting the
> >>>>>>>>>>> existing symbols file with the new one generated as DEBIAN/symbols.
> >>>>>>>>>>>
> >>>>>>>>>>>> I also switched the packaging to d-shlibs a I promised. The good thing
> >>>>>>>>>>>> is that it enforces library packaging policy and thus I was able to spot
> >>>>>>>>>>>> missing "Section" fields in the control file.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks. It makes sense. Admittingly, the thing I've had issues with was
> >>>>>>>>>>> the override option as at the time, did not make sense to me. I'll now
> >>>>>>>>>>> use d-shlibs within future library packages as it does most of the job
> >>>>>>>>>>> itself in a safe way.
> >>>>>>>>>>>
> >>>>>>>>>>>> I'll test this later.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks. It could just be that d-shlib rectifies this as maybe I was
> >>>>>>>>>>> installing the wrong thing. I will try initiate a build as I have not
> >>>>>>>>>>> done so yet.
> >>>>>>>>>>>
> >>>>>>>>>>>> I promised support - specifically when you are tackling such a huge
> >>>>>>>>>>>> beast. ;-)
> >>>>>>>>>>>
> >>>>>>>>>>> Grateful for this :). It makes more sense visualising the current issues
> >>>>>>>>>>> and how you would rectify it, which I can apply to future packaging.
> >>>>>>>>>>>
> >>>>>>>>>>> Kind Regards,
> >>>>>>>>>>> Shayan Doust
> >>>>>>>>>>>
> >>>>>>>>>>> On 23/08/2019 08:36, Andreas Tille wrote:
> >>>>>>>>>>>> Hi Shayan,
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Aug 22, 2019 at 09:09:29PM +0100, Shayan Doust wrote:
> >>>>>>>>>>>>> Just to extend the previous email.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I created another fresh environment and even used pbuilder and the
> >>>>>>>>>>>>> results are the same, that the build succeeds.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Something which is really strange is that dh_makeshlibs created a diff
> >>>>>>>>>>>> for the symbols file. I applied that diff and started a rebuild - no
> >>>>>>>>>>>> idea why this is happening.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I also switched the packaging to d-shlibs a I promised. The good thing
> >>>>>>>>>>>> is that it enforces library packaging policy and thus I was able to spot
> >>>>>>>>>>>> missing "Section" fields in the control file.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Warning: I've commited despite my build did not finished but I wanted
> >>>>>>>>>>>> to push the d-shlibs change soon to get you informed.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Additionally, I have attempted to disable RPATH during the build but the
> >>>>>>>>>>>>> error and RPATH still exist, so I am not too sure. If a solution isn't
> >>>>>>>>>>>>> reached, I will write to debian mentors like you have mentioned. I'm not
> >>>>>>>>>>>>> sure if I can just use dpkg-shlibdeps -l<lib> for each executable but
> >>>>>>>>>>>>> this seems like a fair bit of pain to do for all the libraries and
> >>>>>>>>>>>>> executables that exist and are compiled.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'll test this later.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Many thanks for looking into this & regards,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I promised support - specifically when you are tackling such a huge
> >>>>>>>>>>>> beast. ;-)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Kind regards
> >>>>>>>>>>>>
> >>>>>>>>>>>> Andreas.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
--
http://fam-tille.de
Reply to: