Bug#833193: closed by Sean Whitton <spwhitton@spwhitton.name> (Closing inactive ITP RFSs)
Hi Sean,
Thanks for your feedback. Below are:
* some responses to your questions
* a description of some major changes recently incorporated in the
package
* some questions about the long-term plans of a fully featured
(non-minimal) Chapel package.
Note that we have upgraded our package to Chapel 1.15, and changed
the source package name to 'chapel' rather than 'chapel-minimal',
which can be found here:
https://mentors.debian.net/package/chapel
Responses to questions
----------------------
> > 7. The package is not compliant with the FHS. Almost everything is
> > installed into /usr/lib/chapel, plus a symlink from /usr/bin/chpl into
> > /usr/lib/chapel (only to the 64-bit version?). The main executable
> > needs to be installed into /usr/bin/chpl. Please take a look at Debian
> > policy section 9.1.1 and install the files appropriately.
> The basic distinction is that /usr/share is for arch-independent files,
> /usr/lib for arch-dependent files or a mix of arch-independent and
> arch-dependent if necessary.
> The CHPL_HOME problem ought to be fixed by patching chapel to look in
> different places for these different resources. For example, you can
> have it look in /usr/share/chapel for the modules/ subdir. Is there
> something about chapel's design that makes this difficult?
We're investigating updating Chapel to work with an installation structure
more normal for Debian.
We're currently pursuing:
runtime (.a and .o files) in /usr/lib/chapel/1.15/runtime/lib
runtime includes (.h files) in /usr/lib/chapel/1.15/runtime/include
compiler binary (chpl) in /usr/bin
modules/, utils/, make/ in /usr/share/chapel/1.15
> You are installing lots of *.h, *.c and *.py files into
> /usr/lib/chapel. What is the purpose of this?
With the changes we are working on, we will no longer install .py files
to /usr/lib/chapel. We'll install the .py files to /usr/share/chapel.
We will however continue to install some .h files to /usr/lib/chapel or
/usr/share/chapel. We will explain more in answer to your next questions.
There is a single .c file we might need to also install to
/usr/lib/chapel. See below.
> How is the C source code necessary for compiling chapel programs?
> *.h files are supposed to be in /usr/include/ ... again, could you
> explain how they are necessary for compiling chapel programs?
The Chapel compiler normally functions by translating Chapel source code
to C. The headers in /usr/lib/chapel/runtime/include are necessary for
this generated C code to build with the Chapel runtime that is already
built (and stored in /usr/lib/chapel/runtime/lib). These headers could
be architecture-specific, so we intend to store them in /usr/lib/chapel
rather than /usr/share/chapel.
These headers are not meant to be user-visible for general C program
development. Instead, they are only useful internally to the Chapel
compiler. For this reason, we think it would be a mistake to make them
generally available in /usr/include.
In some configurations (namely with LLVM support), the Chapel compiler
needs a sigle .c file in order to function correctly, as it is building
upon the work of a C compiler. For this reason, we may include a C file
in /usr/lib/chapel or /usr/share/chapel. The other C files you observed
have been removed from the upstream project.
We can see in GCC a precedent for this sort of thing; for example there
is a /usr/lib/gcc/x86_64-linux-gnu/6.3.0/include directory on my system.
Based on discussion in #debian-mentors, we're including a version number
as in /usr/lib/chapel/<vers> and /usr/share/chapel/<vers>. The reason
for this is that since Chapel is a language, people often need to be
able to install more than one version at a time.
> > > 11. I'm not an expert on multi-arch issues but this package seems to be
> > > targeted only at 32-bit and 64-bit Linux machines, just from looking at
> > > the 'linux32' and 'linux64' directories you have a lot of. Debian
> > > supports lots of other architectures, and the package should work on
> > > those. Is that something that can be fixed?
> > I have not made any changes in response to this feedback.
> > This is a way in which Chapel distinguishes platforms, which are independent
> > of the architectures Debian distinguishes. I would expect all Debian-supported
> > architectures to be categorized as either 'linux32' or 'linux64' platforms
> > for Chapel's purposes.
> > For a list of our platforms, see our documentation:
> >
> > http://chapel.cray.com/docs/1.14/usingchapel/chplenv.html#chpl-host-platform
>
> It seems that things have changed since we last spoke. I built the
> package and I now get paths like this:
>
> /usr/lib/chapel/runtime/src/threads/pthreads/gen/linux64.gnu.arch-native.loc-flat.comm-none.tasks-fifo.tmr-generic.unwind-none.mem-cstdlib.atomics-intrinsics.gmp-none.hwloc-none.re-none.wide-struct.fs-none/
>
> According to the docs you linked to, a value of CHPL_HOST_PLATFORM is
> meant to be much shorter than this ... or am I misunderstanding?
The paths of built runtime libraries are encoded with the
build-configuration information. That is what you are observing above.
Note that, of all of that configuration information, 'linux64' is the
only platform-distinguishing configuration. The other path components
reflect the configuration of Chapel that was built.
Do you feel that we have addressed your concerns about the FHS?
Major changes in package
------------------------
As noted above, we've pursued a more traditional 'make install' approach
for Chapel installation to conform to the FHS more closely. This patch
is currently under review upstream as well.
Additionally, we have reduced the amount of contents stripped out of the
source package, and split our package up into several packages, such as
'chapel-minimal-compiler', 'chapel-minimal-runtime', and 'chapel-doc'.
This is anticipating the desire to distribute multiple build
configurations for Chapel that can all be built from a single source
package, e.g. a non-minimal Chapel. The license files within copyright
have been updated to more of the third-party licenses that we are
bundling (see next section for more details).
You can track the progress of these changes at:
github.com/chapel-lang/chapel-packaging.
Do these changes sound good to you?
How to create a non-minimal Chapel package?
-------------------------------------------
1. Chapel uses the Sandia qthreads user-level tasking library by
default. But, it is not enabled in the minimal package we are
building so far. As a result, Chapel programs using the minimal
package will have worse performance and will potentially deadlock if
they use too many tasks.
The process of building a Chapel source release normally compiles
qthreads with non-default configurations for functionality and
performance. Functionality configurations include: guard pages,
scheduler, disable-spawn-cache, and with-hwloc, and numa-vs-flat. A
qthreads Debian package does not exist, and would be very complex,
given the matrix of configurations it would need to support as
packages. For this reason, we believe it would be best for our
source package to rebuild qthreads in the configuration that Chapel
supports.
We could potentially:
* re-build qthreads from source in our package build process
* make a chapel-qthreads package built with our configuration
* ... something else you might know about ?
Note that we've updated the Debian package to includes the qthreads
source code and the appropriate entries in the copyright file. We're
hoping for feedback on that change.
2. Chapel includes a fork of re2 that includes patches necessary for
complete formatted I/O functionality. Since the minimal package does
not include re2 or these patches, use of the Regexp library as well
as some formatted I/O features will not function.
We attempted to contribute these patches to the main re2 project,
but the upstream maintainer doesn't appear to be willing to accept
the changes. How should we handle this situation?
Note that these changes add to the RE2 API but do not affect
existing functionality.
See https://groups.google.com/forum/#!topic/re2-dev/gyZ6ukLJm30
We could potentially:
* re-build re2 from source in our package build process
* make a chapel-re2 package built with our fork
* ... something else you might know about ?
We will need to answer similar questions for other dependencies besides
qthreads and re2. To give a sense of the challenges in this area, we
have created a table that shows how Chapel uses third-party code:
patches = Non-upstream patches applied for functionality/performance
configs = Configured in a non-default way for functionality/performance
dpkg = Debian package exists
sys-support = Chapel can support a system-installed version, e.g.
CHPL_GMP=system. If not, a patch would be required to depend on
an external Debian package.
func = Necessary for functionality
perf = Improves performance
port = Expands portability (to ARM in the one case below)
=======================================================
| 3-party | patches | configs | dpkg | sys-support|
|-----------------------------------------------------|
| qthreads | none | func, perf | N | N |
| jemalloc | none | perf | Y | WIP |
| hwloc | none | default | Y | WIP |
| gasnet | port | port, perf | N | N |
| gmp | perf | default | Y | Y |
| re2 | func | default | Y | N |
| llvm | none* | default | Y | Y |
=======================================================
* the Chapel LLVM support currently uses an LLVM header file that is not
installed.
Note that there are some other minor caveats not covered in this table,
e.g. jemalloc Debian package is 3.6 and Chapel relies on 4.5 for full
functionality.
How should we handle the third-party dependencies as we expand beyond
just chapel-minimal?
Look forward to hearing back from you.
Thanks,
Ben
Reply to: