Re: Endianness of data files in MultiArch
Aron Xu <happyaron.xu@gmail.com> writes:
> On Thu, Feb 9, 2012 at 01:35, Simon McVittie <smcv@debian.org> wrote:
>> On 08/02/12 17:22, Aron Xu wrote:
>>> Some packages come with data files that endianness matters, and many
>>> of them are large enough to split into a separate arch:all package if
>>> endianness were not something to care about. AFAIK some maintainers
>>> are not aware of endianness issues in their packages and then just
>>> ignored it (not sure how many, but if any of them are discovered it
>>> should lead to RC bug).
>>
>> Hopefully Jakub Wilk's automatic checks for conflicting files
>> <http://people.debian.org/~jwilk/multi-arch/> will already be picking
>> this up, in cases where the less-used-endianness architectures aren't
>> broken already.
>>
>> If the less-used-endianness architectures are already broken, that's
>> also a bug (potentially an RC one), just like code that compiles but
>> doesn't work on a particular endianness due to other assumptions - and
>> if nobody has noticed it yet, presumably the package doesn't have any
>> users (or regression tests) on those architectures.
>>
>
> Or some of them just gave up because it is "less-used" architecture.
>
>>> It would be great to have some mechanism to
>>> handle such kind of problems in Debian, to avoid forcing those data to
>>> be placed into arch:any package.
>>
>> If the right endianness is critical: libfoo:i386 Depends:
>> libfoo-data-le, libfoo:powerpc Depends: libfoo-data-be, both data
>> packages arch:all, data files in /usr/share/foo/le and /usr/share/foo/be
>> respectively?
I would have them conflict and use the same directory. Otherwise you
need to use different paths in the binary, docs and maybe even
conffiles (which would then be architecture dependend too).
> This looks not very nice, because we need to maintain a list of
> architectures in debian/control, and when new architectures are added
> the package is potentially broken.
If endian dependend data is really a larger issue then introduce a
dpkg-architecture -qDEB_HOST_ENDIANESS
> Also, arch:all packages are usually generated by the uploading DD on
> one architecture, mostly amd64 and i386 today, how can he managed to
> generate be data files if he doesn't have access to such a machine?
> Adding an option to the data generator/parser and make it able to
> generate be/le data on any architecture seems not to be a reasonable
> approach.
That is indeed the biggest problem currently. Also a problem for the
idea of building arch:all packages on buildds. They might not build on
all archs.
>> Or just make sure the data has an endianness marker, and enhance the
>> reading package to do the right byteswapping based on the endianness
>> marker - e.g. this has been discussed for gettext, which ended up just
>> writing out the same endianness on all platforms. Many formats
>> (particularly those that originated on Windows) are always
>> little-endian, and big-endian platforms reading them just take the minor
>> performance hit; formats that respect "network byte order" have the
>> opposite situation.
>>
>
> This is valid for most-used applications/formats like gettext, images
> that are designed to behave in this way, but on the contrary there are
> upstream that don't like to see such impact, especially due to the
> complexity and performance impact.
>
> Currently I am using arch:any for data files which aren't be affected
> with multiarch, i.e. not "same" or "foreign". For endianness-critical
> data that is required to make a library working, I have to force them
> to be installed into /usr/lib/<triplet>/$package/data/ and mark them
> as "Multiarch: same", this is sufficient to avoid breakage, but again
> it consumes a lot of space on mirror.
>
> I thought about something like /usr/share/$package/data/{be,le} in
> arch:all, but appears to be not a reasonable solution because we need
> to modify the data generator/parser.
It should be possible to build a converter or generator that can output
either endianess. So you could have a single arch:all package with both
/usr/share/$package/data/{be,le} in it or to generate the right
endianness on install. That way the "performance impact" argument is non
existant.
MfG
Goswin
Reply to: