Bug#653061: libgs9 exports symbols conflicting with the same symbols in libjpeg8

[Jonathan Nieder <jrnieder@gmail.com>]

Bill Allombert wrote:

> The short answer is that it is not possible.

Alas.  Thanks for checking.

*checks*  Wait, is the long answer that it _is_ possible? :)  The
"mem" field seems to do exactly what I'd want.

> But you can use versionned symbols.

Do you mean that libgs should explicitly override
jpeg_get_small@LIBJPEG_8.0 et al instead of using unversioned symbols?
If so, I'd agree, but I don't think it would help with this crash.

Please feel free to skip the following background[*] unless morbidly
curious.

Thanks much,
Jonathan

[*] In ghostscript, the context object when using libjpeg is
the jpeg_compress_data struct.  The notable fields for us at the moment
are:

	typedef struct jpeg_compress_data_s {
		...
		gs_memory_t *memory;	/* heap for library allocations */
		...
		struct jpeg_compress_struct cinfo;
		...
	} jpeg_compress_data;

Ghostscript is designed to be _very_ flexible about the environment it
runs in, since one of the target environments is printer firmware and
saving money on processing power or memory in that context is
desirable.  In particular, ghostscript maintains a private heap to
avoid overhead from, for example, unnecessarily maintaining alignment
of strings.

A vtable with memory management routines can be found in the
jpeg_compress_data::memory field.  But the jpeg_compress_struct object
provided by non-ghostscript callers is not part of a larger
jpeg_compress_data, so attempts to downcast and access that field
segfault.

Luckily, libgs's custom allocator support is not part of the public
interface documented in its headers, so we are not too constrained in
solving this.

Possible fixes:

 a) stop using a custom allocator for libgs's jpeg support

 b) set custom allocators in the jpeg_compress_struct::mem field
    instead of globally overriding the system-dependent memory manager

 c) use a magic value for jpeg_compress_struct::client_data to
    distinguish between ghostscript and non-ghostscript callers to the
    memory manager and fall back to the stock libjpeg versions for
    non-ghostscript calls

Of these, (b) seems the most appealing, but maybe (a) is a better
choice.