Re: perl: 64-bit integers and long doubles

[Florian Weimer <fw@deneb.enyo.de>]

* Stefan Fritsch:

> I may be a bit late to this discussion, but aren't 64bit ints (and 
> especially pack/unpack "Q") very useful for 64bit file pointers and 
> such? IMHO, this means that they would also be very useful on 
> "smaller" architectures like arm.

Yes, they are, and that's where I have run into incompatibilities.  In
theory, Perl scalars degrade to doubles when the 32 bit range is
exceeded on 32 bit architectures, which gives you around 52 bits.
This should be sufficient for file sizes.  But the differing internal
representation sometimes becomes visible and causes issues.

> In any case, there isn't really a strong connection with 64bit 
> instructions in the processor. An executable compiled for 32bit cannot 
> use those instructions anyway. So the issue is just perl's memory 
> usage compared to the memory usually available on such systems. Do you 
> have any experience how memory usage increases with 64bit support in 
> real life usage?

On lenny-i386, pahole says this about struct sv:

struct sv {
        void *                     sv_any;               /*     0     4 */
        U32                        sv_refcnt;            /*     4     4 */
        U32                        sv_flags;             /*     8     4 */
        union {
                IV                 svu_iv;               /*           4 */
                UV                 svu_uv;               /*           4 */
                SV *               svu_rv;               /*           4 */
                char *             svu_pv;               /*           4 */
                SV * *             svu_array;            /*           4 */
                HE * *             svu_hash;             /*           4 */
                GP *               svu_gp;               /*           4 */
        } sv_u;                                          /*    12     4 */

        /* size: 16, cachelines: 1 */
        /* last cacheline: 16 bytes */
};      /* definitions: 39 */

So it boils down to malloc granularity (I don't think Perl's allocator
is used on Debian).  For that, I wrote this little test program:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
  unsigned size = atoi(argv[1]);
  unsigned count = atoi(argv[2]);
  unsigned i;

  for (i = 0; i < count; ++i) {
    malloc(size);
  }
  getc(stdin);
  return 0;
}

Running it (again, on lenny-i386, albeit with a squeeze amd64 kernel),
I get:

./a.out 16 10000000               267m
./a.out 20 10000000               267m
./a.out 24 10000000               343m

And at least on i386, long long hasn't got alignment requirements, so
switching the type of IV only increases the size of struct to 20.

#include <stdio.h>

typedef unsigned int U32;
typedef long long IV;
typedef unsigned long long UV;
#define SV void
#define HE void
#define GP void

struct sv {
  void *                     sv_any;
  U32                        sv_refcnt;
  U32                        sv_flags;
  union {
    IV                 svu_iv;
    UV                 svu_uv;
    SV *               svu_rv;
    char *             svu_pv;
    SV * *             svu_array;
    HE * *             svu_hash;
    GP *               svu_gp;
  } sv_u;
};

int main()
{
  printf("%u\n", sizeof(struct sv));
}

So if this analysis is correct, due to malloc granularity, this change
should not have a significant space overhead (the IV type is used in
other places, too, but SVs are the most sensitive place, I guess).
There might be a performance hit if the IV value caches a cache line
boundary, but in my experiment, malloc returns pointers congruent 8
modulo 32, so this should not happen in practice.

It's best to double-check all this by compiling Perl with the
appropriate flags, though.