Bug#682812: calloc (nmemb=0, size=12)

[Johann Klammer <klammerj@a1.net>]

Package: mesa
Version: 7.11.2-1

Hello,

I tried to replace the memory allocator for a program
linked against mesa. With the libc allocators the app shows some 
textured and shaded triangles, but with the custom allocators, it's all 
black triangles. Sorry, I cannot post the program source as it is rather 
large, but will try to come up with a minimal test if this is necessary.

The custom allocator returns NULL for zero-sized requests.
The problem goes away if I change the allocator to return some memory.

Symptoms:
all black triangles.

likely cause:
'calloc (nmemb=0, size=12)' in
st_translate_mesa_program at state_tracker/st_mesa_to_tgsi.c:1181

hardware:
[   33.950374] [drm] nouveau 0000:01:00.0: Detected an NV30 generation 
card (0x034200b1)

driver:
nouveau

gdb backtrace:
(gdb) backtrace
#0  alcAlloc (sz=0, alignment=1, res=false) at alc.c:351
#1  0xb7dee54c in calloc (nmemb=0, size=12) at alc.c:593
#2  0xb6d6384a in st_translate_mesa_program (ctx=0xb6916000, procType=0, 
ureg=0xb560c000, program=0xb573f120,
    numInputs=3, inputMapping=0xbff75768,
    inputSemanticName=0xbff7580c "\001\001\005\265쉈\265\060\240t\267 
\371\202\327\266\060\240t\267\230X\367\277\020{\216\266", 
inputSemanticIndex=0xbff7582c "", interpMode=0xbff756e8, numOutputs=1, 
outputMapping=0xbff757e0,
    outputSemanticName=0xbff7584c "\001", 
outputSemanticIndex=0xbff7586c "", passthrough_edgeflags=0 '\000')
    at state_tracker/st_mesa_to_tgsi.c:1181
#3  0xb6ca3b1e in st_translate_fragment_program (st=0xb68e5380, 
stfp=0xb573f120, key=0xbff758f8)
    at state_tracker/st_program.c:609
#4  0xb6ca49d6 in st_get_fp_variant (st=0xb68e5380, stfp=0xb573f120, 
key=0xbff758f8)
    at state_tracker/st_program.c:667
#5  0xb6d52e37 in update_fp (st=0xb68e5380) at 
state_tracker/st_atom_shader.c:86
#6  0xb6d506e6 in st_validate_state (st=0xb68e5380) at 
state_tracker/st_atom.c:197
#7  0xb6d5756a in st_Clear (ctx=0xb6916000, mask=18) at 
state_tracker/st_cb_clear.c:497
#8  0xb6d15b10 in _mesa_Clear (mask=<value optimized out>) at 
main/clear.c:250
#9  0x0805020b in event_loop (dpy=0xb77d52e0, win=37748738, 
glxw=37748739, sd=0xbff75da4) at main.c:2671
#10 0x08051970 in main (argc=1, argv=0xbffff534) at main.c:3330

possible fix:
replace:

>      t->constants = CALLOC( program->Parameters->NumParameters,
>                             sizeof t->constants[0] );

with:

>      t->constants = CALLOC(
>(program->Parameters->NumParameters==0)?1:program->Parameters->NumParameters,
>                             sizeof t->constants[0] );

But this may just mask a more significant BUG...

JK