Bug#449490: libglu1-mesa: Error in matrix inversion in gluUnProject - bad call to __gluInvertMatrixd(finalMatrix,finalMatrix)

[Brice Goglin <Brice.Goglin@ens-lyon.org>]

forcemerge 440137 449490
thank you


Christian Holm Christensen wrote:
> In the call 
>
>   if (!__gluInvertMatrixd(finalMatrix,finalMatrix)) return(GL_FALSE);
>
> in gluUnProject, there's mistake.  Inspecting the implementation
> of __gluInverMatrixd(const double* src, double* inv) the arrays src and 
> inv cannot overlap.  There should be another intermediate 4x4 matrix
> defined, like 
>
>   GLint GLAPIENTRY
>   gluUnProject(GLdouble winx, GLdouble winy, GLdouble winz,
>                const GLdouble modelMatrix[16],
> 	       const GLdouble projMatrix[16],
> 	       const GLint viewport[4],
> 	       GLdouble *objx,
> 	       GLdouble *objy,
> 	       GLdouble *objz)
>   {
>     double   finalMatrix[16];
>     double   temp[16];	/* New  temporary store. */
>     double   in[4];
>     double   out[4];
>
>     __gluMultMatricesd(modelMatrix, projMatrix, temp);
>
>     if	(!__gluInvertMatrixd(temp,finalMatrix)) return(GL_FALSE);
>
>  
> The rest of the function is as before.   
>
> If this is not done, then gluUnProject will return very small (and
> incorrect) vectors in (*objx,*objy,*objz)
>
> [Please also refer to discussion on
> https://savannah.cern.ch/bugs/index.php?30943]
>   

That's very likely to be already fixed upstream in
http://gitweb.freedesktop.org/?p=mesa/mesa.git;a=commitdiff;h=40438b34405ce88d0e160bc8c3dea24276d63189;hp=e0b80660c8a4cbd9624a9fbe07841541105f712c

Merging with #440137

Mesa 7.0.2 is supposed to be released soon.

Brice