Bug#449490: libglu1-mesa: Error in matrix inversion in gluUnProject - bad call to __gluInvertMatrixd(finalMatrix,finalMatrix)

[Christian Holm Christensen <cholm@nbi.dk>]

Package: libglu1-mesa
Version: 7.0.1-2
Severity: important

*** Please type your report below this line ***

In the call 

  if (!__gluInvertMatrixd(finalMatrix,finalMatrix)) return(GL_FALSE);

in gluUnProject, there's mistake.  Inspecting the implementation
of __gluInverMatrixd(const double* src, double* inv) the arrays src and 
inv cannot overlap.  There should be another intermediate 4x4 matrix
defined, like 

  GLint GLAPIENTRY
  gluUnProject(GLdouble winx, GLdouble winy, GLdouble winz,
               const GLdouble modelMatrix[16],
	       const GLdouble projMatrix[16],
	       const GLint viewport[4],
	       GLdouble *objx,
	       GLdouble *objy,
	       GLdouble *objz)
  {
    double   finalMatrix[16];
    double   temp[16];	/* New  temporary store. */
    double   in[4];
    double   out[4];

    __gluMultMatricesd(modelMatrix, projMatrix, temp);

    if	(!__gluInvertMatrixd(temp,finalMatrix)) return(GL_FALSE);

 
The rest of the function is as before.   

If this is not done, then gluUnProject will return very small (and
incorrect) vectors in (*objx,*objy,*objz)

[Please also refer to discussion on
https://savannah.cern.ch/bugs/index.php?30943]

Yours,

Christian

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.22-3-686 (SMP w/2 CPU cores)
Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages libglu1-mesa depends on:
ii  libc6                         2.6.1-6    GNU C Library: Shared libraries
ii  libgcc1                       1:4.2.2-3  GCC support library
ii  libgl1-mesa-glx [libgl1]      7.0.1-2    A free implementation of the OpenG
ii  libstdc++6                    4.2.2-3    The GNU Standard C++ Library v3

libglu1-mesa recommends no packages.

-- no debconf information