You can try XFree CVS, it has a significantly updated GeForce driver. If the problem persist, maybe you can contact the driver's author (Mark Vojkovich), he may have a clue. I suspect part of the problem may be the EDID DDC block returned by the monitor beeing slightly wrong and OS X containing a fixed "override" version
I tried XFree CVS, it didn't work either.But now I fixed it by flashing the nvidia firmware. As I suspected it was a firmware bug.
Everything works now. Strange that this problem is undocumented and unknown. Andi