Between 2.6.9 and 2.6.10 there has been a rewrite of the atyfb driver. This rewrite left users of powermac 5500, 6500 and Beige g3 with a distorted display, but only when booting from BootX. So, this means that the MacOS ROM is probably tweaking one of the card's registers that would otherwise not get touched on, causing this difference. Unfortunately, I have installed my machine with XFS as root filesystem and cannot boot it with quik.
So here's where you can help – I need you to compile atyfb with debug option – edit the appropriate lines in drivers/video/atyfb/mach64_ct.c and drivers/video/atyfb/atyfb_base.c to make sure DEBUG is defined. Then send me the output of dmesg, once when booting with quik, and once when booting with bootx. If you do that, I might be able to squash this bug once and for all.
Please reply off-list as I'm not subscribed.