On my dual core G5, I have to disable hardware acceleration when using the nouveau driver by adding this to my yaboot parameters.
nouveau.noaccel=1
If that works, you'll want to add this to your /etc/yaboot.conf file as the last line under your default kernel (probably named Linux) as well to make it permanent.
append="nouveau.noaccel=1"
Save those changes, and then run "sudo ybin -v" to apply those changes.
I'm currently trying to get 3D acceleration working again on the nouveau driver by compiling the latest drivers directly from Nouveau's git repository and using a later kernel. Unfortunately, I'm getting a GPU locked up message at boot, which causes my system to fall back to the fbdev driver, but that's an issue for another email on the Debian PowerPC mailing list and possibly a bug report to the nouveau maintainers.
B-rock