Could you try to unload all usb/uhci/ehci modules (e.g. by /etc/init.d/hotplug stop) before starting X? maybe a newer kernel would help, there was some drm/dri merge somewhere in 2.4.23 or 2.4.24 ?