The grsec kernel in backports has the vmmouse kernel driver enabled, making it conflict with the X.org driver xserver-xorg-input-vmmouse, which is used on older kernels. The regular (non-grsec) linux in backports already has the kernel driver disabled. Legimet