I also faced this problem after installing GNOME on Debian stretch running on Virtualbox which uses the vesa driver unless you install the guest additions. I came across this page describing how rootless X11 is supposed to work:
http://hansdegoede.livejournal.com/14268.html
Here's the important bit:
> When the Xorg binary is no longer suid root, UMS drivers will not work. To solve this there is a new suid-wrapper called Xorg.wrap which is suid root, when installed this wrapper will get called in stead of the real X server, and it will check if there are KMS capable cards in the system, if KMS capable cards are found it will drop all elevated rights and execute the X server as a normal user. If no KMS capable cards are found it will execute the X server with root rights, allowing old UMS drivers, such as the proprietary drivers for some cards, or the vesa driver to work.
Can the X11 maintainers please implement something like this? Or borrow from upstream if it already exists?