Usually I install xorg, gnome-session-fallback and gdm3. No network-manager, no pulse-audio, no nothing actually. From there you can build it up.