I was able to get Xfce installed pretty well by using the CD installer and then installing the dependencies of the xfce desktop task but without lightdm. I tried out some other display managers, but in the end I decided to just not use a display manager. That was the main thing that I fixed, but there were lots of other dependencies and recommends that I adjusted slightly.