Have a peek at /etc/X11/xinit/xserverrc, I have an option of '-nolisten' listed there which seems like it could be your problem.