I found out that X freezes in this setting whenever something happens in the console from which I typed startx to bring up the X system. Then I can switch from X to the text console and back without problem as long as I don't touch the console I started X with. I think also a write from another user could freeze everything in X. Kind of a weird workaround but perhaps also good to find out where the problem is.