I'm in this same boat. I disable installing recommends on all but one of my systems, including my laptop's and other devices. If (when) I have issues or check out new software, I take a look at those lists to see what I might want.
Heh.. I remember using gnome, kde, xfce, etc. and they all seemed to have this problem in one way or another. After enough battling, I eventually arrived at the conclusion that these desktop environments were written to be super complete, not super lean. They were for everybody, not for me.
I ended up switching to openbox and a whole different selection of "desktop" utilities and even had to write some myself. Issues like this have completely evaporated for me.