Not quite true. It doesn't break gnome if non-nvidia drivers used for your card (eg: intel, nouveau, radeon, etc).
I suspect it breaks gnome due to nvidia being displaced, which is causing gdm or the like to be removed.
Are you installing the nvidia blob from backports as well, or are you just trying to install it from stretch? The version in stretch seems to have install issues for me atm.