Don't know if there is a better "Debian" way, but you can edit your
/etc/inittab to specify a default runlevel. By default (IIRC) Debian
installs with runlevel 2 (default) and 3 executing the same.
You can for example, change runlevel 2 to not run your display manager
similar to this (man update-rc.d):
update-rc.d gdm start 99 3 4 5 . stop 01 0 1 2 6 .
This will make your default runlevel (2) *not* start gdm. If you
change your default runlevel to 3 then gdm *will* start and you can
"telinit 2" to change to runlevel 2.