I'm not exactly the foremost expert on this but ... my understanding is that the initramfs, or initrd or whatever it is, contains a bunch of useful things the kernel needs to boot the board fully. Primarily these include hardware driver modules needed by the kernel ahead of when the main filesystem becomes available. For instance disk controllers and filesystem drivers.
One way to reduce the use of the initramfs/initrd is to build a kernel that has the drivers you need built into it rather than loaded as modules from the ramfs. You would need to build a custom kernel (relatively easy with debian already on the device and using make-kpkg) with the kernel config customised to achieve this.
I have no idea how much this would speed up boot time, if at all. You also lose some of the other benefits of using an initramfs/initrd, which I'm having trouble remembering right now. (recovery shell? various scripts for mdadm/lve? stuff...)
Hope that helps, I realise I have glossed over a LOT of detail there.
David.