Indeed it was the debug causing the issue. After disabling all DEBUG lines in the .config the kernel is now of the correct size.