So, to sum it up. Your current kernel + root filesystem combo prevents
you to overwrite /etc/group (and I suspect any existing file) with
another file by means of conventional rename(2) syscall, which returns
anomalous return code.
My gut feeling tells me that one should blame filesystem (btrfs)
implementation in cases such as this.
This behavior justifies a bug report with severity 'serious' against
'linux-image' package as I don't see any easy way to fix this short of
kernel patch or replacing the filesystem with something more
conventional (ext4 comes to mind).