After some analysis the issues turned out to be related to missing ed25519 host key. Once it was created the sshd had no issues with running. My custom sshd_config has only specific HostKeys defined and for some reason not having there ed25519 cause this crash. Running: ssh-keygen -A and adding to /etc/ssh/sshd_config line: HostKey /etc/ssh/ssh_host_ed25519_key solved this.