On Debian 9 with latest updates, fail2ban not creating rules when used with nftables:
2017-06-29 01:06:14,217 fail2ban.action [2593]: ERROR nft add set inet filter f2b-sshd \{ type ipv4_addr\; \}
nft insert rule inet filter INPUT tcp dport \{ ssh \} ip saddr @f2b-sshd reject -- stdout: b''
2017-06-29 01:06:14,218 fail2ban.action [2593]: ERROR nft add set inet filter f2b-sshd \{ type ipv4_addr\; \}
nft insert rule inet filter INPUT tcp dport \{ ssh \} ip saddr @f2b-sshd reject -- stderr: b'<cmdline>:1:1-74: Error: Could not process rule: No such file or directory\ninsert rule inet filter INPUT tcp dport { ssh } ip saddr @f2b-sshd reject\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
2017-06-29 01:06:14,218 fail2ban.action [2593]: ERROR nft add set inet filter f2b-sshd \{ type ipv4_addr\; \}
nft insert rule inet filter INPUT tcp dport \{ ssh \} ip saddr @f2b-sshd reject -- returned 1
2017-06-29 01:06:14,218 fail2ban.actions [2593]: ERROR Failed to start jail 'sshd' action 'nftables-multiport': Error starting action
Let me know what more info you need.