On 2020-08-03 16:17, deloptes wrote:
any thoughts on using deduplication? For example I started using borg some time ago. It saves a lot of space and makes it possible to have multiplebackups and longer retention.
ZFS supports de-duplication, but the documents warn about enabling it. So, of course I enabled de-duplication on my ZFS SOHO file server. ;-)
Everything was groovy when utilization was ~30%, but performance for bulk writes degraded precipitously as the pool filled. This includes backup replication jobs. I am fairly certain de-duplication is a major contributing factor. The only way to test this hypothesis is to create a fresh pool using similar hardware, replicate the data without de-duplication, and benchmark.
jdupes looks interesting, and should work on any file system that supports hard links. I expect BorgBackup either calls jdupes or implements similar functionality:
https://linuxcommandlibrary.com/man/jdupes.html David