Re: deduplicating file systems: VDO with Debian?
On Mon, 2022-11-07 at 17:01 +0100, hede wrote:
> Am 07.11.2022 16:29, schrieb hede:
> > Am 07.11.2022 02:57, schrieb hw:
> > > Hi,
> > >
> > > Is there no VDO in Debian, and what would be good to use for
> > > deduplication with
> > > Debian? Why isn't VDO in the stardard kernel? Or is it?
> >
> > I have used vdo in Debian some time ago and didn't remember big
> > problems.
>
> Btw. please keep in mind: VDO is transparent to the filesystem on-top.
> And deduplication (likewise compression) is some non-deterministic task.
>
> Where btrfs' calculation of the real free space is tricky if compression
> and/or dedup is in use, it's quite impossible for a filesystem ontop of
> VDO. It's much wore with VDO. The filesystem on top sees a "virtual"
> size of the device which is a vague guess at best and is predefined on
> creation time. You need to carefully monitor the actual disk usage of
> the VDO device and stop writing data to the filesystem if it fills up.
Yes, I figured that would be a problem.
> It stalls if the filesystem wants to write more data than is available.
> (At least if I remember correctly. Please correct me if I'm wrong here.)
Like NFS? And then what? There isn't really a way to resolve that problem once
you ran into it, is there?
> So if you are expecting issues with space, there's some risk in damaging
> your (file-)system.
Even damage it? That would really suck.
> With something like btrfs or ZFS there's less risk in that. Both do know
> the free space and even if this was indeed a Problem in first days*,
> rebalancing and filled up filesystems are (AFAIK) no longer a problem
> with btrfs.
Well, I'm finding btrfs somewhat disappointing since it doesn't support
deduplication like ZFS does, and even RAID56 is still broken. It feels like the
available file systems haven't been up to the task for almost a decade and might
never catch up.
> *) running out of space on btrfs could render filesystems read-only,
> deleting files was no longer possible. COW means even deleting a file
> needs some space so it got broken. This is AFAIK resolved. For deleting
> files there's always some reserved space.
>
> regards
> hede
>
Reply to: