Re: rm -rf is too slow on large files and directory structure(Around 30000)
Anyone heard of the unlink command?On Wed, 15 Feb 2012, Stan Hoeppner
wrote:
> On 2/15/2012 12:55 AM, Jochen Spieker wrote:
> > Bilal mk:
> >>
> >> I tried to remove 5GB directory. In that directory around 30000 files and
> >> directory. It will take more than 30 min to complete.
> >> There is no other cpu intensive process running. After sometime it goes to
> >> D state and unbale to kii that process.
> >
> > Removing that many files is I/O bound.
>
> This isn't correct. Removing a kernel source tree is fast, and it
> contains on the order of 4500 directories and 50K files.
>
> EXT4 can 'rm -rf' the kernel source in 2-3 seconds. XFS prior to
> delaylog could take a minute or two, with delaylog it's 4 seconds.
>
> So that's 2-4 seconds to remove a directory tree of 50k files. The OP's
> system is taking forever then freezing. So if it's EXT4 he's using,
> this isn't an IO problem but a bug, or something else, maybe bad hardware.
>
> > The CPU doesn't play any
> > significant role in it.
>
> CPU, and memory, play a very significant role here if the filesystem is
> XFS. Delayed logging takes all of the log journal writes and buffers
> them, so duplicate changes to the metadata are rolled up into a single
> physical IO. With enough metadata changes it becomes CPU bound. But
> we're talking lots of metadata if we have a modern fast CPU.
>
> I can't speak to EXTx behavior in this regard as I'm not familiar with it.
>
> > The only way to speed things up is to either get
> > faster storage (an SSD with high IOPS value for random writing) or you
> > can try another filesystem. IIRC XFS is good at what you are doing. I
> > cannot recommend it, though, as I don't have any recent experience with
> > it.
>
> XFS is absolutely *horrible* with this workload prior to kernel 2.6.35,
> when delayed logging was introduced. So if this is your workload, and
> you want XFS, you need mainline 2.6.35, better still 3.0.0.
>
>
----------------------------------------------------------------
Jude <jdashiel-at-shellworld-dot-net>
<http://www.shellworld.net/~jdashiel/nj.html>
Reply to: