[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: rm -rf is too slow on large files and directory structure(Around 30000)

Anyone heard of the unlink command?On Wed, 15 Feb 2012, Stan Hoeppner 

> On 2/15/2012 12:55 AM, Jochen Spieker wrote:
> > Bilal mk:
> >>
> >> I tried to remove 5GB directory. In that directory around 30000 files and
> >> directory. It will take more than 30 min to complete.
> >> There is no other cpu intensive process running. After sometime it goes to
> >> D state and unbale to kii that process.
> > 
> > Removing that many files is I/O bound. 
> This isn't correct.  Removing a kernel source tree is fast, and it
> contains on the order of 4500 directories and 50K files.
> EXT4 can 'rm -rf' the kernel source in 2-3 seconds.  XFS prior to
> delaylog could take a minute or two, with delaylog it's 4 seconds.
> So that's 2-4 seconds to remove a directory tree of 50k files.  The OP's
> system is taking forever then freezing.  So if it's EXT4 he's using,
> this isn't an IO problem but a bug, or something else, maybe bad hardware.
> > The CPU doesn't play any
> > significant role in it. 
> CPU, and memory, play a very significant role here if the filesystem is
> XFS.  Delayed logging takes all of the log journal writes and buffers
> them, so duplicate changes to the metadata are rolled up into a single
> physical IO.  With enough metadata changes it becomes CPU bound.  But
> we're talking lots of metadata if we have a modern fast CPU.
> I can't speak to EXTx behavior in this regard as I'm not familiar with it.
> > The only way to speed things up is to either get
> > faster storage (an SSD with high IOPS value for random writing) or you
> > can try another filesystem. IIRC XFS is good at what you are doing. I
> > cannot recommend it, though, as I don't have any recent experience with
> > it. 
> XFS is absolutely *horrible* with this workload prior to kernel 2.6.35,
> when delayed logging was introduced.  So if this is your workload, and
> you want XFS, you need mainline 2.6.35, better still 3.0.0.

Jude <jdashiel-at-shellworld-dot-net>

Reply to: