Re: rm -rf is too slow on large files and directory structure(Around 30000)
On 2/15/2012 12:55 AM, Jochen Spieker wrote:
> Bilal mk:
>>
>> I tried to remove 5GB directory. In that directory around 30000 files and
>> directory. It will take more than 30 min to complete.
>> There is no other cpu intensive process running. After sometime it goes to
>> D state and unbale to kii that process.
>
> Removing that many files is I/O bound.
This isn't correct. Removing a kernel source tree is fast, and it
contains on the order of 4500 directories and 50K files.
EXT4 can 'rm -rf' the kernel source in 2-3 seconds. XFS prior to
delaylog could take a minute or two, with delaylog it's 4 seconds.
So that's 2-4 seconds to remove a directory tree of 50k files. The OP's
system is taking forever then freezing. So if it's EXT4 he's using,
this isn't an IO problem but a bug, or something else, maybe bad hardware.
> The CPU doesn't play any
> significant role in it.
CPU, and memory, play a very significant role here if the filesystem is
XFS. Delayed logging takes all of the log journal writes and buffers
them, so duplicate changes to the metadata are rolled up into a single
physical IO. With enough metadata changes it becomes CPU bound. But
we're talking lots of metadata if we have a modern fast CPU.
I can't speak to EXTx behavior in this regard as I'm not familiar with it.
> The only way to speed things up is to either get
> faster storage (an SSD with high IOPS value for random writing) or you
> can try another filesystem. IIRC XFS is good at what you are doing. I
> cannot recommend it, though, as I don't have any recent experience with
> it.
XFS is absolutely *horrible* with this workload prior to kernel 2.6.35,
when delayed logging was introduced. So if this is your workload, and
you want XFS, you need mainline 2.6.35, better still 3.0.0.
--
Stan
Reply to: