[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: to lvm or not to lvm?



Greg Folkert wrote:
On Thu, 2007-05-03 at 10:36 +0800, Bob wrote:
Greg Folkert wrote:
8< massive snip

Auto adding of space has been done. Trust me on this one. Don't do it.
I've seen the after effects. A typo in a script can do more harm in 20
seconds (or less) than any one person could do in 20 years when dealing
with disk space. You would be well advised to setup an existing projects
disk space monitoring system and have it urgently mail you that disk
space is becoming a premium commodity on /blah filesystem.
As long as it's not possible for a growing partition to take space away from another partition, I can't see how this would do anything but give you more time to react, you run your messed up script that starts pushing the contents of /dev/zero into a text file and eating harddrive space, if your sitting next to the machine you see the HDD lights come on and hear the seeking, if your remote the first clue you get is an email or SMS saying /blah has been automatically grown by 5GB because it was more than 85% full, [0] you can kill your script, delete the text file, reduce the partition again and nothing crashed, if all it had done was email you a message pointing out /blah was running out of space, and /blah was also required by some other vital process that ran out of space and crashed before you could kill the script, you'd wish you had auto growth.

No, you don't get the fact that saturation of IO is a bad thing. Even on
8 multi-path IO sub-systems. Sometimes things spin and spin and spin.
When they do... it gets icky.
To give you an example:

        I was managing a n-tier (2-tier and 3 to 5 tier) setup. We had
        18 Terabytes of spare mirrored disk allocated to this machine.
        We had only 8 TB of mirrored disk already for DATA. One day the
        Oracle DB starts CRANKING and CRANKING on transaction logs. But
        its not really doing any work. The Oracle DBA had disk
        allocation rights from the spare 18TB of disk. So, he made a
        minutely cronjob, checking for percentage of space free. If the
        LV and Filesystem didn't have X% free add another X amount of
        space.
Unfortuneately, at 8TB, 1% of space is ~85GB. Problem was, that
        he was adding 32GB chunks and extending the filesystem by 32GB
        at a time. Once a minute. The LV extending commands were never
        completing let alone the File System extending. You can guess
        how well that went.
In about 11 hours, all 18TB was allocated but not used and was
        not committed but wasn't recoverable until it completed the
        commit process. And the Filesystem was never extended properly.
        We had to switch over to a fail-over machine and pray the DB was
        good on the hot-copy. It was... so not much was really bad.

So, you see, it is all about understanding scale. When you start talking
about TeraBytes or PetaBytes you need to have completion checks before
the next step or a repeat. And do things in large enough "chunks" to
make a difference.

That sounds like poor implementation, the critical bit is not to let it iterate without it having worked the first time. I still don't see anything wrong with the concept as long as it's implemented right, although I've never dealt with a system on the scale you're talking about.

I think the best way to implement it would be to set limits on how much a partition can grow, so once /blah has eaten 90% of the remaining space, it's not allowed to grow any more and the processes responsible for the expansion, along with any others that require space on /blah, crash hopefully not taking the whole system with it but still leaving a good chunk of space for other vital partitions to grow into, should they need to, so when you get off your train, plain or hobby horse, hopefully you can still ssh in and fix things.

The machine I mentioned was SO BUSY, ssh took over 10 minutes to get a
login and then they key-auth expired before the login process was
finished. Even the console was that slow, but at least it didn't time
out.

I've never seen anything that busy, had a MythTV backed that would sit with the load average up at 3 or 5 and I though that was "getting my moneys worth".

--
Garrr, do your bit for global warming, become a pirate, you can "borrow" my copy of Windows 95 if you want.



Reply to: