Re: LSI MegaRAID SAS 9240-4i hangs system at boot
On Sun, 10 Jun 2012 17:30:08 -0500
Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 6/10/2012 9:00 AM, Ramon Hofer wrote:
> > A situation update: Mounted the mobo with the CPU and RAM, attached
> > the PSU, the OS SATA disk, the LSI and expander as well as the
> > graphics card. There are no disks attached to the expander because
> > I put them again into the old NAS and backing up the data from the
> > 1.5 TB disks to it.
> >
> > Then I installed Debian Squeeze AMD64 without problems. I don't have
> > the over-current error messages anymore :-)
> > But it still hangs at the same time as before.
>
> Try the Wheezy installer. Try OpenSuSE. Try Fedora. If any of these
> work without lockup we know the problem is Debian 6. However...
I didn't do this because it the LSI worked with the Asus mobo and
Debian squeeze. And because I couldn't install OpenSuSE nor Fedora.
But I will give it another try...
> Please call LSI support before you attempt any additional
> BIOS/firmware updates.
I mailed them and got this answer:
"Unfortunately, the system board has not been qualified on the hardware
compatibility list for the LSI MegaRAID 9240 series controllers. There
could be any number of reason for this, either it has not yet been
tested or did not pass testing, but the issue is likely an
incompatibility.
It sounds like the issue is related to the bootstrap, so either to
resolve the issue you will have to free up the option ROM space or
limit the number of devices during POST."
This is what you've already told me.
If I understand it right you already told me to try both: free up the
option ROM and limit the number of devices, right?
(...)
> > Thanks again very much.
> > The air flow / cooling argument is very convincing. I haven't
> > thought about that.
>
> Airflow is 80% of the reason the SAS and SATA specifications were
> created.
You've convinced me: I will mount the expander properly to the case :-)
> > It was the P7P55D premium.
> >
> > The only two problems I have with this board is that I'd have to
> > find the right BIOS settings to enable the LSI online setting
> > program (or how is it called exactly?) where one can set up the
> > disks as JBOD / HW RAID.
>
> I already told you how to do this with the C7P67. Read the P7P55D
> manual, BIOS section. There will be a similar parameter to load the
> BIOS ROMs of add in cards.
Ok, thanks!
> > Sorry I don't understand what you mean by "don't put partitions on
> > your mdraid devices before creating the array".
> > Is it wrong to partition the disks and the do "mdadm --create
> > --verbose /dev/md0 --auto=yes --level=6
> > --raid-devices=4 /dev/sda1.1 /dev/sdb1.1 /dev/sdc1.1 /dev/sdd1.1"?
> >
> > Should I first create an empty array with "mdadm --create
> > --verbose /dev/md0 --auto=yes --level=6 --raid-devices=0"
> >
> > And then add the partitions?
>
> Don't partition the drives before creating your md array. Don't
> create partitions on it afterward. Do not use any partitions at
> all. They are not needed. Create the array from the bare drive
> device names. After the array is created format it with your
> preferred filesystem, such as:
>
> ~$ mkfs.xfs /dev/md0
Ok understood. RAID arrays containing partitions are bad.
> > Hmm, that's a very hard decision.
> > You probably understand that I don't want to buy 20 3 TB drives
> > now. And still I want to be able to add some 3 TB drives in the
> > future. But at
>
> Most novices make the mistake of assuming they can only have one md
> RAID device on the system, and if they add disks in the future they
> need to stick them into that same md device. This is absolutely not
> true, and it's not a smart thing to do, especially if it's a parity
> array that requires a reshape, which takes dozens of hours.
> Instead...
Nono, I was aware that I can have several RAID arrays.
My initial plan was to use four disks with the same size and have
several RAID5 devices. But Cameleon from the debian list told me to not
use such big disks (>500 GB) because reshaping takes too long and
another failure during reshaping will kill the data. So she proposed to
use 500 GB partitions and RAID6 with them.
Is there some documentation why partitions aren't good to use?
I'd like to learn more :-)
> > the moment I have four Samsung HD154UI (1.5 TB) and four WD20EARS (2
> > TB).
>
> You create two 4 drive md RAID5 arrays, one composed of the four
> identical 1.5TB drives and the other composed of the four identical
> 2TB drives. Then concatenate the two arrays together into an md
> --linear array, similar to this:
>
> ~$ mdadm -C /dev/md1 -c 128 -n4 -l5 /dev/sd[abcd] <-- 2.0TB drives
May I ask what the -c 128 option means? The mdadm man page says that -c
is to specify the config file?
> ~$ mdadm -C /dev/md2 -c 128 -n4 -l5 /dev/sd[efgh] <-- 1.5TB drives
> ~$ mdadm -C /dev/md0 -n2 -l linear /dev/md[12]
This is very interesting. I didn't know that this is possible :-o
Does it work as well with hw RAID devices from the LSI card?
Since you tell me that RAIDs with partitions aren't wise I'm thinking
about creating hw RAID5 devices with four equally sized disks.
The -C option means that mdadm creates a new array with the
name /dev/md1.
Is it wise to use other names, e.g. /dev/md_2T, /dev/md_1T5
and /dev/md_main?
And is a linear raid array the same as RAID0?
> Then make a write aligned XFS filesystem on this linear device:
>
> ~$ mkfs.xfs -d agcount=11 su=131072,sw=3 /dev/md2
Are there similar options for jfs?
I decided to use jfs when I set up the old server because it's easier
to grow the filesystem.
But when I see the xfs_grow below I'm not sure if xfs wouldn't be the
better choice. Especially because I read in wikipedia that xfs is
integrated in the kernel and to use jfs one has to install additional
packages.
Btw it seems very complicated with all the allocation groups, stripe
units and stripe width.
How do you calculate these number?
And why do both arrays have a stripe width of 384 KB?
> The end result is a 10.5TB XFS filesystem that is correctly write
> stripe aligned to the 384KB stripe width of both arrays. This
> alignment prevents extra costly unaligned RMW operations (which
> happen every time you modify an existing file). XFS uses allocation
> groups for storing files and metadata and it writes to all AGs in
> parallel during concurrent access. Thus, even though the your
> spindles are separated into two different stripes instead of one
> large stripe, you still get the performance of 6 spindles. Two RAID
> 5 arrays actually give better performance, as you will have two md
> threads instead of one, allowing two CPU cores to do md work instead
> of only one with md RAID6.
Is it also true that I will get better performance with two hw RAID5
arrays?
> So now you've run out of space or nearly so, and need to add more.
> Simple. Using four new drives (so our array geometry remains the
> same), say 3TB models, you'd create another RAID5 array:
>
> ~$ mdadm -C /dev/md3 -c 128 -n4 -l5 /dev/sd[ijkl]
>
> Now we grow the linear array:
>
> ~$ mdadm --grow /dev/md0 --add /dev/md3
>
> And now we grow the XFS filesystem:
>
> ~$ xfs_growfs /your/current/xfs/mount_point
>
> Now your 10.5TB XFS filesystem is 19.5TB and has 9TB additional free
> space, with additional AGs, still aligned to the RAID stripe size of
> the md RAID arrays, which are all identical at 384KB. And unlike an
> md reshape of an 8 drive RAID6 array which can take over 36 hours,
> the XFS grow operation takes a few seconds. Creating the new 4 drive
> array will take much longer, but not nearly as long as a reshape of
> an 8 drive array involving 4 new drives.
>
> > Actually I've just saw that the Samsungs are green drives as well.
>
> I fear you may suffer more problems down the road using WDEARS drives
> in md RAID, or any green drives.
What if I loose a complete raid5 array which was part of the linear
raid array? Will I loose the whole content from the linear array as I
would with lvm?
> > The reason why I bought green drives is that the server provides
> > mythbackend, nas, logitech media server, etc.
> > So it doesn't have much to do but it still should be ready all the
> > time (if I wan't to listen to music I don't want to power the
> > squeezebox radio which triggers the server to start up and only
> > when it started I can listen to music which would probably take >1
> > min. So I thought the drives should manage themselves to save some
> > power.
>
> > I understand that there may be timing problems. But do they make it
> > impossible?
>
> Just make sure you don't have any daemons accessing directories on the
> md array(s) and you should be fine. IIRC the WD Green drives go to
> sleep automatically after 30 seconds of inactivity, and park the heads
> after something like <10 seconds. I'll personally never own WD Green
> drives due to their reputation for failure and anemic performance.
Thanks for the advice!
About the failure: This is why I use raid5 and as I don't need very
high performance this doesn't matter for me.
But I understand that they are risky.
I'm still aware that 3 TB raid5 rebuilds take long. Nevertheless I think
I will risk using normal (non-green) disks for the next expansion.
If I'm informed correctly there are not only green drives and normal
desktop drives but also server disks with a higher quality than
desktop disks.
But still I don't want to "waste" energy. Would the Seagate Barracuda
3TB disks be a better choise?
> > What would you do if you?
> >
> > Let's say I'd "throw away" these disks and go for 3 TB drives. At
> > the
>
> I wouldn't. 3TB drives take far too long to rebuild. It takes about
> 8 hours to rebuild one in a mirror pair, something like 30+ hours to
> rebuild a drive in a 6 drive RAID6. If a 3TB drive fails due to
> age/wear, and your drives are identical, the odds of having two more
> drive failures before the rebuild completes are relatively high. If
> this happens, you better have a big full backup handy. Due to this
> and other reasons, I prefer using drives of 1TB or less. My needs are
> different than yours, however--production servers vs home use.
My needs are probably *much* less demanding than yours.
Usually it only has to do read access to the files. Aditionally copying
bluray rips to it. But most of the the it sits around doing nothing
(the raid). MythTV records almost most of the time but to a non RAID
disk.
So I hope with non-green 3 TB disks I can get some security from the
redundancy and still get a lot of disk space.
> > moment four in a RAID 6 array would be enough. So I'd have 6 TB
> > available.
>
> Never build a RAID6 array with less than 6 drives, as RAID10 on 4
> drives gives vastly superior performance vs a 5/6 drive RAID6, and
> rebuild times are drastically lower for mirrors. Rebuilding a RAID1
> 3TB drive takes about 8 hours, and you're looking at something North
> of 30 hours for a 6 drive RAID6 rebuild.
Because you told me that it's not good to use partitions I won't set up
raid6.
Instead I'll go for raid5 with 4 disks.
> > Then I'd run out of space and want to upgrade with another disk.
> > Probably it'll still be available but will it also be when I'll
> > have 19 disks and want to add the last one?
> > Just as an example to explain my worries ;-)
>
> Start with the 4 drive RAID5 arrays I mentioned above. You have a 20
> cage case, 4 rows of 5 cages each. Put each array in its own 4 cage
> row to keep things organized. You have two rows empty as you have 8
> drives currently. When you expand in the future, only expand 4
> drives at a time, using the instructions I provided above. You can
> expand two times. Using 2TB drives you'll get +6TB twice for a total
> of 22.5TB.
This was exactly what I had in mind at the first place. But the
suggestion from Cameleon was so tempting :-)
Btw I have another question:
Is it possible to attach the single (non raid) disk I now have in my old
server for the mythtv recordings to the LSI controller and still have
access to the content when it's configured as jbod?
Since there are recordings which it wouldn't be very bad if I loose
them I'd like to avoid backing this up.
Cheers
Ramon
Reply to: