Loading ...Mdadm Cheat Sheet
Mdadm is the modern tool most Linux distributions use these days to manage software RAID arrays; in the past raidtools was the tool we have used for this. This cheat sheet will show the most common usages of mdadm to manage software raid arrays; it assumes you have a good understanding of software RAID and Linux in general, and it will just explain the commands line usage of mdadm. The examples bellow use RAID1, but they can be adapted for any RAID level the Linux kernel driver supports.
1. Create a new RAID array
Create (mdadm –create) is used to create a new array:
mdadm --create --verbose /dev/md0 --level=1 /dev/sda1 /dev/sdb2
or using the compact notation:
mdadm -Cv /dev/md0 -l1 -n2 /dev/sd[ab]1
2. /etc/mdadm.conf
/etc/mdadm.conf or /etc/mdadm/mdadm.conf (on debian) is the main configuration file for mdadm. After we create our RAID arrays we add them to this file using:
mdadm --detail --scan >> /etc/mdadm.conf
or on debian
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
3. Remove a disk from an array
We can’t remove a disk directly from the array, unless it is failed, so we first have to fail it (if the drive it is failed this is normally already in failed state and this step is not needed):
mdadm --fail /dev/md0 /dev/sda1
and now we can remove it:
mdadm --remove /dev/md0 /dev/sda1
This can be done in a single step using:
mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1
4. Add a disk to an existing array
We can add a new disk to an array (replacing a failed one probably):
mdadm --add /dev/md0 /dev/sdb1
5. Verifying the status of the RAID arrays
We can check the status of the arrays on the system with:
cat /proc/mdstat
or
mdadm --detail /dev/md0
The output of this command will look like:
cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]
md1 : active raid1 sdb3[1] sda3[0]
19542976 blocks [2/2] [UU]
md2 : active raid1 sdb4[1] sda4[0]
223504192 blocks [2/2] [UU]here we can see both drives are used and working fine – U. A failed drive will show as F, while a degraded array will miss the second disk -
Note: while monitoring the status of a RAID rebuild operation using watch can be useful:
watch cat /proc/mdstat
6. Stop and delete a RAID array
If we want to completely remove a raid array we have to stop if first and then remove it:
mdadm --stop /dev/md0
mdadm --remove /dev/md0
and finally we can even delete the superblock from the individual drives:
mdadm --zero-superblock /dev/sda
Finally in using RAID1 arrays, where we create identical partitions on both drives this can be useful to copy the partitions from sda to sdb:
sfdisk -d /dev/sda | sfdisk /dev/sdb
(this will dump the partition table of sda, removing completely the existing partitions on sdb, so be sure you want this before running this command, as it will not warn you at all).
There are many other usages of mdadm particular for each type of RAID level, and I would recommend to use the manual page (man mdadm) or the help (mdadm –help) if you need more details on its usage. Hopefully these quick examples will put you on the fast track with how mdadm works.
Tags: cheatsheet, mdadm, raid, tips
9th March 2009, 11:51
A couple more helpful pointers:
mdadm(8) can be run in daemon mode to detect and alert about degraded arrays. You can configure it to either send an email or call an external script of your choice.
You can create a new or assemble an existing array without one of the member devices by using the keyword “missing” in place of the block device. Handy for recovery or upgrade procedures.
16th March 2009, 02:35
[...] Mdadm Cheat Sheet | MDLog:/sysadmin This will be utterly useless to anyone who isn’t experimenting with software RAID on 2.26 kernels. For those of you who are just getting your feet wet with mdadm (e.g. YT), this crib-sheet is a nice resource. That might just point out some things you would otherwise have to plumb the dreaded man page for. [...]
18th April 2009, 18:03
[...] Mdadm Cheat Sheet [...]
28th April 2009, 08:52
BTW, did you ever got stuck with rootfs on md array while booting?
I got stuck when tried to upgrade from 2.6.18 (yes, etch+lenny-RC mess) to 2.6.26 (pure lenny).
28th April 2009, 10:34
@Yury: what do you mean by stuck? Be sure you have the raid module in the initrd image you are booting. Not having the raid module loaded can cause this. hth.
30th April 2009, 09:34
I mean “stcuk” in this sense:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=515676
and
http://groups.google.com/group/linux.debian.user/browse_thread/thread/b97fd9b0a450370f#
I.e. I had got into BusyBox/initramfs shell after fail of array assembly.
If I manually run script/lcoal-top/mdadm , all works fine.
2.6.18 kernel did not make me giving up, only 2.6.22 and above.
That’s it.
31st July 2009, 22:41
Does anyone know a solution to this? I am terrified I will lose data
[root@linuxserver ~]# mdadm -D /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Mon Apr 28 18:53:27 2008
Raid Level : raid1
Used Dev Size : 488375872 (465.75 GiB 500.10 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Fri Jul 31 10:55:57 2009
State : active, degraded, Not Started
Active Devices : 0
Working Devices : 2
Failed Devices : 0
Spare Devices : 2
Number Major Minor RaidDevice State
0 22 65 0 spare rebuilding /dev/hdd1
1 0 0 1 removed
2 22 1 – spare /dev/hdc1
13th August 2009, 08:02
Nice cheat sheet! There is a couple of typo in the short version of the create array. The short notations don’t use an “=”, it’s “-l1″ and “-n2″ respectively.
13th August 2009, 19:08
@Nicolas: thanks for the note. I fixed the typo.
Cheers,
- Marius -
14th August 2009, 05:31
2Christos: I guess you have a fail here. Broken rebuild in progress is a real issue.
I suggest you try to mount /dev/hdd1 as an ext2/ext3 FS alone, not as MD device.
Maybe ext2restore (IIRC that name) help you.
14th August 2009, 05:41
Marius, I guess it’s worth mention how to create incomplete arrays (I mean with option “missing drive”). Sometimes it’s very usefull. I used do this while migration from non-md-based systems to md-based. In your case that would be:
mdadm –create –verbose /dev/md0 –level=1 –raid-devices=2 missing /dev/sdb2
consider /dev/sda1 not yet in the array, maybe it conatins prior data/system
then do in the appropriate point:
mdadm –add /dev/md0 /dev/sda1
that’s it!
14th August 2009, 15:26
@Yury: you are soo right. Thank you for your addition! I’ve done this many times also (migrating from a single drive to raid1). Actually I left this out because I wanted to write a special post about it, but because of lack of time it remained in drafts
. I will add your suggestion to the post and also finish my migration to raid1 post and link to it for more details.
Thanks,
- Marius -
4th September 2009, 06:51
[...] Mdadm Cheat Sheet [...]
1st February 2010, 19:08
[...] Mdadm Cheat Sheet | MDLog:/sysadmin – January 30th ( tags: linux raid mdadm sysadmin cheatsheet reference tips tricks ) February 1st, 2010, @ 1:00 pm | Tags: links | Category: delicious links | Comments are closed | Trackback this Post | 0 views [...]
5th May 2010, 23:09
The long version of the create array example is missing “–raid-devices=2″
16th June 2010, 14:54
[...] Reference URLs: ———————- http://en.wikipedia.org/wiki/RAID http://en.wikipedia.org/wiki/Standard_RAID_levels http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/ [...]
24th June 2010, 21:36
Fantastic! This is probably the best tutorial I’ve ever used, I think I’ve referenced it about 100 times.
26th June 2010, 17:22
Nice summary of commands … can you suggest a page which provides steps to replace a failed drive?
I have two physical drives sda and sdb. Each has 3 partitions which are then configured into 3 raid devices (md0,md1, md2). two are degraded and SMART is whining about sdb.
I think the general steps are to fail sdb1, sdb2, and sdb3 and remove them from the raid arrays.
Then replace the physical drive and use the sfdisk above to create the new partitions.
Add the new drive partitions back into their respective raid arrays and wait for the rebuilds?
Anything I’ve missed?
Thanks.
27th June 2010, 02:10
@David: I should write a short post on that, but yes basically it is just as you said:
1. fail the partitions from raid of the bad disk (if the system has not done that already)
2. Replace the disk with a good one
3. Partition it similar to sda
4. Add the new partitions in the array, and wait for the resync.
hth,
- Marius -
30th June 2010, 04:24
@Marius: Thanks for the feedback confirming my plan. Worked like a champ. The sfdisk partition ‘trick’ is sweet!
- David Morris
30th June 2010, 04:36
@David: you are most welcome!
1st July 2010, 04:15
After reflecting further on what I’ve done, I realized that there may be a critical step missing.
I think some kind of GRUB related process is needed such that if /dev/sda dies, the system
can boot from /dev/sdb. My BIOS is set to try both, but I think a MBR is still needed on /dev/sdb.
Is that correct? And if so, is there an easy way to recreate the proper MBR environment
after rebuilding the RAID?
Thanks again,
Dave Morris
1st July 2010, 04:50
David, I believe the easiest way is just copy from another survived drive.
Of course, it works only if both drive are identical.
I mean:
#/sbin/sfdisk -d /dev/sdb | /sbin/sfdisk /dev/sda
Easy as that!
Though I had once managed to copy correct disk layout from one drive to another with different layout. I copy output of sfdisk -d command to a file, edit it apprpriately, then feed it to sfdisk (cat file |sfdisk DRIVE NAME).
And remember, MBR does not comprise all partition information, only frist 1,2,3 or 4.
Others are in a chain on the disk. RTFM.
Hope it helps. Cheers.
1st July 2010, 06:03
@Yury … I don’t believe that sfdisk does any more than set up partitions
and mark one bootable … but from my dim dark past I recall that a RAID
set up required special work to make the system bootable from either drive.
I’m not positive, but I think the partition table is not contained in the MBR
but rather referenced by the MBR …
And in particular the boot code in the MBR or referenced by the MBR needed
to have special code to deal with the fact that the disk contained software
raided partitions.
TFM was too confusing for me to charge ahead and try things on a
running production system. Hence my request for clarification.
1st July 2010, 06:49
Ah!:)
Now I’ve got your point. Though I’m not sure about what is “special work”.
Anyway, to make the disk drive really bootable just execute /usr/sbin/grub-install –no-floppy –recheck /dev/sdb. Personally me never touch big troubles with thta because I always tend to divide /boot (physical partition) and other system (mostly R1 or R5). Works for me. WBR.
1st July 2010, 07:12
@David: you can do something like this to ensure grub is installed on both disks:
grub --device-map=/boot/grub/device.map
grub> root (hd0,0)
grub> setup (hd0)
grub> root (hd1,0)
grub> setup (hd1)
grub> quit
28th July 2010, 05:46
Have you heard of bios settings changes mangling an MD raid? I built a raid with the “RAID autodetect or sata” option set. When I changed to “Raid autodetect or AHCI” option and tried to reboot my newly built Debian linux install on my new Dell T3500 my md0, md1, md2 changed to md125, md126, and md127 (discovered with a knoppix recovery CD). I gave up on that effort and tried to install debian on the RAID array without first deleting all the partitions. The Debian install found the RAIDs; I reassigned them to /boot, swap and / respectively… but now with raids MD125, MD1 and MD127. Then I couldn’t get past the linux (grub) loader. It couldn’t even find /boot. Now (using knoppis CD) mdadm -a -s can’t find the arrays. I’ll try to build them by command just for grins.
Cheers….
What’s the point of this email? A little whining. A little searching for confirmation that this DELL bios setting either mangles software raids or that I just had bad luck, and will succeed if I start from square-1.
13th August 2010, 05:57
with knoppix, the mdadm –assemble –scan changed my “preferred minors”.
Another of my mistakes was not using the amd64 debian to reinstall.
The amd64 install couldn’t create my md0, md1, md2, but knoppix could.
The installation hung trying to install grub. Rebooting the installer into rescue mode was necessary; with a root shell, mounted md0 as /boot, then “setup-grub” command followed by “grub-install” got the system bootable.
27th August 2010, 09:44
AHCI Bios options changes the way disks re represented to the OS, so I’m not surprised it mangled your RAID array. There’s not much you can do but revert to previous BIOS Settings and hope to be able to recover data in circumstances like this. for more info, google/wikipedia “AHCI”
4th October 2010, 10:55
[...] http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/ [...]
5th November 2010, 09:35
Great howto.. I didnt’ know about the watch command. thanks!
7th January 2011, 12:04
[...] you need to troubleshoot a problem or a disk has failed in an array, I suggest referring to the mdadm cheat sheet (that’s what I [...]
12th March 2011, 17:01
Hi
I installed lenny5 on a system with 3 SATA HDD , 2 of the 3 HDD’s where reiserfs , I dont know why when the installation finished I found that the 2 HDD’s became parts of a raid array md0 !!!!!! these 2 HDD’s have a valuable data , I tried to get them back to reiser , i followed stopping and removing md0 , and i write zer0-superbloack to them …. i tried to mount them as reiserfs ..
can i get back the data ?? seems they are empty now !!!!
14th March 2011, 02:17
4. Add a new disk to an existing array
We can add a new disk to an array:
mdadm –add /dev/md0 /dev/sdb1
And then we need to grow the array by one :
mdadm –grow /dev/md0 –raid-devices=4
21st March 2011, 17:21
I tried creating 3 radi5 arrays and I had two bad disks out of the box. I have replaced them and repartitioned many times. dd’ed zeros to the partitions, moved the partitions, and tried re-creating them, but the mdadm –create command says it cannot create because the array is in manage mode. I have no /etc/mdadm directory, no mdadm running, zeroed partitions, and I still cannot create my arrays!!! What the heck do I need to do short of re-installing from scratch?
21st March 2011, 17:40
oh, and I forgot to say that any stop, fail, remove operations claim there is no such array. ‘cat /proc/mdstat’ claims there are no arrays.
21st March 2011, 17:54
@Michael:
mdadm –detail –scan
should tell you what the system sees even if you have no configuration files. You might want to look at zero-superblock if you removed a disk from the raid and want to cleanup raid config on it.
21st March 2011, 18:54
root@nexus:~# mdadm –detail –scan
root@nexus:~#
It returns nothing!
root@nexus:~# mdadm –force –create –verbose –auto=yes /dev/md0 –raid-devices=4 — spare-devices=1 /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sde4 /dev/sdf4
mdadm: –create would set mdadm mode to “create”, but it is already set to “manage”.
root@nexus:~#
21st March 2011, 19:06
Strange. I’ve never seen this error before. In this case, since you don’t have anything on the disks, I would start from scratch. Use zero-superblock to ensure you cleanup all the metadata on the partitions, and start from scratch.
21st March 2011, 19:20
6 disks, sda1 is the root partition and linux is installed there, partition 2 on all drives is swap, and 3 and 4 are to be a 6 disk arraay, partition 1 on c d e f (sdd is currently dead) will be an array. Is there any way I can preserve partition 1 on sda and still do the superblock zero? Is the superblock zero command really suppose to target the whole drive and not a partition?
21st March 2011, 20:46
You should be able to run mdadm –zero-superblock /dev/sda4 (aka on partitions not only on a drive).
21st March 2011, 21:45
mdadm –zero-superblock /dev/sda3
gives me
mdadm: Unrecognized md component device – /dev/sda3
I even tried to zero md0 but it could not open it for write!!
I think I am ready to toss this stupid thing and give up on RAID!!!! My older system (pre-mdadm) worked FINE! I will zero out ALL the disks and re-partition and re-install from scratch — I have already spent FAR too much time on this (4 days!)
25th March 2011, 05:07
Hi – great post thanks !!!
I am having issues though, I create a RAID 1 array (Ubuntu 10.04) using mdadm using /dev/sdc1 and /dev/sdd1 and when I later view the array with mdadm –query –detail /dev/md0 I see that the devices are now specified without their partition, ie. /dev/sdc (without the 1). Is this something to worry about ?
I noticed this when I removed an array member as a test, fdisked the drive and put it back in the array (mdadm –change –add /dev/md0 /dev/sdd1) – now when I look at the array details the first drive is /dev/sdc and the second is /dev/sdd1.
Any ideas ??
25th March 2011, 16:10
@ilium007: that is strange and definitely not normal. It should give you the actual partitions in the raid (just like /proc/mdstat). fdisk -l should also tell you the partitions and their type should be fd – Linux raid autodetect). I can’t tell why you get that; I have many systems running ubuntu lucid and never seen something like that.
25th March 2011, 23:06
I have had so many problems with software RAID and mdadm over the past week – I have spent probably 60 hours building, rebuilding arrays and testing. The worst one now is the loss of LVM logical volumes when I fail a raid1 member and then reboot. The logical vol disappears, in fact all LVM configuration goes. Obviously /dev/md0 is still there, but pvdisplay, vgdisplay and lvdisplay yield nothing.
I think I am just going to have to buy a hardware raid card.
29th April 2011, 05:22
[...] # mdadm –verbose –create /dev/md0 –level=0 –raid-devices=2 /dev/sdc1 /dev/sdd1 repeat procedure source [...]
9th May 2011, 08:48
ilium007- OK just went through a similar thing on the weekend. The problem is in your GRUB2. I think you need to update the GRUB2 configuration so it includes all of your md RAID array devices.
In GRUB2 during booting use the “e” key to edit the boot stanza you are using in GRUB, get rid of “splash” and “noquiet” then boot. You will see some illuminating messages. It may take a minute or 2 until the LVM access fails.
I actually think that your GRUB2 md configuration files are old, and don’t have all of the new disks defined for the md arrays. So, you probably generated your GRUB config when you had an md0 which included only one disk in your RAID1 array (the other was probably not there at that time). You have now added the new disk, but the /etc/mdadm/mdadm.conf file inside GRUB2 probably only has the original disk’s UUID in it. The file is in the “mini-system” inside grub2 which you can look at from the grub command line. So, you’ll have to figure out how to update grub2′s md configuration file, and write it to both disks! I am about to go on the same journey.
Also note that because grub2 will NOT access a VG if *any* of PVs in the VG are missing even if the missing PV has nothing on it all (yeah it is cr@p logic). So using LVM mirroring for anything required during boot is a big fat lose.
9th May 2011, 09:37
OK sorry some info in my previous note was wrong. md should find all of the devices which should be in the array upon boot, by scanning all disk partitions. So you should not have to update your grub configuration. But you need to make sure /etc/mdadm/mdadm.conf has “DEVICE partitions” as one of the lines in it. This means all disk partitions will be scanned. If you have limited the scan you may be in trouble. Also, I think if you have 2 x md devices (md1 and md2) and you have spread your VG across them, and both md1 and md2 are not accessible during grub2 boot, then your VG which contains /boot wont be available.
20th May 2011, 00:37
[...] Mdadm Cheat Sheet | MDLog:/sysadmin [...]
6th July 2011, 10:11
I had to add ‘–raid-devices=2′ to the create command:
mdadm –create –verbose /dev/md0 –level=1 –raid-devices=2 /dev/sda1 /dev/sdb2
hopefully this will help people who are getting:
mdadm: no raid-devices specified.
17th September 2011, 16:10
i second peter tailors comments.
invaluable tutorial!
23rd September 2011, 14:48
[...] These are the instructions for the samba and RAID 5 Ubuntu server configuration that I prefer to use. It’s an easy to manage set up that yields 80+ MB/s with burst speeds of 100MB/s using 5400 rpm Hitachi Drives. This is just a quick rundown of the procedure so that it is not forgotten. For further information visit: http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/ [...]
25th October 2011, 18:14
Thanks for an invaluable tutorial. My datacenter replaced a failing disk tonight and I had resynch the array.
the sfdisk trick was hugely helpful as was the info on adding the partitions to multiple MD raid devices (as per your discussion with David Morris).
I wasn’t quit sure what he meant about the grub changes though … Does this mean that if my other disk fails the system will be unable to boot?
25th October 2011, 20:45
@Ked: please see my reply on comment #26 on what I do to ensure grub is installed correctly on both drives.
28th December 2011, 07:31
[...] These are the instructions for the samba and RAID 5 Ubuntu server configuration that I prefer to use. It’s an easy to manage set up that yields 80+ MB/s with burst speeds of 100MB/s using 5400 rpm Hitachi Drives. This is just a quick rundown of the procedure so that it is not forgotten. For further information visit:http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/ [...]
28th December 2011, 21:31
Thank you for this guide. This was very helpful for me as I was setting up my home media/file server. I am using Ubuntu and Raid 1 for my backup storage and rotating hard drives off site.
31st December 2011, 15:40
[...] thanks to MDLog:/sysadmin. posted by specky in Uncategorized and have No Comments Tags: admin, Linux, [...]
1st February 2012, 06:02
[...] This info is taken from here. [...]