Loading ...
Increase the speed of Linux Software RAID reconstruction
If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ;)), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.
To see your Linux kernel speed limits imposed on the RAID reconstruction use:
cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
1000In the system logs you can see something similar to:
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:
echo 50000 >/proc/sys/dev/raid/speed_limit_minThe results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):
watch cat /proc/mdstatShare This
6th September 2006, 04:39
Thank you for this nice tip. But what value should I set to speed_limit_min? If I give it a too high value (e.g. 100000), will it hurt my raid?
8th September 2006, 14:41
That should be ok and not cause any problems. Probably the system will not be able to reach that speed but by no means it will not hurt the system. If you are doing this with an idle system and wait to complete the RAID build this should be ok. If not, you might see the system responding much slower, as it will dedicate more resource to the RAID restoration.
21st September 2006, 01:01
Ahh, thanks. Didn’t help too much on these linked up IDEs, but 30 MB/s went to about 32, time dropped from 85 to 65 minutes within an instant.
3rd January 2007, 23:26
I’m desperately searching through the web the cause to the slow reconstruction speed of my Ubuntu Edgy raid1 array. It seems that my system is totally unsensible to variation in speed_limit_max and speed_limit_min.
Whatever value I set in those variables, resynch speed stucks between 4000 and 6000 Kb/sec.
Yes, my “diskparm -Tt” is ok while the system is healthy. Yes, those orrible performances are independent on system load and usage.
I’m starting to suspect that there are some issues with Ubuntu itself. I think I’m going to signal it on official forums, but it’s important to spread the web with this message: poor Ubuntu mdadm users: you’re are not alone!
2nd April 2007, 04:02
I cannot seem to write to this file as root:
root ~ # echo 50000 > /proc/sys/dev/raid/speed_limit_min
bash: /proc/sys/dev/raid/speed_limit_min: cannot overwrite existing file
root ~ # ls -l /proc/sys/dev/raid/speed_limit_min
-rwxr–r– 1 root root 0 Apr 2 00:00 /proc/sys/dev/raid/speed_limit_min
I am in the process of resynching.. does this lock the file?
2nd April 2007, 06:53
Steve:
“does this lock the file?” No. You can change it on the fly. The effect is instant (though you will not see this immediately on the real speed of the sync because it will depend on many other factors). Anyway regarding your questions you can change this at any time without anything special and the effect should be instant.
Why you are not able to write on the file? I am not sure… It might be something related to your OS/Kernel/SE/Permissions, etc. I can’t tell with the given information only.
HTH.
13th April 2007, 18:15
Doesn’t work on CentOS 4.4. the speeds always stay the same. I did see them change once but it went from 23000 down to 18000 after setting min and max to 50000. Since 23000 was over 20000 (which was supposed to be the max–according to /proc . . .) I don’t think it’s working at all. This is an old kernel. Centos is using 2.6.9.
4th June 2007, 06:26
I also had the “permission denied” problem on my ubuntu 6.06 machine. The trick is to REALLY switch user to root, as plain sudo does not work…
Try:
sudo -s -H
and then
echo 50000 >/proc/sys/dev/raid/speed_limit_min
and instantly I was down from 7000minutes to 240!! :-))
17th November 2007, 17:57
Great tip! Thank you!
I was struggling with my RAID recovering at no more than 256KB/s (min_speed was set to 100) with almost no I/O nor CPU activity. Now, with min_speed set to 5000, I get ~5MB/s, which is enough. Interesting that the system does seem to care about the max_speed (set to 100000) value too much.
19th February 2008, 03:31
[...] Now’s the time to go get a drink, see a movie, wash the car/dog/cat, file your taxes AND take a nap. This will take a long time, depending on the amount of data on the drives and the speed of your system. On my system, it reported 1500 min (that’s 25 hours) when it started. You can speed this up some (at the expense of load on the system), by following the advise here. [...]