Loading ...Debian Lenny PXE Installation on Dell PowerEdge 1950/2950 servers: bnx2 annoyances
Debian Lenny installer brings in several improvements over the old stable Etch version, but for us this has been a big disappointment. Basically as I mentioned in an older post “bnx2 missing from stock Debian 2.6.24 etchnhalf kernel“, starting with kernel 2.6.24, the debian team removed the bnx2 kernel module out of the kernel due to license restrictions placed on the firmware. I am a long time debian fan and I can fully understand the reason why they have done that, but still, if you have several hundred servers that use the Broadcom NetXtreme II NIC (that uses the bnx2 module) then you have a problem. This post is intended to show how we solved this problem and hopefully help others in the same situation.
As I mentioned we have many Dell PE1950 and PE2950 servers and we deploy all new servers using PXE boot install and use preseeding to setup basic configurations (custom partitions, apt mirrors, etc.). This worked fine for some years now with debian Etch; let’s see what happens with lenny installer: after the initial pxe boot, the installation halts with this error:
This tells us that the network hardware needs non-free firmware files to operate, and asks us to load them using a USB stick or a floppy. Even though this solution is out of the question because we could not automate something like that, we did try to use the DRAC virtual media to load the firmware but that didn’t work either. We are performing the installation over a remote console (using a DRAC card) and we are not near the servers while performing the installation, as this is mostly automated. But let’s assume that if we can put the firmware on a floppy (?!) or USB stick we should be able to move on with the installation. Since this was not acceptable for us in the first place we had to find a solution to this issue.
Adding bnx2 firmware to the installer initrd
Debian has removed the firmware from the initrd, but it is possible for anyone to add it back in and have it working again. For this we have to uncompress the initrd.gz, add the bnx2 firmware and save the result back as initrd. Here is an older post that I wrote describing how we can edit the initramfs debian uses.
We uncompress the image using:
mkdir temp
cd temp/
zcat ../initrd.gz | cpio -iv
Download the firmware-bnx2 debian package:
cd ..
wget http://http.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14_all.deb
and extract the contents with:
dpkg-deb -x firmware-bnx2_0.14_all.deb bnx2
We need to copy the files under lib/firmware/bnx2-06-4.0.5.fw , bnx2-09-4.0.5.fw and bnx2/usr/share/initramfs-tools/hooks/firmware_bnx2 inside the extracted directory of the initrd and compress it back (from the extracted directory):
find . -print0 | cpio -0 -H newc -ov | gzip -c > ../initrd.gz
Automatically install firmware-bnx2
After this, a new pxe boot using the resulted initrd.gz image will allow the installer to set up the network interface and move on with the installation. Please note that the firmware-bnx2 package will not automatically be installed by the installer leaving the installed system with a problem as it can’t use the bnx2 network interface to go out on the network (and eventually install the package afterwards). So the best way to do this is to install the firmware-bnx2 package during installation; this can be done including in the preseeding file:
d-i pkgsel/include string openssh-server firmware-bnx2
This should install the firmware-bnx2 package automatically and have the system work on the network once it finishes the installation.
Disabling usb-storage in the installer
Even if this is not related to bnx2, another annoyance we encountered was caused by the way the hard drives were detected by the installer. Sometimes this is different from the finished installed system, changing sda to sdb and so on. For example in our preseeding file we would partition and format the first drive, and this would fail like this:
Looking at the console of the installer this shows that sda was detected as scsi removable disk (USB DRAC stuff), while the actual system disk was detected as sdb:
I am sure other people use usb storage during installation 
but we don’t need that at all, so I just chose the simplest solution to remove that completely from the initrd (I had one try to disable it using blacklist modules but that didn’t work). From the extracted content of the initrd I removed:
rm -rf lib/modules/2.6.26-1-amd64/kernel/drivers/usb/storage
and created the initrd without the module:
find . -print0 | cpio -0 -H newc -ov | gzip -c > ../initrd.gz
Now the lenny installer is able to run ok on our Dell servers. Hopefully this will be useful to others in the same situation. You can also download the resulted initrd using the above method from here (this is for amd64; if anyone needs it for i386 let me know and I will upload it also).
Update: as requested by email, here is also a i386 initrd image. It should work just fine, but I haven’t tested this as I don’t have any i386 installations available for this. Let me know if all worked as expected…
Update 20090518: apparently several people used the resulted initrd’s, and after lenny’s 5.0.1 upgrade these were no longer working (because of kernel update in the installer). I have fixed this and uploaded updated images for both i386 and amd64. Let me know if you encounter any problems by using them.
Tags: bnx2, Debian, debian-lenny, dell, initramfs, initrd, kernel, pxe
8th March 2009, 17:00
[...] Debian Lenny PXE Installation on Dell PowerEdge 1950/2950 servers: bnx2 annoyances | MDLog:/sysadmin WARN: Lenny’s installer no longer supports Broadcom’s NetXtreme II. Seeing as how this NIC (or one of its family members) is in pretty much every Dell manufactured in the last five years, this is an important "gotcha". Especially if you’re dumpster-diving in Corporate America’s dustbins for your hardware like me. [...]
20th March 2009, 11:49
I got so pissed off at debian i switched to slackware for all my servers because of the broadcomchipset. ahhhhhhhhhhhhhhhhhh frustration with debian!
20th March 2009, 12:18
@some admin: this is exactly the reason why I published the workaround for this problem. This is not at all complicated to be solved, but of course moving away from debian because of this, is your choice. I am sorry you didn’t find my post sooner when you encountered the problems with lenny installer.
Cheers,
- Marius -
21st March 2009, 07:10
[...] información: MDLog:/sysadmin, Ali [...]
31st March 2009, 11:54
We have PowerEdge 2950 servers running Etch, during dist-upgrading to Lenny it got the warnings below:
W: Possible missing firmware /lib/firmware/bnx2-09-4.0.5.fw for module bnx2
W: Possible missing firmware /lib/firmware/bnx2-06-4.0.5.fw for module bnx2
The dist-upgrade went fine and the system seems to work ok, but I’m afraid problems will arise when the system is rebooted. Does anyone has experience with this?
Is it enough when I do the following:
# wget http://http.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14_all.deb
# dpkg -i firmware-bnx2_014_all.deb
Kind regards,
Jurgen
31st March 2009, 12:09
@Jurgen: if you run 2.6.18 and haven’t upgraded to lenny kernel you should be fine.
If you run 2.6.26 from lenny (as you should) you will have to install firmware-bnx2 package before reboot, and that will do the trick. That’s all what is needed.
You can install it either like you said, by downloading it and installing it with dpkg, or if you have in your apt sources non-free you can just install it using regular apt tools, like aptitude install firmware-bnx2
- Marius -
31st March 2009, 12:42
Hi Marius,
Thx for the quick response. We run OpenVZ kernel 2.6.18-12-fza-686-bigmem. Do you also have experience with that?
Kind regards,
Jurgen
31st March 2009, 12:52
@Jurgen: I have no experience with that, sorry. Still I would say that you should have no problem with that. bnx2 was removed from kernel starting with 2.6.24, meaning 2.6.18 should be just fine.
31st March 2009, 12:55
Thx, Marius
16th April 2009, 03:53
We’ve installed Debian etch on a PowerEdge 1950 and upgraded it to Debian Lenny using #aptitude upgrade. After the full upgraded we’ve successfully logged in to Debian Lenny but it cannot detect the Network interface. Is there a way for Lenny to detect the Network interface using the bnx2 firmware?
16th April 2009, 06:49
I installed the bnx2 firmware (see post above) and added ‘bnx2′ to /etc/modules. This works fine for me. NIC is detected and can be used.
16th April 2009, 06:57
@Nani: as @Jurgen said just install the firmware-bnx2 package from the regular non-free repository. For more details, you might find my older post about this useful: http://www.ducea.com/2008/10/21/bnx2-missing-from-stock-debian-2624-etchnhalf-kernel/
14th May 2009, 12:05
I tried the initrd.gz provided here on a Proliant DL-380G5 with no luck. Replacing the Debian kernel in the installer with a vanilla one with a built-in bnx2 driver seemed to do the trick.
14th May 2009, 12:08
@Ilkka Mattila: what installer did you use? 5.0.1? I am publishing an updated version for that shortly.
15th May 2009, 04:28
The scsi subsystem tends to be a large annoyance nowadays, as many drivers use it, leading to much confusion with swapped disk names depending on which drive is detected first. The solution is to use a device-agnostic naming scheme. udev, which Debian uses, provides a few choices for this – take a look at the symlinks in /dev/disk/by-{id,uuid,label,path}. Other device-agnostic naming schemes include LVM, which doesn’t care about *where* it’s lvs reside, just as long as it can find them. Then, of course, there’s good old filesystem labeling (man e2label). By utilizing one/some/all of these methods, you’ll find that your OS no longer cares about which disk it’s data resides on – and neither will you.
4th June 2009, 08:08
As for the USB issues, since you’re already modifying the initrd you might as well remove it there too, but there is a very simple option available too:
add the kernel option ‘nousb’ or ‘nousbstorage’.
Adding nousb will also cripple DRAC. Adding nousbstorage upon booting Linux, will disable all removable media but keeps DRAC functionality.
5th June 2009, 11:05
@Christian: thanks for your suggestion. Actually I hate making changes in initrd (makes it harder to manage and update, etc.), and before doing this I have tried all other possible choices to do this without messing with that. I have tried to use nousbstorage as a pxe kernel option but somehow that was not working regardless on what I tried. This was the reason, why I had to remove the kernel module from the initrd. My original work on this was done with lenny 5.0 installer, but if you have that working (maybe it was fixed in 5.0.1?) please let me know, as it is would be the preferred solution.
Thanks,
- Marius -
2nd July 2009, 16:07
[...] things on dell pe 1950 / 2950 i need to add firmware for broadcom gigabit ethernet cards. based on this article: mkdir /tftpboot/debian-installer/amd64/tmp cd /tftpboot/debian-installer/amd64/tmp # link [...]
8th October 2009, 17:01
[...] utili link:1. http://www.ducea.com/2009/03/02/debian-lenny-pxe-installation-on-dell-poweredge-19502950-servers-bnx…2. [...]
4th November 2009, 14:22
Thanks for this! I encountered the exact same problem a week ago. Ended up booting with the older kernel until I could find a solution. Looks like I can try to “upgrade” again now
13th January 2010, 21:51
preseed.cfg in the root of the initrd.gz will be picked up automatically … took me a bit to find that
8th March 2010, 14:18
First of all, very useful info.. thanks!
It helped me through this mess of our autoinstall procedure being broken. Perhaps useful to mention, we use Dell R410 servers and it is also needed with those broadcom versions that a more current module is included in the ramdisk besides the firmware.
In that case, just download the official driver and compile it for the version kernel you’re pxe booting. cp the module files inside the extracted dir and archive it again like mentioned above
16th March 2010, 06:02
[...] Ggwdg.de Ducea.com [...]
22nd June 2010, 14:55
@Ger
I have the same issue… downloaded http://www.broadcom.com/docs/driver_download/NXII/linux-5.2.55.zip but having trouble compiling it!!! GRRRRR!
Any tips appreciated.
Remember that the debian initrd is in fact an initramfs… so no need to unpack it you can just concatinate another cpio.gz archive to it to get more stuff in the initial ramdisk… see:
http://wiki.debian.org/DebianInstaller/NetbootFirmware
17th November 2010, 03:33
Hi All,
i found the simplest way to solve the problem. I succesfully installed Debian Lenny 5.0.6 (version without bnx2 firmware in the intrd by default) on a PowerEdge 1950 via DRAC console redirection. Here below is the procedure:
1. Download bnx2 firmware debian package from here: http://packages.debian.org/lenny/firmware-bnx2
2. Make an iso file (I used magic iso to do this) with the just downloaded .deb package inside
3. Connect Debian Lenny netinst ISO to the virtual CD-ROM (in DRAC console)
4. Start and Follow the normal installation procedure until it shows the error message (bnx2 firmware not found)
3. In DRAC console disconnect Debian Lenny netinst ISO from the virtual CD-ROM
4. In DRAC console connect as a Virtual Floppy/USB the iso created before (the one containing the .deb package)
5. In the error window (in Debian installation) select “yes” when it asks: “Loading missing firmware from removable media”.
6. The system will load the correct firmware and installation will continue normally
7. After the firmware is loaded disconnect the iso with the bnx2 firmware and re-connect as a Virtual CDROM the Debian Lenny netinst iso (this is very important, otherwise the installation won’t be able to continue).
It worked for me and it saved me a lot of time!
Cheers.
Miticofioz
5th February 2011, 01:37
Thanks for the help!
Is there a way to create a bash script that helps all to create the initrd.gz correctly? thanks again.
Michelangelo
17th July 2011, 23:11
[...] información: MDLog:/sysadmin, Ali Aboosaidi Share and Enjoy: Posted in Cómo hacer? Tags: Broadcom NetXtreme, debian « [...]
12th October 2011, 08:59
Thanks works with http://downloads.julian360.net/Firmware/bnx2%20firmware%20pack%20by%20julian360.net.zip fine!!!!!!!! hope an upgrade work?!
Simon
26th October 2011, 04:09
I’d like to suggest to you to improve for DELL Administrators the tutorial adding the uploading of the drivers by DELL DRAC Card
Regards
2nd December 2011, 11:37
Can you order just the connecting tunnels to enlarge your system?
15th December 2011, 14:16
If you rm -rf usb/storage directory then squeeze installer kernel panics. Removing the contents of directory works.