urls

http://datadisk.co.uk/

Interesting Open Source:
corosync
openfiler
Linux Cluster Management Console:

http://architects.dzone.com/articles/who-follow-open-source-cloud

http://blog.programmableweb.com/

secure http
http://arstechnica.com/business/news/2011/03/https-is-great-here-is-why-everyone-needs-to-use-it-so-ars-can-too.ars

VeriSign ssl
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&actp=CROSSLINK&id=AR198

U Wisc IT help desk
http://kb.wisc.edu/helpdesk/

Apache Wiki
http://wiki.apache.org/httpd/FrontPage
http://www.w3schools.com/

iOS SDK 开发
http://blog.sina.com.cn/iphonesdk

Pocket Guide
http://users.cis.fiu.edu/~tho01/psg/


nodejs
MongoDB
http://jetty.codehaus.org/


Dropbox Startup experience
http://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/

NetApp
https://systems.nic.uoregon.edu/wiki/Netapp_Filers
http://www.netapp.com/us/support/university/learning.html
http://communities.netapp.com/community/netapp-blogs/shadeofblue/blog/2008/10/20/really-thin-provisioning
http://unixfoo.blogspot.com/2009/07/pocket-guide-for-netapp-commands.html

NetBackup
http://www.netbackupcommands.com/
http://www.linuxtopia.org/online_books/linux_lvm_guide/index.html

Xen
http://www.virtuatopia.com/index.php/Building_a_Xen_Virtual_Guest_Filesystem_using_Logical_Volume_Management_%28LVM%29
http://wiki.kartbuilding.net/index.php/LVM_on_XEN_%28dom0%29
http://greg.porter.name/
Backup XenServer VM script http://nchrissos.wordpress.com/2013/09/05/free-backup-for-citrix-xenserver-live-virtual-machines/

PV vs HVM
http://stackoverflow.com/questions/8103314/whats-the-difference-between-paravirtual-and-hvm-in-xen
HVM to PV
http://itproctology.blogspot.com/2009/06/pv-enable-hvm-on-xenserver.html#!/2009/06/pv-enable-hvm-on-xenserver.html

Cisco Aironet stopped working in Ubuntu Hardy

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/189398

Oracle
http://www.oracle-base.com/index.php
http://psoug.org/

Interactive Programming Tutorial http://www.ctothoughts.com/2012/07/interactive-programming-tutorials-net.html

Topic	Interactive Tutorial
REDIS key-value store	http://try.redis-db.com/
MongoDB NOSQL database	http://tutorial.mongly.com/tutorial/index
SQL	http://sqlzoo.net/
Python	http://www.learnpython.org/
Python	http://trypython.org
Ruby	http://tryruby.org
Haskell	http://tryhaskell.org
Scala	http://www.simplyscala.com
Python, Ruby, Javascript	http://labs.codecademy.com/
.NET	http://mottishaked.com/training/dot-net-tutorial
UNIX	http://www.cns.nyu.edu/unix/unix/demo.html#1
Fun social platform for collaborating	https://www.hackerrank.com/
Regular Expressions	http://regexone.com/

configure snapvault

http://indiawebsearch.com/content/how-to-configure-snapvault-using-network-appliance-data-ontap-on-a-fas3050-filer

Before we get started, the prerequisites are to have a "Primary" Filer and a "Seondary Filer" . A "Primary System QTree" and a "Secondary System QTree" . In case you did not remember what a QTree is, QTree in Data ONTAP terms from Network Appliance is simply referred to as a folder on a volume, QTrees are just like traditional or flex volumes, with minor differences such as they can not be cloned. A Qtree is just used to group similar data together, apply quotas etc

A Primary Filer is generally referred to as your "onsite" storage system, whose data you want to back up using Data ONTAP Snapvault (eg; source FAS3050 Filer)

A Secondary Filer is your offsite or another storage system where you want your data backed up to (eg; destination FAS3050 filer)

You will normally be performing the Snapvault data backup from the Primary Qtree to the Secondary Qtree. In a disaster recovery scenario, you will be restoring data from your destination FAS2050 filer's qtree to your source or Primary filer.

To configure Snapvault , we need to perform five simple commands. I will show you the Data ONTAP commands, that will configure SnapVault as your backup on FAS3050 filers. In our example, we will call our Primary Filer as Filer1 and our secondary filer as filer2.

Step 1:

- Configure the Primary Filer for Snapvault, add the Snapvault license by using the following command on a terminal session:

license add sv_primary_license

Step 2:

- Enable snapvault on primary by using the following command

options snapvault.enable on

Step 3:

- Specify the name of the secondary storage system on the primary by using the command

options snapvault.access host=NameOfTheSecondaryFilerHere

Note that you can also use the following command to all all hosts as destination filers

options snapvault.access host=all

Step 4:

- Perform steps 1 and 2 on the secondary filer as well

Step 5:

- Enable the compliance clock on the secondary filer by using the command:

date -c initialize

Step 6:

- You can also optionally create a "log volume" for snapvault logs, but we will by pass it here

Step 7:

- Finally on the secondary filer for each qtree you need to backup on the primary, create a initial baseline transfer

For example, if you want the qtree called /vol/wandata/windowsdata on the primary to be backed up to a qtree on secondary called /vol/sv_wandata/windowsdata , you will give the following command:

snapvault start -S filer1:/vol/wandata/windowsdata /vol/sv_wandata/windowsdata

Note that it is a good practiceto name your volumes with sv_ (indicating snapvault volume).

This initialization can take some time to complete., to check the progress, you can give the command:

snapvault status

That is it. The final step is to create a Snapvault schedule on Primary and secondary. Here are the steps

Step 8:

- Create a schedule on Primary by using the following command:

snapvault snap sched -x vol snap_name count [@day_list][@hour_list]

so for example, it will be:

snapvault snap sched -x filer1:/vol/wandata/windowsdata sv_nightly 150@mon-sat 0

meaning, snapvault will create backups with prefix sv_nightly for the primary qtree /vol/wandata/windowsdata at midnight monday to saturday and will retain 150 copies (one for every night)

Step 9:

Similar to step 8 configure the secondary, I will just demonstrate the command for the secondary

snapvault snap sched -x filer2:/vol/sv_wandata/windowsdata sv_nightly 150@mon-sat 0

NetApp firmware upgrade

Contents:

• NetApp firmwares
• upgrade NetApp disk and shelf firmware
• What triggers an update to Disk, Shelf, or ACP firmware in Data ONTAP 8 7-Mode
• Data ONTAP and Disk Qualification Package (DQP)

---

NetApp firmwares:
https://library.netapp.com/ecmdocs/ECMP1196990/html/GUID-B376C9C6-44F0-4D36-A6C1-0383CF4338C3.html

• Motherboard/BIOS/System firmware
  Begining Data Ontap 8.0, minimum BIOS release required enables automatic BIOS updates. Therefore, no need worry about upgrading BIOS from 8.x. After minimum version is running, subsequent updates take place automatically during the boot sequence whenever Data ONTAP detects a version resident on the boot device is more recent than the running version.
  filer> sysconfig -a
  filer> version -b
• Shelf Firmware
• Disk Firmware
• ACP (Alternate Control Path) Firmware
• SP (Service Processor) Firmware
  ssh to SP IP, login with "naroot" with root password
  SP filer> system console
• DQP (Disk Qualification Package)

-----
upgrade NetApp disk and shelf firmware
http://indiawebsearch.com/content/how-to-upgrade-disk-shelf-firmware-on-a-netapp-filer

This article provides step by step instructions on how to upgrade the Shelf Firmware on a NetApp Filer running Data OnTap version 7.2. The Filer has four storage shelves which need their firmware to be upgraded.

We would be using CIFS to map a drive to the C$ of the Filer from a Windows machine. You can do the same using NFS by mounting the root volume of the Filer on a Unix client

In our case, we would use Filer1 as the name of the filer on which we need to upgrade the Disk Shelf Firmware.

• Download the latest Shelf Firmware as a zip file from NetApps NOW site
• Unzip the contents to a temporary folder e.g.; C:\temp\FilerShelfFirmware
• Map a drive to the C$ share of Filer1 e.g. X:\
• Copy the contents of C:\temp\FilerShelfFirmware to X:\etc\shelf_fw
• Logon to the consoles of Filer1 using Putty
• Give the command “Priv set advanced” to enter advanced mode
• Check the current shelf firmware version by typing “sysconfig –v” in the output look for following lines, it shows the firmware version to be 35
  

  Shelf 1: AT-FCX Firmware rev. AT-FCX A: 35 AT-FCX B: 35
  Shelf 2: AT-FCX Firmware rev. AT-FCX A: 35 AT-FCX B: 35
  Shelf 3: AT-FCX Firmware rev. AT-FCX A: 35 AT-FCX B: 35
  Shelf 4: AT-FCX Firmware rev. AT-FCX A: 35 AT-FCX B: 35

• Type “storage download shelf” to update firmware on both filers
• Wait until you get the following output, it could take 70 seconds per shelf x 4 shelves so close to about 5 minutes:

  Tue Dec 30 12:17:42 PST [sfu.downloadSuccess:info]: [storage download shelf]: Firmware file AT-FCX.3700.SFW downloaded on 0b.shelf4.
  Tue Dec 30 12:17:42 PST [sfu.downloadSuccess:info]: [storage download shelf]: Firmware file AT-FCX.3700.SFW downloaded on 0b.shelf3.
  Tue Dec 30 12:17:42 PST [sfu.downloadSuccess:info]: [storage download shelf]: Firmware file AT-FCX.3700.SFW downloaded on 0b.shelf2.
  Tue Dec 30 12:17:42 PST [sfu.downloadSuccess:info]: [storage download shelf]: Firmware file AT-FCX.3700.SFW downloaded on 0b.shelf1.
  Tue Dec 30 12:17:42 PST [sfu.downloadSummary:info]: Shelf firmware updated on 4 shelves

• Check the new shelf firmware version by typing “sysconfig –v” it should now say 37 instead of 35

---

The following procedure describes how to upgrade the disk firmware on a NetApp Filer running Data OnTap version 7.2

The first step before upgrading the firmware is to check if the correct disk qualification packages are present on the filer. This determined by checking the datecode on the qual_devices_v3 file. Do the following to check the datecode

Log on to Filer Console using Putty Give the following command: “rdfile /etc/qual_devices_v3” The command would have outputted a lot of information, what we are interested in is the header, you might have to enable buffer logging in Putty to look at the buffer output or scroll back in Putty. We are looking for the datecode output in the header which looks like: ######### # This file is auto generated by the script file. # Copyright (c) 2000 Network Appliance, Inc. # All rights reserved. # # Datecode: 20080807

Then we need to install the “Disk Qualification Package” else skip to section for Downloading and installing Disk Firmware Instructions for downloading and installing the Disk Qualification Package

• Download the Disk Qualification Package as a zip file from: http://now.netapp.com/NOW/download/tools/diskqual/ and copy it to a temporary location on your C: drive e.g. C:\temp\FilerDiskFirmware\DiskQualificationPackage
  
• Map a drive to the C$ share of Filer e.g. Q:\
  
• Extract the files from the zip archive to the \etc directory on the Filer, e.g. Q:\etc
  
• Wait 5 minutes for the system to process the newly installed Disk Qualification Packages.
  

Instructions for Downloading and Installing Disk Firmware

• Go to http://now.netapp.com/NOW/download/tools/diskfw/ and click on the link that says “Download All Current Disk Firmware”
  
• On the next page, click on the “Download Zip” button and save the zip file to a temporary location on your C: drive, e.g. C:\Temp\FilerDiskFirmware\all.zip
  
• Extract the contents of the zip file to the /etc/disk_fw directory of the FILER e.g Q:\etc\disk_fw
  
• Check that the following option is enabled on the filer by giving the command:
  “options raid.background_disk_fw_update.enable” it should return options raid.background_disk_fw_update.enable on . If its not on, turn it on
  
• The filer should automatically detect that there is new disk firmware and do the upgrade, no reboot is necessary.
  

---

What triggers an update to Disk, Shelf, or ACP firmware in Data ONTAP 8 7-Mode

https://kb.netapp.com/support/index?page=content&actp=LIST&id=3013267

Disk Firmware:

Disk Firmware will be updated in one of the following three ways: 

In the Background:
This occurs when the following conditions are met:

• options raid.background_disk_fw_update.enable is on
  -or-
  options raid.disk.background_fw_update.raid4.enable is on (where applicable)
• Newer firmware for the disks present is placed in the /etc/disk_fw directory.
   

When manually triggered:
This occurs when the following conditions are met:

• options raid.background_disk_fw_update.enable is off
  -or-
  options raid.disk.background_fw_update.raid4.enable is off (where applicable)
• Newer firmware for the disks present is placed in the /etc/disk_fw directory
• The disk_fw_update command is issued.

Upon next reboot:
This occurs when the following conditions are met:

• options raid.background_disk_fw_update.enable is off
  -or-
  options raid.disk.background_fw_update.raid4.enable is off (where applicable)
• Newer firmware for disks present is placed in the /etc/disk_fw directory
• The system is rebooted.
   

Shelf Firmware:

Shelf Firmware is updated in one of the following three ways:
Note: There are some concerns specific to the AT-FCX modules which is addressed on each shelf FW download page.

When manually triggered:
This occurs when the following conditions are met:

• Newer firmware for the shelf modules present is placed in the /etc/shelf_fw directory.

-and-

• The following commands are issued:
  • priv set advanced
  • storage download shelf
     -or-
    storage download shelf adapter_name
  • priv set

Upon the next reboot:
This occurs when both the following conditions are met:

• Newer firmware for the shelf modules present is placed in the /etc/shelf_fw directory.
• The controller is rebooted.

Upon certain events being encountered:
This occurs when the following conditions are met:

• Newer firmware for the shelf modules present is placed in the /etc/shelf_fw directory.

-and-

• One of the following events occur:
  • The cf giveback command is issued
  • New disk drives are inserted
  • New shelf modules are inserted
  • A-SAS shelf is hot-added
     

ACP Firmware:
ACP Firmware is updated in one of the following two ways:

In the background:
This occurs when the following condition is met:

Newer firmware for the ACP is placed in the /etc/acpp_fw directory.
Note: No other action is required - Data ONTAP polls this directory periodically.

When manually triggered:
This occurs when the following conditions are met:

• Newer firmware for the ACP is placed in the /etc/acpp_fw directory.
• The following commands are issued:
  • priv set advanced
  • storage download acp
  • priv set

For complete information on all the firmware updates, see the Upgrade Guide for the version of Data ONTAP being used.

---

Data ONTAP and Disk Qualification Package (DQP)

Data ONTAP does not recognize a drive correctly.
Data ONTAP recognizes a 15k RPM drive as 10K RPM (default).

1. Where does one install the DQP package?
   Data ONTAP looks for the qual_devices* file in the /etc directory of the root volume.
    
2. Should one copy a specific version of the qual_devices file out of the archive or should one copy the contents of the entire DQP archive in the /etc folder? The minimum requirement is to install the version specific to the current Data ONTAP running on the storage system. The DQP file should be placed in the /etc folder in the root volume (/vol/vol0) on the filer.
   
   However, it is recommended to place all qual_devices* files (in the Disk Qualification Package) in the /etc folder to ensure smooth Data ONTAP upgrades in future.
    
3. How does Data ONTAP determine if the DQP file has changed? And, how frequently does it check for changes in the installed DQP file? Data ONTAP reads the modification time of the DQP file to determine if it has changed or not. If the file modification time is different from the last time it checked, it processes it.
   Data ONTAP checks approximately every 5-10 minutes to see if the file modification time has changed.
    
4. Does one have to reinstall DQP every time one adds a new drive or shelf?
   No, one does not have to install a new DQP every time one adds a new drive or shelf.
   Nonetheless, it is good practice to install the latest DQP, disk firmware, shelf firmware before the new hardware is added. The best time to do this is close to (before) adding new hardware.
    
5. Does Data ONTAP load DQP after a reboot?
   Yes, it loads DQP as a part of the boot up process if it finds unqualified disks that are unknown to the kernel. This may take place a few minutes after the initial boot up and therefore a little delay is expected.
    
6. Does Data ONTAP understand all versions of qual_device or does it understand only specific version depending upon the Data ONTAP revision?  
   Yes, specific revisions of Data ONTAP look for a particular revision of the /etc/qual_devices file as shown below. However, you still need to install/copy allqual_devices* files to the filer, regardless of what version you are on.
   
   qual_devices     For releases prior to 7.2.1
qual_devices_v2  For releases 7.2.1, 7.2.2 and 7.2.3
qual_devices_v2.copy Backup copy of v2 file
qual_devices_v3  For releases 7.2.4 and later
qual_devices_v3.copy Backup copy of v3 file
   
   Note: If the Data ONTAP version is 7.2.4 and later, then refer the latest version of 7.3. For reference, see:
   Burt 270735
   Burt 234290
7. How do I force Data ONTAP to re-read the Disk Qualification File?
   One can reboot the filer to have Data ONTAP re-read the qual_devices* files

CautionDO NOT edit the qual_devices* files or save the changes - any changes in the Qual Devices file will render the file meaningless and prevent Data ONTAP to read it successfully.

The latest qual_devices file is located at http://mysupport.netapp.com/NOW/download/tools/diskqual/

---

nagios notification

Object Inheritance http://nagios.sourceforge.net/docs/3_0/objectinheritance.html

define host {
 name              windows
 contact_groups    +windows
 ...
}
define host {
    name              citrix
    contact_groups    +citrix
    ...
}

For the citrix Windows VM, we want it inherit all the Windows properties, but not the contact group. This would not work.

define contact {
     contact_name      joe
     contact_groups    +windows, !citrix
     ...
}

Work around:

define host {
     use              windows, citrix
     contact_group    citrix
     ...
}

There seems to be a bug that host and service contact_groups overwrite each other. Similiar to bug http://tracker.nagios.org/view.php?id=166

define host {
     use              windows, citrix
     contact_group    citrix
     ...
}

would overwrite

define host {
    contact_group    +windows
    ...
}

end up with only critical group receive alert.

perl tips

all Perl programs should contain the following three lines:

#!/usr/bin/perl -Tw         # force taint checks, and print warnings
use strict;                 # install all three strictures
$|++;                       # force auto flush of output buffer

http://www.cs.mcgill.ca/~abatko/computers/programming/perl/howto/

NetApp Links

• Creating NFS Datastores on NetApp
• NetApp Buying guide
• http://unixfoo.blogspot.com/search/label/netapp
• https://exchangeengine.wordpress.com/category/netapp/
• Basic Concept https://communities.netapp.com/servlet/JiveServlet/previewBody/2999-102-1-3620/NetApp-Basic-Concepts-Quickstart-Guide.pdf
• NetApp cabling and hardware basics  SAS, ACP cabling
• Pocket Survival Guide http://users.cis.fiu.edu/~tho01/psg/netapp.html
• cheatsheet http://www.datadisk.co.uk/html_docs/netapp/
• https://www.youtube.com/user/NetAppKBTV
• http://www.oznetnerd.com/2015/02/15/netapp-ground-beginners-guide-index/
• 
  
  
  
  

NetApp FAS2020 failed to boot

NetApp FAS2020 failed to boot
I went to a customer today to configure a NetApp FAS2020 for SnapMirror replication. For sake of argument, let’s call this filer FLR2. After hooking up the fibre channel interconnect to a second diskshelf (containing SATA-disks), I noticed that all disks from the onboard shelf were missing. They were added to a second FAS2020 (FLR1), which is running all the production VM’s and thus has more benefit from the 300GB 10K SAS disks than FLR2, which was going to do only SnapMirror replication from FLR1 over a 20 Mbit internet connection. Without the SAS-disks in place, which hold aggr0 and the root volume (vol0), there wasn’t much to boot from. Hence, I had myself a very expensive piece of useless storage.
Because my experience with NetApp filers has been relatively minimal up until this point, I needed to dive into it deeply to figure out how the root volume works, why NetApp has placed it onto the ‘data’ disks (and not onto the CompactFlash storage), and most important, how to fix this mess.

The bootlog

I began by examining the output of the boot process. I hookup up the serial connection, and saw the following:

AMI BIOS8 Modular BIOS
Copyright (C) 1985-2006,  American Megatrends, Inc. All Rights Reserved
Portions Copyright (C) 2006 Network Appliance, Inc. All Rights Reserved
BIOS Version 3.0
+++++++++++++++

Boot Loader version 1.3
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
Portions Copyright (C) 2002-2006 Network Appliance Inc.

CPU Type: Mobile Intel(R) Celeron(R) CPU 2.20GHz

Starting AUTOBOOT press Ctrl-C to abort...
Loading:.................0x200000/33899484 0x22543dc/31974692 0x40d2900
 /2557760 Entry at 0x00200000
Starting program at 0x00200000
cpuid 0x80000000: 0x80000004 0x0 0x0 0x0
Press CTRL-C for special boot menu


[fci.initialization.failed:error]: Initialization failed on
 Fibre Channel adapter 0b.

NetApp Release 7.2.6.1: Wed Dec 10 21:26:02 PST 2008
Copyright (c) 1992-2008 NetApp.
Starting boot on Mon Nov  2 15:50:00 GMT 2009

[nvram.battery.turned.on:info]: The NVRAM battery is turned ON.
 It is turned OFF during system shutdown.
[diskown.isEnabled:info]: software ownership has been enabled
 for this system
[coredump.spare.none:info]: No sparecore disk was found.
 WARNING: 0 disks found!

Storage Adapters found:
2 Fibre Channel Storage Adapters found!
1 SAS Adapters found!
0 Parallel SCSI Storage Adapters found!
1 ATA Adapters found!

Target Adapters found:
0 Fibre Channel Target Adapters found!
1 iSCSI Target Adapters found!
1 Unknown Target Adapters found!

WARNING: there do not appear to be any disks attached to the system.

Check that disks have been assigned ownership to this
system (ID 135058219) using the 'disk show' and 'disk assign'
commands from maintenace mode.

No root volume found.
Rebooting... (ctrl-c to break reboot loop)

OK, so the bootloader is still functioning properly, but complaining about the lack of disks, which makes perfect sense, because the ownership of the disks hasn’t been claimed yet. More importantly, even if the filer had claimed the disks, there wouldn’t be anything on those disks, so the boot process would’ve stopped in a similar fashion anyway.

The solution

So, the filer doesn’t have any disks assigned to it and there are no volumes defined on those disks. Let’s fix that! See the red bits in the log below for details:

1. We’ll start out by pressing [CTRL] – [C] to enable the very special boot menu.
2. Select option 4a to assign ownership of a couple of disks, initialize them and create a flexible root volume.

AMI BIOS8 Modular BIOS
Copyright (C) 1985-2006,  American Megatrends, Inc. All Rights Reserved
Portions Copyright (C) 2006 Network Appliance, Inc. All Rights Reserved
BIOS Version 3.0
+++++++++++++++

Boot Loader version 1.3
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
Portions Copyright (C) 2002-2006 Network Appliance Inc.

CPU Type: Mobile Intel(R) Celeron(R) CPU 2.20GHz

Starting AUTOBOOT press Ctrl-C to abort...
Loading:.................0x200000/33899484 0x22543dc/31974692 0x40d2900
 /2557760 Entry at 0x00200000
Starting program at 0x00200000
cpuid 0x80000000: 0x80000004 0x0 0x0 0x0

Press CTRL-C for special boot menu
Special boot options menu will be available.

[fci.initialization.failed:error]: Initialization failed on
 Fibre Channel adapter 0b.

NetApp Release 7.2.6.1: Wed Dec 10 21:26:02 PST 2008
Copyright (c) 1992-2008 NetApp.
Starting boot on Mon Nov  2 15:57:08 GMT 2009

[nvram.battery.turned.on:info]: The NVRAM battery
 is turned ON. It is turned OFF during system shutdown.
[diskown.isEnabled:info]: software ownership has
 been enabled for this system

(1)  Normal boot.
(2)  Boot without /etc/rc.
(3)  Change password.
(4)  Assign ownership and initialize disks for root volume.
(4a) Same as option 4, but create a flexible root volume.
(5)  Maintenance mode boot.
Selection (1-5)? 4a

The system has 0 disks whereas it needs 3 to boot, will try to assign the
required number.

[diskown.changingOwner:info]: changing ownership for disk 0a.18
 from unowned (ID -1) to  (ID 135058219)
[diskown.changingOwner:info]: changing ownership for disk 0a.23
 from unowned (ID -1) to  (ID 135058219)

[diskown.changingOwner:info]: changing ownership for disk 0a.27
 from unowned (ID -1) to  (ID 135058219)

Zero disks and install a new file system? y
This will erase all the data on the disks, are you sure? y
Zeroing disks takes about 280 minutes.

[coredump.spare.none:info]: No sparecore disk was found.

[raid.disk.zero.done:notice]: Disk 0a.18 Shelf 1 Bay 2
 [X269_SMOOS01TSSX NA01]: disk zeroing complete
[raid.disk.zero.done:notice]: Disk 0a.27 Shelf 1 Bay 11
 [X269_SMOOS01TSSX NA01]: disk zeroing complete
[raid.disk.zero.done:notice]: Disk 0a.23 Shelf 1 Bay 7
 [X269_SMOOS01TSSX NA01]: disk zeroing complete

[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.27 Shelf 1 Bay 11 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully
[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.23 Shelf 1 Bay 7 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully

[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.18 Shelf 1 Bay 2 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully

[wafl.vol.add:notice]: Aggregate aggr0 has been added to the system.
[fmmbx_instanceWorke:info]: no mailbox instance on local side
[fmmb.current.lock.disk:info]: Disk 0a.18 is a local HA mailbox disk.
[fmmb.current.lock.disk:info]: Disk 0a.23 is a local HA mailbox disk.
[fmmbx_instanceWorke:info]: normal mailbox instance on local side

After configuring the hostname, network settings for e0a and the bmc-adapter, I was ready to hit NetApp FilerView to configure the rest. But the web server that should be running FilerView wasn’t running. Back to the console log once more:

[init_java:warning]: Java disabled:  Missing /etc/java/rt131.jar.
[httpd_servlet:warning]: Java Virtual Machine not accessible

[asup.config.minimal.unavailable:warning]: Minimal Autosupports
 unavailable. Could not read /etc/asup_content.conf
[sysconfig.sysconfigtab.openFailed:notice]: sysconfig: table of
 valid configurations (/etc/sysconfigtab) is missing.

Hmm, it seems the filer is still missing a lot of files, all in the /etc folder, which should reside on the root volume. Most importantly, the Java bits needed to use FilerView weren’t there. It looks like the Data ONTAP installation still isn’t recovered fully. As I noticed this filer’s ONTAP version (7.2.6.1) was old anyway, I decided to upgrade the software in the usual fashion. I browsed to the NOW page for ONTAP v7.3.2, Checked out the release notes, changed features page, the requirements for running this version, read up on the known problems and limitations and important cautions.
I was convinced that this release was the right one for me. As this was the first time I did an actual upgrade of ONTAP, I decided to read up on the Data ONTAP Upgrade Guide as well. While we’re at it, I downloaded the complete set of documentation for Data ONTAP here. Lastly, I grabbed the ‘System Files‘ and ‘Netboot Image‘.
Upgrading itself seemed easy: the filer wasn’t performing any tasks or I/O and isn’t in any cluster. This means I can simply jank out the power cables if I wanted to. Not that that’s such a good idea when upgrading firmware, but still. The second thing that made the upgrade easier was that the BIOS/NABL (v3.0), Diagnostics (v5.3.7) and the BMC controller firmware (v1.2) are all up to date.
Since the wipe-out of ONTAP by removing the SAS disks also destroyed any reference to licenses, I could not place the ONTAP upgrade software on the filer using CIFS or NFS, or even iSCSI or FCP. The only option I could use was to download it through HTTP. So I placed the System File onto a HTTP server, ready for the filer to pick up the bits and do its magic.

flr2> software update http://webserver/ontap/732_setup_e.exe

software: copying to 732_setup_e.exe software:
100% file read from location.
software: /etc/software/732_setup_e.exe has been copied.
software: installing software, this could take a few minutes...
software: installation of 732_setup_e.exe completed.

[rc:info]: software: installation of 732_setup_e.exe completed.
 Please type "download" to load the new software, and "reboot"
 subsequently for the changes to take effect.

flr2> download 
[download.request:notice]: Operator requested download initiated
 download: Downloading boot device download: If upgrading from a
 version of Data ONTAP prior to 7.3, please ensure download: there
 is at least 3% of available space on each aggregate before upgrading.
 Additional information can be found in the release notes.

Version 1 ELF86 kernel detected.............
download: Downloading boot device (Service Area).........
[download.requestDone:notice]: Operator requested download completed

flr2> reboot 
[kern.shutdown:notice]: System shut down because : "reboot".

Wow, that’s way easier than I expected. Copy the file to the filer, install the .exe image, activate the new code on the filer’s boot device and actually reboot the unit. During reboot, a couple of messages about updating software on one of the shelfs flew by:

[kern.version.change:notice]: Data ONTAP kernel version was changed from
 NetApp Release 7.2.6.1 to NetApp Release 7.3.2.

[sas.adapter.firmware.download:info]: Updating firmware on SAS adapter 0c
 from version 1.24.03.00 to version 1.26.03.00.
[dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
[sfu.firmwareDownrev:warning]: Disk shelf firmware needs to be updated on
 1 disk shelf.
[sfu.ctrllerElmntsPerShelf:info]: [storage download shelf]: 1 ES
 controller element can be updated on 0c.shelf0.
[sfu.downloadStarted:info]: Update of disk shelf firmware started
 on 1 shelf.
[sfu.downloadingController:info]: [storage download shelf]: Downloading
 SAS.0500.SFW on disk shelf controller module B on 0c.shelf0.
[sfu.suspendDiskIO:info]: Suspending disk I/O to reboot shelf modules for
 shelf firmware update.
[sfu.rebootRequest:info]: Issuing a request to reboot disk shelf
 0c.shelf0 module B.
[ses.shelf.fw.update.pfu.nrdnt:info]: SCSI Enclosure Services (SES)
 shelf firmware update will be performed using Package Firmware
 Update (PFU) due to not having redundant paths.

[sfu.adapterSuspendIO:info]: Suspending I/O to SAS adapter 0c for
 40 seconds while shelf firmware is updated.
[sfu.resumeDiskIO:info]: Resuming disk I/O after shelf firmware update.
[sfu.downloadSuccess:info]: [storage download shelf]: Firmware file
 SAS.0500.SFW downloaded on 0c.shelf0.
[sfu.downloadSummary:info]: Shelf firmware updated on 1 shelf.
[sfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk shelves.

Nothing shocking to notice here, just confirmation that the ONTAP kernel version has been upgraded, that the SAS Adapter 0c and ES shelf controller 0c firmware will be upgraded and rebooted afterwards. Also, good to see the extra confirmation that all disk shelves and disk drives have current firmware versions after upgrading!

Concluding

Don’t ever take out the disks if the root volume is running on them. Period.
If you do, like this customer did, you’ll have yourself a couple of hours o’ fun restoring and upgrading ONTAP, as well as a heart-to-heart with the people who removed them in the first place, which is likely to take up the remainder of the day.