perl tips

all Perl programs should contain the following three lines:

#!/usr/bin/perl -Tw         # force taint checks, and print warnings
use strict;                 # install all three strictures
$|++;                       # force auto flush of output buffer

http://www.cs.mcgill.ca/~abatko/computers/programming/perl/howto/

NetApp Links

• Creating NFS Datastores on NetApp
• NetApp Buying guide
• http://unixfoo.blogspot.com/search/label/netapp
• https://exchangeengine.wordpress.com/category/netapp/
• Basic Concept https://communities.netapp.com/servlet/JiveServlet/previewBody/2999-102-1-3620/NetApp-Basic-Concepts-Quickstart-Guide.pdf
• NetApp cabling and hardware basics  SAS, ACP cabling
• Pocket Survival Guide http://users.cis.fiu.edu/~tho01/psg/netapp.html
• cheatsheet http://www.datadisk.co.uk/html_docs/netapp/
• https://www.youtube.com/user/NetAppKBTV
• http://www.oznetnerd.com/2015/02/15/netapp-ground-beginners-guide-index/
• 
  
  
  
  

NetApp FAS2020 failed to boot

NetApp FAS2020 failed to boot
I went to a customer today to configure a NetApp FAS2020 for SnapMirror replication. For sake of argument, let’s call this filer FLR2. After hooking up the fibre channel interconnect to a second diskshelf (containing SATA-disks), I noticed that all disks from the onboard shelf were missing. They were added to a second FAS2020 (FLR1), which is running all the production VM’s and thus has more benefit from the 300GB 10K SAS disks than FLR2, which was going to do only SnapMirror replication from FLR1 over a 20 Mbit internet connection. Without the SAS-disks in place, which hold aggr0 and the root volume (vol0), there wasn’t much to boot from. Hence, I had myself a very expensive piece of useless storage.
Because my experience with NetApp filers has been relatively minimal up until this point, I needed to dive into it deeply to figure out how the root volume works, why NetApp has placed it onto the ‘data’ disks (and not onto the CompactFlash storage), and most important, how to fix this mess.

The bootlog

I began by examining the output of the boot process. I hookup up the serial connection, and saw the following:

AMI BIOS8 Modular BIOS
Copyright (C) 1985-2006,  American Megatrends, Inc. All Rights Reserved
Portions Copyright (C) 2006 Network Appliance, Inc. All Rights Reserved
BIOS Version 3.0
+++++++++++++++

Boot Loader version 1.3
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
Portions Copyright (C) 2002-2006 Network Appliance Inc.

CPU Type: Mobile Intel(R) Celeron(R) CPU 2.20GHz

Starting AUTOBOOT press Ctrl-C to abort...
Loading:.................0x200000/33899484 0x22543dc/31974692 0x40d2900
 /2557760 Entry at 0x00200000
Starting program at 0x00200000
cpuid 0x80000000: 0x80000004 0x0 0x0 0x0
Press CTRL-C for special boot menu


[fci.initialization.failed:error]: Initialization failed on
 Fibre Channel adapter 0b.

NetApp Release 7.2.6.1: Wed Dec 10 21:26:02 PST 2008
Copyright (c) 1992-2008 NetApp.
Starting boot on Mon Nov  2 15:50:00 GMT 2009

[nvram.battery.turned.on:info]: The NVRAM battery is turned ON.
 It is turned OFF during system shutdown.
[diskown.isEnabled:info]: software ownership has been enabled
 for this system
[coredump.spare.none:info]: No sparecore disk was found.
 WARNING: 0 disks found!

Storage Adapters found:
2 Fibre Channel Storage Adapters found!
1 SAS Adapters found!
0 Parallel SCSI Storage Adapters found!
1 ATA Adapters found!

Target Adapters found:
0 Fibre Channel Target Adapters found!
1 iSCSI Target Adapters found!
1 Unknown Target Adapters found!

WARNING: there do not appear to be any disks attached to the system.

Check that disks have been assigned ownership to this
system (ID 135058219) using the 'disk show' and 'disk assign'
commands from maintenace mode.

No root volume found.
Rebooting... (ctrl-c to break reboot loop)

OK, so the bootloader is still functioning properly, but complaining about the lack of disks, which makes perfect sense, because the ownership of the disks hasn’t been claimed yet. More importantly, even if the filer had claimed the disks, there wouldn’t be anything on those disks, so the boot process would’ve stopped in a similar fashion anyway.

The solution

So, the filer doesn’t have any disks assigned to it and there are no volumes defined on those disks. Let’s fix that! See the red bits in the log below for details:

1. We’ll start out by pressing [CTRL] – [C] to enable the very special boot menu.
2. Select option 4a to assign ownership of a couple of disks, initialize them and create a flexible root volume.

AMI BIOS8 Modular BIOS
Copyright (C) 1985-2006,  American Megatrends, Inc. All Rights Reserved
Portions Copyright (C) 2006 Network Appliance, Inc. All Rights Reserved
BIOS Version 3.0
+++++++++++++++

Boot Loader version 1.3
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
Portions Copyright (C) 2002-2006 Network Appliance Inc.

CPU Type: Mobile Intel(R) Celeron(R) CPU 2.20GHz

Starting AUTOBOOT press Ctrl-C to abort...
Loading:.................0x200000/33899484 0x22543dc/31974692 0x40d2900
 /2557760 Entry at 0x00200000
Starting program at 0x00200000
cpuid 0x80000000: 0x80000004 0x0 0x0 0x0

Press CTRL-C for special boot menu
Special boot options menu will be available.

[fci.initialization.failed:error]: Initialization failed on
 Fibre Channel adapter 0b.

NetApp Release 7.2.6.1: Wed Dec 10 21:26:02 PST 2008
Copyright (c) 1992-2008 NetApp.
Starting boot on Mon Nov  2 15:57:08 GMT 2009

[nvram.battery.turned.on:info]: The NVRAM battery
 is turned ON. It is turned OFF during system shutdown.
[diskown.isEnabled:info]: software ownership has
 been enabled for this system

(1)  Normal boot.
(2)  Boot without /etc/rc.
(3)  Change password.
(4)  Assign ownership and initialize disks for root volume.
(4a) Same as option 4, but create a flexible root volume.
(5)  Maintenance mode boot.
Selection (1-5)? 4a

The system has 0 disks whereas it needs 3 to boot, will try to assign the
required number.

[diskown.changingOwner:info]: changing ownership for disk 0a.18
 from unowned (ID -1) to  (ID 135058219)
[diskown.changingOwner:info]: changing ownership for disk 0a.23
 from unowned (ID -1) to  (ID 135058219)

[diskown.changingOwner:info]: changing ownership for disk 0a.27
 from unowned (ID -1) to  (ID 135058219)

Zero disks and install a new file system? y
This will erase all the data on the disks, are you sure? y
Zeroing disks takes about 280 minutes.

[coredump.spare.none:info]: No sparecore disk was found.

[raid.disk.zero.done:notice]: Disk 0a.18 Shelf 1 Bay 2
 [X269_SMOOS01TSSX NA01]: disk zeroing complete
[raid.disk.zero.done:notice]: Disk 0a.27 Shelf 1 Bay 11
 [X269_SMOOS01TSSX NA01]: disk zeroing complete
[raid.disk.zero.done:notice]: Disk 0a.23 Shelf 1 Bay 7
 [X269_SMOOS01TSSX NA01]: disk zeroing complete

[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.27 Shelf 1 Bay 11 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully
[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.23 Shelf 1 Bay 7 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully

[raid.vol.disk.add.done:notice]: Addition of Disk
 /aggr0/plex0/rg0/0a.18 Shelf 1 Bay 2 [X269_SMOOS01TSSX NA01]
 to aggregate aggr0 has completed successfully

[wafl.vol.add:notice]: Aggregate aggr0 has been added to the system.
[fmmbx_instanceWorke:info]: no mailbox instance on local side
[fmmb.current.lock.disk:info]: Disk 0a.18 is a local HA mailbox disk.
[fmmb.current.lock.disk:info]: Disk 0a.23 is a local HA mailbox disk.
[fmmbx_instanceWorke:info]: normal mailbox instance on local side

After configuring the hostname, network settings for e0a and the bmc-adapter, I was ready to hit NetApp FilerView to configure the rest. But the web server that should be running FilerView wasn’t running. Back to the console log once more:

[init_java:warning]: Java disabled:  Missing /etc/java/rt131.jar.
[httpd_servlet:warning]: Java Virtual Machine not accessible

[asup.config.minimal.unavailable:warning]: Minimal Autosupports
 unavailable. Could not read /etc/asup_content.conf
[sysconfig.sysconfigtab.openFailed:notice]: sysconfig: table of
 valid configurations (/etc/sysconfigtab) is missing.

Hmm, it seems the filer is still missing a lot of files, all in the /etc folder, which should reside on the root volume. Most importantly, the Java bits needed to use FilerView weren’t there. It looks like the Data ONTAP installation still isn’t recovered fully. As I noticed this filer’s ONTAP version (7.2.6.1) was old anyway, I decided to upgrade the software in the usual fashion. I browsed to the NOW page for ONTAP v7.3.2, Checked out the release notes, changed features page, the requirements for running this version, read up on the known problems and limitations and important cautions.
I was convinced that this release was the right one for me. As this was the first time I did an actual upgrade of ONTAP, I decided to read up on the Data ONTAP Upgrade Guide as well. While we’re at it, I downloaded the complete set of documentation for Data ONTAP here. Lastly, I grabbed the ‘System Files‘ and ‘Netboot Image‘.
Upgrading itself seemed easy: the filer wasn’t performing any tasks or I/O and isn’t in any cluster. This means I can simply jank out the power cables if I wanted to. Not that that’s such a good idea when upgrading firmware, but still. The second thing that made the upgrade easier was that the BIOS/NABL (v3.0), Diagnostics (v5.3.7) and the BMC controller firmware (v1.2) are all up to date.
Since the wipe-out of ONTAP by removing the SAS disks also destroyed any reference to licenses, I could not place the ONTAP upgrade software on the filer using CIFS or NFS, or even iSCSI or FCP. The only option I could use was to download it through HTTP. So I placed the System File onto a HTTP server, ready for the filer to pick up the bits and do its magic.

flr2> software update http://webserver/ontap/732_setup_e.exe

software: copying to 732_setup_e.exe software:
100% file read from location.
software: /etc/software/732_setup_e.exe has been copied.
software: installing software, this could take a few minutes...
software: installation of 732_setup_e.exe completed.

[rc:info]: software: installation of 732_setup_e.exe completed.
 Please type "download" to load the new software, and "reboot"
 subsequently for the changes to take effect.

flr2> download 
[download.request:notice]: Operator requested download initiated
 download: Downloading boot device download: If upgrading from a
 version of Data ONTAP prior to 7.3, please ensure download: there
 is at least 3% of available space on each aggregate before upgrading.
 Additional information can be found in the release notes.

Version 1 ELF86 kernel detected.............
download: Downloading boot device (Service Area).........
[download.requestDone:notice]: Operator requested download completed

flr2> reboot 
[kern.shutdown:notice]: System shut down because : "reboot".

Wow, that’s way easier than I expected. Copy the file to the filer, install the .exe image, activate the new code on the filer’s boot device and actually reboot the unit. During reboot, a couple of messages about updating software on one of the shelfs flew by:

[kern.version.change:notice]: Data ONTAP kernel version was changed from
 NetApp Release 7.2.6.1 to NetApp Release 7.3.2.

[sas.adapter.firmware.download:info]: Updating firmware on SAS adapter 0c
 from version 1.24.03.00 to version 1.26.03.00.
[dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
[sfu.firmwareDownrev:warning]: Disk shelf firmware needs to be updated on
 1 disk shelf.
[sfu.ctrllerElmntsPerShelf:info]: [storage download shelf]: 1 ES
 controller element can be updated on 0c.shelf0.
[sfu.downloadStarted:info]: Update of disk shelf firmware started
 on 1 shelf.
[sfu.downloadingController:info]: [storage download shelf]: Downloading
 SAS.0500.SFW on disk shelf controller module B on 0c.shelf0.
[sfu.suspendDiskIO:info]: Suspending disk I/O to reboot shelf modules for
 shelf firmware update.
[sfu.rebootRequest:info]: Issuing a request to reboot disk shelf
 0c.shelf0 module B.
[ses.shelf.fw.update.pfu.nrdnt:info]: SCSI Enclosure Services (SES)
 shelf firmware update will be performed using Package Firmware
 Update (PFU) due to not having redundant paths.

[sfu.adapterSuspendIO:info]: Suspending I/O to SAS adapter 0c for
 40 seconds while shelf firmware is updated.
[sfu.resumeDiskIO:info]: Resuming disk I/O after shelf firmware update.
[sfu.downloadSuccess:info]: [storage download shelf]: Firmware file
 SAS.0500.SFW downloaded on 0c.shelf0.
[sfu.downloadSummary:info]: Shelf firmware updated on 1 shelf.
[sfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk shelves.

Nothing shocking to notice here, just confirmation that the ONTAP kernel version has been upgraded, that the SAS Adapter 0c and ES shelf controller 0c firmware will be upgraded and rebooted afterwards. Also, good to see the extra confirmation that all disk shelves and disk drives have current firmware versions after upgrading!

Concluding

Don’t ever take out the disks if the root volume is running on them. Period.
If you do, like this customer did, you’ll have yourself a couple of hours o’ fun restoring and upgrading ONTAP, as well as a heart-to-heart with the people who removed them in the first place, which is likely to take up the remainder of the day.

NetApp CIFs Tricks

http://blog.aarondelp.com/2010/01/netapp-cifs-tricks.html
I recently had to perform the following during a CIFs (Windows File Sharing) installation from a NetApp storage controller.  The chances of me remembering this again aren't very good so I wanted to post it here for later.  We had two issues that caused us some grief.

Issue #1 - For whatever reason when looking for a domain controller it wasn't "attaching" to the local domain controller.  The system would ask for a list of domain controllers but then try to communicate with remote AD servers, some of which were behind firewalls.  NetApp is nice enough to allow us to "pin" the storage to a preferred list of domain controllers to correct this behavior.  From the command line, use the following commands:

• cifs domaininfo - lists which domain controllers the NetApp is communicating with.  The preferred list is a list you specify, the favored list is the list AD thinks are closet to you, and then the rest are listed.
• cifs prefdc - This command allows you to populate a list of the domain controllers you want to communicate with first.  More than one can be entered in the command seperated by spaces in the format: cifs prefdc add (domain) (dc1) (dc2) (etc...)
• cifs resetdc - After a dc is added you need to reset the connection
• cifs prefdc print - Shows the list

 Issue #2 - The site admin wasn't a domain admin.  This leads to many permission related issues because by default when a NetApp is added to AD only the local NetApp admin (created during CIFS setup) and the Domain Admins are in the machine administrators group.  We needed to add the site admin into the Administrators group on the NetApp.  This was achieved using the useradmin command.  Here is the syntax: useradmin domainuser add (username) -g Administrators

After these two steps were complete, we were able to proceed.

jsh
An extremely neat command (undocumented) in NetApp DataOntap OS is "java netapp.cmds.jsh" which enables a command shell much like a stripped down version of linux. in this shell, you can use commands like "cd", "ls", "cat", etc.

NetApp Setup Cheat Sheet

http://blog.aarondelp.com/2009/05/netapp-setup-cheat-sheet.html 
This is getting a little closer to something I am happy with.  I have tried to list both telnet(SSH) and FilerView for everything up to the volume level.  I will fill in the rest later.  After that, my next major update will be to add Systems Manager for each step.  I am waiting on Systems Manager version 1.1 to be released before I update that piece.  I have also noted where vifs will be changed to interface groups (ifgrp) in ONTAP8.


Latest Revision Date: 10/28/2009 tested on ONTAP 7.3.1

Setting Up a Filer:

1. Check version and licenses

• (telnet) license to list
• (telnet) license add (key) to add a license
• (FilerView) Filer -> Manage Licenses
  

2. Setup Network Interfaces (set up single-mode and multi-mode vifs or ifgrps)

• NOTE: You can't modiy a vif once it is created.  I usually have to delete the vif and start over to modify it.  This also means network service will go down on that port group!
  
• (telent) ifconfig -a to list the interfaces
• (telnet) ifconfig command to manage the interfaces
  
• (telent) To manage vifs in ONTAP 7: vif command, in ONTAP 8: ifgrp
• (telnet) You can also run setup, will probably require a reboot!
  
• (FilerView) Network -> Manage Interfaces -> i.e. vif1a or vif1b

3. Enable SSH on the filer

• (telnet)  secureadmin setup ssh
  
• (FilerView) Secure Admin ->SSH -> Configure -> Generate Keys -> OK -> Apply

4. Set Snap reserve on Aggr0 if not MetroCluster

• The default for this is 5%.  I have seen some set this to 0% if not using MetroCluster.  I set it to 3% because I have seen issues if set to 0% and NetApp support likes at least a little reserve at this level for disaster recovery
• (telnet) snap reserve -A aggr0 3
• (FilerView) I don't think this is possible from FilerView
  

5. Resize Vol0

• (telnet) vol size vol0 15g for 2000 series (20gb on 3000 series)
• (telnet) vol size vol0 to check
• (FilerView) Volumes -> Manage -> Click on vol0
• Click Resize Storage
• Click Next -> Click Next -> Enter New Volume Size -> Next -> Commit -> Close
  

6. Iscsi & FCP status

• (telnet) iscsi status to check
• (telnet) iscsi start to enable
• (telnet) same commands for fcp
  
• (FilerView) LUNs -> Enable/Disable ->Check Enable Box
  

7. NTP Setup

• (telnet)  to list all time options: options timed
• (telnet) options timed.servers 0.us.pool.ntp.org,1.us.pool.ntp.org,2.us.pool.ntp.org (or enter your time servers for your site)
  
• (telnet) options timed.proto ntp (if not already set to ntp)
• (telent) options timed.log on (if you would like the updates to go to console and log)
• (telnet) options timed.enable on
  
• (FilerView) Filer -> Set Date/Time -> Modify Date/Time
• Choose Time Zone -> Click Next
• Choose Automatic -> Click Next
• Change Protocol to ntp -> Click Next
• Enter ntp servers 0.us.pool.ntp.org (1&2) - or any other ntp server you have
• Click Commit -> Click Close
  

8. Enable and Test Autosupport

• (telnet) to list all options: options autosupport 
• (telnet) options autosupport.from (userid) - sets the userid autosupport is sent from
• (telnet) options autosupport.mailhost (host ip or name)
• (telnet) options autosupport.to (user1,user1)
• (telnet) options autosupport.enable on
• (telnet) options.autosupport.doit test - generates a test autosupport
  
• (FilerView) Filer -> Configure Autosupport
• Change Autosupport enabled to Yes
• Enter mailhosts
• Enter To: fields -> Click Apply
• Click Test Autosupport on the left -> Click Send Mail
  

9. Check Cluster Failover

• (telnet) cf status – Check Cluster Failover Status
• (telnet) cf partner - Lists the partner head
  
• (telent) cf monitor – more details on failover
• (telnet) cf  takeover - This will reboot the host you are taking over!
• (telnet) cf giveback - when partner is ready to receive back
  
• (FilerView) Cluster - Click the buttons!  :)
  

10. Disk Commands (Assign Drives to Controller)

• Typically I would assign the odd disks to the first controller and the evens to the second controller.  I have also assigned the majority of drives to one controller and a minimum (3-4 drives) to the other controller to maximize capacity
  
• (telnet) disk show –n - show disks NOT owned
• (telnet) disk show –o (controller name) - show disks owned by a controller
• (telnet) disk show –v - show all disks owned and not owned
• (telnet) disk assign (drive#)
• (telnet) disk remove_ownership (drive#) (This is a priv set advanced command!)
  
• (telent) disk zero spares
• (FilerView) - I don't think this is possible from FilerView
  

11. Aggregate Commands (Assign Drives to Aggregate)

• In RAID-DP environments, if there are 2 hot spares, I often add one back in.  You already have 2 drive failure with RAID-DP and this way you have one more drive capacity
• (telnet) aggr status -v - shows status with volume information
  
• (telnet) aggr status –r - shows which drives are in the aggr already including raid groups
  
• (telnet) aggr options (aggr) - shows all options for the aggr (raid groups, raid type, etc.) 
• (telnet) aggr show_space -m OR -g
  
• (telnet) aggr add (aggr) –d (drive#)
• (FilerView) Aggregates -> Click on the aggregate
• Click Show Raid to see existing disk configuration
• Click Add Disks to add new disks and follow the wizard

12. Volume Commands (Create Volumes From Aggregates)

• (telnet) vol status - shows all volumes and options
• (telnet) vol create (volumename) -s none (aggr) (size) - create volume with no guarantee
• (telnet) vol create (volumename) -s volume (aggr) (size) - create volume with volume guarantee
• (telnet) vol size (volume) - check the volume size
  
• (telnet) vol size (volume) (size) - set volume to new size
• (telnet) vol autosize (volume) on - turn on volume autogrow, the volume will grow by default in 5% increments until 20% growth achieved by default
  
• (telnet) vol autosize volume -m (new max size) -i (new increment amount) - notice that even though the defaults are 20% max and 5% increments, if you change the value the new values have to be in space (MB or GB)!
• (telnet) vol autosize (volume) - reports the current setting
• (telnet) vol options (volume) - list the volume options
  
• (telnet) vol options (volume) guarantee volume OR none - sets the volume guarantee to volume or none
  (telnet) vol options (volume) fractional_reserve (value) - sets fractional reserve to (value) but only works if volume guarantee is volume.
• (telnet) vol options (volume) try_first snap_delete OR volume_grow - sets the value to which method a volume will use when it runs out of space.  It will either grow the volume or delete snap shots.  The defaults is to grow the volume.
  
• (FilerView) Too much here to list.  Just click on volumes and poke around.  The following aren't possible in FilerView that I can tell: volume autosize, volume try_first option, changing fractional reserve
  

13. Check and Modify Snapshot Settings as Needed per Volume

• NOTE: Disable the snapshot schedule on the volume if SnapManager is protecting the volume
  
• (telnet) vol options (volume) nosnap on - Turn off the SnapShot Schedule but leave the schedule in place. I do this to disable snapshots in favor of the following command
  
• (telnet) snap sched (volume) 0 0 0 - Disable SnapShots by modifying the schedule.  I like the first option better.
• (telnet) snap reserve (volume) (#) - Sets the snap reserve to #% of the volume
  
• (telnet) snap autodelete (volume) - shows the snapshot autodelete settings
• (telnet) snap autodelete (volume) on - turns on snapshot autodelete (Check the settings!  Too many default settings to list here!)
• (FilerView) Too much here to list.  Just click on Volumes -> Snapshots and poke around.  The snap autodelete option is not possible in FilerView that I can tell.
  

14. Set DeDupe Settings per Volume

• DELETE EXISTING SNAPSHOTS FIRST ON THE TARGET VOLUME!
• Note: For the sis command you need the full volume path, /vol/vol0 for instance
  
• (telnet) sis on (full volume path) - enables dedupe on that volume
  
• (telnet) sis start (full volume path) - runs dedupe on that volume now
  
• (telnet) df –sh - To check space saving on DeDupe
• (telnet) sis config and sis status to check settings
• (telnet) sis config -s sun,mon-fri,sat@0,3-9,12 (full volume path) - The -s option allows you to change the schedule.  You can list the days separated by commas or a range by dash, same with hours.
  
• (FilerView) This is not possible in FilerView
  

15. Cifs setup (set up ntp first!)

• (telnet) cifs setup
• OR (Filer View) CIFS -> Configure -> Setup Wizard

16. To Set-up a CIFS Share (Add Volume First!)

• (Filer View) CIFS -> Shares -> Add
• (Filer View) CIFS -> Shares -> Manage -> Change Access
  

17. NFS Setup (Add Volume First!)

• Need to Add NFS Export Permissions

18. LUN Setup (Add Volume First!)

• Need more information here
  
• (telnet) lun setup - wizard to create the lun and map the igroup all in one
• (telnet) lun set reservation (full path to lun) enable OR disable - disable space reservation to thin provision the LUN
  
• (Filer View) LUNS -> Add (Name Lun with aggr.LUN file convention)
• Add an igroup and assign initiators
• LUNS -> Manage -> No Maps
  

Other Common Information Commands That I Still Need to Add and Document:
Sysconfig

snap reclaimable
snap delta
aggr show_space (aggr) -g
shelfchk
Wrfile /etc/hosts
Rdfile /etc/hosts
Rdfile /etc/rc
rdfile /etc/messages

Priv set advanced
Led_on (drive#) (get nmber from disk show) (need priv set advanced first)
(priv set) disk shm_stats 
(need to add deswizzle status command)
sysstat
statit (disk utilization)
stats (performance data)
reallocate measure or
reallocate start
to see raid groups: sysconfig -r OR aggr status -R
fcstat device_map - command to show shelf layout
sysconfig -d
interface alias process

storage show
vif favor
----------

•  Create the role, group, and user on each NetApp controller. Enter each line from the command line
• useradmin role add vsc-role -a login-http-admin,api-aggr-list-info,api-cf-get-partner,api-cf-status,api-disk-list-info,api-ems-autosupport-log,api-fcp-adapter-list-info,api-fcp-get-cfmode,api-license-list-info,api-lun-get-vdisk-attributes,api-lun-list-info,api-lun-map-list-info,api-nfs-exportfs-list-rules,api-qtree-list,api-snmp-get,api-snmp-get-next,api-system-get-info,api-system-get-version,api-volume-autosize-get,api-volume-list-info,api-volume-options-list-info
• useradmin group add vsc-group -r vsc-role
• useradmin user add vsc-user -g vsc-group