NetApp - Simple Steve

Objects in mirror ...

I always like a good simplicity story, even if the story makes me look a little silly.

I recently got a car with a backup camera.  When backing up, it helps you see if there are children or other objects hidden behind the car.

The car is parked in a place where the camera directly faces the daily sunset.  I immediately started worrying about possible burn-in of the camera CCD.  I imagined the camera lens acting like a magnifying glass in the hot sun, with the poor camera smoking away ...

Sure enough, after a few weeks of use, the camera developed a dark area where the sun had been shining.  It looked a little like the picture to the right.

Certainly this was caused by the setting sun burning away, just as I had predicted. 

I immediately started performing web searches, looking for other users who had experienced similar problems.  I considered whether this problem could be covered by the vehicle warranty.  I thought perhaps I should try to order a replacement camera, presuming that this problem would become progressively worse over time.

Then, after a little additional thought, I went outside and wiped off the camera lens.

The small piece of dirt clinging to the lens fell to the ground.  The spot on the camera promptly disappeared.

It can be dangerous to let your intuition and expectations carry you toward an unnecessarily complex solution. 

Sometimes the simplest answer is also the easiest.

(Thanks for reading - Steve Klinkner)

One Candle

After a year of blogging for NetApp, I offer a few reflections.

Blog Apologies

One thing I never like to see is what I'll call a "blog apologist":

Hello loyal fans!  Sorry I haven't blogged in 6 months.  I was busy ripping MP3s and optimizing BitTorrent for World of Warcraft patches [...]

Yawn.

I will continue trying hard to avoid that.  But I do wish I had more time for good posts.

!Blogging.Easy() == true

Last year, Dave Hitz asked me if I'd like to write a NetApp blog.  How could I say No?

Yet I was initially quite reluctant, in spite of the opportunity.

I had just started a largish new project, and my first priority was the engineering team.  I knew that writing a blog would come on top of piles of other work. 

But I observed that my daily schedule had an under-utilized slot from 0200 to 0400 hours.  I figured I could somehow squeeze in some writing.

Writing a decent blog is hard!

Authoring a good article from scratch takes several hours, minimum.  Cutting and pasting from elsewhere is lame, and leaves me with pangs of plagiarism-filled guilt.  I struggle to find a happy medium.

Me?  A Blogger?!?

I didn't get a chance to ask Dave why he asked me to write a blog. 

I assume he was hoping I'd write a lot about customer-facing issues, since I work on our storage management tools.  He may have thought I have a clue.  (Heh.)

When I told colleagues I'd be writing a blog, many reacted like this

Ha!  Ha!  Ha!  That is really funny Steve!

[pause, dabbing eyes]

You are joking, aren't you?

Clearly my colleagues are among my top supporters :-)

At NetApp I apparently have a reputation for a certain degree of ... irreverence.  When blogging, I try to keep it in check, but it just wouldn't be right to not let it sneak out once in a while, would it?

The Future

Since we started the NetApp blogs, the field of NetApp bloggers has expanded substantially.  Please have a look!  Many are very good.

(You can also see that a number of the bloggers are "directors" and "vice presidents", so I privately think of mine as the "budget blog".  Wink, wink.)

I'm nearly unique among our bloggers as someone who is working in NetApp engineering day-to-day to design, create and support NetApp's products.  Going forward - in addition to all the other topics - I'll try to give a little more of a flavor showing what it's like to work in NetApp engineering.

If there are other topics you've particularly enjoyed (or not) in the past year, I'd love to hear your feedback.

(Thanks for reading - Steve Klinkner)

A Brief History of Unified Storage

I have been meaning to writing about unified storage for some time.  With this article, I'm covering another item in my temporary flashback to ONTAP 6.3.

Though NetApp's unified storage architecture did not originate with ONTAP 6.3, it took a Great Leap Forward with support for SAN.

Early Days - NFS and CIFS

In 1993, NetApp shipped its first NAS appliance for NFS. 

NetApp followed this in 1996 (ONTAP 4.0), shipping unified storage for Windows (CIFS/SMB) and Unix (NFS) protocols.

A clustering solution on the same architecture was introduced in 1998.

Unified NAS + SAN

2002 saw the introduction of support for SAN protocols (customers can create LUNs on NetApp systems using either Fibre Channel or iSCSI).

In 2003 NetApp introduced the gFiler (now V-Series).  This technology allows ONTAP to run in front of existing storage arrays, where the NetApp system can offer up any of its unified protocols - allowing customers to flexibly re-use existing SAN deployments. 

All on the same ONTAP operating system.

But Wait!  There's More ...

In describing NetApp's unified storage, it's hard to know exactly where to begin, and where to end, since it truly is Just One Operating System.

The same architecture supports cool technologies like Snapshots and RAID-DP, and makes possible FlexVols, FlexClones and Deduplication.

NearStore and SnapLock products are built on the same operating system, along with many other features.

Who Cares?

A skeptic (or competitor) might ask "Who cares?" 

It's a legitimate question to ask whether a unified approach brings substantial benefits as well as trade-offs.

From a simplicity and training perspective, the advantages seem pretty clear: 

• There is only one core architecture to deploy and learn. 
• One set of user interfaces works across all the products. 
• You don't need to learn about new technologies until you choose - and then, they are conveniently available on the same platform you already know.

Our competitors like to suggest that a unified architecture doesn't scale, or might be difficult to extend or maintain.

They've been saying that as long as I can remember!  Yet we continue to successfully extend ONTAP to include new and useful capabilities.

(In my humble opinion, mostly they are reasoning with a python.)

Deciding for Yourself

As always, I advise you to look at our products, compare against our competitors', and decide for yourself which works better for you.

Our web site has five things to think about if you're considering a unified storage solution.

In the future I'll write a little more about unified storage from the perspective of storage management simplicity.

Reasoning with a Python

I've had a recurring image in my head for some time.

I imagine a misguided academic, reasoning with a python.  He thinks he's got a solid proof that the python can't possibly eat him. 

Or perhaps he's just engaged in a fatal self-delusion.  The details, shall we say, are academic.

The python could care less - it's all good food to him, and he eats the academic, proof and all:

Click the image to enlarge.

I'm sure I'll be referring to this cartoon in the future, because the academic reminds me of some of our competitors.

There seems to be no end to the stream of proofs and other silliness offered up by others via blogs, white papers and other sources as FUD.  Invariably the stuff turns out to be baloney and NetApp moves forward. 

That is, the python gets to eat tasty misguided academics.

You can try reasoning with the snake, but it doesn't change the reality (you'd best be right, or he's eating you for lunch).  Solutions which work in real customer environments disprove the FUD by their existence.

In other words, the proof is in the python.

ManageONTAP SDK

As I mentioned in my last few blogs, I've been pondering a few issues which each had their origins back in ONTAP 6.3. 

Today I'm thinking about the ManageONTAP SDK.  Our first bundle of management APIs first shipped along with ONTAP 6.3.

Genesis

In the beginning, there was only the ONTAP CLI and system registry.

NetApp engineers, customers, and corporate partners each had only limited capability to remotely manage NetApp storage systems.  Scripting based on RSH access gets old after a while ...

The need for structured and supported remote management of ONTAP motivated the creation of the Manage ONTAP APIs and SDK.

What is it?

Manage ONTAP is a collection of SDK infrastructure and API documentation which facilitates remote management of NetApp storage systems. 

Features include:

• structured input, output and error constructs
• authenticated transport using HTTP, HTTPS and RPC
• APIs forward compatible
• SDK support for Windows, Linux, Solaris, HP-UX, AIX and VMware ESX
• multiple language bindings (C/C++, Java, Perl)

Who uses it?

The APIs are highly leveraged within NetApp engineering.  FilerView utilizes the APIs internally to ONTAP.  Operations Manager makes extensive use of the APIs for remote management, as do other NetApp products. 

Our Professional Services organization uses the APIs to create custom solutions to meet customer requirements.

Most importantly, the APIs are used by corporate partners and customers who create scripting and management applications of their own.

Usage

An easy and low-risk way to get started with API experimentation is to install an ONTAP simulator.  It's a full-featured software simulation of ONTAP.

You can then grab an SDK download and get started in one of your favorite languages.

Trends

Although API support in the ONTAP 6.X releases was somewhat spotty, ONTAP 7.X has seen a fairly broad fill-out of functionality supported in the APIs.  An SDK has recently been offered in support of DataFabric® Manager as well.

How to get it?

You can access the NetApp Communities page for Interfaces and Tools, which has a number of downloads of interest, including Manage ONTAP SDK Introduction and Download Information.

There are also a number of presentations available providing technical and product overviews.  These should be available to users with NOW site access.

Although it isn't required to use the SDK, if your level of interest is sufficiently high, you might consider becoming a NetApp Solution Partner.

SAN Usability

My mind is still stuck temporarily back in 2002, musing on ONTAP 6.3. 

In that release, NetApp introduced SAN support, and we conducted several usability studies around SAN use cases.  These studies were structured on the ONTAP CLI, FilerView and SnapDrive.

I remember coming away from those usability studies with a few observations which I'll share. 

SAN is not simple

I mean "not simple" in a relative sense.  Setting up and managing a fibre channel environment is more complicated than an equivalent NAS solution.

In addition to worrying about all the usual IP-based technologies, SAN administrators must concern themselves with host software and hardware, additional switches, port mappings, and so on.

One of the interesting corollaries is that, even if SAN storage management is simplified, it does not necessarily make SAN simple.  You are still at the mercy of all the other details (switches, hosts, HBAs, ...).

Hence short of controlling all the hardware and software end-to-end, it is difficult for a vendor to deliver a holistically simplified SAN solution.

NetApp's SAN solution

Our usability test subjects included customers of other vendors' hardware, as well as existing NetApp customers and SAN novices.  Naturally our SAN usability tasks included unstructured tasks around the creation of LUNs. 

The test subjects who had the hardest time with the NetApp solution were the experienced SAN administrators.  Their prior experience tended to predispose them to seek more complicated work flows.

They spent time poking around trying to figure out how to cobble together a LUN from multiple disks, not initially realizing that NetApp provides an abstraction layer allowing volumes and aggregates to be carved into multiple LUNs.

Once they realized this, the existing SAN administrators were pleased with the relative simplicity of LUN creation on NetApp storage systems.

(The abstraction layer also makes possible products like our storage virtualization V-Series and host consistent SnapShots using SnapDrive).

Cut & Paste matters, too

Two of the most persistently annoying usability issues had little to do with storage and everything to do with multiple vendors and user interface design.

One involved terminology - an intermingling of port names, node names, WWNs, WWPNs, WWNNs, initiators.  The labels were not always consistently applied, and varied between vendors (this is an issue that a single vendor can't fix).

The other involved cut and paste issues.  A node might be presented as

00A0B301-03DE0000

in one user interface, but another vendor's user interface might insist that it be formatted as

00:A0:B3:01:03:DE:00:00

requiring an infuriating diversion to Notepad for a simple cut/paste operation.

This leads to a simple user interface directive:

Format consistently for presentation, but be permissive for input rules

The more things change ...

In my most recent engineering project, I've been re-examining some SAN work flows and have become reacquainted with each of these issues.

I thought they'd be interesting to share.

Fetching Logs and Cores

As I previously mentioned, I was musing on ONTAP 6.3 recently.  Although NetApp's systems are frequently multi-protocol, supporting SAN meant that some systems might not allow file-based access.

In addition to the introduction of an improved upgrade technique, SAN-only storage systems created the need to pull log and core files without the help of NAS protocols. 

(Naturally, I hope that you won't need to pull core files often, but - like seat belts in an automobile - this capability exists despite hopes that you'll never need it.)

Files via HTTP

ONTAP contains an HTTP server which supports a number of administrative functions (notably FilerView and Manage ONTAP APIs). 

Since ONTAP 6.4, you can use HTTP to retrieve logs and core files from a storage system (if authenticated), like this:

http://your-fas-system/na_admin/logs

http://your-fas-system/na_admin/cores

httpd.autoindex.enable

Also worth noting, HTTP directory indexing was disabled by default starting with ONTAP 7.2. 

You can fetch files directly by URL if you know their names, else if you want to browse for them you'll need to set

fas> options httpd.autoindex.enable on

in ONTAP 7.2 and later, else you'll see HTTP 403 errors.

ONTAP Software Install

ONTAP Installation Techniques

If you look at our current Upgrade Guide, it describes roughly three techniques for software installation and upgrade for ONTAP:

1. TAR file using a Unix (NFS) client
2. EXE file using a Windows (CIFS) client
3. EXE file pulled to the storage system using HTTP

We introduced the 3rd technique in ONTAP 6.4 in the form of the ONTAP software command.

Opportunity Knocks

Although we shipped the release some years ago, I was recently musing on ONTAP 6.3.  In that release, NetApp introduced SAN support - a significant extension to unified storage.

The introduction of SAN support in ONTAP meant the possibility of systems which lacked file-based protocols (neither NFS nor CIFS access).  This meant that installation of ONTAP shouldn't rely on file access for upgrades.

This presented a functional need as well as an opportunity:

• introduce a software installation "pull" (in addition to the existing "push" model) to satisfy the SAN-only requirement
• simplify the unbundling and installation process

Both were addressed with the (then new) software command.

Birth of the "software" Command

Borrowing concepts from a similar NetCache command (called install), we implemented the ONTAP software command as an HTTP client which can fetch, unbundle and install software packages.

It lets you do installation like this.  The self-extracting EXE is treated as a ZIP file by the storage system:

> software get http://myserver/ontap/7.3/setup.exe 7.3

> software install 7.3

As long as you've got an HTTP server holding the ONTAP software, this is generally a simpler install/upgrade experience than using NFS or CIFS client access.

Software Install Tricks

In NetApp engineering, I have access to all the ONTAP releases ever produced, on all different platforms (if you like, you can collect a similar set with corresponding NOW site downloads).

It's an easy matter for me to cross-mount a series of directories and write a CGI script on top to provide URL access to various ONTAP installations. 

For example the following URL on our internal (sorry - doesn't work outside the firewall) web server

http://web.netapp.com/~klink/ontap.cgi?release=7.3&arch=pc_elf

pulls the software for the pc_elf build of ONTAP 7.3. 

I can use this URL directly with the software command, making it really handy for pulling various versions of ONTAP onto different systems.  That doesn't happen too often in the field, but since I work in NetApp engineering, we swap versions all the time for testing.

More sophisticated scripts can interrogate the storage system for its architecture, formulate a URL, then download & install ONTAP, and finally reboot.

Software Command Usage

I don't have very good direct evidence, but I understand that the software command is now used fairly widely.  However I expect that many customers still upgrade using NFS and CIFS client access, probably due to established procedures.

Otherwise the ONTAP software command seems to be a nice example of a feature which met an immediate functional need but had the bonus of enhancing overall ease-of-use in other situations as well.

Sometimes capability and simplicity can be improved at the same time.

ONTAP 6.3 Déjà Vu

Remember back to 2002 ? 

A lot has changed since then, but that was the year NetApp shipped ONTAP 6.3.

I worked on several parts of that release, and it was notable in my mind for several reasons:

• NetApp expanded unified storage with SAN support
• I worked on several usability studies to improve NetApp's SAN ease of use
• We started adding a number of new wizards to FilerView
• We introduced the Manage ONTAP SDK 
• New commands like software install were planned to support SAN environments

I've been busy lately working on a new product.  I'd really like to tell you more about that product - but then, as the saying goes, NetApp would have to kill me.

In the meantime, I've run into a couple of situations that reminded me of issues we first faced back in 2002.

So next I'll take a little time to describe a few of those.

Half Empty?

Is your storage glass half empty? 

When it comes to deduplication, half empty seems like a good thing. 

Mike Shea's TechOntap article Use 50% Less Storage in Your Virtual Environments-Guaranteed mentions the NetApp 50% Virtualization Guarantee. 

This article also includes several nice resources for additional reading.

 

Virtualization Guarantee White Paper

There's a nice white paper by Jack McLeod explaining some technical details of the guarantee:

• Whitepaper: 50% Virtualization Guarantee Program Technical Guide (PDF)

 

VMware VI3 Storage Best Practices

Also referenced is a technical report by Vaughn Stewart, Michael Slisinger and Larry Touchette highlighting procedures and best practices for VMware storage.  Lots of useful screen shots, instructions and examples:

• TR-3428: NetApp and VMware VI3 Storage Best Practices (PDF)

 

Deduplication Implementation and Best Practices

Another nice implementation guide with details and examples, this time for deduplication, is this technical report:

• TR-3505: Deduplication Implementation and Best Practices (PDF)

 

Storage Efficiency Calculator

Last, there's a link to a tool you can use to estimate your savings:

• Storage Efficiency Calculator

 

We'd like to help make your storage glass "more empty".