Responsibility vs. Hysteria

Anytime someone ends a blog posting with the phrase Centera is and remains the answer, you just know there has been a disruption in the force which simply must be corrected :)

Clearly things in EMC's archiving and compliance division are getting desperate, since 'Zilla and EMC's Backup Dude have turned into Storage Chicken Little's with their creative interpretation of a routine NetApp Customer Support Bulletin regarding a benign SnapLock vulnerability which has now been closed.

Kostadis did a great job of putting this all into the proper context.  He demonstrated NetApp's continued commitment to responsible vendor behavior by continuously broadening the scope of regression testing of our solutions for potential vulnerabilities.  This enables us to follow widely acknowledged vendor security best-practices by proactively altering our installed base before they are exposed.

I'd also like to add that this is a routine fix for all SnapLock customers since they can stay within their Data ONTAP version to apply this patch (often non-disruptively).  I.e. 7.0.x customers can patch it with 7.0.7, while 7.1.x customers can do the same with 7.1.3 and finally 7.2.x customers can patch this with 7.2.5.1 at their earliest convenience.  As with any Data ONTAP patch, no user data is touched or migrated in any way, making this a very simple update.

Allegedly having to upgrade all versions of Data ONTAP to 7.2.5.1 is yet another complete misunderstanding by EMC regarding how simple compliance storage can be.  Perhaps they're drawing on their own ugly EMC Centera experience which mandates upgrading only to the latest & greatest release to fix bugs and patch security vulnerabilities?

H2 ain't just a dying breed of gas-guzzling SUV's

Now It's one thing to artificially attempt to induce (H1) hysteria via irresponsible assertions about a competitor's products.  EMC has a long and established history of that, but ...

It's quite another to (H2) hypocritically proclaim one of the most flawed products in the history of storage is somehow an antidote.

Remember, this is the same product which:

• Required forklift migration of their first-generation customers over to a new version in order to overcome serious data corruption issues due to silent data loss, then
• Saw the founder of your CAS technology painstakingly lay out all the reasons your entire product strategy is flawed, and finally,
• And finally had the Group CTO follow suit and confirm there is no future in this product line.  Wow, there's a winning formula if ever I saw one :)

Do you feel lucky punk?

Perhaps my favorite Centera anecdote comes from a huge multi-national investment banking customer who replaced all their EMC Centera's with SnapLock last year.  When they are asked why, they respond as follows:

"EMC once told us the odds of losing data silently due to MD5 hash collisions and other design flaws like CAS API incompatibilities were even less than winning the lottery...

... Well after having won this unfortunate lottery about five times, we decided we were just too lucky to continue archiving precious compliance data to this porous platform!"

Comments

[Scott Waterhouse]

That was an awful lot of talk about vendor responsibility. I think we are all familiar with the maxim: it is not whether you have problems, it is how you handle them. You can either take full responsibility for them, work through the difficult times with the customer, fix the problems, take the proverbial beating (and sometimes incur significant costs to do so) or you can walk away. Good vendors do the former, vendors you probably dont want as a partner opt for the later. I will just leave you with one quote: "NetApp cannot stand by the SnapLock user agreement unless the upgrade is performed." After opting for security through obscurity (yeuch), after casting stones at others, I will let you draw your own conclusions as to the implications of that quote.

I would also like to venture a guess that if this data was stored in an insecure manner for a duration of its lifecycle, it will never be secure or compliant again. You can't stand by SnapLock now, which means the data cannot be guaranteed compliant now. Which means that a smart lawyer will point out that it is, well, garbage. Now and forever.

Has EMC had problems, like everybody else? Of course. Have we stood by our customers 100% of the time, in my experience? You bet.

By the way, Val, the world is a small place. I was actually the recipient of a 1 on 1 technical briefing by a guy from Ottawa about 10 years ago. Same name as you. He was an oustanding technical presenter, one of the better ones I have seen during my career. Nice to run into you again.

[Val Bercovici]

Thanks for the comment Scott. Nice to see yet another EMC Canuck in the storage blogosphere! Guess we must have met some years ago in Calgary, eh? :)

I think you, Zilla and the other Chicken Little’s are reading WAY too much into that one quote out of context relative to the entire customer support bulletin involved. Yes we’re encouraging our compliance customers to upgrade ASAP, but there is no firm deadline involved, and we’re not holding a gun to anybody’s head until they do. We’re giving them options and stating the proper sense of urgency.

Do you *really* think we’re about to abandon thousands of customers due to your biased interpretation of that one sentence?

Most compliance regulations (SEC, PCI, FRCP, HIPAA, etc..) require that the supporting IT systems be kept uptodate when security patches are available from the respective vendors, so why the extraordinary hysteria from EMC regarding this one? Is it EMC’s policy to leave known security holes open at will? Or maybe after years of industry scorn against Centera, EMC is unleashing all that repressed energy as a result of finally being able to pounce on a rare SnapLock vulnerability?

Bottom-line - No SnapLock customer has been exposed to this vulnerability, their data has always been safe - and our sense of urgency is focused on keeping it exactly that way.

P.S. “Security thru obscurity” is actually a central part of EMC sales teams’ pitch for CAS advantages vs NAS, so I find it ironic you would attempt to label NetApp with it as well. As for EMC being open and honest regarding security vulnerabilities, you may not recall the events of August 23rd, 2004, but I sure do. On that day we exposed to the media how to silently delete data archived on a Gen1 Centera via MD5 hashes.
What was EMC’s response? To vehemently deny any problem exists while simultaneously rushing a CentraStar software update featuring “MD5 Plus” for no apparent reason. If Roy Sanford is still around, feel free to ask him off the record what *really* went down that day, both literally (think online developer Centera’s on cascommunity.org) and figuratively :)

Yes the world is a small place Scott, so I would be careful about speaking in absolutes like EMC always being open and forthright regarding security vulnerabilities. After all, a wise Jedi once said “… only a Sith deals in absolutes …”

P.P.S Not to pile on too much here, but I've attempted to leave two comments on your blog over the past few weeks and none of them have appeared. Is this yet another example of EMC's "openness"?

[Scott Waterhouse]

Well, one good comment deserves another.

With respect to your comments on my blog, the first was blatantly ad hominem, and not in any way related to the post it was appended to (or my blog for that matter, it was a personal attack on Chuck Hollis). So no, it didn't get posted.

The second I thought about, and concluded that since it was devoid of content (just a link to the post above). At least your colleague included comments in his post. That seems to me to be the professionally courteous method. I am clearly not going to post just a link to a post rife with inaccuracies. :)

Several of those come up in your comments too: CAS is not advertised nor sold as security through obscurity. No more so than the fact that WAFL is not open source is NetApp therefore subscribing to security through obscurity.

Second--notice the quote "in my experience". A lot of crap gets thrown at EMC, especially in the blogosphere. All I *know* at the end of the day, is what I have experienced at EMC. So, while I find your accusations about as amusing as all the rest who are content to take cheap shots (and there are a lot of you) all I can speak to with passion and surety is what I have experienced. In my experience EMC has gone to extraordinary lengths to keep customers happy and secure. I am not about to indulge in or respond to gossip about events of which I have no direct (or even indirect in this case) knowledge. Surely the posts and comments of a disgruntled ex-employee who starts up a competitive company fall squarely into that category.

Finally, I think if you pursue my blog you will see that I am largely focused on discussing EMC's approach to various problems, and highlighting competitive approaches where they have significant weaknesses and where that weakness is revealing and useful to understand because it helps us better understand the underlying technology paradigms and capabilities. I don't see much of that here, to be candid. I see a lot of mudslinging (to be brutally honest). But I don't see a lot of "why are we [NetApp] great." I do see a lot of "why do others suck."

I reckon that in this small world we live that is the only way in 10 years that I will be able to live with what I write today. So while I may put absoluteles in my blog, I think about the only occassions on which I do are when things are clearly contextual (we have the fastest is semantically equivalent to we have the fastest on the day I wrote this post) or involve basic characteristics of a technology (data with a very high change rate dedups poorly compared to data with a low change rate). No jedi mind tricks, no Sith analogies, just my thoughts for today.

[Val Bercovici]

Have it your way Scott. EMC's reputation for FUD is well documented and acknowledged across the storage industry. You have to look no further than your own Mr. Hollis, Anarchist or the Zilla man himself to see the "shoot first, ask questions later" mentality in action.

I find it very telling that you choose to personally censor comments on your blog. That not only goes against modern blog etiquette, but speaks volumes about the insecurity and level of control you feel necessary regarding the claims you make. If you can't stand open scrutiny, perhaps you should consider a different medium more akin to the monologue you seek?

[Val Bercovici]

Just ran across Tony's take on this from the IBM perspective.

[Scott Waterhouse]

Ad hominem attacks are not tolerated. I don't indulge in them, and I don't post them. Suggesting they should be allowed in the name of "etiquette" is absurd. Post content and I will be happy to put it up. Post opinions that are not an attack on the person but an attack on an idea, and I will put them up. I don't care how rancorous somebody cares to be but this is about ideas, concepts, truth, and so forth. If you choose to make it personal, I won't put it up. Call it censorship if that is what you wish, but there it is. I call it simple civility.

[Val Bercovici]

I don't have to "call it" censorship, when you've proven that's just what it is.

So when Chuck Hollis attacks NetApp out of the blue in your blog about Data Domain, that's fair game to you. Yet when I try to point out the inappropriateness of the out of context slur, you unilaterally deem it a personal attack and suppress the comment.

OK, I get it now. Thanks for clearing that up Scott. You're doing wonders for EMC's attempts to improve their image of a close-minded, controlling and deceptive organization.

Actually, on second thought - keep it up. Your actions and statements are the kind of negative publicity I couldn't even dream up for EMC! :)

[Mike]

----------------------
But I don't see a lot of "why are we [NetApp] great." I do see a lot of "why do others suck."
----------------------

Scott,

It's laughable that you would make such a statement, considering your original post was all about "why do others suck" and nothing about "why are we [EMC]great." In your attempt to stay "largely focused on discussing EMC's approach to various problems" the only sentence in your entire original post that even mentions your product was "Centera is and remains the answer."

Should I believe what you say or what you do?

[Can't We All Just Get Along?]

Great point Mike. Even though I think you're mixing quotes from EMC's StorageZilla and Scott, your point is solid nevertheless.

EMC is the bully of the storage industry. And anyone who clinically studies bullies knows the true cowardice of their nature!

[Nigel Wauters]

What a piece of work! So let me get this straight:

1. On behalf of EMC, Scott attacks NetApp with a slanderous claim about their products, then
2. He gets proven wrong here and elsewhere on the blogoshpere, so
3. He continues his rant here, and
4. Refuses any responses on his own blog?

That's the kind of moral & intellectual cowardice money can't buy. People who live in glass houses...