EMC Centera Customers - Consider Your Whistle Blown

LOST is one of my favorite TV shows, but it is a work of fiction. Reality can be much harsher.

If you left your coat with the coat check service at a restaurant or theater, you would naturally expect to get it back upon producing the claim ticket. How upset would you be if the restaurant or theater lost your coat in this scenario? What are your legal options or reimbursement expectations in that case?

Now what if this same scenario involves your car left with a valet parking attendant? The stakes just got much higher. But it doesn't stop there. These metaphors can sadly escalate into nightmarish proportions when dealing with corporate data worth billions of dollars - or in the most extreme cases - patient data which of course is priceless.

Familiarity Breeds Contempt - of Court

The coat check and valet parking examples above should be familiar to those of us who sat through EMC's introduction of the concept of Content Addressable Storage (CAS) earlier this decade as they rolled out their Centera archiving solution.

Some of us more mathematically inclined were naturally skeptical of EMC's weak assurances given regarding complete data integrity using brittle hashing technology. We attempted a public debate on the issue but EMC's knee-jerk reaction was to attack the whistleblowers instead of addressing the underlying issue:

• http://searchstorage.techtarget.com/news/article/0,289142,sid5_gci1001882,00.html
• http://www.byteandswitch.com/document.asp?doc_id=58160
• http://www.techworld.com/security/news/index.cfm?newsid=2121
• http://www.crn.com/storage/46200136

Chinks in the Centera Armor

By their very nature, online archiving solutions like NetApp's SnapLock and EMC's Centera are designed to store the most sensitive digital data around. As you may expect, companies under government regulation or in active litigation are not overly inclined to publicly disclose issues with their archiving implementations. Neither are healthcare institutions storing precious patient information in an era of rampant malpractice lawsuits.

Yet despite EMC's denials and their customers' hesitations to go public with their archiving problems, evidence emerged which raised questions about the integrity of EMC's commitment to the integrity of their customers' archives.

Evolution of Digital Archiving

Market conditions have changed dramatically since the early days of digital archiving 1.0. Today the adverse impact of data loss has become significantly greater. Signs of data loss (intentional or otherwise) are a litigator's dream and aggressive regulator's door opener. The advent of "four-letter words" such as FRCP mean more and more businesses are being asked to search and retrieve their "active archives" for legal discovery purposes. Industries prone to litigation will run into these challenges sooner rather than never.

Some examples of violations which have meant heavy penalties and embarrassing publicity for their organizations:

• Morgan Stanley faced $1.45 billion fines and penalties, due in part to mishandled emails. http://www.nytimes.com/2005/05/19/business/19perelman.html and http://online.wsj.com/article/SB111620910505034309.html?mod=home_whats_news_us. The fines were eventually reduced to roughly $15 million, but various stakeholders may have had a change of underwear in the interim :)
• Intel had some "document retention lapses” with email during litigation with AMD. http://www.ciolaw.org/litigation-holds/amd-v-intel-part-one-a-look-at-the-%E2%80%98largest-electronic-production-in-history%E2%80%99/
• Qualcom v. Broadcom. This patent litigation case resulted in sanctions for not properly complying to judge’s FRCP-related orders. http://www.ediscoverylaw.com/2008/02/articles/news-updates/qualcomm-accepts-sanctions-issued-by-magistrate-judge-and-pays-entire-856863324-sanction-to-broadcom/

More currently, the new Obama administration has strongly signaled that more regulation, oversight and auditing will be the norm rather than the exception.

Finally, a lot of first generation archiving systems have reached the end of their useful life (or have been formally retired by their vendors) and customers need to migrate their critical archives onto new systems.

Sadly it's only as data is retrieved during these scenarios will customers start to realize some of that data has disappeared. Permanently.

You Can't Keep a Good Whistleblower Down

Enter Symantec. As the market leader for Email archiving they command authority on this subject. As an experienced anti-virus vendor and respected corporate citizen, Symantec shares NetApp's dedication to responsible disclosure. NetApp has a enjoyed a long and mutually beneficial alliance with Symantec offering optimized joint solutions across the data protection, high availability and of course archiving markets. In short - when Symantec publicly declares that EMC Centera (and only EMC Centera) is vulnerable to data loss, the entire industry - and most importantly archiving customers need to stand up and listen.

In case you're wondering, this is NOT an issue with Symantec software. The root cause is the complex EMC Centera API and brittle internal Centera architecture which Symantec Enterprise Vault and other similar applications are forced to utilize for proper archiving functionality on that platform.

UPDATE Feb 3rd:  After research & consultation with Symantec, they have posted an update to their Symantec KnowledgeBase article 302229. This revised KB article clarifies that this rare condition occurs *before* EV calls to the Centera API.

Since comments on this post are getting rather numerous, I recommend continuing the discussion regarding the ongoing issues with the Centera’s archiving architecture and EMC’s resulting veil of secrecy over on my more recent The Ironic Tension of Whistleblowing blog.”

Next Steps

If you are an EMC Centera customer (using Symantec Archiving software or any other) you need to prepare yourself. Some recommendations:

• Read the Symantec KnowledgeBase article 302229 carefully in order to interpret your own vulnerability to the data loss issues described. You may need some technically-savvy advisors to clarify any details you don't fully understand.
• Meet with your records management department to assess the scope of your data loss exposure, particularly for regulated data.
• Develop a procedure (either manual and/or technical) to audit your archiving system ASAP in order to determine whether you can successfully retrieve ALL of your previously archived data.
• Based on your audit results, interpretation and exposure, consult your legal council to determine if there is any resulting risk to ongoing or anticipated legislation.
• Consult with EMC on any possible remediation actions. My own technical (not legal) interpretation of this KB is that due to Centera's silent data corruption problem, EV data discovered to be lost is irretrievable and gone completely. For your sake I hope I'm wrong but I fear I'm right.
• Along the same lines of risk, EMC also needs to clarify the current status and future of their Centera division, which is coming under increasing scrutiny due to their actions of late.

There is a Safe Harbor

NetApp has never knowingly sacrificed the integrity of data archived by SnapLock and consequently enjoys a reputation of responsibility in this regard. We also pass the layman's Google Good Housekeeping Seal for data loss.

Having anticipated this unfortunate outcome almost 4 years ago, NetApp has partnered with experienced specialists in this area such as Procedo to help archiving customers in distress migrate their remaining accessible data non-disruptively over to a safe archiving platform they can trust via transparency.

There is no shame in being a victim of Centera's well-hidden silent data corruption problem. With George W Bush out of office now, we can all return to the proper version of the old adage, Fool me once...

Don't leave yourself in a position to be fooled twice!

Comments

[Storagezilla]

This is a known EV bug (Of many) where the data isn't stored on the Centera in the first place.

I'd ask about NetApp's archiving credibility but I'm not sure if the company has any.

Still deleting items out of your inbox when you hit your quota? You might want to buy some email archiving software, I'll do you a good deal! :-D

[Chuck Hollis]

Mark is right -- the bug is a Symantec one, not a Centera one. I doubt you'll correct this blog post, though. Telling the truth wouldn't be any fun, would it?

Val, you don't seem to be able to write anything unless you're attempting to beat up someone, usually EMC.

You've gotten in trouble before with this sort of behavior. Looks like you're heading down the exact same road again.

-- Chuck

[Lee Razo]

Hey Val,

Great post. I just checked to see what Google's "Good Housekeeping" check turned up for EMC Centera:

http://www.google.com/search?hl=en&q=emc+centera+data+loss&btnG=Google+Search&aq=f&oq=

Ouch... I'd hate to have been the Symantec developer who was tasked with porting their software to that beast.

[Anonymous]

I also say kudos to Symantec for posting the exposure, although it does certainly read like they're pointing the finger to EMC. EMC regularly posts concerns and issues to their customers, so I'm not sure why it should be exploited against them even if it isn't their issue -- problems occur with products, even NetApp's. It's a social responsibility to find them, report them, and fix them.

[the storage anarchist]

Val -

Your misdirection-laden post indicates that you apparently did not take the time to carefully and objectively read through Symantec's report.

In their report, Symantec basically admits it was they who were the cause of any "silent corruption" because their software did not report its failure to successfully store the "lost" objects into Centera. Whether their code inadvertently did not attempt to store these objects in the first place, or their software simply failed to check and handle an error condition isn't clear. But your (unfounded) assertion of guilt aside, I can find no language from Symantec's report indicating that the Centera either lost or corrupted a successfully stored object.

As innovative and reliable as Centera has proven to be in the market, it alas cannot retrieve objects that were not first stored in it.

In your haste and glee to attack a market-respected competitor, it seems you have abandoned objectivity and misrepresented the facts. IMHO, you've demonstrated a questionable lack of integrity and judgment in this post.

[marc farley]

Were blogging now!

[Mike Rley]

I think the debate strategy here is to attack the author rather than examine the issue. I'd like to get this discussion focused back on the issue.

EMC did a great job marketing the Centera - right time, right product. In fact, EMC's virtual monopoly on the market allowed them to be very up front with where they thought the Centera fit and where it didn't. Kudos to EMC on that (and it shouldn't take a monopoly to inspire that type of honesty). I think the up-take on the product even surprised many at EMC. The downside of that initial success was exposing the flaws of a rigid and proprietary CAS approach in an area where flaws were not to be tolerated, Compliance. EMC basically took an architecture that in its previous life was largely responsible for storing adult entertainment on the web and turned it into a compliance solution targeted at meeting federal regulations. The two worlds couldn't be more different when it comes to retaining "critical" data. (Losing a picture of Pam Anderson is different than losing a customer's insurance claim). We also know that back in 1996 RSA (now part of EMC's security division) issued an advisory warning customers to stay away from storing important information using MD5. (You would think EMC would listen to its own security division). "Cracks" to MD5 were consistently reported (one fellow did it on an Acer laptop over his Christmas break) but EMC continued to rely on this approach not just for data organization but for data integrity - big difference. (EMC acquiesced and included a stronger hash ID but the same argument applies as hash technology erodes over time.) Good and bad, this is all part of the history of the product and issues with the technology it was built upon so I'm not sure how you spin this one away from the facts.

On a more subjective note, EMC also clung to the notion (at least outwardly) that a proprietary API interface would ensure tight integration and offer enhanced security. Jon Toigo has often commented on the vendor lock-in strategy as the true driving force. I agree. Regardless, Paul Carpentier (one of the founding fathers of CAS, founder of Filepool) put it best in his interview with Jon Toigo: "Complexity on the outside reflects what happens on the inside..." Putting aside the marketing spin that the Centera was somehow "self-healing" it is this complexity that has lead to these issues. We know customers that have had to recover "orphaned" data. Many a script has been run by many a "tiger team" to reconnect C-Clips and objects. We know issues with the Garbage Collection process have tossed out the baby with the bathwater - objects that cannot be recovered. Now we have a KB article from Symantec specifically regarding the Centera. Yet another indication, yet another story to add to a growing list of the potential issues that reflect the complexity of "what happens on the inside."

Let me say that CAS is not in and of itself a bad idea. I think Paul's new company, Caringo, is extremely interesting and has done some great work to open up and simplify that architecture. I'm a fan. I think EMC's implementation of CAS is flawed. EMC has essentially built themselves a rigid and complex architecture. So much so that when it came time to look into "cloud" architectures, they chose to acquire new companies to build this next generation CAS solution rather than simply leverage the CAS architecture underneath their noses. (We're also seeing an increased focus on Celerra as a compliance platform - that evil NAS/file system approach decried by EMC years ago). It does make you ponder the comment made by Filepool's other founder, Jan Van Riel, when he left EMC to join Paul at Caringo: "With EMC scaling down the Centera unit and the future of Centera unclear the chance to join Caringo, which understands the potential of CAS, and partner once again with Paul Carpentier was too good of an opportunity to pass up." Best of luck to Jan at Caringo but RIP Centera? EMC will say that reports of Centera's death are greatly exagerated but with new CAS platforms not based on Centera and compliance functionality creeping into Celerra, one has to wonder.

So, whether the fault lies with Symantec or EMC, both should be commended for taking action to notify customers. As Anonymous mentions, all products will have help desk cases opened against them. However, history seems to indicate that the Centera is particularly susceptible to these types of errors and unfortunately it is in an industry where errors are simply not tolerated. I hope what isn't lost in all of this huffing and puffing from EMC are the remediations Val recommended. Those are solid suggestions whether you move to NetApp or not.

Now given this as a backdrop, I do think it's fair for us to turn up the heat on CAS 2.0 (aka. Atmos) given the track record of CAS 1.0 (aka. Centera). What lessons were learned or will EMC propagate the same design? What does that new design tell us about Centera deficiencies? I'm not sure why customers would take a leap of faith that CAS 2.0 has to be better than CAS 1.0. We're talking about the life blood of any company here so I'm not sure the "trust me" sales strategy works.

[Val Bercovici]

Woot! Nothing like a blogfight to drive traffic and comments... :)

Now let's take 'em on one by one (when I'm not doing real work :)

[Val Bercovici]

First off - Hi Zilla. Thanks for the comment.

Were it not so predictable, I would have written it myself and saved you the trouble of the attempt at wit :)

On a more serious note, I think it will be very interesting to see the opinions of other more experienced subject-matter experts on this topic. What will they "expose"?

[Val Bercovici]

Hi Chuck. Welcome back to Exposed.

True to form, it's nice to see you immediately attempt to personalize an evidence-based product discussion. Please spend more time here and look around. You'll see I only address EMC every now and then - and only when you have it coming :)

Also, please remind me of the "trouble" I got into last time? I seem to recall many independent users and even more objective expert analysts agreed with my position(s) rather than yours.

It seems the only trouble I get into is with you and your selectively delicate sensibilities, forcing you to autocratically censor comments from me (with condescending reminders no less) which you don't want appearing on your blog.

Regardless, I consider it a compliment either way.

I look forward to observing how much EMC is willing to sacrifice your relationship with Symantec here (by throwing them under the bus) in order to cover up the truth about Centera's obvious data porosity.