Niel Armstrong is the CIO of Activision, and we gave a talk together at Oracle Open World. Niel uses pretty much the full suite of Oracle products, including the database itself, much of the middleware stack, the eBusiness Suite, and Hyperion. He also uses pretty much the full suite of NetApp products.
My favorite story was about the speed up Niel saw in making copies of his production database for test and development. Before he started working with NetApp, test and dev copies were a real pain. It took a “long week” to create a full copy. (A “long week” is when you start at close of business Friday, keep working for the whole week, and don’t finish till Sunday night the weekend after. Or maybe Monday morning.) It was so painful that they did a process-re-engineering project that drove the time from roughly nine days down to four. Niel said, “We felt great about cutting the time in half. This was a big project for us, and we worked really hard at it.”
When Niel switched to NetApp storage, he started using NetApp’s copy-on-write clones (FlexClone) to make test and dev copies, and he cut the time from 4 days to 4 hours. Actually, he said it’s usually 2 hours, but he says 4 just to give himself some buffer.
This completely changed their workflow model. Niel said, “Before NetApp, we used to really fight these clones. We just didn’t want to do them. No more than one or two per large Oracle project if possible. Now we do them all the time. I’ve got over a dozen clones right now just for the UK Oracle implementation. One guy wanted a clone for a training class, and we gave it to him. We never would have done that before. It would have taken too long and used up too much storage.”
Clones are handy during normal operations, but they are especially important when you are making a major change to your environment. “Without NetApp clones, we would never have finished our upgrade in time.”
What I love most about Niel’s story is that it is almost exactly the same thing I wrote about last year after Oracle Open World (read here), except then I focused on the demo we were giving in our booth, and now I’m describing a customer who is in production and who stood on stage next to me to describe his experiences.
Let me give you a sense of Niel’s environment. Activision has game design studios all around the world. A big problem was getting reliable backups at the studios, so they implemented remote disk-to-disk copies (SnapMirror) from each studio to their disaster recovery center in Burbank, as well as replicating all of the headquarters data. A few Sundays ago, disaster recovery was “tested” by two back-to-back power failures, courtesy of SoCal Edison. They recovered in less than 4 hours, with no business interruption, which earned them attaboys from both the Chairman and the CEO.
I asked Niel whether he mostly makes clones from his DR systems, or his production systems, or what. His response: “We make clones pretty much everywhere. It’s clones all over.”
Activision's next game: SEND IN THE CLONES!
Posted by: anonymous | December 06, 2007 at 06:36 PM
Even 2 hours is long. In our dev/qa environments, one of the key advantages of having clones is the ability to continually update the backing SnapMirror without disrupting normal operations.
A snapmirror update brings over a quiesced copy of the DB's from the production datacenters, whether its needed or not. Besides keeping the snapshot utilization low on the production side, this also means that when QA requests a refresh, there's data from <12h ago ready to go. Combined with some ZAPI scripts in the hands of the DBA team, this is now 100% hands-off from the storage administration side.
Obviously the incredible space efficiency allows scaling out the number of parallel QA environments far further than would have otherwise been feasible.
We've held off from clones supporting production so far, as there's presently no way to reserve a "reasonable" amount of space for a clone from the aggregate. As-is, with multiple clones in an aggr, the potential for "a clone gone wild" starving the aggr and affecting the other instances isn't acceptable.
Posted by: Kevin Graham | December 07, 2007 at 12:54 PM
i just want to know: is this david hitz who was in the u.s.n., went to boot camp in florida? if this is him, we went to boot camp together. is there a way to contact you?
j
[Dave replies: I'm afraid I'm not that guy! Good luck finding him.]
Posted by: james harkness | January 02, 2008 at 09:01 AM