Chuckle for today

Bryan Cantrill of Sun made a scathing indictment of the SPEC SFS benchmark earlier this week. I infer from Bryan's post that Sun's Fishworks team will not be posting SPEC SFS numbers.

My reactions:

• Of course SPEC SFS does not model every real workload. No benchmark can model every customer's needs. Show me one million storage customers, and I'll show you one billion workloads (assuming you'll give me the storage to capture all the packet traces). You have to pick one, and start there for a basis for comparison.
• About eight years ago Brian Pawlowski made the observation to me that those who criticize the SPEC SFS benchmark and instead attempt to push their own custom benchmark methodology are those who are probably producing SPEC SFS numbers that are not good enough to publish.
• As a corollary to the above two points, if a vendor can't excel at SPEC SFS, it probably can't excel at much else. SPEC SFS performance is the minimum bar for entry into the NAS business.
• SPEC's benchmarks are a product of the efforts of the companies that contribute. From our association at Sun, I know Bryan Cantrill is a very smart guy. Last I checked, Sun was still a member of SPEC and the SFS subcommittee. I'm sure the subcommittee could use his contributions on the follow on to SPEC SFS 2008.
• On the lack of a price/performance metric in SPEC SFS. Back in 1992 when LADDIS was being worked on, I recall this was a topic of discussion. The reason why the price/performance metric was not added was because:
1. Like many industries, few storage companies have fixed pricing. As much as heads of sales departments would prefer to charge the same highest price to every customer, it isn't going to happen. Storage is a buyers' market. And for storage devices that serve NFS and now CIFS, the easily accessible numbers on spec.org are yet another tool for buyers. I just don't understand why a storage vendor would advocate removing that tool.
2. In storage, the cost of the components to build the device falls continuously. Just as our customers have a buyers' market, we storage vendors are buyers of components from our suppliers and also enjoy a buyers' market. Re-submitting numbers after a hunk of sheet metal declines in price is silly.

• On caching .... Tom Haynes made a comment that NetApp appears to agree with Bryan's assertion that because SPEC SFS does not encourage external caching, vendors do not sell external caches. Actually, for about a year now NetApp has been selling external cache appliances. And NetApp is not the only storage vendor with an external cache appliance. In Bryan's post we see that ACCESS operations were increased in the SPEC SFS 2008 workload relative to SPEC SFS 3.0. I can't seriously think that Bryan believes storage vendors want to handle more ACCESS calls. The percentage increased between 3.0 and 2008 because customers switched from NFS clients that had a good internal cache of ACCESS results to ones that did not. I.e. customers stopped using Solaris NFS clients. This was unfortunate for Sun, but believe me, it was unfortunate for NetApp too ... I'm still having nightmares from last fall's customer escalation caused by a really bad ACCESS cache on the client. But nonetheless, the philosophy of SPEC SFS has always been to model reality as opposed to the idealist reality dream where a storage device never has to process a request. P.S., in an earlier blog post, I made the argument that SPEC SFS 2008's differences from SPEC SFS 3.0, show the caching on NFS clients has improved.

Comments

[Bryan Cantrill]

Hey Mike,

You write: "If a vendor can't excel at SPEC SFS, it probably can't excel at much else."

Given SPEC SFS's exclusive focus on the read-side and on latency, that's an entirely incorrect statement. Take, for example, bandwidth -- which SPEC SFS does not exercise at all. We have explicitly described our streaming performance here:

http://blogs.sun.com/brendan/entry/my_sun_storage_7410_perf

This was not using opaque, synthetic benchmarks but using straightforward techniques like dd -- every customer is welcome to take a Sun 7410 and repeat this experiment and verify that our results are not cooked.

So to back up your claim that SPEC SFS is the "minimum bar", how about you performing the same experiment on a 3170 or 6080? It's a very simple question, really, and it's one that SPEC SFS does not even attempt to answer: what is the maximum streaming read bandwidth from disk? What is the maximum streaming write bandwidth to disk? Actually, let's boil it something even simpler -- a yes or no question: can you keep 10 GigE fed?

I trust you won't be blogging the answers to these questions (as the answers reveal that there are products that have admirable SPEC SFS numbers that entirely fall down when examining bandwidth), but perhaps it will soften your position that SPEC SFS is the "minimum bar" for performance.

[Mike Eisler]

Hi Bryan,

Vendors that don't publish SPEC SFS numbers don't stay in the NFS storage business. Until there's a counter example, my claim stands.

You raise good points about other workloads being of interest. If they were of as much interest as SPEC SFS (which captures a combination of I/O and metadata, as it should since CIFS and NFS are file access protocols, and files have metadata and data), I'm sure the market would have led the storage industry to producing benchmarks with workloads different than SPEC SFS.

Thus I'm not interested in your challenges. Get SPEC, SPC, or a similar benchmark organization to adopt the workloads behind your challenges, and I'll personally express interest. While I cannot speak for what NetApp will do if that happens, note that NetApp has published results for the SPC-1 benchmark, which is an I/O-only workload.

Thanks for visiting.

[Bryan Cantrill]

You and I both know that you don't need some fancy apparatus to test bandwidth, and you certainly don't need to wait until there is an "official" benchmark to express an interest in something so fundamental. But perhaps I shouldn't be trying so hard to shake you out of your SFS-induced slumber -- the complete disinterest in the competitive threat that our products pose is actually quite helpful...

[max]

Bryan,

As a customer of both Sun and Netapp, I've tested your new box using our real life work loads and Spec. While your system is nice, it doesn't stand up in terms of performance vs. Netapp filers; and frankly from what I'm actually getting for my dollar you aren't that much more cost competitive than Netapps midrange.

I've asked my account manager with Sun to please release Spec benchmarks, no matter how bad they might be before I put your kit in production. I don't understand why you think the new FISHWorks NAS is so special; even Sun's Procomm acquisition and 280Rs have Spec numbers. :)

[Bryan Cantrill]

Max,

If you have made a request via your account manager to have us report SPEC SFS numbers for the 7000 Series, that request has not yet reached me -- and I'm the person that that would need to reach. So if you'd like, comment on my blog at http://blogs.sun.com/bmc and leave your e-mail address (it won't be shown) and I will contact you and we can figure out what you need. I'm also curious about our appliance not standing up in terms of performance; your experience would be atypical in that regard, so I'd like to use analytics with you to understand your workload -- or have you already done this?

[Max]

Hi Bryan,

First, on the topic of the the request not reaching you, that is very frustrating to me.

I'll happily post on your blog; but I'm going to respond here as well, so here goes:

Workload: Mixed Block (iscsi/FCP) running databases on various OSs (MySQL, MSSQL) VMware images over NFS, Maildirs over NFS (For roughly 600,000 mailboxes), Large scale webhosting (content over NFS). The problem, in a nutshell, is some wildly inconsistent response times esp. with RaidZ.

BEGIN RANT

What boggles my mind is the distinct difference in tone between what your position is and what Sun's own sales people are telling me. They don't even bother to challenge the results of our benchmarking, they state that the Sun solution is not meant to compete directly with our filer, but rather it's positioned to be good second tier to NetApp, and I doubt this position is being taken to spare the poor customers feelings.

The problem from a customer perspective with not publishing spec numbers them is that raises more questions than it answers.

In my years of experience playing this game, I've made the mistake of going with other vendors who claimed very vocally to a more cost effective solution than Ntap, (Google "Unappliance" if you want one example of who I'm talking about) only to have it fall down hard in production.

The one common and very apparent thread with these vendors is that they don't publish spec numbers, and blow smoke by attacking the benchmark itself.

END RANT

Other than that, I owe you a blog post!

~Max

[Bryan Cantrill]

Hey Max,

Interesting! I've got a bunch of follow-up questions about your test config: how many disks/JBODs? What storage config? (Double parity RAID? Single parity RAID with narrow stripes?) How many log devices? How much read cache? What's your streaming/random breakdown from analytics on disk basis? (How many ops are you seeing per disk?) I'm certainly impressed that you were able to throw such a varied multi-tenancy workload at a test rig; the analytics must be really beautiful!

In terms of your rant about the difference in positioning: remember that this is a disruptive product in a new domain for Sun, and as such, there are different ideas on both how to use it and how to sell it. But if you're curious as to how I advocate positioning the product vis a vis competitive products, see this presentation:

http://blogs.sun.com/bmc/resource/disruptive7000.pdf

As I outline in that presentation, your reps aren't necessarily wrong -- we have seen tremendous uptake in Tier 2 (and beyond) where other products can't fit because of the economics. But we also believe that we can also go head-to-head in Tier 1 for NAS -- and the product is ultimately designed to be a Tier 1 product.

Anyway, hope that explanation does more good than harm. ;) Do comment on my blog entry or just shoot me an e-mail (my first name dot my last name at sun dot com), and we can talk about our product and your workload!

[Tom Haynes]

Actually my comment about caches is pretty simple, I argue that NetApp agrees that Spec SFS is not very interesting as a performance indicator because they use caches.

I follow that up by talking about Darrell Suggs, NetApp's performance troubleshooter extraordinaire. His approach to solving performance issues is to investigate the NFS server in the context of the environment enclosing it.

Which is a fancy way of saying the best performance metric is to run your application on it.