Fibre Channel over Ethernet (FCoE): Hurry Up and Wait

The other day a customer told me that he was all excited about FCoE (Fibre Channel Over Ethernet), and he asked if I thought he should be doing anything about it right now. My response: “FCoE is best for people who would rather wait.” The standard isn’t even done yet, so it clearly isn’t for people in a hurry.

NetApp is a big fan of any kind of storage over Ethernet. We were the first big NAS vendor, we were an early iSCSI proponent, and NetApp is the market share leader of the “storage over Ethernet” market as a whole. We even participated in an FCoE proof of concept demo at the Storage Networking World conference last fall.

That said, my first reaction to FCoE was skepticism. Given that many people are already using iSCSI and NAS for mission critical applications—SAP, Oracle, as well as Exchange and SQL Server—I wondered why we needed yet another protocol for storage over Ethernet. If you are really interested in storage over Ethernet, why wait? Why not start saving money now? I even wondered whether FCoE was an evil plot by Fibre Channel vendors who were hoping to sow confusion and thereby slow down adoption of Ethernet storage.

Since then, I’ve been educated about the benefits of FCoE, and I’ve gotten over my conspiracy theory. The big advantage of FCoE is that, from a storage management perspective, it looks almost exactly like traditional Fibre Channel SAN. For many customers, the management differences between NAS, iSCSI and SAN aren’t a problem. The number one feedback I’ve gotten from customers who have just installed iSCSI is: “It just worked.” On the other hand, change can be painful and expensive for customers with large SANs, especially if they have invested heavily in documenting their environment and training their employees. For them, it’s a big win if FCoE can make the transition to Ethernet easier.

Some people also argue that FCoE will have a performance advantage. I see that as less of an issue because Ethernet can already handle the vast majority of applications today. Oracle has one of the largest data centers in the world, and they run almost everything on NFS over Ethernet. They have even optimized the Oracle database  specifically to work well with NFS. Nevertheless, Fibre Channel has a latency advantage for a small handful of very performance-sensitive applications, and FCoE will allow even this handful to run over Ethernet.

So what is my advice? That depends on your situation. If you have an existing Fibre Channel infrastructure that meets your needs, the best option may be to do nothing. If you are considering a major Fibre Channel upgrade in 2010, or maybe even 2009, then you really ought to investigate FCoE to see whether it will be in time to meet your needs. On the other hand, if you want to start using Ethernet storage right away, then you should look closely at iSCSI and NAS. The best strategy may be to deploy iSCSI and NAS for new projects, and then use FCoE to convert your existing Fibre Channel infrastructure to Ethernet over time.

Maybe the most interesting thing about FCoE is that everyone seems to be taking it very seriously. Despite my initial skepticism, it appears that Fibre Channel switch vendors really do intend to convert their existing customer base to Ethernet. Now that Brocade has purchased Foundry, all of the major Fibre Channel switch vendors also have an Ethernet solution.

The end-game has become clear. The future of Fibre Channel is Ethernet.

Comments

[John Martin]

I still think its an evil plot :-).

One of the nice things about iSCSI is that assumes an unreliable Layer 2 network, and performs error correction on top of this. This allows for a much greater margin of error, which means faster deployments and lower purchase costs.

In all likelihood FCoE will assume a completely reliable Layer 2 network, which means the kinds of end to end QA and compatability matrices that plague FC deployments.

[Shibin Zhang]

Dave,

I am a big fan of your blog too. There is another great blogger named Chuck. Comparing your blog to Chuck's, I feel yours is more straightforward. I like your blog style a little bit better.

BTW, I remember Netapp CTO Dr. Pawlowski has a blog site too. In one of his blog, he mentioned that he was even managing an advanced developer team by himself. (Is he still managing the team?). Say "Hi" to him if you get a chance.

Thanks,

Shibin

[Shibin Zhang]

'Say "Hi" to him if you get a chance.'

-- A silly comment from me. I know you say "Hi" to him everyday if you both are working in the same building. What I really meant is: I would ask him "Hi, is your advanced developer group still growing?" if I got a chance.

[Shibin Zhang]

Dave,

I have two more comments:

1. You mentioned conspiracy, so my first comment is about conspiracy. There are two kinds of conspiracies: good conspiracy and bad conspiracy. I.e. I admit that data center consolidation is a conspiracy of green IT, but it’s a good one. It’s good to everyone. Data center consolidation doesn’t mean the vendors making less money. First, consolidation doesn’t mean price cut. If you consolidate two disks into one and double the price, you are making the same amount of money. Second, “next generation” is a driving force of replacing old equipment and “green” will be one of the factors driving “next generation”.
2. Your entire article shows very strong preference. It looks like you were forcing yourself thinking on NetApp side. Using some standard, you are not an independent thinker. Is there anything wrong? The answer is no. If I am working for NetApp, I will be thinking on NetApp side too. However, if I am working for EMC, I will be thinking on EMC side. I would say it’s an acceptable preference. From your blog, I never saw you were nice to EMC. Can I draw a conclusion that you are un-nice? The answer is clear: no!

Shibin

[Ashish Batwara]

Dave,
The introduction of FCoE in the same fabric particularly SAN will need Ethernet as well as FC management as compare to current FC. Do you think it may delay or decline the adoption of FCoE? Or standard like SMI-s help in this?

Ashish

[Netapp Fan]

Data Center Ethernet (DCE) solves the problem of too many diverse cables on server side and this could very well be true for Netapp as well. Today we got way too many cables - "Ethernet", "Fibre channel" and "Infiniband" on servers. With a single DCE physical link (having multiple virtual lanes), you can run "classical ethernet", "fibre channel" protocols. You won't need "Infiniband" any more as DCE is supposed to support Messaging and RDMA that Infiniband provides.

In short, DCE in short should cleanup all the cable mess we got in server farms today. This is going to be the "USB" of desktops.

[FCoE and pNFS]

Hi,
I read the white-paper by Mike E and Joshua K on Tech-ONTAP about pNFS.

Reading your end note,
"The end-game has become clear. The future of Fibre Channel is Ethernet"

I would say NETAPP is at an advantage with its NAS dominance and future developments w.r.t FCoE and pNFs go.

What would you like to comment on pNFS and FCoE?