NetApp - Eisler's NFS Blog

IETF 72 in Dublin Starts Monday, July 28, 2008

I have a slot at the NFSv4 working group meeting to talk about what comes after NFSv4.1.

Part II: Since NFSv4 is Stateful It Must Be Less Robust, Right?

It turns out I'm going to make this at least a three part series because I've received questions about CIFS that warrants separate treatment for this comparison of state recovery in NFSv4 and CIFS

Let's start with a logic error:

Jane's arms are not strong enough to pull her weight. Jane cannot climb ropes. Jill's arms are also not strong enough to pull her weight. Therefore Jill cannot climb ropes either.

Obviously, lack of arm strength could make it harder to climb ropes, but if one understands how many people climb a rope hung from the ceiling of a gymnasium, one knows that arm strength has little to do with it.

Now lets replace Jane with CIFS, Jill with NFSv4, "arms are not strong enough to pull her weight" " with "has state", "climb ropes" with "recover from network or server failure":

CIFS has state. CIFS cannot recover from network or server failure. NFSv4 also has state. Therefore NFSv4 cannot recover from network or server failure either.

The above is false, but throw in the nouns state and recover, and some people cannot immediately see the illogic.

Let's revisit Jane and Jill, and change the passage a bit:

Jane's arms are not strong enough to pull her weight, and she does not wear gym shoes. Jane cannot climb ropes. Jill's arms are also not strong enough to pull her weight, but she wears gym shoes. Jill can climb ropes.

No logic errors this time. Jane is probably trying to pull herself up using just her arms. Like Jane, I could not in junior high school, and still cannot as an adult, vertically pull myself up the full length of 30 foot rope attached to a ceiling. In my gym class there was one kid named Wade who could do it. Wade was heavy into weight training, had arms as thick as the thighs of most men, and could easily lift weights totaling more than his own weight. And by the time he pulled himself up to the top of the Samuel Crowther School's gym ceiling he was red faced and exhausted, and I'm surprised he had any stamina left to safely climb back down. Impressive nonetheless.

But the rest of us 90 pound weaklings managed to climb by using our shoes. Gym shoes have soles with great traction designed for running but also great for trapping the rope between bottom of one shoe and the top of the other. By holding the rope with the hands first, bending the legs to grab and trap the rope between the feet, and unbending the legs, once can inch up the rope with not more more effort needed than to climb stairs. One web site calls this the brake and squat method, http://www.powerathletesmag.com/archives/Girevik/Second/articleropeclimbing.htm . The web site says to wear pants when climbing this way, but Samuel Crowther's gym teachers would send us to detention if we didn't have gym shorts, so I guess we suffered or something.

So let's replace "wear gym shoes" with "possess state recovery":

CIFS has state, and it does not possess state recovery. CIFS cannot recover from network or server failure. NFSv4 also has state, but it does possess state recovery. NFSv4 can recover from network or server failure.

Again no logic errors.

Indeed, CIFS ties its state to the continued existence of the TCP connection between the CIFS client and server. Break the TCP connection, and all the state objects, locks, opens, sessions, etc. disappear. That is zero state recovery. Break the TCP connection in NFS, regardless of whether it is NFSv2, v3, v4, v4.1, ... and the NFS client just re-connects. State is not lost because there is no TCP connection.

So using CIFS as evidence that NFSv4 cannot recover from failure is false logic. Part I of this series explained how NFSv4 state recovery from server failure works. Part III will explore the impact on applications.

Dave Hitz and Brian Pawlowski Interviewed by Harvard Professor Margo Seltzer on the Topic of the Present and Future of SAN/NAS

I haven't seen Dave or Brian mention it on their NetApp blogs but the June, 2008 issue of ;login:  has an interesting interview of Dave and Brian by Margo. The abbreviated interview is at http://usenix.org/publications/login/2008-06/openpdfs/netapp.pdf and the full interview, facilitated by Rik Farrow, Editor of ;login: is at http://usenix.org/publications/login/2008-06/netappinterview.pdf .

Part I: Since NFSv4 is Stateful It Must Be Less Robust, Right?

From me that is a rhetorical question.

But lately I've been fielding this question from folks far less immersed in NFS. So it is optimal for me and hopefully whoever is reading this to give the short answer and the long answer.

The short answer is no. In fact NFSv4 protocol is more robust than NFSv3. The long answer follows. This will be a three part discussion; part I will be in this blog posting, and other parts will compare NFsv4 with CIFS, and explore the implications for applications.

When NFS was invented 1984, it was termed a "stateless" protocol, because server and/or client restart did not require any recovery.

When byte-locking locking (to support the fcntl() API from UNIX) was added to NFS clients and servers, it came in the form of two "sideband" protocols: The Network Lock Manager (NLM) and Network Status Monitor (NSM) protocols.

NLM was the "steady state" protocol for granting and releasing locks. NSM was the recovery protocol for dealing with NLM client or server restart. The issue for client restart is that because all processes that had locks no longer exist, the locks are no longer needed by the client, and so the server needs to know that. The NSM protocol requires that clients that restart notify all NLM servers that the client does not need the locks it had, and so each NLM server releases that client's locks. The issue for server restart is that the because the NLM protocol did not mandate that lock state survive a server restart, the server can forget what locks each NLM client had. Instead the NLM protocol mandates that the server keep track of the list of NLM clients that had locks, and the NSM protocol is used to notify all such clients when the server restarts. The clients then had to reclaim all the locks they had before server restarted. The server holds off granting new, non-reclaim locks until the a grace period expires.

Can you spot the holes in these recovery protocols? The first problem is what if the client crashes but fails to restart? The client now has locks on the server that it will never release, thus requiring manual intervention on the server. The second problem is what if the NSM message from the server fails to reach the client and another client comes after the grace period expires to request the same lock the first client had? Now two clients think they have the same lock on the same file, and if locking is being used to synchronized modifications, data corruption can results.

The addition of locking, even as sideband protocols, made NFS as whole less robust, albeit only if processes on the client side acquired locks.

Enter NFSv4, which fixes these problems by dispensing with the NSM protocol, and making locks subject to expiration if the client fails to maintain contact with the server (in other words, locks are "leased" to clients).

The NFSv4 client is required to renew its leased in a fixed, known time before the term of the lease expires. If the client fails to renew in time the locks it has are subjection to revocation. So lets say the client crashes and never restarts. Then the locks it had will automatically expire, allowing other clients to acquire the locks.

When an NFSv4 server restarts, as with NLM, there is a grace period for reclaim. But there is no NSM-like notifications to the client nor are any needed. Because the client has to renew its lease, it will eventually find out about the server restart on a renew attempt. The grace period is at least as long as the period of the lease, allowing any client that had locks sufficient time to find out that it has lost its lease and its locks. Thus it is not possible for to clients to think they have the same lock on the same file.

So when taking to the consideration client that performs both file and lock access, NFSv4 is more robust.

A follow up blog posting will discuss robustness issues for applications.