Experiments in NFS Troubleshooting

While attempting to mount my NFS shares in the Ubuntu VMs in my DMZ, I ran into some trouble where my DMZ servers, acting as NFS clients, were unable to reach my NFS server across the ACL.

It’s difficult to troubleshoot a problem when you don’t know what “good” looks like and there doesn’t seem to be a lot of information on the internet about it either. Therefore, once I got it working, I thought I’d do a little experiment to make it easier to troubleshoot NFS mounts in the future.

I began by removing all of my NFS ports from my access control list (ACL), thus effectively blocking all NFS connections and then ran the following:

sudo mount -vvv -o proto=udp,vers=3 10.0.30.5:/mnt/user/backup/blogContent /mnt/backupNFS

By manually mounting the NFS share, with verbosity enabled -vvv, we can gain insight into how an NFS client connects to an NFS server. Simultaneously, by identifying what each step in the command’s verbose stdout corresponds to, we can use the above command to identify where our NFS connection is failing.

So let’s take a look!

Troubleshooting NFS Mounts with Verbose Output:

No ports permitted through ACL (not even port 111):

mount.nfs: trying text-based options 'proto=udp,vers=3,addr=10.0.30.5'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Timed out

Port 111 (rpcbind / sunrpc) Permitted (2049 and 32766-32768 still blocked by ACL):

mount.nfs: trying text-based options 'proto=udp,vers=3,addr=10.0.30.5'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: trying 10.0.30.5 prog 100003 vers 3 port UDP port 2049
mount.nfs: portmap query failed: RPC: Timed out

Now, a pattern is starting to emerge. So mount.nfs/the NFS client, in verbose mode, will show the step it’s on as it executes; the appearance of the line in no way means that it completed successfully unless another step occurs after it.

Take the case above, we have mount.nfs: trying 10.0.30.5 prog 100003 vers 3 port UDP port 2049, but port 2049 is currently blocked, and so the mount fails with mount.nfs: portmap query failed: RPC: Timed out.

Port 111 and 2049 (NFS) Permitted (32766-32768 still blocked by ACL):

mount.nfs: trying text-based options 'proto=udp,vers=3,addr=10.0.30.5'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: trying 10.0.30.5 prog 100003 vers 3 port UDP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.0.30.5 prog 100005 vers 3 port UDP port 32767
mount.nfs: portmap query failed: RPC: Timed out

Again, we see here that the last step was to attempt connection through port 32767, but since it’s blocked, it fails.

Ports 111, 2049, and 32766-32768 (All NFS Ports) Enabled:

mount.nfs: trying text-based options 'proto=udp,vers=3,addr=10.0.30.5'
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: trying 10.0.30.5 prog 100003 vers 3 port UDP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.0.30.5 prog 100005 vers 3 port UDP port 32767

And we have connection. Note the last step is still mount.nfs: trying 10.0.30.5 prog 100005 vers 3 port UDP port 32767, but more importantly, the lack of error mount.nfs: portmap query failed: RPC: Timed out.


Analysis

We have a phrase in medicine, “Nothing tells you more about how something works than what happens when it breaks.” That’s why pathologists are usually the smartest.

In the above experiment, by purposefully breaking the connection, we learn a lot about NFS client/server connections. From the experiment, we can infer that in a client connection to an NFS server, the following steps occur:

  1. The client first contacts the server on port 111 (i.e. the port mapper service; rpcbind). The portmap service then directs the client to the correct port for the NFS service.
  2. The client then contacts the NFS server on the NFS port 2049.
  3. Finally, the client establishes the mount using the NFS mount daemon on port 32767, rpc.mountd.

Pretty interesting stuff. On a more practical level, hopefully seeing the steps a client takes to establish a mount with an NFS server will help you troubleshoot any problems you come across.

Let me know if you have any thoughts or questions!

-TorqueWrench