[Leaplist] Any suggestions on resolving "Stale NFS file handle" errors?

[Bryan J Smith]

Stepping back, looking at the obvious ...

On Tue, 2008-12-09 at 09:46 -0500, Damien McKenna wrote:
> I'll see what I can do to find out more about the SAN setup.

It makes 100% difference.

Furthermore, what are you doing about "fencing" your storage?  And
"coherency"?  No network protocol solves that on its own.

In other words, when I re-read your message, and saw that you were
mounting web content "rw" and not "ro," then red flags went way up.  If
you're round-robin serving, then your servers are stomping on each
others write locks, and rather quickly.  ;)

On Tue, 2008-12-09 at 09:46 -0500, Damien McKenna wrote:
> The errors have really only started since we started pulling down  
> hundreds of megs of static files from a 3rd party service for display  
> on our sites.  There's 9840 directories at the top level, each  
> directory then has a few sub directories and only one actual file  
> (index.html) per directory.  In total there are 18000 directories and  
> roughly the same number of files.  Every hour the server runs an rsync  
> command to pull down updates to these files from an off-site server,  
> which can be anywhere from hundreds to thousands of updated  
> directories & files each time.

Is there _any_ reason you mounting this data "rw" then?  Just curious.
Given Apache can locate resources anywhere, as well as follow symlinks
(including both crossing and not crossing filesystems), why aren't you
mounting this data "ro"?

On Tue, 2008-12-09 at 09:46 -0500, Damien McKenna wrote:  
> I have worked out that they're using Ubuntu on the cluster and are  
> using nfs-common v1.1.2-2ubuntu2.1 along with kernel 2.6.24-19-server.

Okay, you've got things to consider ...  

1.  How are you doing NFS serving/fencing on the storage-end?
2.  How are you preventing your NFS clients from stomping on each
other's write locks?

Virtually 10 out of 10 "HOWTOs" I've seen on NFS-based load balancing
use:  

1.  A single NFS server, all other servers fenced.  There are other
options though, with hardware+software-based, lower-level file services
(Veritas, GFS, etc...)

2.  *READ-ONLY* mounts.  Unless you are guaranteeing writes are atomic
and singularities, mounting "rw" and round-robin your access via web
servers are going to cause a mess with locking.  Normally you use a
combination of "ro" mounts for shared data, and "rw" for exclusive or
otherwise minimal writes where servers won't be stomping on one another.

I almost want to throw the RH436 manual** at whomever designed this,
although I'm making assumptions based on your use of "rw" in your NFS
client options.


On Tue, 2008-12-09 at 09:40 -0500, Richard F. Ostrow Jr. wrote:
> I've found that linux in general tends to have lots of issues with NFS
> with other operating systems. I've got a FreeBSD RAID server that I've
> tried to tie multiple machines into (all linux, one vista over NFS)... and
> the only machine that works well is the vista machine (aside from some
> permissions issues, the performance and stability is much higher than with
> the linux machines).
> This is odd, as the Vista NFS support is rather seriously lacking in any
> form of documentation (all I've found is a few scraps of info from online,
> nothing in the internal "help" structure).

NT is probably failing to do a lot of locking and other details.  In
other words, just because you're not getting errors doesn't mean it's
"unsafe."  The Linux clients have NFS stale handles not because it's
"buggy" (although that can be the root cause at some level), but because
it's telling you, "hey, this schtuff is not coherent, it's changed from
my last access."

Yes, there are many immature Linux NFS servers and clients.  But in
kernel 2.6, on select distros, it's quite mature.  I would argue NFS
2.6.1x is mature -- e.g., FC 6 / RHEL 5 (at least with their patches).
There may be other distros, although some distros ship things that are
purposely a compatibility nightmare with NFS services (and you have to
read the release notes to find that out).

On Tue, 2008-12-09 at 09:40 -0500, Richard F. Ostrow Jr. wrote:
> The linux machines cannot handle large amounts of I/O... they end up
> hanging (hard) with no log message or anything I've been able to trace
> back. This is awkard because one of these is my mythtv backend running
> diskless. I've managed to make it work because the thing boots fine (as
> long as I bypass a lot of NFS necessities like using nfs.lockd and
> nfs.statd, neither of which work properly at boot time, which means that
> the whole fs lacks file locking support) and stores its recordings locally
> (3 TB software RAID-0). I boot it diskless so it can take advantage of the
> massive redundancy of my central file server, even though it has its own
> local disks. The OS is fault-tolerant... and I can live with it if I lose
> all my recordings.

What distros?  It does make a bit of difference.  Several distros
honestly don't care about NFS, and use the stock kernel.  I say that to
several kernel maintainers at a select distributor, and they fall into
shock.  ;)

On Tue, 2008-12-09 at 09:40 -0500, Richard F. Ostrow Jr. wrote:
> For my diskless machines (I've had more than one in the past, but now I'm
> just down to one), I've semi-overcome this issue by putting all locations
> that may have lock files into a tmpfs partition (which does support file
> locking), which would include things like /var/run, /var/lock, /tmp, and
> /var/tmp. I've also had to create some scripts to create some directories
> and change some permissions on boot-up, as mysql keeps its own folder in
> /var/run that it can't auto-create.
> In general, I'm getting tired of the linux implementation of NFS, and
> especially the common init-scripts implementation of NFS (like mounting
> the root file system R/W before starting nfs.statd and nfs.lockd, which
> makes booting with locking support impossible).

I think you're talking about legacy issues, although there are several
reasons for that order.



-- 
Bryan J  Smith                Professional, Technical Annoyance
Mugshot Homepage:  http://mugshot.org/person?who=58wDcGKx6NcZAb
---------------------------------------------------------------
           Fission Power:  An Inconvenient Solution            


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.