[Leaplist] Any suggestions on resolving "Stale NFS file handle" errors?

[Damien McKenna]

On Dec 9, 2008, at 8:26 PM, Bryan J Smith wrote:
> In other words, when I re-read your message, and saw that you were
> mounting web content "rw" and not "ro," then red flags went way up.   
> If
> you're round-robin serving, then your servers are stomping on each
> others write locks, and rather quickly.  ;)

The files which are causing the major problem are all in one  
subdirectory (/opt/httpd/htdocs/mysite/coolstuff/), but the entire  
directory is being mapped several levels up (/opt/httpd/htdocs) so I  
don't know if it's going to be feasible to redo it given there are  
several sites running on the same cluster, or if they're going to want  
to.

The "coolstuff" directory is pulled down on one of the servers (say,  
server A) and then through the glory of NFS is accessible to all of  
the other servers, so maybe I could talk with them about moving the  
cron-update task to the SAN.  Would that make any difference?

> Is there _any_ reason you mounting this data "rw" then?  Just curious.
> Given Apache can locate resources anywhere, as well as follow symlinks
> (including both crossing and not crossing filesystems), why aren't you
> mounting this data "ro"?

Key reasons are that for our CMS (drupal):
* admins upload content (in this case specifically images),
* some parts of the static content (CSS, JS) are optimized & merged  
into single files,
* image thumbnails are generated dynamically as needed,
and these actions can happen on any of the servers.

What I could do is see about writing a plugin to pre-generate some of  
the content on a specific server, so that in theory only one server  
would be writing to the cluster for those instances, but I don't know  
how feasible that is given the number of hits the site receives.

> 1.  How are you doing NFS serving/fencing on the storage-end?

I don't know, but I'll talk with the adin.

> 2.  How are you preventing your NFS clients from stomping on each
> other's write locks?

That I don't know.  The majority of the NFS errors are showing up for  
a directory which is only written to on one server.  That said, given  
that there are a few other sites on the server and several scenarios  
where any of the servers could be trying to write to the directory  
structure, I'll have to see if they can work out which directory /  
files are triggering the failure.

Thanks Bryan.

-- 
Damien McKenna - Husband, father, geek.
damien at mc-kenna.com - http://www.mc-kenna.com/


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.