[Leaplist] Any suggestions on resolving "Stale NFS file handle" errors?

[Damien McKenna]

Back to this again.

On Dec 13, 2008, at 8:46 AM, Bryan J Smith wrote:
> On Sat, 2008-12-13 at 01:43 -0500, Damien McKenna wrote:
>> The files which are causing the major problem are all in one
>> subdirectory (/opt/httpd/htdocs/mysite/coolstuff/), but the entire
>> directory is being mapped several levels up (/opt/httpd/htdocs) so I
>> don't know if it's going to be feasible to redo it given there are
>> several sites running on the same cluster, or if they're going to  
>> want
>> to.
>
> My apologies in advance, but I'm not following your logic here ...
>  "but the entire directory is being mapped several levels up"

I meant that it wasn't just this one site's htdocs directory that was  
mounted, the mount point covers all of the sites mounted.   
Specifically, the mount point goes to /opt/httpd/htdocs while my  
site's directory is in /opt/httpd/htdocs/mysite, with another 30-ish  
sites in there too.  This is a four node cluster, all running Ubuntu  
8.04.

> Understand you can map directories upon directories with NFS.  So you
> could map in separate rw directories for each system.

I'm starting to think that the solution to this one problematic  
directory structure is to control it locally per server rather than  
through the NFS, and just use Apache directory aliases to pull the  
content.

> You _must_ maintain coherency between clients.  The very likely reason
> you're getting stale NFS handles is because the NFS clients _are_  
> trying
> to enforce some coherency.  The very likely reason the Windows  
> client is
> not is because it doesn't bother and honor locks, checking state,  
> etc...

There are no Windows clients, that was someone else.

>> The "coolstuff" directory is pulled down on one of the servers (say,
>> server A) and then through the glory of NFS is accessible to all of
>> the other servers, so maybe I could talk with them about moving the
>> cron-update task to the SAN.  Would that make any difference?
>
> If they are mounting it read/write, and possibly using write locks on
> the files, hell yes.  The servers are liking running into locking
> issues.  But even then, you still haven't talked about how the clients
> maintain coherency.  I'd really like to see this architecture drawn  
> up,
> the type of interactions, etc....

I'm trying to find out how the clients and server are configured, the  
network structure, etc, but am not receiving any answers to the  
questions.

>> Key reasons are that for our CMS (drupal):
>
> BTW, Acquia Drupal is an official ISV's which Red Hat supports with
> SLAs, so I assume they have to have a "reference architecture" for
> clustering:
>  http://rhx.redhat.com/rhx/catalog/productdetail.jspa?productId=1027
>
> I'll see if I can get some info on this internally.

Have you dug up anything?  We're not eligible for Acquia's support  
plans yet, our sites are still on Drupal 5 while Acquia only supports  
Drupal 6.  And I'm not sure what the plan is on migrating (though  
several of us have been pushing for it).

>> * admins upload content (in this case specifically images),
>> * some parts of the static content (CSS, JS) are optimized & merged
>> into single files,
>> * image thumbnails are generated dynamically as needed,
>> and these actions can happen on any of the servers.
>
> You have an atomicity problem then.  You have to solve that first'n
> foremost.  You are getting NFS stale handle issues because content is
> changing, and it's happening mid-access as you get hits.  A _proper_  
> NFS
> client will tell you that.  A _poorly_ designed NFS client will gladly
> serve mixed-state content without.

Just to nitpick: new content gets added, but there are never changes  
to existing files, e.g. if a CSS file is being regenerated it uses a  
new filename so that visitors are sure of getting the new file.

FYI this is in addition to a large directory structure with over  
10,000 directories and files (mostly one index.html file per  
directory, though there are exceptions).

Drupal and its plugins are not intelligent enough to be told "only do  
action X on server Y", so we're limited to trying to improve the  
situation.

> When it comes to production sites, 10 out of 10 solve this with a
> combination of SQL and directory revisioning.  All dynamic content is
> stored in SQL, and seelct static content is rotated (e.g., new
> directory, change the alias -- version numbers, such as with a version
> control checkout revision, work well),  and a "graceful" is done to  
> load
> the new, static content.

The files that appear (my guess) to be causing this massive problem  
are updated a few times a week and there can be hundreds or thousands  
of changes each time.  The 3rd party company was reluctant to work out  
a way to programatically insert the content into Drupal itself (what I  
wanted) due to the volume of content changes, so we're stuck with the  
current separation for at least the next few months.

>>> 1.  How are you doing NFS serving/fencing on the storage-end?
>>
>> I don't know, but I'll talk with the adin.
>
> How their clustering architecture is designed would be of great  
> interest
> to myself -- client, storage, etc...  Does it follow any "reference
> design"?  Again, I'm going to see what we have with Acquia.

I'll work through the Drupal High Performance SIG to see if they have  
recommendations.

> 1.  Designate only 1 server, and _only_ mount the NFS export read/ 
> write
> there.  If this causes your CMS system and/or other software to break,
> then your CMS system was _only_ designed for a single server
> (unfortunately).

My guess is that, with how all of the pieces seem to fit, that it  
really is only designed for a single server.

> 2.  Find out what the "reference design" is for clustering with  
> Drupal.

Here's what they're doing for the upcoming drupal.org architecture  
revamp:
http://drupal.org/node/155850#comment-626523
It's from 2007 and I'm checking to see if there's an update, but at  
least it is a starting point.

Their discussions/graphs don't differentiate between read-only vs read/ 
write, so should I assume read/write?

One thing I see in that thread is that it was recommended to use  
proto=tcp, whereas our servers are using proto=udp.

Thanks.

-- 
Damien McKenna - Husband, father, geek.
damien at mc-kenna.com - http://www.mc-kenna.com/


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.