Prev: Solaris Patching on Mirrored Disks
Next: Sandisk Sansa "Clip" mp3-player: can load-up via unix/solaris?
From: Bernie Ohls on 16 Jun 2008 23:26
Good questions:

Greg Andrews wrote:
> What kind of NFS mounts are you currently using? v2, v3 or v4?

Generally v2. I should have added that though the NFS server is Solaris,
the clients can be Solaris or Linux or (potentially, down the road)
something else. This is one of the reasons for v2, since Linux
historically has had a hard time with anything newer though I don't know
how true that remains.

> As I recall, NFS v3 added a stat() speedup by sending the stat() data
> along with the directory listing when a directory is opened and read.
> I.e., the server would anticipate stat() calls for the items listed in
> the directory and deliver the data to the client ahead of time. Is
> your NFS client machine using old v2 mounts, perhaps?

I will look into it but believe they are mostly v2.

> What pattern of directory reads and stats is the app performing?
> (i.e. Does it open and read a directory, then perform stat calls on
> the items listed there?

If you mean does it stat all the files in a given directory, the answer
is no. It's a more-or-less random subset, but the number of containing
directories is generally relatively small (see below).

> Does it perform stat calls on absolute file
> paths without opening and reading the directories containing the
> files?

Yes, it currently *always* uses an absolute path, even if some of the
files are in the same, even the current, directory. Would it be faster
to handle all files in the same directory consecutively? Would using
relative paths help? Would it make sense to cd to a directory and stat
all the files we care about within it using unqualified names? Is it
necessary to use opendir/closedir to trigger the v3 optimization?

FWIW I just did a test run on a small data set. There were ~200K total
stat calls on ~900 total files contained in ~40 unique directories. Some
of these repeated stats were within the same process but most were not.
In other words a relatively small set of files is getting hammered with
a storm of repeated stat calls from a sequence of processes, so
process-space caching will not help much but OS/NFS caching could help a
lot.

> Is the application doing a lot of waiting for the automounter to mount
> the filesystem (e.g. the app scans people's home directories, and must
> wait for the automounter to mount the next one)?

No, the mounts are explicit (in /etc/vfstab or equivalent). Automounter
is not an issue.

> Are you sure the application absolutely, positively *must* be run on
> an NFS client rather than the NFS server?

I am positive that it must be done so *sometimes* - i.e. I am not in a
position to dictate that NFS is not an option. I can document that NFS
is costly but that will not cause the problem to go away, unfortunately.

Bernie O.
From: Stefan Krüger on 17 Jun 2008 04:58
Bernie Ohls wrote:
> I'm wondering if there are any good methods for tuning NFS to
> minimize the cost of stat()?

hm after reading this cachefs comes to my mind, though I've never used
it and I can't tell if it'll help you with your stat()-"problem"...
From: AGT on 17 Jun 2008 11:20
On Mon, 16 Jun 2008 12:28:20 -0400, Bernie Ohls wrote:

> I'm running an application which does a LOT of stat() system calls. I've
> observed that it runs much more slowly on NFS file systems, and after
> some work with truss have concluded that the slowness is primarily due
> to the fact that stat() calls are much more costly over NFS. I
> understand that this may be "just the way it is", and am looking into
> whether the application can be changed to do fewer stats, but in the
> meantime I'm wondering if there are any good methods for tuning NFS to
> minimize the cost of stat()? Also, is it possible newer versions such as
> NFSv4 may do better?

I know this wont fly but if you MUST use NFS especially in a mixed
environment I would try to sell the concept of a file server like
a netapp or even a Sun NAS product? to someone. Not being a fan of NFS my
opinion is no doubt tainted...

From: Rex Mottram on 18 Jun 2008 10:50
Bernie Ohls wrote:
> Yes, it currently *always* uses an absolute path, even if some of the
> files are in the same, even the current, directory. Would it be faster
> to handle all files in the same directory consecutively? Would using
> relative paths help? Would it make sense to cd to a directory and stat
> all the files we care about within it using unqualified names?

For the record, it sounded like there was a potential speedup in using
relative pathnames (on the theory that statting an absolute path incurs
a cost for every component) so I threw together a quick Perl script to
test it. Unfortunately, though there is a win, it turns out to be pretty
minor. Here's the output of my script:

### Elapsed for rel/229779: 329 seconds (698.41641337386 stat/sec)
### Elapsed for abs/229779: 341 seconds (673.838709677419 stat/sec)

which says that traversing 229779 NFS files it was able to do 698 stats
per second using relative paths (after cd-ing into each directory) and
673 stats per second using absolute paths.

Bernie O.
 | 
Pages: 1
Prev: Solaris Patching on Mirrored Disks
Next: Sandisk Sansa "Clip" mp3-player: can load-up via unix/solaris?