Samba 3.0.22 - slow performance - Really urgent help
in [Samba]
|
Prev: Failed to join domain: failed to precreate account in ou (null): Out of memory
Next: [Samba] How to stop mount.cifs remembering password
From: Volker Lendecke on 1 Apr 2010 05:50 On Thu, Apr 01, 2010 at 05:24:26PM +0800, Alan Pek wrote: > Have been struggling with this for the past 10 days, we are running Samba > 3.0.22 on VCS zone, we have end users saving files > onto Samba mapped drive, and complained that it 5mins to save 300 files, > now it takes 30 mins. There is recently a change in the > topology. > > Before > > Enduser ----------- samba mapped -------------- server (local > attached storage) > physical > Solaris 8 > Now > > Enduser-------------samba mapped ------------ server > ------------------------------ SFCFS/HA > virtual VCS NFS > cluster > Solaris 10 > > > We have reverted the NFS from version 4 to 3 > > Fine tune Samba, wb and rb are at 64k > > Is there in samba log somewhere to tell me why is the bottleneck ? > > If I move the Samba server which is less loaded in Memory > consumption , will it help. > Is the above degrading expected ? Some slow-down is expected if you re-export NFS because the data will travel over the network twice instead of once. A factor of 6 is a bit more than is expected though. To see where the bottleneck is, it might be interesting to look at simultaneous network traces for the SMB and NFS side of your Samba server to see what NFS requests are triggered by what SMB requests. Doing a truss with high-resolution timestamps on the smbd processes might also give hints. Next would be to look at "top" (pstat?) if there is a local CPU bottleneck on the Samba server. There's just many things that can go slow, and in a moderatly complex environment (HA clusters fall into this category), this can be an interesting hunt :-) Volker
From: Alan Pek on 1 Apr 2010 06:30 Hi Volker, Thanks for the useful feedback. a) Though the CPU utilisation is not high but there memory used and swapped are very high. This will slow it down further. b) Think every samba connection will consume a considerable amout of memory as well. c) We did a test, by saving a huge number of files onto local drive (on enduser PC) and the copy into the mapped drives This is much faster, probably by factor 6 or more, then using Excel marco and writing directly into the samba share over NFS. d) Looking at the system, and samba processes, how should I do a truss with high-resolution timestamps on the smbd processes ? Every smbd process ? Just do : truss -p 20995 without any option ? apps 28824 14805 1 13:50:58 ? 34:40 /opt/DBsamba/sbin/smbd -D root 14808 14807 0 Mar 27 ? 0:00 /opt/DBsamba/sbin/winbindd root 23698 23434 0 18:13:59 pts/3 0:00 grep -i mb root 21729 14805 0 18:01:18 ? 0:01 /opt/DBsamba/sbin/smbd -D root 29259 14805 0 11:32:38 ? 2:07 /opt/DBsamba/sbin/smbd -D root 14809 14805 0 Mar 27 ? 0:00 /opt/DBsamba/sbin/smbd -D root 20629 14807 0 Mar 29 ? 0:00 /opt/DBsamba/sbin/winbindd root 14807 21996 0 Mar 27 ? 0:01 /opt/DBsamba/sbin/winbindd apps 27073 14805 0 17:18:58 ? 0:04 /opt/DBsamba/sbin/smbd -D root 11568 14805 0 Mar 29 ? 76:51 /opt/DBsamba/sbin/smbd -D root 14801 21996 0 Mar 27 ? 0:06 /opt/DBsamba/sbin/nmbd -D root 22898 14805 0 18:09:42 ? 0:00 /opt/DBsamba/sbin/smbd -D root 14805 21996 0 Mar 27 ? 0:02 /opt/DBsamba/sbin/smbd -D root 11509 14805 0 Mar 29 ? 77:25 /opt/DBsamba/sbin/smbd -D root 29253 14805 0 11:32:25 ? 1:37 /opt/DBsamba/sbin/smbd -D root 23144 14805 0 18:10:55 ? 0:00 /opt/DBsamba/sbin/smbd -D apps 20995 14805 0 17:56:25 ? 0:04 /opt/DBsamba/sbin/smbd -D apps 3805 14805 0 14:26:03 ? 0:15 /opt/DBsamba/sbin/smbd -D e) Network tracing , meaning application profiling by putting a sniffer, or/and snoop on both samba and nfs side ? Thanks again. Regards Alan Volker.Lendecke(a)SerNet.DE 04/01/2010 05:48 PM Please respond to Volker.Lendecke(a)SerNet.DE To Alan Pek/db/dbcom(a)DBAPAC cc samba(a)lists.samba.org Subject Re: [Samba] Samba 3.0.22 - slow performance - Really urgent help On Thu, Apr 01, 2010 at 05:24:26PM +0800, Alan Pek wrote: > Have been struggling with this for the past 10 days, we are running Samba > 3.0.22 on VCS zone, we have end users saving files > onto Samba mapped drive, and complained that it 5mins to save 300 files, > now it takes 30 mins. There is recently a change in the > topology. > > Before > > Enduser ----------- samba mapped -------------- server (local > attached storage) > physical > Solaris 8 > Now > > Enduser-------------samba mapped ------------ server > ------------------------------ SFCFS/HA > virtual VCS NFS > cluster > Solaris 10 > > > We have reverted the NFS from version 4 to 3 > > Fine tune Samba, wb and rb are at 64k > > Is there in samba log somewhere to tell me why is the bottleneck ? > > If I move the Samba server which is less loaded in Memory > consumption , will it help. > Is the above degrading expected ? Some slow-down is expected if you re-export NFS because the data will travel over the network twice instead of once. A factor of 6 is a bit more than is expected though. To see where the bottleneck is, it might be interesting to look at simultaneous network traces for the SMB and NFS side of your Samba server to see what NFS requests are triggered by what SMB requests. Doing a truss with high-resolution timestamps on the smbd processes might also give hints. Next would be to look at "top" (pstat?) if there is a local CPU bottleneck on the Samba server. There's just many things that can go slow, and in a moderatly complex environment (HA clusters fall into this category), this can be an interesting hunt :-) Volker [attachment "signature.asc" deleted by Alan Pek/db/dbcom] --- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
From: Volker Lendecke on 1 Apr 2010 06:50 On Thu, Apr 01, 2010 at 06:22:34PM +0800, Alan Pek wrote: > a) Though the CPU utilisation is not high but there memory used and > swapped are very high. This will slow it down further. > > b) Think every samba connection will consume a considerable amout of > memory as well. The rough estimate here is that Samba should have 2-3MB real RAM per client. If your smbds use a lot more, you might be hitting a memleak that might be fixed in later versions. 3.0.22 is very old, I would really recommend looking at 3.4.7 or 3.5.2 (next week). > c) We did a test, by saving a huge number of files onto local drive (on > enduser PC) and the copy into the mapped drives This is much faster, > probably by factor 6 or more, then using Excel marco > and writing directly into the samba share over NFS. Do you have oplocks disabled? This would severely impact performance. If you don't have concurrent NFS clients accessing the same data, you might also want to try posix locking = no > d) Looking at the system, and samba processes, how should I do a truss > with high-resolution timestamps on the smbd processes ? Every smbd process > ? > Just do : > > truss -p 20995 without any option ? Not sure how truss really works, in Linux you would do a strace -ttT -p 20995 -o smbd.out You pick a single smbd with "smbstatus" and run your test. The truss output will need some interpretation though. > e) Network tracing , meaning application profiling by putting a sniffer, > or/and snoop on both samba and nfs side ? Yes. Best done on the samba server box itself. BTW, before you send stuff to the list -- both the truss and the network sniffer output will probably contain sensitive data. Volker
From: Christoph Beyer on 1 Apr 2010 07:40 Hi, if you do 'truss -f -p <smbd-PID>' do you see a lot of: 19702: fcntl(17, F_SETLKW64, 0xFFBFE640) = 0 ? If yes this is a problem that is present in most of the recent samba versions (tried 5 or 6 versions off the different branches) it's a solaris related bug but that's all I can say. It might be fixed in 3.5.1 ... cheers christoph On Thu, 1 Apr 2010, Volker Lendecke wrote: > On Thu, Apr 01, 2010 at 06:22:34PM +0800, Alan Pek wrote: > > a) Though the CPU utilisation is not high but there memory used and > > swapped are very high. This will slow it down further. > > > > b) Think every samba connection will consume a considerable amout of > > memory as well. > > The rough estimate here is that Samba should have 2-3MB real > RAM per client. If your smbds use a lot more, you might be > hitting a memleak that might be fixed in later versions. > 3.0.22 is very old, I would really recommend looking at > 3.4.7 or 3.5.2 (next week). > > > c) We did a test, by saving a huge number of files onto local drive (on > > enduser PC) and the copy into the mapped drives This is much faster, > > probably by factor 6 or more, then using Excel marco > > and writing directly into the samba share over NFS. > > Do you have oplocks disabled? This would severely impact > performance. If you don't have concurrent NFS clients > accessing the same data, you might also want to try > > posix locking = no > > > d) Looking at the system, and samba processes, how should I do a truss > > with high-resolution timestamps on the smbd processes ? Every smbd process > > ? > > Just do : > > > > truss -p 20995 without any option ? > > Not sure how truss really works, in Linux you would do a > > strace -ttT -p 20995 -o smbd.out > > You pick a single smbd with "smbstatus" and run your test. > The truss output will need some interpretation though. > > > e) Network tracing , meaning application profiling by putting a sniffer, > > or/and snoop on both samba and nfs side ? > > Yes. Best done on the samba server box itself. > > BTW, before you send stuff to the list -- both the truss and > the network sniffer output will probably contain sensitive > data. > > Volker > best regards ~ christoph -- /* Christoph Beyer | Office: Building 2b / 23 *\ * DESY | Phone: 040-8998-2317 * * - IT - | Fax: 040-8998-4060 * \* 22603 Hamburg | http://www.desy.de */ -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
From: Volker Lendecke on 1 Apr 2010 07:50
On Thu, Apr 01, 2010 at 01:32:02PM +0200, Christoph Beyer wrote: > if you do 'truss -f -p <smbd-PID>' do you see a lot of: > > 19702: fcntl(17, F_SETLKW64, 0xFFBFE640) = 0 > > ? > > If yes this is a problem that is present in most of the recent samba > versions (tried 5 or 6 versions off the different branches) it's a > solaris related bug but that's all I can say. It might be fixed in 3.5.1 > ... Those many fcntl calls are just normal Samba operation. They should be *very* fast. There might be a problem with the messages.tdb, notify.tdb and notify_onelevel.tdb in current Samba version, where there is one fcntl read lock being held per smbd on each of these databases. This might be a performance problem if you have thousands of connected clients. But normally, fcntl calls should be very fast. Volker -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba |