Prev: outbound spam filtering
Next: 答复: Question about multiple postfix instance, thank you very very much
From: Victor Duchovni on 5 Nov 2009 19:28
On Fri, Nov 06, 2009 at 02:19:34AM +0300, devel anaconda wrote:

> user: |/usr/local/bin/script
>
> The flow is about 250-300 mails per second.

Stop right there. At 200 msgs/sec, fork/exec of scripts via local(8)
is never going to give you the right performance. You need (in addition
to the faster disks already mentioned), to use a resident (pre-forked)
SMTP or LMTP server script to receive the mail in question.

- Do not use local(8)
- Do not start a new interpreter process for each message

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo(a)postfix.org?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

From: Stan Hoeppner on 6 Nov 2009 01:02
devel anaconda put forth on 11/5/2009 5:19 PM:
> Hello everybody!
>
> I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:

How many free (empty) SCSI disk bays do you have in this server? Is it
a 1U server chassis? Post the make/model of the server please, or
preferably a link to a photo of the server on the manufacturer's
website. The answer to this may or may not limit your options in
increasing your disk throughput, and thus solving your problem.

If it's a 1U server, I can tell you already that you are screwed, and
that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
disks given your load of 250-300 msgs/sec.

--
Stan

From: devel anaconda on 6 Nov 2009 01:11
05.11.09, 17:58, "Kenneth Marshall" <ktm(a)rice.edu>:

> On Fri, Nov 06, 2009 at 02:19:34AM +0300, devel anaconda wrote:
> > Hello everybody!
> >
> > I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> > This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:
> >
> > user: |/usr/local/bin/script
> >
> > The flow is about 250-300 mails per second.Everything is going fine, if there is no queue. But when queue grows to 10000+ letters (for some reasons), postfix loses control of it. deferred queue is null, active queue is almost null, but incoming queue is growing and growing. When incoming queue is about 100k+ letters, postfix just can't pick it up, until I stop incoming traffic. I'm wondering, why such a server can't handle this workload? Even if I do the following:
> >
> > user: /dev/null
> >
> Your system is not a high-performance server I/O-wise. Your two disks can only
> handle 200-300 fsync's to disk per second and postfix will always sync your
> mail to disk before passing it on for local processing. You will need a
> battery backed caching RAID controller or fast SSD drive for the
> /var/spool/postfix directories to allow you to go faster. You could maybe
> move the spool directory to a RAM or tmpfs file system as well. You lose
> the safety net for messages if you have a power or other hardware problem.


Firs of, thank you Kenneth, Wietse, Victor and Corey for quick answer.

So I must confess, all looks like the bottle neck for now is a IO performance of my system. Unfortunately, I cannot move all the postfix's spool to tmpfs or /dev/shmem because of large size of queue (sometimes about a 80-100Gb), and I cannot move only incoming queue to RAM, because of postfix's specific file moves (I need to patch postfix, so it can move messages in split-partitions-queue). Please, correct me, if I'm wrong, but if the current bottle neck is limited fsync, so I can do the following (at my own risk):


--- postfix-2.5.9/src/global/mail_stream.c.orig 2009-11-05 23:27:23.000000000 +0300
+++ postfix-2.5.9/src/global/mail_stream.c 2009-11-05 23:27:34.000000000 +0300
@@ -292,9 +292,9 @@
|| (want_stamp && stamp_path(VSTREAM_PATH(info->stream), want_stamp))
#endif
|| fchmod(vstream_fileno(info->stream), 0700 | info->mode)
-#ifdef HAS_FSYNC
+/* #ifdef HAS_FSYNC
|| fsync(vstream_fileno(info->stream))
-#endif
+#endif */
|| (check_incoming_fs_clock
&& fstat(vstream_fileno(info->stream), &st) < 0)
)

?

It disables fsync() on each incoming mail. Plus, if I mount my ext3 partition with option commit=30 or even commit=100, can it helps a bit?



> It probably never needed more than 5-8 local processes to sink only 200-300
> messages per second. That is why you do not see more. Add a "sleep 1" to
> your script and see what happens then. :)

Yes, it did the trick :)

From: devel anaconda on 6 Nov 2009 01:18
06.11.09, 00:02, "Stan Hoeppner" <stan(a)hardwarefreak.com>:

> devel anaconda put forth on 11/5/2009 5:19 PM:
> > Hello everybody!
> >
> > I have a high-performance server (dual quad-core Xeon 2.8Ghz + 16GB RAM + 2SCSI disks 140Gb), RedHat Enterprise Linux 5.4, software RAID1 + Postfix 2.5.9.
> > This server serves only smtp traffic. The only thing postfix should do - receive mails for one user and send it to local script via pipe. Like this:
> How many free (empty) SCSI disk bays do you have in this server? Is it
> a 1U server chassis? Post the make/model of the server please, or
> preferably a link to a photo of the server on the manufacturer's
> website. The answer to this may or may not limit your options in
> increasing your disk throughput, and thus solving your problem.
> If it's a 1U server, I can tell you already that you are screwed, and
> that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
> disks given your load of 250-300 msgs/sec.

This is an 1U PowerEdge server with 4 slots for SCSI disks

From: Victor Duchovni on 6 Nov 2009 01:58
On Fri, Nov 06, 2009 at 12:02:45AM -0600, Stan Hoeppner wrote:

> If it's a 1U server, I can tell you already that you are screwed, and
> that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
> disks given your load of 250-300 msgs/sec.

Does this application really need Postfix and a queue? Why not just
turn the script into an SMTP server that pre-forks a fixed number of
copies and loops receiving/processing email? For 200-300 msgs/sec on
an I/O constrained server with a single "mailbox", one really does not
need Postfix, and can't afford the I/O cost of a local queue.

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo(a)postfix.org?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: outbound spam filtering
Next: 答复: Question about multiple postfix instance, thank you very very much