Prev: outbound spam filtering
Next: 答复: Question about multiple postfix instance, thank you very very much
From: devel anaconda on 6 Nov 2009 02:16
06.11.09, 01:58, "Victor Duchovni" <Victor.Duchovni(a)morganstanley.com>:

> On Fri, Nov 06, 2009 at 12:02:45AM -0600, Stan Hoeppner wrote:
> > If it's a 1U server, I can tell you already that you are screwed, and
> > that you need a bigger chassis with more SCSI/SAS disk bays, at least 8
> > disks given your load of 250-300 msgs/sec.
> Does this application really need Postfix and a queue? Why not just
> turn the script into an SMTP server that pre-forks a fixed number of
> copies and loops receiving/processing email? For 200-300 msgs/sec on
> an I/O constrained server with a single "mailbox", one really does not
> need Postfix, and can't afford the I/O cost of a local queue.

Actually, I didn't tell the complete story.
The point is:

MTA (postfix) recieves the message. Then I need to put this message to 3 scripts (I must notify three different services about incoming mail). Now it looks like this:

mail comes to user(a)localhost on my Postfix, on aliases file I have:

usrer: user1,user2,user3

user1: |/usr/local/bin/script1
user2: |/usr/local/bin/script2
user3: |/usr/local/bin/script3

probably it'll be more scripts soon. When script1 fails with exit 75, but script2 and script3 have success, I must return mail for script1 to queue and try again later, and remove from queue messages for script2 and script3. That's why I use postfix and that's why I can't use pre-queued filter. I thought about different scheme, but postfix is the "easy way", so I'm trying speedup things a bit :)

From: Victor Duchovni on 6 Nov 2009 02:32
On Fri, Nov 06, 2009 at 10:16:35AM +0300, devel anaconda wrote:

> Actually, I didn't tell the complete story.
> The point is:
>
> MTA (postfix) recieves the message. Then I need to put this message
> to 3 scripts (I must notify three different services about incoming
> mail). Now it looks like this:
>
> mail comes to user(a)localhost on my Postfix, on aliases file I have:
>
> user: user1,user2,user3
>
> user1: |/usr/local/bin/script1
> user2: |/usr/local/bin/script2
> user3: |/usr/local/bin/script3
>
> probably it'll be more scripts soon. When script1 fails with exit 75,
> but script2 and script3 have success, I must return mail for script1 to
> queue and try again later, and remove from queue messages for script2 and
> script3.

Well, the Postfix "local" delivery agent does not do that for you,
it will retry all the scripts again, because local alias expansion is
done on each delivery attempt. The queue file stores only the original
recipient "user", so that can either be marked completed or re-processed
each time.

You could attempt to create appropriate "owner-alias" entries to make
this work, but with complex alias trees, I can't remember how. Far
easier to make your alias expansion to 3 users happen in virtual(5)
(pre-queue) not aliases(5) (post-queue).

For the throughput you are expecting, and especially running 3 three
(and later more) scripts for each message, you are not getting the job
done on the hardware you propose without serious optimization and
a design that reduces (rather than increases) the number of scripts
going forward if at all possible.

> That's why I use postfix and that's why I can't use pre-queued
> filter. I thought about different scheme, but postfix is the "easy way",
so I'm trying speedup things a bit :)

You are not going to succeed without more spindles and likely more CPUs.
Just launching a "Perl" hello-world script takes >10ms on a fast machine,
do that 100 times a second and you've burned a CPU, and non-trivial
scripts are slower still.

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo(a)postfix.org?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

From: Seth Mattinen on 7 Nov 2009 04:18
devel anaconda wrote:
>
> It disables fsync() on each incoming mail. Plus, if I mount my ext3 partition with option commit=30 or even commit=100, can it helps a bit?
>


Have you tried a filesystem other than ext3 like Reiser or XFS? The
performance of ext3 really, really sucks if you have a lot of files in
one directory as a mail spool is bound to have.

~Seth

From: Stan Hoeppner on 7 Nov 2009 14:41
Seth Mattinen put forth on 11/7/2009 3:18 AM:
> devel anaconda wrote:
>> It disables fsync() on each incoming mail. Plus, if I mount my ext3 partition with option commit=30 or even commit=100, can it helps a bit?
>>
>
>
> Have you tried a filesystem other than ext3 like Reiser or XFS? The
> performance of ext3 really, really sucks if you have a lot of files in
> one directory as a mail spool is bound to have.

I'd stay away from Reiser, for multiple reasons, but XFS would
definitely be an improvement over ext3. However, changing file systems
alone isn't going to solve his problem--it won't even double his
throughput, and he needs something like at least a 3x increase in disk
throughput at a minimum.

Unless he changes his filters so as to not write so darn many spool
files, he will absolutely require more spindles. SSD is an option for
speed, but I doubt he'd want to spend the money required for 100+ GB of
SSD. I'd say he needs at least 3 more SCSI drives in a Linux RAID 0
setup, or 6 more drives in a Linux RAID 10 setup, with XFS atop those.

IMHO, his next step should be adding a couple of 74GB 10K or 15K RPM
drives into his two empty disk bays, configure for RAID 0 with XFS atop
that, and test. If that's still not enough, then he should go with
something like that cheap 14 drive external array I previously
mentioned, or SSD if he's flush with cash.

--
Stan

From: Victor Duchovni on 7 Nov 2009 17:48
On Sat, Nov 07, 2009 at 01:18:35AM -0800, Seth Mattinen wrote:

> Have you tried a filesystem other than ext3 like Reiser or XFS? The
> performance of ext3 really, really sucks if you have a lot of files in
> one directory as a mail spool is bound to have.

This is not so in most cases. Ext3 is just fine for Postfix.

--
Viktor.

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.

To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majordomo(a)postfix.org?body=unsubscribe%20postfix-users>

If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: outbound spam filtering
Next: 答复: Question about multiple postfix instance, thank you very very much