From: Emmanuel Bailleul on
Hello,

I'm currently working on a test box to verify the behavior of some features I'd like to implement on a production mail relay server.
Among those are recipient_bcc_maps/sender_bcc_maps (to keep copies of certain emails) and after-queue content_filter (to be able to block certain file types).

These two features alone work as expected/documented, but when used altogether produce the non desired effect of generating 2 identical bccs : the first one seems to occur at first submission and the second one after "resubmission" by the content_filter.
I have tried to override the 'recipient_bcc_maps' setting in master.cf but to no avail.

Isn't this setting overridable by any means ? or maybe the same goal could be achieved another way ?

Please find below the output of 'postconf -n', an extract of my master.cf, my recipient_bcc file and related logs from a test (sending an email via smtp from localhost) :

### postconf -n ###
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/sbin
data_directory = /var/lib/postfix
debug_peer_level = 2
html_directory = no
mail_owner = postfix
mailq_path = /usr/bin/mailq
manpage_directory = /usr/man
message_size_limit = 1024000
mydestination = $myhostname, zencore.$mydomain, localhost
mydomain = blah.org
myhostname = zencore.blah.org
mynetworks = 10.36.6.219/32
newaliases_path = /usr/bin/newaliases
queue_directory = /var/spool/postfix
readme_directory = /usr/doc/postfix-2.6.5/readme
recipient_bcc_maps = hash:/etc/postfix/recipient_bcc
relay_domains = testdom.fr
sample_directory = /etc/postfix
sender_bcc_maps = hash:/etc/postfix/sender_bcc
sendmail_path = /usr/sbin/sendmail
setgid_group = postdrop
smtpd_recipient_restrictions = permit_mynetworks check_recipient_access hash:/etc/postfix/denied_recipients reject_unauth_destination
smtpd_sender_restrictions = permit_mynetworks reject_unknown_sender_domain reject_non_fqdn_sender check_sender_access hash:/etc/postfix/sender_access permit
transport_maps = hash:/etc/postfix/transport
unknown_local_recipient_reject_code = 550
###

### recipient_bcc file ###
@testdom.fr root
###

### master.cf ###
(snip)
smtp inet n - n - - smtpd
-o content_filter=filter:dummy
-o recipient_bcc_maps=
(snip)
filter unix - n n - - pipe
flags=Rq user=manu argv=/tmp/filter.sh -f ${sender} -- ${recipient}
###

### maillog ###
Jun 11 00:46:36 (none) postfix/smtpd[1979]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
Jun 11 00:46:36 (none) postfix/smtpd[1979]: connect from localhost[127.0.0.1]
Jun 11 00:47:04 (none) postfix/smtpd[1979]: B3DE110CE3: client=localhost[127.0.0.1]
Jun 11 00:47:27 (none) postfix/cleanup[1984]: B3DE110CE3: message-id=<20100610224704.B3DE110CE3(a)zencore.blah.org>
Jun 11 00:47:27 (none) postfix/qmgr[1956]: B3DE110CE3: from=<user(a)bonjour..fr>, size=338, nrcpt=2 (queue active)
Jun 11 00:47:28 (none) postfix/pickup[1955]: 1DED810CE5: uid=1000 from=<user(a)bonjour.fr>
Jun 11 00:47:28 (none) postfix/cleanup[1984]: 1DED810CE5: message-id=<20100610224704.B3DE110CE3(a)zencore.blah.org>
Jun 11 00:47:28 (none) postfix/pipe[1985]: B3DE110CE3: to=<me(a)testdom.fr>, relay=filter, delay=31, delays=31/0.02/0/0.16, dsn=2.0.0, status=sent (delivered via filter service)
Jun 11 00:47:28 (none) postfix/pipe[1985]: B3DE110CE3: to=<root(a)zencore.blah.org>, relay=filter, delay=31, delays=31/0.02/0/0.16, dsn=2.0.0, status=sent (delivered via filter service)
Jun 11 00:47:28 (none) postfix/qmgr[1956]: B3DE110CE3: removed
Jun 11 00:47:28 (none) postfix/qmgr[1956]: 1DED810CE5: from=<user(a)bonjour..fr>, size=453, nrcpt=3 (queue active)
Jun 11 00:47:28 (none) postfix/local[1991]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
Jun 11 00:47:28 (none) postfix/local[1991]: 1DED810CE5: to=<root(a)zencore.blah.org>, relay=local, delay=0.2, delays=0.13/0.04/0/0.03, dsn=2.0..0, status=sent (delivered to mailbox)
Jun 11 00:47:28 (none) postfix/local[1991]: 1DED810CE5: to=<root(a)zencore.blah.org>, relay=local, delay=0.26, delays=0.13/0.07/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 11 00:47:28 (none) postfix/smtp[1992]: connect to 10.36.6.219[10.36.6.219]:25: Connection refused
Jun 11 00:47:28 (none) postfix/local[1993]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
Jun 11 00:47:28 (none) postfix/smtp[1992]: 1DED810CE5: to=<me(a)testdom.fr>, relay=none, delay=0.26, delays=0.13/0.11/0.01/0, dsn=4.4.1, status=deferred (connect to 10.36.6.219[10.36.6.219]:25: Connection refused)
Jun 11 00:47:29 (none) postfix/smtpd[1979]: disconnect from localhost[127.0..0.1]
###

As seen in the logs, same email is delivered twice to root's mailbox (which I can verify by opening it).

Note 1 : the 'connection refused' message is due to no real mail server being present at the IP address. This is just for a test and I don't think it's relevant for my problem.

Note 2 : for the test, I use a "NOOP" filter shell script which does nothing but resubmitting mails via 'sendmail' command. I observed the same behavior with amavisd-new and smtp "resubmission".

Thanks for your help

Emmanuel

From: Jeroen Geilman on
On 06/11/2010 12:59 AM, Emmanuel Bailleul wrote:
> Hello,
>
> I'm currently working on a test box to verify the behavior of some features I'd like to implement on a production mail relay server.
> Among those are recipient_bcc_maps/sender_bcc_maps (to keep copies of certain emails) and after-queue content_filter (to be able to block certain file types).
>
> These two features alone work as expected/documented, but when used altogether produce the non desired effect of generating 2 identical bccs : the first one seems to occur at first submission and the second one after "resubmission" by the content_filter.
> I have tried to override the 'recipient_bcc_maps' setting in master.cf but to no avail.
>
> Isn't this setting overridable by any means ? or maybe the same goal could be achieved another way ?
>
> Please find below the output of 'postconf -n', an extract of my master.cf, my recipient_bcc file and related logs from a test (sending an email via smtp from localhost) :
>
> ### postconf -n ###
> command_directory = /usr/sbin
> config_directory = /etc/postfix
> daemon_directory = /usr/sbin
> data_directory = /var/lib/postfix
> debug_peer_level = 2
> html_directory = no
> mail_owner = postfix
> mailq_path = /usr/bin/mailq
> manpage_directory = /usr/man
> message_size_limit = 1024000
> mydestination = $myhostname, zencore.$mydomain, localhost
> mydomain = blah.org
> myhostname = zencore.blah.org
> mynetworks = 10.36.6.219/32
> newaliases_path = /usr/bin/newaliases
> queue_directory = /var/spool/postfix
> readme_directory = /usr/doc/postfix-2.6.5/readme
> recipient_bcc_maps = hash:/etc/postfix/recipient_bcc
> relay_domains = testdom.fr
> sample_directory = /etc/postfix
> sender_bcc_maps = hash:/etc/postfix/sender_bcc
> sendmail_path = /usr/sbin/sendmail
> setgid_group = postdrop
> smtpd_recipient_restrictions = permit_mynetworks check_recipient_access hash:/etc/postfix/denied_recipients reject_unauth_destination
> smtpd_sender_restrictions = permit_mynetworks reject_unknown_sender_domain reject_non_fqdn_sender check_sender_access hash:/etc/postfix/sender_access permit
> transport_maps = hash:/etc/postfix/transport
> unknown_local_recipient_reject_code = 550
> ###
>
> ### recipient_bcc file ###
> @testdom.fr root
> ###
>
> ### master.cf ###
> (snip)
> smtp inet n - n - - smtpd
> -o content_filter=filter:dummy
> -o recipient_bcc_maps=
> (snip)
> filter unix - n n - - pipe
> flags=Rq user=manu argv=/tmp/filter.sh -f ${sender} -- ${recipient}
> ###
>
> ### maillog ###
> Jun 11 00:46:36 (none) postfix/smtpd[1979]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
> Jun 11 00:46:36 (none) postfix/smtpd[1979]: connect from localhost[127.0.0.1]
> Jun 11 00:47:04 (none) postfix/smtpd[1979]: B3DE110CE3: client=localhost[127.0.0.1]
> Jun 11 00:47:27 (none) postfix/cleanup[1984]: B3DE110CE3: message-id=<20100610224704.B3DE110CE3(a)zencore.blah.org>
> Jun 11 00:47:27 (none) postfix/qmgr[1956]: B3DE110CE3: from=<user(a)bonjour.fr>, size=338, nrcpt=2 (queue active)
> Jun 11 00:47:28 (none) postfix/pickup[1955]: 1DED810CE5: uid=1000 from=<user(a)bonjour.fr>
> Jun 11 00:47:28 (none) postfix/cleanup[1984]: 1DED810CE5: message-id=<20100610224704.B3DE110CE3(a)zencore.blah.org>
> Jun 11 00:47:28 (none) postfix/pipe[1985]: B3DE110CE3: to=<me(a)testdom.fr>, relay=filter, delay=31, delays=31/0.02/0/0.16, dsn=2.0.0, status=sent (delivered via filter service)
> Jun 11 00:47:28 (none) postfix/pipe[1985]: B3DE110CE3: to=<root(a)zencore.blah.org>, relay=filter, delay=31, delays=31/0.02/0/0.16, dsn=2.0.0, status=sent (delivered via filter service)
> Jun 11 00:47:28 (none) postfix/qmgr[1956]: B3DE110CE3: removed
> Jun 11 00:47:28 (none) postfix/qmgr[1956]: 1DED810CE5: from=<user(a)bonjour.fr>, size=453, nrcpt=3 (queue active)
> Jun 11 00:47:28 (none) postfix/local[1991]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
> Jun 11 00:47:28 (none) postfix/local[1991]: 1DED810CE5: to=<root(a)zencore.blah.org>, relay=local, delay=0.2, delays=0.13/0.04/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
> Jun 11 00:47:28 (none) postfix/local[1991]: 1DED810CE5: to=<root(a)zencore.blah.org>, relay=local, delay=0.26, delays=0.13/0.07/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
> Jun 11 00:47:28 (none) postfix/smtp[1992]: connect to 10.36.6.219[10.36.6.219]:25: Connection refused
> Jun 11 00:47:28 (none) postfix/local[1993]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled
> Jun 11 00:47:28 (none) postfix/smtp[1992]: 1DED810CE5: to=<me(a)testdom.fr>, relay=none, delay=0.26, delays=0.13/0.11/0.01/0, dsn=4.4.1, status=deferred (connect to 10.36.6.219[10.36.6.219]:25: Connection refused)
> Jun 11 00:47:29 (none) postfix/smtpd[1979]: disconnect from localhost[127.0.0.1]
> ###
>
> As seen in the logs, same email is delivered twice to root's mailbox (which I can verify by opening it).
>
> Note 1 : the 'connection refused' message is due to no real mail server being present at the IP address. This is just for a test and I don't think it's relevant for my problem.
>
> Note 2 : for the test, I use a "NOOP" filter shell script which does nothing but resubmitting mails via 'sendmail' command. I observed the same behavior with amavisd-new and smtp "resubmission".
>
>
Those don't work at all in the same way.

The point of the transport is to resubmit the mail to a *different* smtp
listener, that has bcc_maps unset.
If you disable it on the main smtp listener, it won't do what you think.

Create a new smtp transport for your content filter, unset *_bcc_maps,
and re-inject via smtp.

That's how amavis does it (and not how you describe it).

J.

From: Emmanuel Bailleul on
> -----Message d'origine-----
> De : owner-postfix-users(a)postfix.org [mailto:owner-postfix-
> users(a)postfix.org] De la part de Jeroen Geilman
> Envoyé : vendredi 11 juin 2010 01:06
> À : postfix-users(a)postfix.org
> Objet : Re: recipient_bcc_maps override
>
> On 06/11/2010 12:59 AM, Emmanuel Bailleul wrote:
> > Hello,
> >
> > I'm currently working on a test box to verify the behavior of some
> features I'd like to implement on a production mail relay server.
> > Among those are recipient_bcc_maps/sender_bcc_maps (to keep copies of
> certain emails) and after-queue content_filter (to be able to block
> certain file types).
> >
> > These two features alone work as expected/documented, but when used
> altogether produce the non desired effect of generating 2 identical bccs :
> the first one seems to occur at first submission and the second one after
> "resubmission" by the content_filter.
> > I have tried to override the 'recipient_bcc_maps' setting in master.cf
> but to no avail.
> >
> > Isn't this setting overridable by any means ? or maybe the same goal
> could be achieved another way ?
> >
> > Please find below the output of 'postconf -n', an extract of my
> master.cf, my recipient_bcc file and related logs from a test (sending an
> email via smtp from localhost) :
> >
> > ### postconf -n ###
> > command_directory = /usr/sbin
> > config_directory = /etc/postfix
> > daemon_directory = /usr/sbin
> > data_directory = /var/lib/postfix
> > debug_peer_level = 2
> > html_directory = no
> > mail_owner = postfix
> > mailq_path = /usr/bin/mailq
> > manpage_directory = /usr/man
> > message_size_limit = 1024000
> > mydestination = $myhostname, zencore.$mydomain, localhost
> > mydomain = blah.org
> > myhostname = zencore.blah.org
> > mynetworks = 10.36.6.219/32
> > newaliases_path = /usr/bin/newaliases
> > queue_directory = /var/spool/postfix
> > readme_directory = /usr/doc/postfix-2.6.5/readme
> > recipient_bcc_maps = hash:/etc/postfix/recipient_bcc
> > relay_domains = testdom.fr
> > sample_directory = /etc/postfix
> > sender_bcc_maps = hash:/etc/postfix/sender_bcc
> > sendmail_path = /usr/sbin/sendmail
> > setgid_group = postdrop
> > smtpd_recipient_restrictions = permit_mynetworks check_recipient_access
> hash:/etc/postfix/denied_recipients reject_unauth_destination
> > smtpd_sender_restrictions = permit_mynetworks
> reject_unknown_sender_domain reject_non_fqdn_sender check_sender_access
> hash:/etc/postfix/sender_access permit
> > transport_maps = hash:/etc/postfix/transport
> > unknown_local_recipient_reject_code = 550
> > ###
> >
> > ### recipient_bcc file ###
> > @testdom.fr root
> > ###
> >
> > ### master.cf ###
> > (snip)
> > smtp inet n - n - - smtpd
> > -o content_filter=filter:dummy
> > -o recipient_bcc_maps=
> > (snip)
> > filter unix - n n - - pipe
> > flags=Rq user=manu argv=/tmp/filter.sh -f ${sender} -- ${recipient}
> > ###
> >
> > ### maillog ###
> > Jun 11 00:46:36 (none) postfix/smtpd[1979]: warning: dict_nis_init: NIS
> domain name not set - NIS lookups disabled
> > Jun 11 00:46:36 (none) postfix/smtpd[1979]: connect from
> localhost[127.0.0.1]
> > Jun 11 00:47:04 (none) postfix/smtpd[1979]: B3DE110CE3:
> client=localhost[127.0.0.1]
> > Jun 11 00:47:27 (none) postfix/cleanup[1984]: B3DE110CE3: message-
> id=<20100610224704.B3DE110CE3(a)zencore.blah.org>
> > Jun 11 00:47:27 (none) postfix/qmgr[1956]: B3DE110CE3:
> from=<user(a)bonjour.fr>, size=338, nrcpt=2 (queue active)
> > Jun 11 00:47:28 (none) postfix/pickup[1955]: 1DED810CE5: uid=1000
> from=<user(a)bonjour.fr>
> > Jun 11 00:47:28 (none) postfix/cleanup[1984]: 1DED810CE5: message-
> id=<20100610224704.B3DE110CE3(a)zencore.blah.org>
> > Jun 11 00:47:28 (none) postfix/pipe[1985]: B3DE110CE3:
> to=<me(a)testdom.fr>, relay=filter, delay=31, delays=31/0.02/0/0.16,
> dsn=2.0.0, status=sent (delivered via filter service)
> > Jun 11 00:47:28 (none) postfix/pipe[1985]: B3DE110CE3:
> to=<root(a)zencore.blah.org>, relay=filter, delay=31, delays=31/0.02/0/0.16,
> dsn=2.0.0, status=sent (delivered via filter service)
> > Jun 11 00:47:28 (none) postfix/qmgr[1956]: B3DE110CE3: removed
> > Jun 11 00:47:28 (none) postfix/qmgr[1956]: 1DED810CE5:
> from=<user(a)bonjour.fr>, size=453, nrcpt=3 (queue active)
> > Jun 11 00:47:28 (none) postfix/local[1991]: warning: dict_nis_init: NIS
> domain name not set - NIS lookups disabled
> > Jun 11 00:47:28 (none) postfix/local[1991]: 1DED810CE5:
> to=<root(a)zencore.blah.org>, relay=local, delay=0.2,
> delays=0.13/0.04/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
> > Jun 11 00:47:28 (none) postfix/local[1991]: 1DED810CE5:
> to=<root(a)zencore.blah.org>, relay=local, delay=0.26,
> delays=0.13/0.07/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
> > Jun 11 00:47:28 (none) postfix/smtp[1992]: connect to
> 10.36.6.219[10.36.6.219]:25: Connection refused
> > Jun 11 00:47:28 (none) postfix/local[1993]: warning: dict_nis_init: NIS
> domain name not set - NIS lookups disabled
> > Jun 11 00:47:28 (none) postfix/smtp[1992]: 1DED810CE5:
> to=<me(a)testdom.fr>, relay=none, delay=0.26, delays=0.13/0.11/0.01/0,
> dsn=4.4.1, status=deferred (connect to 10.36.6.219[10.36.6.219]:25:
> Connection refused)
> > Jun 11 00:47:29 (none) postfix/smtpd[1979]: disconnect from
> localhost[127.0.0.1]
> > ###
> >
> > As seen in the logs, same email is delivered twice to root's mailbox
> (which I can verify by opening it).
> >
> > Note 1 : the 'connection refused' message is due to no real mail server
> being present at the IP address. This is just for a test and I don't think
> it's relevant for my problem.
> >
> > Note 2 : for the test, I use a "NOOP" filter shell script which does
> nothing but resubmitting mails via 'sendmail' command. I observed the same
> behavior with amavisd-new and smtp "resubmission".
> >
> >
> Those don't work at all in the same way.
>
> The point of the transport is to resubmit the mail to a *different* smtp
> listener, that has bcc_maps unset.
> If you disable it on the main smtp listener, it won't do what you think.
>
> Create a new smtp transport for your content filter, unset *_bcc_maps,
> and re-inject via smtp.
>
> That's how amavis does it (and not how you describe it).
>
> J.


Hi,

Thanks for your reply.
The content_filter I use on my test platform is the simple example from http://www.postfix.org/FILTER_README.html#simple_filter

I tried to override the *_bcc_maps on the main smtp daemon because with this filter, mails are resubmitting via the 'sendmail' command so I don't have this second smtp transport to perform my test on. Though I may try the override setting on the pickup entry of the master.cf file (?)

Even if this test setup seems "awkward", shouldn't the first bcc be ignored anyway ?

BTW I have already tried a similar setup with amavis (with 2 different smtp transports) and had the same problem.

I will update my test platform and make a new test with amavisd-new and "smtp-resubmission" (thus a different transport) and will post my results accordingly.

Thanks for your time.

Emmanuel