Mail package TPmail for Unix systems

Language

[eng] [rus]

Module milter_connect. Description and usage.

Module milter-connect
Features list of milter-connect
Switches of module milter-connect
Configuration file for milter-connect
Enable the basic features in the configuratuin file
Static white and black lists
Algorithm LMTA
Algorithm RHTA
Exceiptions mechanism as (accounts)
Auto dynamic white/black lists
Dynamic grey lists
Dynamic checked white lists
Local recipients list
Order to lookup of various lists during work with message's envelope
Regular and simple expressions
Sample configuration file
Examples of connection control
Examples of accounts usage
Diagnostic program mc_client

The general notes about the milter modules you must see in the section of the module milter-agent.
The module milter-connect is designed for an effective control of the connections before the beginning of the message receiving (before DATA phase of the protocol SMTP). The module is included the built-in antispam filter based on the algorithms LMTA and RHTA. The module milter-connect is provided the abilities to control the static white/black lists and the automatic dynamic lists using the results of a work of the heuristic algorithm LMTA. The module is allowed to expand of the functionality of the program sendmail by the addition the various filtering methods for the mail messages connections. The built-in is the effective and high-performance filter unlike the tradional systems of the mail content filter and our filter is not used the global servers of black lists at all.

The module milter-connect has the following options for the messages control (connections):

Dynamic re-configuration. The module is automatically loaded the new configuration if the syntax errors are not found in this new configuration.
Loading by demand. The configuration can be loaded by the external signal (USR1).
The syntax of the configuration file is simple and transparent to understand and use.
The support of the regular expressions for the lists (IEEE Std. 1003.2).
The output of the current configuration and state of the module.
No stop the processing if fatal error encountered - disk overflow and son on.
Detail info level of events for logs.
The local recipients lists for the cases of the transparent mail relaying.
Blocking "black" lists to reject always.
Accepting "white" lists to accept without any checks.
Reject/accept grey lists based on LMTA.
Rreject/accept checked lists based on LMTA.
Dynamic "white/black" lists (by sender and receiver).
Heuristic algoritm of legal sender status (LMTA - Legacy Mail Testing Algorithm).
Algoritm of checking of service headers of message (RHTA - Received Header Testing Algorithm).
Addition of results of checking by algorithms as message's headers.
Reject in receiving by algorithm LMTA (without real message's data receiving).
Caching of results of checking by the algoritm LMTA to improve the performance (reduce DNS server's loading).
White and black lists are used on the envelope processing phase using the various schemes of combinations of names, addresses and hosts.

The program milter_connect is supported the following options of the command line
(they are described in the standard manual man for milter_connect):

Option	Purpose
-h	print short help
-V	print version
-v	more detail info output
-c config	use this file as configuration file (default=/etc/mail/milter-connect.conf)
-p pipe	local socket (Unix domain socket) to data exchange with sendmail (default=unix:/var/spool/milter-connect/sock)
-i socket	local socket (Unix domain socket) to exchange data with user (default=unix:/var/spool/milter-connect/cmd)
-t timeout	operations timeout for socket of sendmail (default=0)
-u user	username, which rights using milter-connect (default=_milter-connect)
-l vlevel	debugging print level (default=0)
-d mode	working mode (0=daemon,>0=debug terminal)
-A	use a blocking to access of configuration
-C	dynamic control of configuration changes
-E	stop on any error
-B	allow static black lists
-W	allow static white lists
-X	enable support of regular expressions (POSIX.2)
-H	allow to add the service headers into message
-L	enable caching of results of algorithm LMTA
-E	allow rejecting by algorithm LMTA
-R	allow a work of algorithm LMTA
-T	enable testing output of LMTA × ÖÕÒÎÁÌ
-Z	allow a work of algorithm RHTA
-z	allow the rejecting by algorithm RHTA
-Y	enable profiling of algorithms LMTA/RHTA
-r	enable ouput of RHTA for information as header of message
-S	enable statistics ouput by headers of RHTA into log
-P	enable output results of checking by algorithms as service headers of message
-Q	use last in recipients list on message processing (default=first from list)
-a	check authenticaed users (SASL and other protocols)
-1	simplify configuration with LMTA (options -HYPUMSXWBRL)
-2	simplify configuration with RHTA (options -HYPUMSZrey)
-O	allow dynamic white lists
-e	allow ouput of found errors as header of message (RHTA)
-M	allow writing of timestamp and hostname as header of message
-U	allow writing about spam as subject of message
-y	allow writing of other flags of RHTA as header of message
-w	get info from configuration of sendmail for dynamic white lists
-x	get info from local config of host for dynamic white lists

The general parameters of configuration file of milter-connect are the same as the options for the command line. Remember that by default practically almost all options of the module are disabled.
All configuration files are read and processed in the sequential order. The sections parsing order is also sequential. It is allowed the administrator to define the own order of processing. All rules in the section are combined by the logical AND.

Example 1. Run of module milter-connect.
# /usr/local/libexec/milter_connect -c /etc/mail/milter/milter-connect.conf -u mailnull -vrAHCTSRWBXZPYL -l 1

Enable the basic options in the configuration file.
Always use the program mc_client to check the results of enabling/disabliing the wanted options!

[General_Parameters]
### Reject algorithm print mode (only print requests, for developers) (LMTA)
test_rej_algorithm = 1
### Reject algorithm (mandatory, run but no real action on reject) (LMTA)
run_rej_algorithm = 1
### Reject algorithm (run and reject as all in real life!) (LMTA)
### (Mandatory run_rej_algorithm=1)
enable_reject = 1
### Enable LMTA to cache results to reduce DNS and network loading
enable_lmta_cache = 1
### Print execution time profiling
time_profiling = 1

### Add message headers but not reject message
enable_spam_info_header = 1
enable_spam_info_subject = 1
enable_spam_info_timestamp = 1

### Enable algorithm (RHTA) to work
### (Mandatory) enable_algo2 = 1
### Reject all if not accepted by algorithm (RHTA)
### (Not recommended!)
##reject_all_by_algo2 = 1
### add message header about RHTA results
enable_spam_rhta_info = 1
### add message header about RHTA detected errors
rhta_errors_to_header = 1
### Print meaning of flags from algorithm (RHTA)
print_flags_algo2 = 1

### Enable auto white/black list collection to reduce user's questions
enable_auto_collect_lists = 1
### Auto_list bw state file (/var/spool/milter-connect/auto_list.state)
auto_list_state_file = "/var/spool/milter-connect/auto_list.state"
### Auto_list cache state file (/var/spool/milter-connect/auto_cache.state)
auto_cache_state_file = "/var/spool/milter-connect/auto_cache.state"

### Enable grey_lists to enable pass for some people on accounts (based on LMTA)
enable_grey_lists_cache = 1
### Enable random reject after grey lists passing
enable_grey_lists_random_reject = 1
### Enable random delay [min..max] instead fixed
enable_grey_lists_random_delay = 1

### Enable local recipients check against database enable_local_recipients_check = 1 ### Enable local recipients reject if not found in database enable_local_recipients_reject = 1
### Enable checked_lists to enable pass for some people on accounts (based on LMTA)
### (Not recommended!)
enable_checked_lists_cache = 1

The static black and white lists are the tradional way to fight with spam. It is possible to see them as partly obsolest (same as grey lists, method of checking of sender by reverse call, global servers of random lists and so on). But the black and white lists can be used in some cases if these lists are implemented so good as in milter-connect. A such good examples can be seen the reject with sender's addresses same as the domain-receiver (forged mail) or accept the service mail from trusted hosts. Generally, the white and black lists can be used as the exceiption or the effective addition for more powerful method of check and reject of spam.

The algorithms LMTA and RHTA were created recently as a new generaton tools to fight with the unwanted mail (spam or like something). Both algorithms are based on the principle: "think globally - act locally".

The algorithm LMTA is maybe the single effective algorithm rejecting the spam without the message's content filtering. The algorithm LMTA gets on its entry four parameters (really, enough two parameters): the IP address of the sender's host, the sender's email address (from the envelope), the domain name of the sender's host, and the heloname taken on stage helo of protocol SMTP. Using these parameters the algorithm tries to determine the real place of the sender. The effectivness of the algorithm on the various kinds of the traffic will be from 50 upto 99 percents. But, unlike the other mail systems, the mail package TPmail has the statistics module sma_stat which will be allowed the administrators to see the real results. Note the most important, that the algorithm LMTA is allowed to reject of the message receiving yet on stage of the envelope checking, i.e. before the message's body receiving. If we really don't accept the full messages the we don't spend our CPU power and disk/network capacity for the further processing of the spam messages. This is the absolute advantage when the sspam messages are thousands and thousands. But for some cases the algorithm LMTA is a too power weapon, for example, the user sends a messages with the email address of its ISP, but this messages comes from the address space of other ISP (Internet Service Provider). In these cases it is possible to use the static white lists with the user's authentication. If the organization must get all mail then it is possible to disable for all or some users the option of reject with the usage of algorithm LMTA, but it can be enabled the writing the service headers with the info of the checking results by the algorithm LMTA. This information could be used later, for example, in the filters of user's mail clients.

The algorithm RHTA checks the correctness of the message's service headers to confirm of the standards RFC 2821 and 2822. After this the algorithm ouputs the checking results, including a possible printing in the text format with the errors indication. The algorithm has the built-in recognizer of the messages from the popular mail servers as Microsoft Exchange, IBM/Lotus Domino, and some others, which unfortunately, have not full standard confirmance implementation or have the wrong settings done by these mail servers administrators. The messages from such servers will come regularly, therefore the algorithm RHTA will indicate these errors permanently. But unlike of the algorithm LMTA the algoritm RHTA don't use (by default) the auto reject and the administrator can add the corresponding filter into milter-agent to block messages containing the reject by the algorithm RHTA. The example of such filter can be found in the package of the module milter-agent. In general case it is recommended to do a such check byt the user's mail programs. The algoritm RHTA is required the full conformance to the standards RFC.

For some special cases, when some users are required the exceiptions from the acting rules there is the mechanism of the local exceiptions - accounts. The accounts are allowed to use the other policy for the selected users. For example, LMTA is enabled to check and reject globally, but the user's account is allowed to no check and no reject after the LMTA's work. True and vice versa. It is possible to check and reject with the algorithm LMTA on the user's group, but for other users the antispam filter will be disabled. The accounts are only used to receive the mail.
In the user's accounts it is possible to enable/disable the following options:
- run of algorithm LMTA;
- reject by algorithm LMTA (if run is enabled);
- remove from global white/black lists;
- add info about checking results by LMTA and RHTA as service header or subject of message;
- use lists of delayed accept (grey lists) on basis of algorithm LMTA;
- use lists white checked lists on basis of algorithm LMTA;

The automatic dynamic white/black lists are designed for cases, where it is impossible to use the static lists and for the initial collection of the statistics about the potential candidats to add into the static list. The auto lists are used two tables: auto cache (supplemental) and auto list (main). To store of the intermediate states between the loadings it is used the state file for each list. Give some principal explanation by example. It is assumed the it is enabled the global reject by the algorithm LMTA. It must be enabled the auto lists options and additionally it must be described the local domains (for the internal senders). The domain is all symbols right after a symbol @. The external message don't accepted by LMTA (for example, this messages comes from state Texas with the email domain of Ukraine as the sender's domain). In this case, if there are not used other rules, the connection information is copied to the auto cache list. If during the one day (default lifetime) the internal user writes to this external sender, the the following from this external sender will be accepted by the matching pair in the dynamic white list. Same will be true if local user writes early and the external user answers later. Now if this pair of abonents will exchange the messages, then each message from any side will increase the lifetime in the auto list by one day (default value), but not more than one month.

The lists of delaying accept or "grey" lists are different from the implementations in the other systems. At first, they are acted only for user's accounts, at second, the reject on receiving is not random, but it is based on the algorithm LMTA. It is allowed to avoid the silly total rejects like the massive rejects in the other mail systems that also given great overhead on sending mail servers. Because these lists are implemented only for the accounts then it is possible to use them selectively. A method of grey lists is not technically powerfull method, it is effective due the well-known reasons.
The current implementation of grey lists has two additional options in grey lists policy:
(1) random_reject - this is random reject (RNG) on message coming after initial reject period expiration;
(2) float window of initial reject period (random_delay) - selection of initial period from interval;

The dynamic checked white lists are designed to reject the massive resending for the defined time interval. The implementation of these lists is also based on the results of the algorithm LMTA. The work principle is simple. Assume that the local user is started to give a massive mail bulk for a short time. The given messages are successfully accepted by the algorithm LMTA, but the messages parameters are changed (envelope addresses, IP address of sender's host). Then we can assume with a little hope that is the mass bulk of the spam. In this case we are accepted only the first messages, but the following messages are rejected before the lifetime of this item in the cache is expired (each rejected message increases this time by a period). The checked lists can be used only for the user's accounts.

In some cases it is impossible to have the local users, for example, on the intermediate mail server or for the mail transit relaying (from external server to internal). If we must to avoid the server's messages about the unknown users then we can used the local recipients lists. Of course, we can have the exceiptions from this list for hosts, domains and IP addresses. The local recipients list is the global and it is acted for all addresses without exceiptions on the checking of the recipient's address. Be careful!

Lookup order of various lists during processing of the envelope's message

During the processing of the message's envelope all lists are checked in the following order:
(1) white/black dynamic_lists;
(2) static white lists;
(3) static black lists;
(4) accounts - if LMTA not enable for account, then goto next stage of message receiving;
(5) get results of work of algorithm LMTA;
(6) check accounts (reject by LMTA if enabled for account, grey lists with temporary reject);
(7) check accounts (white checked lists with temporary reject, if LMTA is given positive answer);
(8) if enabled global reject by LMTA, then reject on negative answer from algorithm LMTA;
(9) checki local recipients list, if this checking enabled;
Increase the detail info level output for the module, it is possible to see the order of the lists lookup during the envelope message's processing.

The regular expressions are allowed for the most lists.
See for a short help by the regular expressions re_format(7) and regex(3).
It is possible to use the simple expression instead the regular. The simple expressions
are required the exact match from the buffer beginning. In them there are allowed the lists with space.
Unlike the simple expressions on the usage with the regular expressions is searched the match
with a patterm in all buffer, not only at buffer's beginning. The following expressions are equivalence:
(1) "user1@domain.com" (simple) and "^user1@domain.com$" (regular);
(2) "user1@domain.com user2@host.domain.net" (simple) and "^(user1@domain.com|user2@host.domain.net)$" (regular);
Especially for the administrators the module has now the service program check_regex to check the regular expressions.

Sample configuration file (available also in package): milter-connect.conf.sample

Examples of connections control (white and black lists).

Example 1. Deny to accept connections from host (static black list).
[Filter]
reject_stage = connect
reject_message = "Prohibited by administrative policy."
reject_hostaddr = "192.168.0.2"

Example 2. Allow to accept mail upto this date (static white list with defined lifetime).
[Filter]
accept_stage = envrcpt
accept_hostname = "host1.domain.ru"
accept_date = "31122003"
accept_envfrom = "user1@host1.domain1.ru user2@host1.domain.ru"
accept_envrcpt = "user3@host3.domain3.ru"

Example 3. Deny to accept mail between users (static black list).
[Filter]
reject_stage = envrcpt
reject_envfrom = "user1@host2.domain.ru user2@host2.domain.ru"
reject_envrcpt = "user1@host3.domain.ru"
reject_message = "Non-legacy mail not accepted"

Example 4. Allow to accept mail between users (static white list).
[Filter]
accept_stage = envrcpt
accept_envfrom = "(user1|user2)@host2\.domain\.ru"
accept_envrcpt = "user1@(()|.*\.)host3\.domain\.ru"

Example 5. Allow to accept all mail from given user (static white list).
[Filter]
accept_stage = envrcpt
accept_envfrom = "user2@foreigndomain.com"
accept_envrcpt = ".*"

Example 6. Allow to accept all mail for given user (static white list).
[Filter]
accept_stage = envrcpt
accept_envfrom = ".*"
accept_envrcpt = "user1@mydomain.ru"

Example 7. Deny to accept all mail from given user (static black list).
[Filter]
reject_stage = envrcpt
reject_envfrom = "user2@foreigndomain.com"
reject_envrcpt = ".*"

Example 8. Deny to accept all mail for given user (static black list).
[Filter]
reject_stage = envrcpt
reject_envfrom = ".*"
reject_envrcpt = "user1@mydomain.ru"

Example 9. Allow to accept mail between given users (static white list).
[Filter]
accept_stage = envrcpt
accept_envfrom = "(user1@mydomain.ru|user2@foreign.com)"
accept_envrcpt = "(user2@foreign.com|user1@mydomain.ru)"

Example 10. Deny to accept mail between given users (static black list).
[Filter]
reject_stage = envrcpt
reject_envfrom = "(user1@mydomain.ru|user2@foreign.com)"
reject_envrcpt = "(user2@foreign.com|user1@mydomain.ru)"

Example 11. Initial settings for local net (static white/black list).
# Accepted hosts
##Added: 16.09.2004
[Filter]
accept_stage = envrcpt
accept_hostaddr = "192.168.0.1 - 192.168.0.14"
accept_envfrom = "@(()|.*\.)ho(me|ney)\.ru"
accept_envrcpt = ".*"

# Local hosts
##Added: 16.09.2004
[Filter]
accept_stage = envrcpt
accept_hostaddr = "127.0.0.1"
accept_envrcpt = ".*"

# PROHIBITED FORGED SENDERS OF OUR DOMAINS
##Added: 16.09.2004
[Filter]
reject_stage = envrcpt
reject_message = "Rejected forged senders of our domains"
reject_envfrom = "@(()|.*\.)ho(me|ney)\.ru"
reject_envrcpt = "@(()|.*\.)ho(me|ney)\.ru"

Example 12. Allow to accept all mail from authenticated users (static white list).
[Filter]
accept_stage = envrcpt
accept_envfrom = "@ourdomain\.ru"
accept_envrcpt = ".*"
auth_user_credentials = ".*"
auth_nonauth_users_also = 0 #auth_user_credentials = "(user1|user2|user3)"
#auth_user_alt_names = ".*"

Examples for accounts

Example 1. Allow to accept all mail for given user (like static while list).
[Account]
account_recipient_name = "user1@mydomain.ru"
account_global_run_reject_algorithm = 0
account_global_enable_reject = 0
account_local_spam_info_header = 0

Example 2. Allow to check by algorithm LMTA without rejecting, to mark messages.
[Account]
account_recipient_name = "user@domain.ru"
account_global_run_reject_algorithm = 1
account_global_enable_reject = 0
account_local_spam_info_header = 1

Example 3. Allow to check by algorithm LMTA with possibe rejecting, to mark messages.
[Account]
account_recipient_name = "(user1|user2)@domain\.ru"
account_global_run_reject_algorithm = 1
account_global_enable_reject = 1
account_local_spam_info_header = 1

Examples 4. Enable all checks by algorithm LMTA with possible rejecting and full marking of messages.
[Account]
account_recipient_name = "(user1|user2)@domain\.ru"
account_global_run_reject_algorithm = 1
account_global_enable_reject = 1
account_local_spam_info_header = 1
account_local_spam_info_subject = 1
account_global_enable_grey_lists = 1
###account_global_enable_checked_lists = 1

Example 5. Like example 4, only it uses all options of grey lists.
[Account]
account_recipient_name = "user3@host.domain.net"
account_global_run_reject_algorithm = 1
account_global_enable_reject = 1
account_local_spam_info_header = 1
account_local_spam_info_subject = 0
account_global_enable_grey_lists = 1
account_grey_lists_enable_random_reject = 1
account_grey_lists_enable_random_delay = 1
#account_grey_lists_cache_rec_ttl = 3600
#account_grey_lists_initial_reject_time = 900
#account_grey_lists_init_reject_time_min = 900
account_grey_lists_init_reject_time_max = 1500

The program mc_client interacts with the loaded module milter-connect through local socket (Unix domain socket). For a short help it is possble to run mc_client without parameters.

Usage example 1. Get current global settings of milter-quota.
# mc_client /var/spool/milter-connect/cmd 1

Usage example 2. Get current state of LMTA cache of milter-connect.
# mc_client /var/spool/milter-connect/cmd 5

Last modified: $Date: 2008-01-04 19:19:32+03 $

Mail package TPmail for Unix systems

Language

Home

Documentation

Support

News

Resources

Contacts

Module milter_connect. Description and usage.

Contents

Lookup order of various lists during processing of the envelope's message