Mail package TPmail for Unix systems

Language

[eng] [rus]

Module milter_agent. Description and usage.

ïÇÌÁ×ÌÅÎÉÅ

General notes about milter modules
Module milter-agent
Features list of milter-agent
Switches and options of module milter-agent
Configuration file for milter-agent
Filters classification by purpose
Filters theory
Regular expressions
Sample configuration file
Examples of addresses filters
Examples of messages filters
Diagnostic and control program ma_client
Creation of dynamic libraries for module milter-agent

Generals notes about milter modules.
These modules are the intermediata agents during the mail processing, i.e they can changed the body and headers of the message, the recipients list, and also they can suggest the sendmail to accept or reject the message. But these agents are not the local delivery agents or the transport agents. The most effective purposes of the milter are the following: check the message for the virus presence, the antispam filtering, the transparent encryption/decryption, the mail archive storage. Other words, a such modules are provided the possibilities for the centralized processing of the mail messages.

The module milter-agent is implemented an idea of the pipeline processing of the information in the mail stream of sendmail. A key idea of the module is the ability to use the other programs to the message processing. Often in the other systems a such programs are called the external filters. The filter running to process can be done always or the matching of some defined conditions. For example, one filter can be inspected the message for the virus presence, other filter can be done the content checking for the spam presence and so on. The main difference of the filter mechanism in milter-agent from other like implementations are the following: simplicity, clearity and functionality of the call mechanism of the external programs (like the scripts running in the Unix shell). During the filter creation it is possible to use any development tools:
shell, C, C++, Perl, Basic, Lisp, Prolog, Haskel, Ruby and so on.

Module milter-agent has the following features to the message processing:

- Dynamic configuration. The module is automatically is loaded a new configuration if it is not found the syntax errors;
- Load by demand. The configuration can be loaded by the external signal (USR1);
- The syntax of the configuration file is a very simple and transparent to understand;
- To message processing it is possible to run the external program. It can used the defined conditions for this filter execution, for example, the match of sender's name or recipients's name, the match of message subject;
- To message processing it is possible to run the internal module (dynamically loaded library). The abilities are the same as for the external programs, but, of course, these modules are worked faster than the external programs;
- Add/Remove the users from the recipients list of the message (for example, automatic copy of some mail lists);
- Changing of the service headers of message (for example, the knowledge of that the messages had passed a some control;
- Changing of message body (for example, removing of unwanted part of message, i.e. mail censority)
- Spport of regular expressions for lists (IEEE Std. 1003.2);
- Print of current configuration and state of module;
- Don't stop the processing on the fatal errors - disks space exhaustion and so on;
- Ability to terminate "hanged" processes by timeout or disable "failed" dynamic libraries or same "bad" processes when the errors threshold is exceeded;
- Detail info level of events for logs;
- Processing of return codes from called programs/modules to accept the message or to reject the message;
- Easy changed parameters of configuraton for filters;
- Ability to run one (or no more than N) instances of application if this appalication is used the resources in exclusive mode (for example, the antivirus scanners);
- Support of time/date in extended format of daemon cron;

The program milter_agent is supported the following options of the command line
(they are also described in the standard manual man for milter_agent):

Option	Purpose
-h	print a short help
-V	print a version
-v	more detail info ouput
-c config	use this file as configuration (default=/etc/mail/milter-agent.conf)
-p pipe	local socket (Unix domain socket) to data exchange with sendmail (default=unix:/var/spool/milter-agent/sock)
-i socket	local socket (Unix domain socket) to data exchange with user (default=unix:/var/spool/milter-agent/cmd)
-t timeout	operations timeout fpr socket sendmail (default=0)
-u user	user's name, which rights will be used by milter-agent (default=_milter-agent)
-l vlevel	level of debugging print (default=0)
-d mode	working mode (0=daemon,>0=debugging terminal)
-m err_level	errors threshold during external programs and internal modules running (default=0)
-A	allow to add recipients for message
-R	allow to remove recipients for message
-H	allow to add service headers for message
-G	allow to change service headers for message
-B	allow to change a message's body
-a	use a blocking access for configuration
-C	close all open files on external program running (after fork, before exec)
-E	stop on any error
-F	dynamic control of configuration change
-T	allow to terminate an external program after timeout expiration
-M	allow to terminate an internal module after timeout expiration
-X	enable the support of regular expressions (POSIX.2)
-I	poutput info about temporary file (for archive systems)
-1	enable all these options -ARHGITMX (for more simple configuration)
-Q	use last recipient in list for message processing (default=first from list)

The general parameters of the configuration file milter-agent are the same as the options of the command line. Remember, that by the default almost all options of the module are disabled.

Example 1. Run a module milter-agent.
# /usr/local/libexec/milter_agent -c /etc/mail/milter/milter-agent.conf -u mailnull -HIBFGARXavTM -m 5 -l 1

Now the module is supported a two types of filters by the purpose: by addresses and by the messages.

The address filter is designed to control of the recipients list or the message headers. Its main abilities - the simplicity of the centralized address lists, the duplicatin of the mail stream, the redirection and other like tasks.

The message filter is designed to run the external applications or the dynamic libraries to work with a message and to report the results of processing to sendmail to accept or reject a given message.

All configuration files are read and processing in the sequential order. The filters running order is also sequential. It allows the administrator to define an own order. All rules in the filter are combined by logical AND.
The regular expressions (see re_format(7) and regex(3)) are allowed for the most lists.

Let's see the most important moments that is neccessary to understand before the right usage of the filters.

All filters are running in the process context of milter-agent. This means that the external programs and internal modules will have just the same rights as milter-agent. Therefore if you are used some catalogs to access or store of the programs or data, exceipt maybe of the catalog /tmp, it is neccessary to sure that you don't have the problems with the access rights. The running programs or the dynamic libraries will have the same problems with the access rights. Be careful with the access rights!

Milter-agent is running all filters sequentually. If some filter is rejected (reject,tempfail) or accepts (accept,discard) the message, using the mechanism of return codes, then all followed filters don't be called. Otherwise, a filters running will continued till the end of the filters list. Sendmail is provided an ability to the filter to report the reject reason on the message receiving. It is possible to do this by the directives reject_message and tempfail_message.

The usage of the directive run_single_instance is allowed to run no more than N instances of the application, in which there is an internal limits for some resources, for example, it is used an exclusive file access mode. The examples of a such applications can be the antivirus scanners KAV, DrWeb, ClamAV and some other programs.

The support of the dynamic re-configuration in milter-agent is allowed the administrator "on fly" to add the external programs and internal modules for message processing. Namely this option is allowed the administrators to use any programs as filters not afraiding to stop a mail system on the errors or failures of these programs. Such reliabiliry is achieved that each filter is running as the separate process. In this case that fatal errors of these applications cannot stop the message processing. The timeout mechanism is provided the monitoring tool to watch the state of the process. When the defined time interval is expired (timeout expired), by default the timeout is equal one minute, the task is terminated forcely by the signal TERM or it is just ignored in the hope that the application will be terminated itself later. But in the last case it is possible the apparence of the large number of the non-terminated applications if the applicaton is never stopped.

A minimal filter consists of two parameters - the task name, the application and the command line for the application.

The main disavantage of the external programs usage is the small performance on the processing. To achieve a more processing performace it can used the internal modules (dynamically loaded libraries). The interaction interface is consisted of only one function process_message_by_module. The samples of such module can be found on the package media. Note the most important. A syntax for modules is almost the same as for the applications. Only in the line to run exec_cmd it is neccessary to write the parameters line without the module's name. The module name to execute is indicated by the directive module_name. As so these modules are worked in the address space of milter-agent, then any fatal errors will be terminated milter-agent. It is recommened the careful and strong testing of the modules code before their real work. The single option for the external errors control of the modules is given the directive module_safe_use. By this directive usage the module is running as the separate thread with the state monitoring using the timeout. If the module termination is enabled then milter-agent is disjointed this thread and is tried to terminate this thread using the thread cancellation mechanism.

For the external applications and internal modules the administrator can set the errors threshold, which will blocked the further calls of application or module when this threshold is exceeded. The standard errors are the following: wrong program or module location, wrong parameters on call, wrong rights on some catalog or file, absence of function in module by dynamic binding. If other packages are required the strong definite order of arguments on the task call, but in milter-agent the administrator can used any order of arguments on the call (i.e. the order is defined by the task's logic). Before the task's call milter-agent parses the call line and replaces a found formal variables as %f and others to the current values for this message, i.e., for example, %s is replaced by the address of the sender from the message's envelope. If it is neccessary to use these parameters without a replacing than to screen is used %, i.e. %%f will be %f on the real call. Such powerful and flexible mechanism had allowed to solve simply and effectively the almost all practical main tasks for the message processing. The reject or accept of the message by return codes, the execution conditions of filter for all or defined senders/recipients, writing the results of filter's work in temporary file (for example, this is used by the antivirus scanners to indicate what viruses found on message checking), writing of new body of message into file (transparent encryption/decryption, saving of "cured" message) and other properties are made the milter-agent the powerful and useful tool on the intermediate processing of the mail.

Let's see the practial example as the illustration of the work of milter-agent: replacing of dynamic library "on-the-fly". The administrator comments the desciption the selected filter in the configuration file. At the moment of a new message incoming milter-agent, detecting the change of the configuratuon, loads the configuration, effectively removing the commented filter from the filters list. So as this flter is used the dynamic library then this library is unloaded before the removing from the list. Now it is possible to load a new module. The administrator writes the neccessary changes for the filter description and uncomments the module running line. When a new message incomes the milter-agent loads the new configuration and adds this filter into the list. On the first call of this filter milter-agent is dynamically loaded the indicated library and it is executed the dynamic binding of the address of the processing function before the call of this function. A such non-complex scheme is allowed, not stopping work of mail server: remove the obsolest software, upgrade the current versions and expand the functionality, add the new filters (programs or modules).

By the default all options of milter-agent are disabled, therefore it is neccessary to enable them from the command line or to use the configuration file. At anytime the administrator with the helo of the utility ma_client can see the current settings of milter-agent, and also which filter are active and their statistics. A such selfdiagnostics is allowed the administrator to see the system's state and even to build with the returned back link. For example, if some filter was exceeded the threshold error and was disabled, then it is possible to get a notice message about it. If some filter is not used for a given time period, then it is possible aslo to get a notice message and make something. The ouput of the current state with a help ma_client is the natual ability for the system's architecture where the administrator is promised the clearity and transparency.

One of interesting features of milter-agent is the following - it is to get the text answers from the messages filters and to return these answers to sendmail as the rejection reason. Namely there were implemented the wrapper-scripts working with the antivirus scanners. The messages about the found viruses are transferred back into sendmail with the suggesting return code. Sendmail writes the return code and the reason into the log maillog (and,of course, it reports them to other side). Later when the module sma_stat is processed the statistics, then it gives the detail statistics: how much, where, from and to. The results are reliable and clear. On the virus epidemias these results are very good. Of course, a such usage of a pipeline idea can be used not only for antiviruses, but for the other useful programs.

Regular expressions in milter-agent are implemented on the basis of the standard POSIX 1003.2 using the standard system library (see re_format(7) and regex(3)). But to simplify the lookup always firstly it used the comparison to exact simple match, and after this it is used the search (comparison) to match with the pattern of the regular expression.

Example 1. Simple exact match
run_message_recipient = "test@home.jp"

Example 2. List for simple exact match
run_message_recipient = "test@home.jp test@home.us"

Example 3. Match by regular expression pattern
run_message_recipient = "test@home\.(ru|jp|us)"

Sample configuration file (it contains on package media): milter-agent.conf.sample

Examples of addresses filters.

Example 1. Add watcher to pair
[Address]
message_sender = "user1@domain1.com"
message_recipient = "user2@domain2.com"
add_recipient = "user3"

Example 2. Redirection
[Address]
message_recipient = "^user1@"
message_forward_recipient = "user2 user3"
message_forward_recipient = "user4@domain4.net"

Example 3. Duplication of mail traffic
[Address]
message_recipient = "user56@domain99.com"
add_recipient = "watching_user"

Example 4. Forwarding of mail from one account to another account with service mark
[Address]
message_recipient = "user1@domain1.com"
message_forward_recipient = "user2@domain2.org"
add_header_name = "X-Forward-Mail-From"
add_header_value = "user1@domain1.com"

Examples of messages filters.

Example 1. Mail archiving
[Filter]
task_name = mail_archive_task
exec_cmd = "/usr/local/sbin/mail_archive.sh %f /usr/local/tmp/mail_archive"

Example 2. Accept only text messages
[Filter]
task_name = test_html_mail_task
task_timeout = 0
reject_exit_codes = "1"
reject_message = "HTML mail or attachments not accepted."
run_message_sender = "user1"
run_message_recipient = "user2"
exec_cmd = "/usr/local/sbin/test_html_mail.sh %f"

Example 3. Cure "infected" mail
[Filter]
task_name = virus_cure_kav
run_single_instance = 1
exec_cmd = "/usr/local/sbin/kav50_cure_uue.sh %f %n"

Example 4. Check mail by antivirus ClamAV
[Filter]
task_name = virus_check_clamav
run_single_instance = 1
exec_cmd = "/usr/local/sbin/clamavscanner.sh %f /usr/local/share/clamav /usr/local/tmp/clamav_archive %r %h"

Example 5. Check mail by various antiviruses
[Filter]
task_name = virus_check
#reject_exit_codes = "1"
run_single_instance = 1
#skip_recipient = "root"
#skip_message_subject = "ALERT! Virus Scanner Problem!"
exec_cmd = "/usr/local/sbin/virus_scanner.sh /usr/local/tmp/avir_archive %f %r %n %h"

Example 6. Testing module
[Filter]
task_name = module_sample_task
module_name = "/usr/local/libexec/module_sample.so"
exec_cmd = "%f"

Example 7. Module to check messages for RHTA mark presence
[Filter]
task_name = test_rhta_mail_module_task
module_name = "/usr/local/libexec/module_test_rhta_mail.so"
exec_cmd = "%f"

The program ma_client is interacted with the loaded module milter_agent through the local socket (Unix domain socket). For a short help just call ma_client without parameters.

Usage example 1. Get current global settings of milter-agent.
# ma_client /var/spool/milter-agent/cmd 1

Usage example 2. Get current configuration of all messages filters of milter-agent.
# ma_client /var/spool/milter-agent/cmd 2

The module milter-agent has in the package delivery ship all neccessary to build the dynamic libraries with a new functionality.
Below is followed the short description of the purpose of supplied files:

makefile.modules - make file for all modules
module.h - interface definition of module
module_dummy.c - filter-module (empty)
module_sample.c - filter-module (sample)
module_test_rhta_mail.c - filter-module for detect of reject by RHTA
module_test_html_mail.c - filter-module for detect mail in HTML format or mail with attachents
module_mail_archive.c - filter-module for archive of all incoming mail

Download archive with modules samples for milter-agent.

Last modified: $Date: 2008-01-04 19:17:10+03 $

Mail package TPmail for Unix systems

Language

Home

Documentation

Support

News

Resources

Contacts

Module milter_agent. Description and usage.

ïÇÌÁ×ÌÅÎÉÅ