">
Qmail-Scanner - An Content Scanner for Qmail
Last Updated:
Copyright 2000/2004 Jason Haar. This software is distributed
under the terms of the GNU General Public License. See COPYING for additional information.
Description
Qmail-Scanner is an add-on that enables a Qmail e-mail server to scan all gateway-ed
e-mail for certain characteristics (i.e. a content scanner). It is typically used for its
anti-virus protection functions, in which case it is used in
conjunction with external virus scanners. but also enables a site (at a server/site level) to
react to e-mail that contains specific strings
in particular headers, or particular attachment filenames or types
(e.g. *.VBS attachments). It also can be used as an archiving tool for
auditing or backup purposes. Qmail-Scanner is integrated into the
mail server at a lower level than some other Unix-based virus
scanners, resulting in better performance. It is capable of scanning
not only locally sent/received e-mail, but also e-mail that crosses the
server in a relay capacity.
Features
-
Uses almost any external Unix command-line virus scanner.
-
Can call more than one virus scanner for each mail message
-
Has its own internal scanner that can be used to
pick up virii for which scanner updates are not yet available
-
The internal scanner can also be used to block e-mail based on attachment types,
or e-mail with certain e-mail headers... Need to stop *.mp3 files or "Subject:
ILOVEYOU" e-mail getting onto and off your LAN - can do! :-)
- Internal engine scans for poorly formatted messages that are known to be used by trojans/virii to infect clients. As such, this is independent of any virus scanner, and can successfully operate against future virii/trojans. Such messages are quarantined immediately. Known to block such major virii as Klez and Aliz, and as a side effect, stops a fair amount of spam too! Format checks include:
- broken MIME continuation headers
- use of comments within standard headers (e.g. "Content-T(xxxxxx)ype:" is *identical* to "Content-Type:" according to the RFCs - but some virii use this as it circumvents some anti-virus scanners). Valid use of this is never seen in the wild - so it's blocked
- repeated occurrences of MIME headers makes Q-S rename the latter ones to nullify them
- MIME boundaries over 250 chars are blocked
- differing definitions of a particular attachment filename causes it to be blocked
- double-defining the same MIME boundary is blocked
- certain MIME types containing windows executable extensions are specifically blocked (e.g. an "audio/wav" of filename "xxx.exe" could only be a virus)
- broken headers within a MIME attachment are blocked
- windows executable attachments that aren't marked as being of MIME type "application/....." are blocked.
- attachment filenames over 256 chars are blocked
- somedouble-barrelled filenames are blocked (e.g. file.gif.exe)
- CLSID file extensions are blocked
- Password-protected zip files can be blocked if you wish ("--block-password-protected yes"). This would stop any future viruses stuffed inside password-protected zip files from getting through, but of course would also stop any legitimate usage. Turned off by default, but perhaps useful to turn on during a new outbreak, and turned off again once an AV update occurs that can catch it.
- defaults to always running any AV you may have over messages first, then runs the internal scanner (perlscan) checks. This means if you block ".PIF" files due to them normally containing viruses, then any .PIF files that do contain a virus known to your AV system will be flagged as such, and any that were missed (perhaps they were a Day-Zero virus) are then tagged as being blocked. This differentiation is then used by the alerting system. It defaults to not notifying the sender that a virus has been found, but will still notify them of attachments been blocked (see below for more detail).
- Can integrate with SpamAssassin to provide comprehensive anti-spam tagging for an entire site - no more running spamc from procmail/whatever!
-
Auto-detects e-mail from "postmaster"-style and mailing-list
addresses - and doesn't send virus reports to them (i.e. attempts to act
more like a responsible net citizen)
- Knows of the virii which forge the From headers - so that the virus appears to come from some poor innocent. Qmail-Scanner will not send alerts to the sender for those types of virii.
- Due to the fact that over 99% of all e-mail-borne viruses are now sent using forged sender information, Q-S now defaults to NOT alerting the sender that a message has been quarantined, unless it was due to a Policy/Perlscan block. This can be turned back to the "old" style by using "--notify sender" instead of the newer default of "--notify psender"
-
Each message is tagged via a new Received: header
with a virus report showing whether it is clean or not and virus scanner
version numbers/etc
-
Messages with virii are moved into a "maildir" mail folder
for later perusal by the appropriate staff
-
Can optionally add a descriptive header: X-Qmail-Scanner
to every e-mail that passes through the system to allow users to see that
a scanner has run over their messages
-
Messages caught by Qmail-Scanner generate an e-mail message (currently
supports English, Italian, Afrikaans, Polish, Swedish, Czech, German, Spanish, Turkish,
Lithuanian, French, Portuguese, Dutch and Chinese messages) to a configurable combination of the sender, recipients and a "quarantine-admin" address explaining why their message was rejected
-
Can archive all processed e-mail into an archive maildir.
Useful when debugging e-mail-based apps, for backup purposes and for audit
policy reasons. Currently the mail envelope headers (the "rcpt to:" and "mail from:" headers) are appended to the bottom of each message.
- Can report via syslog or to a file, a one-line description of each processed message, giving information such as subject line, attachment filenames, sizes, etc.
-
Redundant scanning. Not only does it unpack each message
before running the scanners over it, it can also scan the original e-mail
message as well as the unpacked messages (if you think a particular scanner
can do a better job than Qmail-scanner's internal systems allow.)
- Reporting: in the contrib directory there's qs2mrtg.pl. A perl script for monitoring your syslog files for qmail-scanner records. It then graphs how Qmail-Scanner is processing your e-mails. It creates different graphs for incoming vs outgoing e-mail, as well as the flow of spam and viruses.
Download
The latest release is 1.25 (via http),
and is kindly housed by SourceForge. GnuPG signature of qmail-scanner-1.25.tgz.asc is also available. Of course, you'll be needing my GPG Public Key to verify that.
Requirements
-
Qmail 1.03 (there's a patched src RPM for Linux users available that contains the QMAILQUEUE patch amongst other things - just "rpm --rebuild" as root to build your own i386.rpm. NOTE: I cannot vouch for it - I do not use it. Please ensure you know how it works before installing Qmail-Scanner.)
- Create a separate account under which to run Qmail-Scanner: defaults to username and groupname "qscand". For extra security, create it with a normal home directory (e.g. "/home/qscand"), but with a "fake" shell (e.g. "/bin/false") - as it's never logged into directly.
-
reformime from Maildrop
1.3.8+
-
Perl 5.005_03+
-
Perl module Time::HiRes
-
Perl module DB_File (most distributions come with it pre-installed, although the latest Perl doesn't)
- Perl module Sys::Syslog (most distributions come with it pre-installed)
- Barely Optional: Mark Simpson's TNEF unpacker. Can decode those annoying MS-TNEF MIME attachments that Microsoft mail servers just love to use. If you don't have this, there are several classes of e-mail that you basically won't be able to detect virii in.
Patches
Bruce Guenter's QMAILQUEUE
patch is required to enable Qmail to call a different qmail-queue program
than the one compiled in by default. Qmail-scanner's
qmail-scanner-queue.pl
perl script is used instead of Qmail's
qmail-queue binary. After
qmail-scanner-queue.pl
has run, it calls the original qmail-queue binary to resubmit the
message back into the system.
-
Note 1: Bruce Guenter also provides a patched qmail-1.03 RPM for Linux systems that contains the above patch plus a bunch of other bits and pieces. Linux users may find it easier to use that. As mentioned above, please ensure you have Qmail working before installing Qmail-Scanner. On the Qmail-Scanner mailing-lists, we are seeing too many cases of users having "problems" - which end up being Qmail configuration/understanding issues...
Supported Virus Scanners
The following virus scanners are known to work with qmail-scanner.
Other Unix-based scanners should be simple to add support for.
CHANGES
There is a separate page listing changes that have been made between releases
TODO
There is a separate TODO page.
FAQ
There is a separate FAQ page.
Performance/Resource Usage
Adding content/virus scanning to an e-mail server will
considerably add to the resource usage of that server. As this
"wrapper" is written in perl instead of low-level C, quite a lot of
memory and file opens/stats occurs just to get it going. Adding to
this the actual scanners memory and CPU usage and it becomes quite
complicated (certainly the debugging info shows that the scanner
harness spends more time running the external scanners than it does
doing things itself [that is to be expected as they do quite a lot of
thinking...]). As a "rule of thumb" I'd suggest you look at how
many simultaneous SMTP sessions you are willing your box to have going
at any one point in time. Each SMTP session can invoke up to 'n'
different virus scanners (although they run one after the other - not
simultaneously) and I'd estimate that leads to around 5-6Mb of memory
usage per SMTP session. Thus if your dedicated SMTP host has 256Mb
RAM + 256Mb swap - that should mean you can handle - well heaps ;-)
The scanners cause the CPU to be thrashed while they're running, so
I'm making sure for our site that our Qmail server will only accept up
to 30 incoming SMTP sessions at any one time - that way I know the box
will handle it. As this leads to an increased memory usage, don't
forget Qmail's memory limits will need to be increased to deal with it (set
via ulimit or softlimit calls with Qmail system startup scripts).
One thing you should test for is what happens if connectivity
between this server and another local SMTP server is down for any
length of time (due to failure/power outage). When the link is
restored, can your server handle the other trying to dump 1,000's of
e-mail msgs onto it at once? You need to use softlimit and tcpserver's
limit options to ensure your box doesn't get killed. Note that this
resource issue isn't caused by Qmail-Scanner.
The same thing will happen with a pure, untouched Qmail (or any other)
system - it will just happen sooner...
After that scare-mongering I should say that I have tested
Qmail-Scanner under ridiculously low resource conditions - and it reacts as
it should - so at worst your system should start deferring e-mail. Thankfully
DJB's layering of programs is such that this is easy to accomplish :-)
Installation
- IMPORTANT: Ensure all anti-virus scanners and/or SpamAssassin are installed and operational before attempting to install Qmail-Scanner. Ensure these products are usable by non-root accounts (some people have had problems with permissions on some AV scanners in the past)
-
Unpack Qmail-Scanner and run ./configure --help.
This will show you what
options
are available to you.
-
Run ./configure ... [with your options], it will autodetect
what software is installed on your system, and will generate a script specific
to your system. If you don't see any errors reported, then the build is (probably) successful.
-
Run ./configure again, this time include "--install" along with the options you chose, this will do the same as the previous line, but will also create the directory structure
required, and install qmail-scanner-queue.pl
-
If you want to manually install it, see the Manual
Installation page.
-
Before going any further, you can test the installation by running ./contrib/test_installation.sh. This will send four e-mails: one normal, two "infected" with the EICAR test virus, and one obvious SPAM - to "root". Obviously Qmail-Scanner should let one through,catch the viruses, and tag the SPAM as "spammy" (if SpamAssassin is installed of course!). As Qmail-Scanner now defaults to not notifying anyone when a virus is caught, you may have to depend on the logs (e.g. syslog if you used "--log-details=syslog") to see what Qmail-Scanner did.
At this stage qmail-smtpd will need to be "told" that Qmail knows to use qmail-scanner-queue.pl
instead of qmail-queue. This is done via the tcpserver control files for smtp. Look to see where tcpserver for qmail-smtpd gets its rules from - it's the file after the "-x" option (well, that's the CDB version actually - find the text file yourself! ;-). Edit that file and tell qmail-smtpd which IP address ranges (corresponds to SMTP client IP addresses) you want Qmail-Scanner to be invoked on - typically all of them.
#/etc/tcpserver/smtp.rules
#
# No Qmail-Scanner at all for mail from 127.0.0.1
127.:allow,RELAYCLIENT="",RBLSMTPD="",QMAILQUEUE="/var/qmail/bin/qmail-queue"
# Use Qmail-Scanner without SpamAssassin on any mail from the local network
# [it triggers SpamAssassin via the presence of the RELAYCLIENT var]
10.:allow,RELAYCLIENT="",RBLSMTPD="",QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"
#
# Use Qmail-Scanner with SpamAssassin on any mail from the rest of the world
:allow,QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"
The above example means from now on all SMTP mail will be scanned, but
with different characteristics. Mail from the LAN (10. network) will
be scanned by the supported virus scanners, whereas mail from the
Internet will be scanned for virii AND tagged by SpamAssassin. This
finer control allows you a lot of versatility, e.g. virus scanning
only performed on mail coming from your Exchange server, and not from your Unix servers.
You must increase the amount of memory your system allows qmail-smtpd
to run with, as it it now running the entire perl interpreter PLUS
virus scanners. Typical installs of Qmail have system rc/startup
scripts (e.g. /etc/rc.d/init.d/qmail or /service/smtp/run) that limit the amount of RAM qmail-smtpd can use via ulimit or
softlimit. You must increase that to around 5-11Mb (totally dependent on your OS and choice of anti-virus scanner). If you don't
qmail-smtpd will crash with a "qq" error on the receipt of the very
first message... The actual amount is dependent on the OS in question as well as the virus scanners being used, so be prepared to experiment a little. Whatever you do, don't just set it to something stupid like 100M "just to be sure". The whole point about limiting RAM usage is so that "unusual" mail messages (e.g. from spammers or hackers) can't cause your system to become unusable by making it run out of RAM. |
To scan all mail sent by local shell users, the QMAILQUEUE
will also need to be defined within /etc/profile or the like so
that when they send mail, it will be affected as well.
Although as they are obviously Unix users,
you may want to save your system the effort and explicitly NOT do that!
:-) Also, think twice before running Qmail-Scanner in front of any mailing-list servers. Do you really think it's a good idea to have 10,000 messages banging away at your anti-virus system at the same time? Either put your mailing-list servers beyond the reach of your Qmail-Scanner servers, or put the mailing-list on the Qmail-Scanner servers themselves - that way each message is only scanned once and the load issues disappear. |
If "$DEBUG=1" (the default) is set within qmail-scanner-queue.pl, then every transaction
will be logged to
/var/spool/qmailscan/qmail-queue.log - so you'll
see how it goes. Regardless of debugging, errors (and attachment info if
enabled) should also be recorded in the qmail logs (probably via syslog)
- just look for entries containing the string "X-Qmail-Scanner".
Any SMTP sessions that are dropped (due to network outages/etc)
may lead to files lying around in /var/spool/qmailscan . Running
/var/qmail/bin/qmail-scanner-queue.pl -z at least once daily will
ensure such files are deleted when they're
over 30 hours old - make a cronjob to do that. Also realize that /var/spool/qmailscan/qmail-queue.log will grow without bounds. At some stage turn debugging off ($DEBUG=0) and delete the logfile. Personally, I like the logfile, so I run a cronjob that just does "mv -f qmail-queue.log qmail-queue.log.1" at 3am every morning. That way logs don't grow without bound, but you still end up with the logs from the past two days. The file can be safely deleted at any time if it becomes a disk-hog, but unless "$DEBUG=0" is set, it'll just get re-created the next time a message comes through.
Qmail-Scanner contains an internal scanner which allows you to reject e-mail
based on attachment filenames and/or e-mail headers. Read the minimal document on it for details.
Philosophy behind Quarantining...
When Qmail-Scanner decided to quarantine a message, it moves
it into a local mail folder (maildir format) - by default
/var/spool/qmailscan/quarantine/.
This means the message can be read in its pure "adulterated" state (e.g. still containing virii) by maildir
clients like mutt - or via IMAP (if
maildir format supported - you'll have to work that out for yourself).
At worse you can just read it with an editor - it's just a MIME file...
If you want a good IMAP server that supports maildir natively
- try Courier-IMAP.
I made the decision to write it into maildir format for
performance and reliability reasons - and it expressly makes it difficult
for any Windows admin to click on it with their vulnerable Windows mailer
and read it :-) Qmail actually comes with a program called /var/qmail/bin/maildir2mbox
which can do just that... (you could run it from cron to automatically
suck all the new mail messages from /var/spool/qmailscan/quarantine/new/
into a mbox.)
Also note that Qmail-Scanner only quarantines. It
doesn't drop or "clean" messages. Cleaning a message is
complicated and is becoming less of an issue. Personally I feel
that if someone sends a virus-laden message through
Qmail-Scanner, then it should be blocked - not cleaned. Why
should Qmail-Scanner "fix" someone else's problem? People need to
take responsibility for their own problems. They get an alert
telling them they're infected, so they will need to disinfect
their system before attempting to send mail through your site
again. Virus scanners that "clean" just make these people think
they don't have to worry about their problem - someone else is
fixing it.
Qmail-Scanner also doesn't distinguish between
"blocking" attachments (via the perlscanner module) and
virii. i.e. if a zip file contains a MP3 file, and you are
blocking MP3s - then the message will be quarantined. Some
people think that Qmail-Scanner should only block files if they
are "direct" attachments - not contained within archive
attachments. Again, if you have a policy saying "no MP3" - then
it should apply no matter how it's sent...
Finally, I am finding
the majority of virii are now trojans - which means that there
is no actual human-generated message to allow through
anyway. There is nothing more annoying than being behind some
other vendors anti-virus scanner when 1,000 copies of the latest
virus comes through. Receiving 1,000 copies of a "cleaned"
trojan is only marginally better than receiving the trojan! (in
fact it's worse for Unix users like me! ;-) In fact, but the
10'th copy, all the users receiving mail are now paranoid to
open any mail as they don't feel secure about what's
going on. The better response is that the ORIGINATOR receive
1,000 alerts telling them they're infected - maybe they'll do
something about it.
Also this event is logged in /var/spool/qmailscan/quarantine.log
in a tab-delimited format (for post-processing). A good script is needed
to convert this file into some nice graphs for management :-). See QSS for an example of one way of generating stats.
If Qmail-Scanner was configured with the "--log-details" option, then a
one-line summary of every message processed is recorded either in mailstats.csv or via syslog. e.g:
Aug 14 16:22:41 srvname qmail-scanner[30802]: Clear:RC:1(1.2.3.4): 0.030769 11569 root@x.y jdoe@y.z More_Power! <20020814042234.27902.qmail@x.y> 1029298961.30804-0.srvname:10649
Aug 14 16:23:17 srvname qmail-scanner[30820]: Clear:RC:0(1.2.3.4): 0.033618 2021 root@x.y jdoe@y.z Cron__run-parts_/etc/cron.daily <20020814042243.28092.qmail@x.y> 1029298997.30822-0.srvname:895
The format is as follows:
- [standard syslog stuff]
- qmail-scanner[PID]
- message status: "Clear" or description of quarantine event
- SpamAssassin is recorded as "SA:1" when SA tags the message as Spam, and "SA:0" otherwise
- The "RC:" bit refers to whether or not the e-mail came from a RELAYCLIENT or not: i.e. "1" says the message was from a "local" SMTP client and "0" means it was from an Internet one. The IP address of the SMTP client is then shown in brackets (127.0.0.1 if the message was generated locally). There are extremely useful if (say) you want to trigger a page if a local SMTP client sends a virus...
- If you have "--log-crypto" enabled, "CR:XXX will appear in the log record of any message that uses PGP, S/MIME or contains a password-protected ZIP file (e.g. CR:PGP(encrypted)). Options are:
- PGP(signed) or PGP(encrypted) for digitally signed or encrypted PGP/MIME e-mails. There is also "old-signed"/etc to cover the "older" method of doing PGP
- SMIME(signed) or SMIME(encrypted) for digitally signed or encrypted S/MIME e-mails
- CR:ZIP(encrypted) for password-protected ZIP files
- time taken (sec) to process the message
- "raw" size of message
- sender (i.e. "mail from")
- recipient (i.e. "rcpt to")
- Subject: header
- Message-ID: header
- space-delimited listing of attachment filenames plus their individual sizes appended to the filename
Note: fields are space-delimited when syslog used (with spaces within fields replaced by underscores), and tab-delimited in mailstats.cvs format
Support
This software is released under the GPL as found in the COPYING
file enclosed.
This package is housed on SourceForge.
Any questions, suggestions, etc to the mailing-list set up to discuss this, subscribe via http://lists.sourceforge.net/mailman/listinfo/qmail-scanner-general ,
or subscribe to the announcements-only list via http://lists.sourceforge.net/mailman/listinfo/qmail-scanner-announce.
Last Updated: