Mail

From FyshyWyky
Revision as of 14:16, 11 August 2007 by Athan (talk | contribs) (→‎Correcting Spamassassin and/or Bogofilter Registration of Email: Rewordifying for the making sense.)
Jump to navigationJump to search
Please note that whilst every effort is made to ensure the accuracy of the information on these pages it is entirely possible that things like the specific configuration of parameters of services have changed since the page you're looking at was last edited.

Some doubt can possibly be alleviated by checking the 'last updated' date/timestamp at the bottom of the page, or looking at the history for it (linked at the top of the page).

If in any doubt at all about a setting that might be affecting your use of a service please send a query to The Fyshy Admin, with all pertinent information.

 

Introduction

bowl.fysh.org hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email The Fyshy Postmaster with details for us to configure things correctly.

We currently use exim4 as the MTA that handles all email. This is configured to allow relaying only for authorised users and also employs a number of Anti-Spam Countermeasures to help protect our users from the deluge that is the norm on the Internet today.

In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a MUA locally through a shell login if they prefer.

SMTP

mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.

Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. To do so you will first have to set up a password specifically for this.

Configure your Email Client with the following details:

Parameter Value
Mail/SMTP Server Name mail.fysh.org
Port 25 (NB: You may also use the SMTP Submission Port, 587, for this.)
Secure Connection Enable 'TLS' for preference, or 'SSL' if your client only supports that.
Authentication Your bowl.fysh.org username and the password you setup specifically for this, not your shell login password.


Reading Mail

POP3

mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. To reduce the risk of this being snooped please be sure to enable any 'SSL' or 'TLS' options in your client.

Parameter Value
Server Name mail.fysh.org
Port 110
User Name Your bowl.fysh.org username
Password Your shell login password, not the SMTP AUTH one.
Secure Connection Use TLS if available (or SSL if your client only offers that).
Use Secure Authentication OFF (the server doesn't support this)


IMAP

mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. To reduce the risk of this being snooped please be sure to enable any 'SSL' or 'TLS' options in your client.

Parameter Value
Server Name mail.fysh.org
Port 993
User Name Your bowl.fysh.org username
Password Your shell login password, not the SMTP AUTH one.
Secure Connection Use SSL
Use Secure Authentication OFF (the server doesn't support this)


Webmail

We offer access to email held on mail.fysh.org via HTTP using Squirrelmail. The address for this service is https://squirrelmail.fysh.org/, or if for some reason you're unable to use encrypted HTTP (HTTPS) use http://squirrelmail.fysh.org/ but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.

URL https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location
Username/Password Your bowl.fysh.org shell login details.


Reading Mail Locally

Many Fysh.Org users prefer to read their email by logging in to bowl.fysh.org and then using a Mail User Agent directly there. The main supported program for this is Mutt, although users should be able to get Emacs to read mail as well. Note that whilst Pine is installed support for it is minimal due to the lack of updates in Debian, which is in turn due to the license of Pine precluding the distribution of binaries.


Maillists

fysh.org is available for hosting multi-user maillists using MailMan as the software. There is a list of those hosted maillists that allow public viewing of details. Please contact The Fyshy Postmaster if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.


Email Spam

If you're really unenlightened as to what it is then the Wikipedia Email Spam page may prove useful.

In short it's unsolicited email being sent either to promote some product or in an attempt to distribute malware. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.


MTA Counter-Measures

mail.fysh.org employs some measures at the MTA level to reduce the amount of spam that ends up in our users' inboxes.

NB: None of these measures are employed if one of the following conditions is true:

  • The email is being submitted to us on port 587, the Mail Submission port which requires all senders to be authenticated.
  • The connection is otherwise authorised and thus trusted.
  • Email being sent by mail.fysh.org itself, i.e. locally generated email.
  • Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.


Mail From Dialup/DSL Hosts

We use a list of regexes which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do not put such regexes as:

.*\.cn$

in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.

Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.

NB: Users with a fysh.org shell account login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have authorised themselves first, which is necessary anyway to relay through mail.fysh.org.


Realtime Black Lists

The single measure we find reduces locally-delivered spam the most is to make use of the Spamhaus SBL-XBL RBL. This is basically a list of hosts known to send email spam. We also still use the relays.ordb.org RBL, but it very seldom catches anything that SBL-XBL doesn't. Whilst there is some risk of false positives in such things in our experience this either never happens with the SBL-XBL, or at so low a rate as to be unnoticeable.

If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to The Fyshy Postmaster with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.


Greylisting

The other primary anti-spam counter-measure that we employ is called Greylisting.

Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our Greylisting lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the Envelope To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the greylist listing, with the time and attempts so far recorded, and then a temporary failure is returned to the sending MTA. With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved. So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again note the attempt, the email stays on the greylist listing, bump the attempt count and update the time of last attempt. When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our greylist list and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the whitelist list. Alternatively if there is no attempt to retry delivery the tuple will be expired from the greylist list after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).

Now, if an email is received where the data matches a tuple already on the whitelist list then we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the whitelist list for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.


Problems With Greylisting

There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:

bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com
bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com

Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the whitelist list. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please forward us the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded whitelisted hosts list. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.

There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to inform us of the IP(s) of the sending host(s) so we can add them to our whitelisted hosts list for greylisting.

Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes). If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email The Fyshy Postmaster with any requests to have either your entire domain or just a set of full addresses skip greylisting.

User Counter-Measures

There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.

Procmail

Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.

Any email delivered to a user on bowl.fysh.org will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.

NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header. This is a header added during final delivery on bowl.fysh.org denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from Fysh-Announce.

There is an example usage of procmail available on bowl.fysh.org in ~athan/etc/procmailrc-example . Please note you WILL have to edit this to suit your own email needs, do NOT put it in place (~/.procmailrc) as-is !

Exim Filters

As we use exim (version 4) for the MTA on mail.fysh.org you can make use of Exim Filters in your .forward file.

Spamassassin

Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.

We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of Mail#Procmail procmail to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:

# Pipe the mail through spamassassin (replace 'spamassassin' with
# 'spamc' if you use the spamc/spamd combination)
#
# The condition line ensures that only messages smaller than 250 kB
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam
# isn't bigger than a few k and working with big messages can bring
# SpamAssassin to its knees.
#
# The lock file ensures that only 1 spamassassin invocation happens
# at 1 time, to keep the load down.
#
:0fw: spamassassin.lock
* < 256000
| spamassassin

# All mail tagged as spam (eg. with a score higher than the set
# threshold) is moved to "spam".
:0
* ^X-Spam-Status: Yes
spam

Please be sure to read the Spamassassin documentation (man spamassass and the files under bowl:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.

Bogofilter

Bogofilter is another means of automatically determining if an email is spam. It uses something called Bayesian filtering to attempt to determine if the text of an email is spam or not. Mail#Spamassassin actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin and Bogofilter over all their email, as each will catch some spam emails that the other doesn't.

Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.


## Run through bogofilter
# -p, --passthrough         - passthrough.
# -e, --ham-true            - in -p mode, exit with code 0 when the mail
#                             is not spam.
# -u, --update-as-scored    - score message as spam or non-spam and register
#                             accordingly.
:0fw: bogofilter.lock
| bogofilter -e -p -u

# if bogofilter failed, return the mail to the queue, the MTA will
# retry to deliver it later
## NB: This causes more headaches than it is worth, when combined with
## using 'formail -D' to de-dupe emails.  If we return it to the queue
## for later delivery and it hits the de-dupe it'll get dropped as a
## dupe.  The filter rule above DOES still pass the original through
## unchanged if bogofilter failed for some reason (at least if it
## doesn't recognise a command line option), so we're safe to continue.
## At worse we'll not filter some spam.  That's as opposed to plain
## losing email due to the bad interaction with de-duping! 
#:0e
#{
#       EXITCODE=75
#}

## If bogofilter thinks it's spam, save
:0:
* ^X-Bogosity: Yes, tests=bogofilter
spam

Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:

## Any rules to have email skip the spamassassin and bogofilter tests here:
## i.e. uncomment these and edit to taste
### ## Absolutely whitelisted email addresses
### :0 H
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)
### {
###         :0
###         * ^TO(abuse|postmaster)@fysh\.org
###         fysh
### 
###         :0 B
###         * !http://www\.bodybouncer\.com/
###         inbox
### }

## Run through bogofilter
# -p, --passthrough         - passthrough.
# -e, --ham-true            - in -p mode, exit with code 0 when the mail
#                             is not spam.
# -u, --update-as-scored    - score message as spam or non-spam and register
#                             accordingly.
:0fw: bogofilter.lock
| bogofilter -e -p -u

# if bogofilter failed, return the mail to the queue, the MTA will
# retry to deliver it later
## NB: This causes more headaches than it is worth, when combined with
## using 'formail -D' to de-dupe emails.  If we return it to the queue
## for later delivery and it hits the de-dupe it'll get dropped as a
## dupe.  The filter rule above DOES still pass the original through
## unchanged if bogofilter failed for some reason (at least if it
## doesn't recognise a commandline option), so we're safe to continue.
## At worse we'll not filter some spam.  That's as opposed to plain
## losing email due to the bad interaction with de-duping! 
#:0e
#{
#       EXITCODE=75
#}

# Pipe the mail through spamassassin (replace 'spamassassin' with
# 'spamc' if you use the spamc/spamd combination)
# 
# The condition line ensures that only messages smaller than 250 kB
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam
# isn't bigger than a few k and working with big messages can bring
# SpamAssassin to its knees.
# 
# The lock file ensures that only 1 spamassassin invocation happens
# at 1 time, to keep the load down.
#
:0fw: spamassassin.lock
* < 256000
| spamassassin

## If bogofilter thinks it's spam, save
:0:
* ^X-Bogosity: Yes, tests=bogofilter
{
        ## Make sure SpamAssassin's filtering learns it as spam
        :0c
        |sa-learn --spam

        ## Something we don't filter/recognise, we'll look at it
        :0
        spam
}

# All mail tagged as spam (eg. with a score higher than the set
# threshold) is moved to "spam".
:0
* ^X-Spam-Status: Yes
spam

This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.

To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (the -M option assumes the email folder is in mbox format):

bogofilter -M -n /var/mail/${USER}

or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:

bogofilter -M -s ~/Mail/spam

Correcting Spamassassin and/or Bogofilter Registration of Email

If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:

## spam
# Register as Spam, unregister as Ham (non-Spam)
#
macro index S "<enter-command>unset wait_key\n\
<tag-entry>\
<tag-prefix>\
<clear-flag>N\
<tag-prefix>\
<pipe-entry>/usr/bin/spamassassin -r\n\
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\
<tag-prefix>\
<save-message>=savedspam\n\
<untag-pattern>.*\n\
<enter-command>set wait_key\n"
macro pager S "<enter-command>unset wait_key\n\
<tag-entry>\
<tag-prefix>\
<clear-flag>N\
<tag-prefix>\
<pipe-entry>/usr/bin/spamassassin -r\n\
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ 
<tag-prefix>\
<save-message>=savedspam\n\
<enter-command>set wait_key\n"
# Register as Ham (non-spam), unregister as Spam
macro index H "<enter-command>unset wait_key\n\
<pipe-entry>/usr/bin/sa-learn --ham  --no-sync\n\
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\
<enter-command>set wait_key\n"
macro pager H "<enter-command>unset wait_key\n\
<pipe-entry>/usr/bin/sa-learn --ham  --no-sync\n\
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\
<enter-command>set wait_key\n"

This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.

Sending Spam from mail.fysh.org

It should go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we will discover such activity, your account will be closed forthwith pending investigation.

In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.

Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your web space that could possibly be insecure and allow for sending of email to arbitrary addresses.