https://www.fysh.org/wiki/api.php?action=feedcontributions&user=Athan&feedformat=atomFyshyWyky - User contributions [en]2024-03-29T08:55:26ZUser contributionsMediaWiki 1.39.5https://www.fysh.org/wiki/index.php?title=Mail&diff=2067Mail2023-12-07T19:47:35Z<p>Athan: /* Having us handle email for your domain */ Add SPF, DKIM and DMARC information</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
It should be noted that '''outbound email purporting to come from a fysh.org email address must be sent through mail.fysh.org or directly on river.fysh.org'''. Attempt to send such from any other email server risks it being rejected if the receiving server checks SPF and/or DKIM.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== SPF (Sender Policy Framework) records ===<br />
[https://datatracker.ietf.org/doc/html/rfc7208 Sender Policy Framework] helps prevent spammers from sending email pretending to be from you.<br />
<br />
So long as you're confident that all holders of email accounts within your domain have access to send through mail.fysh.org you can specify an SPF record that copies the one for fysh.org. Check the current one with, e.g.:<br />
host -t txt fysh.org | grep v=spf1<br />
We recommend initially ending your record with " ~all", not " -all" until you confirm it is working as intended.<br />
<br />
If we host the DNS for your domain we can take care of this at your request.<br />
<br />
=== DKIM (DomainKeys Identified Mail) ===<br />
[https://www.rfc-editor.org/rfc/rfc6376.html DomainKeys Identified Mail] allows email to be signed at the originating email server to attest that the email is from a genuine user of the domain. This entails setting up a signing key and a DNS record with the public half of the key pair for that so recipients can verify the signatures.<br />
<br />
If we control the DNS for your domain (as that makes it much easier for us to perform the necessary setup, we can enable outbound DKIM signing of email for your domain. As with SPF, you really want mail.fysh.org to be the only place that email from the domain originates, i.e. all users send email through there. If not you risk genuine email going out unsigned and thus looking like possible spam.<br />
<br />
=== Domain-based Message Authentication, Reporting, and Conformance (DMARC) ===<br />
[https://www.rfc-editor.org/rfc/rfc7489 Domain-based Message Authentication, Reporting, and Conformance] mostly allows you to define email address(es) to receive reports about email received elsewhere purporting to be from your domain, with the information pertaining to the passing or failure of both SPF and DKIM checks. It also allows for allowing or disallowing that your SPF and DKIM setups also apply to any subdomain of your domain.<br />
<br />
It does ''not'' allow you to alter the soft/hard fail nature of your SPF or DKIM setup.<br />
<br />
If we control your DNS then we can set up the necessary DNS record for this. You'll need to either tell us the email addresses you'd like the reports to go to, or tell us that you're happy for us to receive them (which can help us diagnose issues with SPF and/or DKIM on your domain).<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably ''do not'' want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== SPF checks on inbound email ====<br />
As of 2023-11-30 mail.fysh.org will reject any email if the SPF check returns a hardfail. This means the alleged sending domain (envelope from):<br />
<br />
1. Has declared an SPF record.<br />
2. That record does not list the sending IP as allowed.<br />
3. That record ends with "-all" to denote that any other IP should cause rejection of the email.<br />
<br />
==== DKIM Signature Verification ====<br />
<br />
As of 2022-08-07 we have implemented MTA level checking of DKIM signatures '''but do not reject any email based on this'''.<br />
<br />
You can check what the outcome of this by checking the ''top-most'' (in case someone nefarious tried to pre-add any of them) "X-DKIM" and "Authentication-Results" headers in any email delivered by mail.fysh.org.<br />
<br />
Examples:<br />
<br />
1. Valid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv<br />
<br />
2. INvalid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc reason="signature_incorrect"<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv reason="signature_incorrect"<br />
<br />
3. No signature on a domain we've seen valid ones on in the past (manually curated list).<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for required domain: twitch.tv)<br />
<br />
4. No signature on a domain we do ''not'' have in that list.<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for example.com)<br />
<br />
SpamAssassin can also do its own verification of DKIM signatures, and on river.fysh.org will do so by default. Further more the current webmail, roundcube, can display the result of this if you enable the "Auth. Result" column in the mailbox view.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2066Mail2023-12-07T19:33:34Z<p>Athan: /* Introduction */ No longer secondary MX. Outbound fysh.org email must go through mail.fysh.org</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
It should be noted that '''outbound email purporting to come from a fysh.org email address must be sent through mail.fysh.org or directly on river.fysh.org'''. Attempt to send such from any other email server risks it being rejected if the receiving server checks SPF and/or DKIM.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably ''do not'' want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== SPF checks on inbound email ====<br />
As of 2023-11-30 mail.fysh.org will reject any email if the SPF check returns a hardfail. This means the alleged sending domain (envelope from):<br />
<br />
1. Has declared an SPF record.<br />
2. That record does not list the sending IP as allowed.<br />
3. That record ends with "-all" to denote that any other IP should cause rejection of the email.<br />
<br />
==== DKIM Signature Verification ====<br />
<br />
As of 2022-08-07 we have implemented MTA level checking of DKIM signatures '''but do not reject any email based on this'''.<br />
<br />
You can check what the outcome of this by checking the ''top-most'' (in case someone nefarious tried to pre-add any of them) "X-DKIM" and "Authentication-Results" headers in any email delivered by mail.fysh.org.<br />
<br />
Examples:<br />
<br />
1. Valid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv<br />
<br />
2. INvalid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc reason="signature_incorrect"<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv reason="signature_incorrect"<br />
<br />
3. No signature on a domain we've seen valid ones on in the past (manually curated list).<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for required domain: twitch.tv)<br />
<br />
4. No signature on a domain we do ''not'' have in that list.<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for example.com)<br />
<br />
SpamAssassin can also do its own verification of DKIM signatures, and on river.fysh.org will do so by default. Further more the current webmail, roundcube, can display the result of this if you enable the "Auth. Result" column in the mailbox view.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2065Mail2023-12-07T19:30:30Z<p>Athan: /* Email Spam */ Inbound email now SPF checked, including hardfail causing rejection.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably ''do not'' want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== SPF checks on inbound email ====<br />
As of 2023-11-30 mail.fysh.org will reject any email if the SPF check returns a hardfail. This means the alleged sending domain (envelope from):<br />
<br />
1. Has declared an SPF record.<br />
2. That record does not list the sending IP as allowed.<br />
3. That record ends with "-all" to denote that any other IP should cause rejection of the email.<br />
<br />
==== DKIM Signature Verification ====<br />
<br />
As of 2022-08-07 we have implemented MTA level checking of DKIM signatures '''but do not reject any email based on this'''.<br />
<br />
You can check what the outcome of this by checking the ''top-most'' (in case someone nefarious tried to pre-add any of them) "X-DKIM" and "Authentication-Results" headers in any email delivered by mail.fysh.org.<br />
<br />
Examples:<br />
<br />
1. Valid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv<br />
<br />
2. INvalid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc reason="signature_incorrect"<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv reason="signature_incorrect"<br />
<br />
3. No signature on a domain we've seen valid ones on in the past (manually curated list).<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for required domain: twitch.tv)<br />
<br />
4. No signature on a domain we do ''not'' have in that list.<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for example.com)<br />
<br />
SpamAssassin can also do its own verification of DKIM signatures, and on river.fysh.org will do so by default. Further more the current webmail, roundcube, can display the result of this if you enable the "Auth. Result" column in the mailbox view.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Main_Page&diff=2064Main Page2023-07-25T11:48:57Z<p>Athan: /* Contacting Admins During Downtime */ Ath no longer does twitter</p>
<hr />
<div>== Introduction ==<br />
Welcome to the Fyshy Wiki. The initial intent of this Wiki is to document the various services available on the Fysh.Org [[Main_Page#Servers|servers]].<br />
<br />
'''NB: If you don't know what Fysh.Org is, and do not have an account, do not ask for one. This is in no way, shape, or form an open or subscribable service if you didn't already know what Fysh.Org is.'''<br />
<br />
== Servers ==<br />
The principal server for Fysh.Org services is [[river.fysh.org]]. Amongst other things it provides [[Mail|email]] and [[Shell Accounts|shell accounts]] to all its users. [[DNS|DNS / Domains]] service also resides on [[river.fysh.org]].<br />
<br />
[[Web|Web hosting]] is on the separate machine [[stickelback.fysh.org]].<br />
<br />
There are also a few other machines that various [[Fysh]] use to provide peripheral services, such as [[SSH#ssh.fysh.org|ssh.fysh.org]].<br />
<br />
Fysh.Org services moved again on 29th June 2013. The new shell machine is [[river.fysh.org]]. The old machines [[lake.fysh.org]] and [[guppy.fysh.org]] ''no longer exist''.<br />
<br />
Note that the old UK2-hosted servers, [[bowl.fysh.org]] and [[www.fysh.org]] ''no longer exist''. We took them down on 17th April 2008, a few days before UK2 were due to turn them off, so as to securely erase all user data on them before we lost access.<br />
<br />
The panix-hosted servers [[pond.fysh.org]] and [[fyn.fysh.org]] ceased to exist mid-March 2010 after our move to Redstation.<br />
<br />
== Services ==<br />
A list of all our service is available [[:Category:Services|here]].<br />
<br />
== Weekends are At-Risk time for Reboots ==<br />
<br />
All UK weekend hours should be considered 'at risk' for unannounced reboots. There may be as little as 30 or even 15 minutes warning of a quick reboot, usually for a new kernel.<br />
<br />
In general such a reboot would take less than 5 minutes, that being the extent of the disruption to service. There is usually no reason for more than one such reboot in any given weekend.<br />
<br />
If planned work is intended to take longer than this then it will be announced at least 3 days in advance, preferably a week. Such announcement would be on the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist, as outlined below. Obviously if there is some pressing security concern work then quick reboots, or extended work, may be carried out at any time without warning.<br />
<br />
You may want to take note of details in the section [[Main_Page#Contacting Admins During Downtime|Contacting Admins During Downtime]] below in case of extended outage.<br />
<br />
== Notification of Service Changes ==<br />
If you wish to know about any changes to Fysh.Org services, which may include scheduled outages, you should be subscribed to the [http://www.fysh.org/mailman3/postorius/lists/fysh-announce.fysh.org/ Fysh-Announce] maillist. We will automatically subscribe any new accounts to this list. Subscriptions require confirmation from the given email address as well as our approval. Anyone may subscribe, but if we don't recognise the email address as being that of one of our users we may check who it is first, or just deny it, so it's best to also email the [mailto:postmaster@fysh.org Fyshy Postmaster] so we know who you are.<br />
<br />
'''NB: The gating of Fysh-Announce emails to the local fysh.announce newsgroup which is then used for the login announcements is disabled as of 2020-08-12. Any old announcements not yet seen will still be shown as outlined below, but for the time being no new announcements will be shown via this mechanism.'''<br />
<br />
By default when you login to a [[river.fysh.org]] [[Shell Accounts|shell account]] you will be shown any new announcements since your last login. If you wish to disable this simply issue this command at a shell prompt on [[river.fysh.org]]:<br />
<br />
touch $HOME/.no_announce<br />
<br />
and to enable it once more if needed:<br />
<br />
rm $HOME/.no_announce<br />
<br />
Note that the file $HOME/.last_announce is used to keep track of when you were last shown new announces. Removing this will cause you to be shown ''every'' existing announcement on next login! Which announces to show you are determined by comparing the 'last update' timestamp on this file against the timestamp on the files holding the announces. So if you change that timestamp on $HOME/.last_announce you may see more, or less, announces than you should.<br />
<br />
== Discussion of Fysh.Org Services ==<br />
If you wish to generally discuss Fysh.Org services you may use the [http://www.fysh.org/mailman/listinfo/Fysh-Discussion Fysh-Discussion] maillist, but note we do not automatically subscribe anyone to this which means you may have to subscribe yourself first and may find that not many people are reading the list itself.<br />
<br />
Alternatively any queries about the Services may be directed to the [mailto:root@fysh.org Fyshy Admin] or you may edit [[Fyshy Suggestions]] here on this wiki.<br />
<br />
== Contacting Admins During Downtime ==<br />
If Fysh.Org services are down and you need to get in touch with an admin then try the following email addresses. Note these aren't guaranteed to be up to date, or even checked routinely. But it's better than only having Fysh.Org-reliant contact details for us:<br />
<br />
{| border="1"<br />
|+ Alternative Fysh.Org Admin contact details<br />
|-<br />
! Admin<br />
! Contact Via<br />
|-<br />
| Athan<br />
| Email: Suisanahta (at) gmail (dot) com<br />
<br />
Mastodon: https://techhub.social/@AthanSpod<br />
|}</div>Athanhttps://www.fysh.org/wiki/index.php?title=River.fysh.org&diff=2063River.fysh.org2023-03-11T16:55:50Z<p>Athan: /* The Fysh Server Called River */ IPv6 available on river.fysh.org again, addition of river46.fysh.org</p>
<hr />
<div>'''NB''': ''All the information provided here-in about the fysh.org services is made available on a best-effort basis. If you're in in doubt as to if the information provided here is up to date please check the 'last modified' entry at the bottom of, or the History link at the top of, the page.''<br />
<br />
= The Fysh Server Called River =<br />
<br />
[[river.fysh.org]] is currently a KVM-hosted virtual machine running on [[stream.fysh.org]], a server rented from and hosted by [http://www.ovh.co.uk/ OVH]. This is paid for via subscriptions charged to the principal users of the machine. Any queries about payments, or obtaining an account, should be directed to [[Mike Ashton]].<br />
<br />
The machine generally runs the current Stable version of the [http://www.debian.org/ Debian] distribution of [http://www.linux.org/ Linux]. The majority of the software installed on it is that which is directly supported by Debian, as this makes day-to-day administration and keeping things up to date very easy. We will, however, install software by hand if sufficient need is demonstrated. Our aim is to offer a ''stable'' service first and foremost, so don't necessarily expect cutting-edge versions of software.<br />
<br />
''Note that this is a 64-bit installation, with some 32-bit compatibility libraries''. If you're trying to get something to work from source yourself you might encounter problems if it is not 100% 64-bit clean. You can try compiling with the '-m32' flag, but it would be best to get the source fixed.<br />
<br />
Between [[river.fysh.org]] and [[stickelback.fysh.org]] we provide a wide range of services, ranging from email and web pages, through hosting of domains to the more esoteric such as [http://www.goteamspeak.com/ TeamSpeak] servers.<br />
<br />
== IPv4 vs IPv6 Connectivity==<br />
As of 2023-03-11 we once more have an IPv6 AAAA DNS record for [[river.fysh.org]]. In case we find reason to remove that in the future, there is also now river46.fysh.org which has both A and AAAA records, thus allowing user to decide which connectivity options they want to allow through using the appropriate hostname.<br />
<br />
{| class="wikitable" style="text-align: center;"<br />
|+ river.fysh.org hostnames and choosing IPv4 vs IPv6 resolution<br />
|-<br />
!Hostname !!IPv4 / A Record !!IPv6 / AAAA record<br />
|-<br />
| river.fysh.org || Y || Y<br />
|-<br />
| river6.fysh.org || N || Y<br />
|-<br />
| river46.fysh.org || Y || Y<br />
|}<br />
<br />
==Problems, Queries, Requests==<br />
If you have a problem with, or a query about, any of the services on fysh.org, or would like to request an additional service, please send email to [mailto:root@fysh.org The Fyshy Admin].<br />
<br />
==Current Supported Services==<br />
<br />
* [[Mail|Mail]]<br />
* [[Shell Accounts|Shell Accounts]]<br />
* [[DNS|DNS / Domains]]<br />
* [[News|Usenet / NetNews]]<br />
* [[FTP|FTP]]<br />
* [[Databases|Databases]]<br />
<br />
Note that the following services are now on separate machines:<br />
<br />
* [[Web|Web Services]] on [[stickelback.fysh.org]]<br />
<br />
= History =<br />
Initially fysh.org was intended to be primarily for the benefit of certain graduates from [http://www.warwick.ac.uk The University of Warwick]'s [http://www.dcs.warwick.ac.uk/ Department of Computer Science] who were just some of the self-proclaimed [[Fysh]]. Indeed it was initially hosted, as the server crucigera.fysh.org, in the office of a friendly member of Warwick DCS staff. Since then it's been hosted for free on various DSL lines (even before those were offered publically in the UK) and at the premises of a few companies with which one of the Fyshy Admin had associations at the time before finally settling down as a paid-for dedicated server rented from [http://www.uk2.net/ UK2.net].<br />
<br />
[[bowl.fysh.org]] took over from [[crucigera.fysh.org]] in June 2001 after repeated hardware problems with the latter. The main server kept that name through another hardware change or two, including the move to the rented machine from UK2.net (where the hard disks have had to be moved to fresh hardware at least once).<br />
<br />
In November 2007 we suffered a successful hack attack against [[bowl.fysh.org]], and as a result started running services in VMware instances. We renamed the machine itself to [[tank.fysh.org]] and brought up an initial four VMware instances: [[bowl.fysh.org]], [[www.fysh.org]], [[dns.fysh.org]] and [[surfers.fysh.org]]. This rapidly proved to be too much for the hardware to cope with. We put Surfers back on [[bowl.fysh.org]] along with DNS (now chrooted) so as to run only two VMware instances. However this was still too much for the machine to cope with and things were further exacerbated when [[tank.fysh.org]] developed new hardware problems. This gave us the final push to move to [http://www.panix.com/ PANIX].<br />
<br />
So, starting from 13th March 2008 we been used two Virtual Co-locations with [http://www.panix.com/ PANIX]. The first of these was [[pond.fysh.org]] which runs the majority of services. The second was [[fyn.fysh.org]] which handles [[Web|Web Services]].<br />
<br />
By early 2010 we were encountering some problems with running newer kernels on the PANIX V-Colos so decided to change back to a dedicated machine on which we'd run virtualisation ourselves. This lead to us renting a new machine, [[reef.fysh.org]] from [http://www.redstation.co.uk/ Redstation]. We then use [http://www.linux-kvm.org/ KVM] to run two virtual machines, [[lake.fysh.org]] which replaced [[pond.fysh.org]], and [[guppy.fysh.org]] which replaced [[fyn.fysh.org]]. We migrated to these new machines during the evening of 8th March 2010.<br />
<br />
In June 2013 the Fyshy Admins decided that the Redstation server was no longer value for money and decided to move to OVH. This lead to us renting [[stream.fysh.org]]. We continued to use [http://www.linux-kvm.org/ KVM] to run virtual machines for user-facing services, this time via [http://libvirt.org/index.html libvirt]. The two VMs are [[river.fysh.org]], which replaced [[lake.fysh.org]], and [[stickelback.fysh.org]], which replaced [[guppy.fysh.org]]. This move was completed on 29th June 2013.<br />
<br />
In July 2015 it was time to move to better hardware (and away from failing hard disks). So some initial work was done to move all services onto an OVH 'FailOver' IP which can be moved between servers within OVH. Once that was completed a new machine was rented and work carried out in early August 2015 to move to it. This culminated in services all being moved to the new machine on 9th August 2015. As this mostly entailed just copying the exact same VMs over to the new machine, and then switching the FailOver IP to point at it instead, no new machine names resulted from this move.</div>Athanhttps://www.fysh.org/wiki/index.php?title=Web&diff=2062Web2023-02-12T13:49:34Z<p>Athan: /* HTTPS and SSL Certificates */ Cite our listing of TLS certificate details</p>
<hr />
<div>= Introduction =<br />
[[stickelback.fysh.org]] runs web servers under the primary hostname of [[www.fysh.org]]. We currently utilise [http://httpd.apache.org/ Apache] 2.4.x (as supplied in Debian 'stable') for this purpose, with a single instance running on both port 80 for http access and port 443 for https access.<br />
<br />
We may also run additional instances of apache, or other web server software, on other ports as needed.<br />
<br />
= Webmaster =<br />
All enquiries about [[stickelback.fysh.org]] and other web services should be directed to the [mailto:webmaster@fysh.org Fyshy Webmaster].<br />
<br />
= User Pages =<br />
Any user with an account on [[river.fysh.org]] can create a page on the [http://en.wikipedia.org/wiki/World_wide_web WWW] simply by placing the requisite files in the directory ''/var/www/user/username/public_html/'' (where 'username' is theirs). This will yield a URL of http://www.fysh.org/~username/ for example. If you find that this directory does not yet exist, or file permissions deny you access, please contact [mailto:webmaster@fysh.org The Fyshy Webmaster] to correct this.<br />
<br />
Please note that /var/www/user on [[river.fysh.org]] is NFS mounted on [[www.fysh.org]] '''read-only'''. Web scripts and CGIs ''cannot'' write to any files. Thus any web page that needs to store data will have to utilise a [[Databases|Database]]. In some circumstances we will set up a separate area for a user to place web pages that absolutely ''require'' file write access, but we would much rather they use a proper [[Databases|Database]] instead. '''Furthermore''' files located in /var/www/user on [[river.fysh.org]] will appear under /home on [[www.fysh.org]]. This is to satisfy the requirements of SuExec when CGIs are run from a /~username/ URL. Be sure to write any scripts/CGIs to take account of this different path! It is noted that at this time you can in fact use the same /var/www/user/... path on [[stickelback.fysh.org]] as on [[river.fysh.org]], as a side effect of river:/var/www being mounted onto stickelback:/var/www, but this should not be relied upon.<br />
<br />
= User Domains =<br />
Additionally we will host the web pages for any domain that a user owns or controls and can get the DNS changed to point to our web server. Please note that our preference is that you use a CNAME record pointing to "www.fysh.org." (the trailing dot is important, don't miss it out!), as this allows us to move your domain webhosting about if we need to. The one practical exception to this is where you use solely the domain name for your URL, i.e. if your domain was 'example.com' and you wanted to use http://example.com/ instead of, or as well as, http://www.example.com/. You can't have a CNAME record and any other records for the same name, and a domain itself requires at least an SOA record. Thus in this configuration you would have to use an A record pointing to the same IP as www.fysh.org.<br />
<br />
Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] to enquire about setting this up. Note that you should ensure that you have a valid and working webmaster email account under your own domain when this is set up.<br />
<br />
The same caveat as for User Pages about '''read-only''' file access will apply.<br />
<br />
If you wish URLs of the form http://www.example.com/~username/ (but for your domain) to be served from /var/www/user/username/public_html/ (as is set up for www.fysh.org URLs) please let us know and we'll adjust the configuration for your domain. By default we assume you don't want just any Fysh.Org user to be able to utilise such a URL. We can configure things such that only certain users have URLs of this form working for your domain, so let us know which users to allow. As always it's the [mailto:webmaster@fysh.org Fyshy Webmaster] you need to contact for this.<br />
<br />
=== HTTPS and SSL Certificates ===<br />
<br />
Since February 2018 we have offered use of [https://letsencrypt.org/ LetsEncrypt] SSL Certificates on the fysh.org web service. Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] if you wish us to set this up for any of your domains we host. You won't need to do anything else as the method we use to obtain LetsEncrypt certificates relies solely on control of the website as proof of legitimacy for the domain, which of course we have.<br />
<br />
Do note that a single IP will still be used, so this relies on clients making use of [http://en.wikipedia.org/wiki/Server_Name_Indication Server Name Indication] to indicate which domain they want to access (so that the correct certificate is used), which has some limitations (mostly to do with rather old browsers and operating systems), but this is becoming less and less of an issue.<br />
<br />
If you ever need to double-check that a presented certificate is the one we have configured then consult [https://www.fysh.org/ssl/] where we maintain basic details of all the certificates we use. In particular you can use the 'Serial' and 'SHA256' (or other) 'Fingerprint' values to check what your browser or email client is seeing.<br />
<br />
= Dynamic Content =<br />
Going beyond simple static content we support use of [http://en.wikipedia.org/wiki/Common_gateway_interface CGI]s, written using any of the installed development languages, and PHP scripts directly via an Apache module.<br />
<br />
== PHP Configuration ==<br />
Our default PHP configuration is somewhat paranoid and as a result you may find you need to adjust some PHP settings, either by getting [mailto:webmaster@fysh.org us] to change the central config files, or by you making use of a .htaccess or .user.ini file as appropriate.<br />
<br />
=== Current Version ===<br />
We currently primarily support version 7.0.x of PHP as supplied by Debian 'stable'. Files with extensions of .php3, .php4, .php7, .php, .pht or .phtml are all handled by this version for web use, and bare 'php' on the command-line will use 7.0.x.<br />
<br />
=== Legacy Version ===<br />
After the upgrade to Debian Stretch we currently still have version 5.6.x available, but these packages are from the older Debian Jessie and will be entirely removed no later than 11th May 2018. Please ensure any of your code is updated to use PHP 7.0 by this date. Currently to use 5.6.x you'll need to either rename files to have a .php5 extension (for web use), or call them with the php5 program if on the command-line.<br />
<br />
=== Obsolete (and Removed) Versions ===<br />
PHP4 hasn't been supported for a long time as the PHP developers themselves dropped all support for it as of the end of 2007[http://www.php.net/downloads.php#v4].<br />
<br />
== CGIs ==<br />
In the case of CGIs there are two ways to ensure the file will run. Either the filename will have to end with .cgi or .pl, or the file must be inside the correct directory. For any domain we host this is its /cgi-bin/ directory. For user pages this will be river:/var/www/user/username/public_html/cgi-bin/ for the URL http://www.fysh.org/~username/cgi-bin/filename. Files outside that directory whose name have no extension or other than .cgi or .pl will have their contents displayed instead of being run.<br />
<br />
'''NB: river:/var/www/user is mounted on [[stickelback.fysh.org]] as /home'''. This means that whilst you will want to create a /~username/cgi-bin/file.cgi as river:/var/www/user/username/public_html/cgi-bin/file.cgi it will appear on [[stickelback.fysh.org]] as /home/username/public_html/cgi-bin/file.cgi and any reference within it to itself or associated files should use that /home/username/... path, ''not'' the /var/www/user/... one!<br />
<br />
Things wouldn't be so complicated if Apache's SuExec didn't treat /~username/ URLs as a special case (as we had everything under /var/www and had hoped that would be enough, but instead SuExec insists on checking the CGI is under the user's home directory AND has 'public_html' in its path). We could compile a local copy of SuExec, but we have done so in the past for other reasons and it leads to maintainability problems when the Debian Apache2 package is updated.<br />
<br />
We will not simply mount river:/home as www:/home, even read-only, as this potentially allows web access to all of everyone's world-readable files, and if [[stickelback.fysh.org]], but not [[river.fysh.org]], is ever compromised the hacker would have full access (even if read-only).<br />
<br />
= Sending EMail from WWW Scripts =<br />
You may send email from scripts on the WWW host, but there are some caveats.<br />
<br />
The 'envelope from' address will always be www-data@stickelback.fysh.org, even if you set a 'From: ' header in the email itself. In April 2016 configuration was adjusted so this should no longer cause issues with remote MTAs trying to perform anti-spam checks.<br />
<br />
If any email sent in this way bounces or otherwise fails the notification will come to the [mailto:webmaster@fysh.org Fyshy Webmaster], not to any email address you set in the 'From: ' header.<br />
<br />
= References =<br />
<references/><br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Main_Page&diff=2061Main Page2023-02-12T13:47:02Z<p>Athan: /* Contacting Admins During Downtime */ Add Athan's mastodon</p>
<hr />
<div>== Introduction ==<br />
Welcome to the Fyshy Wiki. The initial intent of this Wiki is to document the various services available on the Fysh.Org [[Main_Page#Servers|servers]].<br />
<br />
'''NB: If you don't know what Fysh.Org is, and do not have an account, do not ask for one. This is in no way, shape, or form an open or subscribable service if you didn't already know what Fysh.Org is.'''<br />
<br />
== Servers ==<br />
The principal server for Fysh.Org services is [[river.fysh.org]]. Amongst other things it provides [[Mail|email]] and [[Shell Accounts|shell accounts]] to all its users. [[DNS|DNS / Domains]] service also resides on [[river.fysh.org]].<br />
<br />
[[Web|Web hosting]] is on the separate machine [[stickelback.fysh.org]].<br />
<br />
There are also a few other machines that various [[Fysh]] use to provide peripheral services, such as [[SSH#ssh.fysh.org|ssh.fysh.org]].<br />
<br />
Fysh.Org services moved again on 29th June 2013. The new shell machine is [[river.fysh.org]]. The old machines [[lake.fysh.org]] and [[guppy.fysh.org]] ''no longer exist''.<br />
<br />
Note that the old UK2-hosted servers, [[bowl.fysh.org]] and [[www.fysh.org]] ''no longer exist''. We took them down on 17th April 2008, a few days before UK2 were due to turn them off, so as to securely erase all user data on them before we lost access.<br />
<br />
The panix-hosted servers [[pond.fysh.org]] and [[fyn.fysh.org]] ceased to exist mid-March 2010 after our move to Redstation.<br />
<br />
== Services ==<br />
A list of all our service is available [[:Category:Services|here]].<br />
<br />
== Weekends are At-Risk time for Reboots ==<br />
<br />
All UK weekend hours should be considered 'at risk' for unannounced reboots. There may be as little as 30 or even 15 minutes warning of a quick reboot, usually for a new kernel.<br />
<br />
In general such a reboot would take less than 5 minutes, that being the extent of the disruption to service. There is usually no reason for more than one such reboot in any given weekend.<br />
<br />
If planned work is intended to take longer than this then it will be announced at least 3 days in advance, preferably a week. Such announcement would be on the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist, as outlined below. Obviously if there is some pressing security concern work then quick reboots, or extended work, may be carried out at any time without warning.<br />
<br />
You may want to take note of details in the section [[Main_Page#Contacting Admins During Downtime|Contacting Admins During Downtime]] below in case of extended outage.<br />
<br />
== Notification of Service Changes ==<br />
If you wish to know about any changes to Fysh.Org services, which may include scheduled outages, you should be subscribed to the [http://www.fysh.org/mailman3/postorius/lists/fysh-announce.fysh.org/ Fysh-Announce] maillist. We will automatically subscribe any new accounts to this list. Subscriptions require confirmation from the given email address as well as our approval. Anyone may subscribe, but if we don't recognise the email address as being that of one of our users we may check who it is first, or just deny it, so it's best to also email the [mailto:postmaster@fysh.org Fyshy Postmaster] so we know who you are.<br />
<br />
'''NB: The gating of Fysh-Announce emails to the local fysh.announce newsgroup which is then used for the login announcements is disabled as of 2020-08-12. Any old announcements not yet seen will still be shown as outlined below, but for the time being no new announcements will be shown via this mechanism.'''<br />
<br />
By default when you login to a [[river.fysh.org]] [[Shell Accounts|shell account]] you will be shown any new announcements since your last login. If you wish to disable this simply issue this command at a shell prompt on [[river.fysh.org]]:<br />
<br />
touch $HOME/.no_announce<br />
<br />
and to enable it once more if needed:<br />
<br />
rm $HOME/.no_announce<br />
<br />
Note that the file $HOME/.last_announce is used to keep track of when you were last shown new announces. Removing this will cause you to be shown ''every'' existing announcement on next login! Which announces to show you are determined by comparing the 'last update' timestamp on this file against the timestamp on the files holding the announces. So if you change that timestamp on $HOME/.last_announce you may see more, or less, announces than you should.<br />
<br />
== Discussion of Fysh.Org Services ==<br />
If you wish to generally discuss Fysh.Org services you may use the [http://www.fysh.org/mailman/listinfo/Fysh-Discussion Fysh-Discussion] maillist, but note we do not automatically subscribe anyone to this which means you may have to subscribe yourself first and may find that not many people are reading the list itself.<br />
<br />
Alternatively any queries about the Services may be directed to the [mailto:root@fysh.org Fyshy Admin] or you may edit [[Fyshy Suggestions]] here on this wiki.<br />
<br />
== Contacting Admins During Downtime ==<br />
If Fysh.Org services are down and you need to get in touch with an admin then try the following email addresses. Note these aren't guaranteed to be up to date, or even checked routinely. But it's better than only having Fysh.Org-reliant contact details for us:<br />
<br />
{| border="1"<br />
|+ Alternative Fysh.Org Admin contact details<br />
|-<br />
! Admin<br />
! Contact Via<br />
|-<br />
| Athan<br />
| Email: Suisanahta (at) gmail (dot) com<br />
<br />
Twitter: AthanSpod (tag #fyshorg on relevant posts)<br />
<br />
Mastodon: https://techhub.social/@AthanSpod<br />
|}</div>Athanhttps://www.fysh.org/wiki/index.php?title=Main_Page&diff=2060Main Page2022-09-17T16:29:32Z<p>Athan: /* Introduction */ fysh.org is nott open to randos</p>
<hr />
<div>== Introduction ==<br />
Welcome to the Fyshy Wiki. The initial intent of this Wiki is to document the various services available on the Fysh.Org [[Main_Page#Servers|servers]].<br />
<br />
'''NB: If you don't know what Fysh.Org is, and do not have an account, do not ask for one. This is in no way, shape, or form an open or subscribable service if you didn't already know what Fysh.Org is.'''<br />
<br />
== Servers ==<br />
The principal server for Fysh.Org services is [[river.fysh.org]]. Amongst other things it provides [[Mail|email]] and [[Shell Accounts|shell accounts]] to all its users. [[DNS|DNS / Domains]] service also resides on [[river.fysh.org]].<br />
<br />
[[Web|Web hosting]] is on the separate machine [[stickelback.fysh.org]].<br />
<br />
There are also a few other machines that various [[Fysh]] use to provide peripheral services, such as [[SSH#ssh.fysh.org|ssh.fysh.org]].<br />
<br />
Fysh.Org services moved again on 29th June 2013. The new shell machine is [[river.fysh.org]]. The old machines [[lake.fysh.org]] and [[guppy.fysh.org]] ''no longer exist''.<br />
<br />
Note that the old UK2-hosted servers, [[bowl.fysh.org]] and [[www.fysh.org]] ''no longer exist''. We took them down on 17th April 2008, a few days before UK2 were due to turn them off, so as to securely erase all user data on them before we lost access.<br />
<br />
The panix-hosted servers [[pond.fysh.org]] and [[fyn.fysh.org]] ceased to exist mid-March 2010 after our move to Redstation.<br />
<br />
== Services ==<br />
A list of all our service is available [[:Category:Services|here]].<br />
<br />
== Weekends are At-Risk time for Reboots ==<br />
<br />
All UK weekend hours should be considered 'at risk' for unannounced reboots. There may be as little as 30 or even 15 minutes warning of a quick reboot, usually for a new kernel.<br />
<br />
In general such a reboot would take less than 5 minutes, that being the extent of the disruption to service. There is usually no reason for more than one such reboot in any given weekend.<br />
<br />
If planned work is intended to take longer than this then it will be announced at least 3 days in advance, preferably a week. Such announcement would be on the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist, as outlined below. Obviously if there is some pressing security concern work then quick reboots, or extended work, may be carried out at any time without warning.<br />
<br />
You may want to take note of details in the section [[Main_Page#Contacting Admins During Downtime|Contacting Admins During Downtime]] below in case of extended outage.<br />
<br />
== Notification of Service Changes ==<br />
If you wish to know about any changes to Fysh.Org services, which may include scheduled outages, you should be subscribed to the [http://www.fysh.org/mailman3/postorius/lists/fysh-announce.fysh.org/ Fysh-Announce] maillist. We will automatically subscribe any new accounts to this list. Subscriptions require confirmation from the given email address as well as our approval. Anyone may subscribe, but if we don't recognise the email address as being that of one of our users we may check who it is first, or just deny it, so it's best to also email the [mailto:postmaster@fysh.org Fyshy Postmaster] so we know who you are.<br />
<br />
'''NB: The gating of Fysh-Announce emails to the local fysh.announce newsgroup which is then used for the login announcements is disabled as of 2020-08-12. Any old announcements not yet seen will still be shown as outlined below, but for the time being no new announcements will be shown via this mechanism.'''<br />
<br />
By default when you login to a [[river.fysh.org]] [[Shell Accounts|shell account]] you will be shown any new announcements since your last login. If you wish to disable this simply issue this command at a shell prompt on [[river.fysh.org]]:<br />
<br />
touch $HOME/.no_announce<br />
<br />
and to enable it once more if needed:<br />
<br />
rm $HOME/.no_announce<br />
<br />
Note that the file $HOME/.last_announce is used to keep track of when you were last shown new announces. Removing this will cause you to be shown ''every'' existing announcement on next login! Which announces to show you are determined by comparing the 'last update' timestamp on this file against the timestamp on the files holding the announces. So if you change that timestamp on $HOME/.last_announce you may see more, or less, announces than you should.<br />
<br />
== Discussion of Fysh.Org Services ==<br />
If you wish to generally discuss Fysh.Org services you may use the [http://www.fysh.org/mailman/listinfo/Fysh-Discussion Fysh-Discussion] maillist, but note we do not automatically subscribe anyone to this which means you may have to subscribe yourself first and may find that not many people are reading the list itself.<br />
<br />
Alternatively any queries about the Services may be directed to the [mailto:root@fysh.org Fyshy Admin] or you may edit [[Fyshy Suggestions]] here on this wiki.<br />
<br />
== Contacting Admins During Downtime ==<br />
If Fysh.Org services are down and you need to get in touch with an admin then try the following email addresses. Note these aren't guaranteed to be up to date, or even checked routinely. But it's better than only having Fysh.Org-reliant contact details for us:<br />
<br />
{| border="1"<br />
|+ Alternative Fysh.Org Admin contact details<br />
|-<br />
! Admin<br />
! Contact Via<br />
|-<br />
| Athan<br />
| Email: Suisanahta (at) gmail (dot) com<br />
<br />
Twitter: AthanSpod (tag #fyshorg on relevant posts)<br />
|}</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2059Mail2022-08-07T17:20:47Z<p>Athan: /* DKIM Signature Verification */ SpamAssassin and Roundcube also 'do' DKIM</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably ''do not'' want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== DKIM Signature Verification ====<br />
<br />
As of 2022-08-07 we have implemented MTA level checking of DKIM signatures '''but do not reject any email based on this'''.<br />
<br />
You can check what the outcome of this by checking the ''top-most'' (in case someone nefarious tried to pre-add any of them) "X-DKIM" and "Authentication-Results" headers in any email delivered by mail.fysh.org.<br />
<br />
Examples:<br />
<br />
1. Valid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv<br />
<br />
2. INvalid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc reason="signature_incorrect"<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv reason="signature_incorrect"<br />
<br />
3. No signature on a domain we've seen valid ones on in the past (manually curated list).<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for required domain: twitch.tv)<br />
<br />
4. No signature on a domain we do ''not'' have in that list.<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for example.com)<br />
<br />
SpamAssassin can also do its own verification of DKIM signatures, and on river.fysh.org will do so by default. Further more the current webmail, roundcube, can display the result of this if you enable the "Auth. Result" column in the mailbox view.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2058Mail2022-08-07T17:17:02Z<p>Athan: /* MTA Counter-Measures */ DKIM signature checking</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably ''do not'' want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== DKIM Signature Verification ====<br />
<br />
As of 2022-08-07 we have implemented MTA level checking of DKIM signatures '''but do not reject any email based on this'''. You can check what the outcome of this by checking the ''top-most'' (in case someone nefarious tried to pre-add any of them) "X-DKIM" and "Authentication-Results" headers in any email delivered by mail.fysh.org.<br />
<br />
Examples:<br />
<br />
1. Valid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc<br />
Authentication-Results: river.fysh.org; dkim=pass header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv<br />
<br />
2. INvalid DKIM signature(s)<br />
X-DKIM: Exim 4.94.2 on river.fysh.org<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=twitch.tv header.i= header.s=wcdtjomfpuxxefyhk6a7dbwy5fy6ofyc reason="signature_incorrect"<br />
Authentication-Results: river.fysh.org; dkim=fail header.d=amazonses.com header.i= header.s=ndjes4mrtuzus6qxu3frw3ubo3gpjndv reason="signature_incorrect"<br />
<br />
3. No signature on a domain we've seen valid ones on in the past (manually curated list).<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for required domain: twitch.tv)<br />
<br />
4. No signature on a domain we do ''not'' have in that list.<br />
X-DKIM: Exim 4.94.2 on river.fysh.org (no dkim signature for example.com)<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2056Mail2022-02-28T13:00:59Z<p>Athan: /* Pull to Google Mail */ I wish mediawiki just used MarkDown</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably ''do not'' want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2055Mail2022-02-28T12:59:55Z<p>Athan: /* Reading Mail */ blindly forwarding email considered harmful</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== Forwarding email elsewhere to read it ===<br />
<br />
We '''STRONGLY ADVISE AGAINST BLINDLY FORWARDING ALL EMAIL TO ANOTHER PROVIDER IN ORDER TO READ IT THERE'''. Why? Because you will inevitably be forwarding some amount of spam and ''that WILL cause the other provider to consider us as a source of spam''. That will then not only mean that your forward either doesn't work at all, or encounters delays, '''but also that any other email from mail.fysh.org users to that provider will be adversely affected'''.<br />
<br />
'''NO''', it is not sufficient to use, e.g. [[Mail#Spamassassin|Spamassassin]], to filter email on mail.fysh.org and only forward what it doesn't think is spam. Google often treats email as suspicious that Spamassassin doesn't.<br />
<br />
An unfortunate example of this is setting a fysh.org account up to forward all email to a Google Mail (gmail.com/googlemail.com) account. Eventually enough email sent to Google in this manner will cause them to, at best, rate limit how fast mail.fysh.org can send email to the entire Google Mail service. It has been observed that this can cause email from mail.fysh.org to all Google Mail accounts to be '''delayed by over 40 hours'''.<br />
<br />
We have no reason to believe that other major email providers, e.g. Microsoft's Outlook webmail, won't exhibit a similar issue.<br />
<br />
==== Actually just use the other email provider ====<br />
<br />
In the first instance, if you prefer this other email provider (and remember we do have both [[Mail#Webmail|Webmail]] and [[Mail#POP3|POP3]]/[[Mail#IMAP|IMAP]] service available), then just use them. Update your email address with anyone who utilises it and stop using mail.fysh.org in this manner.<br />
<br />
==== Set up the other provider to ''pull'' from mail.fysh.org instead ====<br />
<br />
Even if you decide to just use this other email provider you might want to ensure you still know about email arriving on mail.fysh.org for you. In some cases you will be able to set up the other email provider to check your mail.fysh.org account in a 'pull' manner.<br />
<br />
What follows is best-efforts advice on how to set this up, but is not guaranteed to be accurate or work when you're reading this. Many other email providers might have the functionality, but not in a free tier that we could access and test. In general you're looking for an option to utilise 'POP', 'POP3', or 'IMAP' to retrieve email from another account.<br />
<br />
===== Pull to Google Mail =====<br />
<br />
# Access the settings for your account. As of 2022-02-28 this can be found via: Cog > See all settings<br />
# Select 'Accounts and Import'<br />
# Scroll down to 'Check email from other accounts'<br />
# Select 'Add an email account', which will open a popup<br />
# Email address: <fysh user>@fysh.org (yes, even if technically the email is for another domain)<br />
# Import emails from my other account (POP3)<br />
# Username: <should be autofilled if you got #2 correct)<br />
# Password: <The fysh.org account password>. Yes, you'll have to tell Google this. For *this* purpose we approve of this.<br />
# POP Server: <should have autofilled as 'mail.fysh.org' from #2><br />
# Port: 995<br />
# Do '''NOT''' tick "Leave a copy of retrieved message on the server."<br />
# '''YOU MUST TICK''' "Always use a secure connection (SSL) when retrieving mail"<br />
# Your choice of "Label incoming messages". Perhaps the mail.fysh.org account is actually receiving email under a different domain, so you will want to change this to reflect that.<br />
# You probably *do not* want to tick "Archive incoming messages (Skip the Inbox), because you want to be aware of the new email, yes ?<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2054Mail2022-01-31T14:09:17Z<p>Athan: /* MTA Counter-Measures */ special local parts no longer skip MTA anti-spam measures for locally delivered domains</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to 'postmaster', 'abuse', or 'dmarcreports' at any domain we are relaying email for. They get to make their own decision on the primary MX. This does mean that email to those local parts for domains delivered locally on mail.fysh.org might be rejected due to one of our MTA-level antispam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2053Mail2020-09-15T09:48:20Z<p>Athan: Add section about us handling a domain's email.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== Having us handle email for your domain ==<br />
We're happy to handle email for your domain directly on fysh.org. Email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details, including any requirement for extra accounts for picking up email to particular aliases on the domain.<br />
<br />
Note that you'll be given control of the aliases file for your domain, but please follow these guidelines:<br />
<br />
=== We currently use 'DB' files ===<br />
We currently use Berkely .db files for historical reasons. As a consequence you ''cannot make use of wildcards in aliases, other than a single "*" overall wildcard as a catchall.'' This also means that order in your aliases file doesn't matter. There is a tentative plan to review aliases for all locally delivered domains and change this.<br />
<br />
=== Be careful about a domain "*" wildcard catchall ===<br />
Spammers have an unfortunate habit of plain guessing at local parts for domains. This can be them simply selling lists to others that put all the local parts they've ever seen on every domain, or some error in their scraping of email addresses.<br />
<br />
As such using a "*" wildcard will likely result in you receiving ''more spam'' when without it these attempts would just bounce with "User unknown".<br />
<br />
=== Final delivery is to <account>@fysh.org ===<br />
In order to cause actual local delivery the target of an alias must be <account>@fysh.org. Using just <account> will keep email directed within your domain and will ''not'' lead to an actual delivery.<br />
<br />
=== Make use of :blackhole: not :fail: ===<br />
If you have an alias that you've decided to abandon and want to fail silently please use ":blackhole:" as the target. If you don't need it to fail silently then simply remove the alias so that attempts get "User unknown".<br />
<br />
Likewise do '''NOT''' specify custom ":fail:" messages to spammers. Just use ":blackhole:" instead. This both avoids potentially causing backscatter to spoofed from addresses ''and also makes automated processing of our logs easier''.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2052Mail2020-09-15T09:37:19Z<p>Athan: /* Procmail */ Don't use EXITCODE=67</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
'''Do NOT''' ''Make use of EXITCODE=67 to cause a bounce if you've decided you don't want the email. The likely outcome of this will be back-scatter to a spoofed from address and/or you just getting a "we couldn't deliver that" bounce back to you.'' Simply direct the email into /dev/null.<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2051Mail2020-08-14T09:28:15Z<p>Athan: /* Maillists */ Update URLs</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [https://list.org/ MailMan] as the software. There is a [https://www.fysh.org/mailman3/postorius/lists/ list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Main_Page&diff=2050Main Page2020-08-12T18:49:11Z<p>Athan: /* Notification of Service Changes */ No new login announcements for now.</p>
<hr />
<div>== Introduction ==<br />
Welcome to the Fyshy Wiki. The initial intent of this Wiki is to document the various services available on the Fysh.Org [[Main_Page#Servers|servers]].<br />
<br />
== Servers ==<br />
The principal server for Fysh.Org services is [[river.fysh.org]]. Amongst other things it provides [[Mail|email]] and [[Shell Accounts|shell accounts]] to all its users. [[DNS|DNS / Domains]] service also resides on [[river.fysh.org]].<br />
<br />
[[Web|Web hosting]] is on the separate machine [[stickelback.fysh.org]].<br />
<br />
There are also a few other machines that various [[Fysh]] use to provide peripheral services, such as [[SSH#ssh.fysh.org|ssh.fysh.org]].<br />
<br />
Fysh.Org services moved again on 29th June 2013. The new shell machine is [[river.fysh.org]]. The old machines [[lake.fysh.org]] and [[guppy.fysh.org]] ''no longer exist''.<br />
<br />
Note that the old UK2-hosted servers, [[bowl.fysh.org]] and [[www.fysh.org]] ''no longer exist''. We took them down on 17th April 2008, a few days before UK2 were due to turn them off, so as to securely erase all user data on them before we lost access.<br />
<br />
The panix-hosted servers [[pond.fysh.org]] and [[fyn.fysh.org]] ceased to exist mid-March 2010 after our move to Redstation.<br />
<br />
== Services ==<br />
A list of all our service is available [[:Category:Services|here]].<br />
<br />
== Weekends are At-Risk time for Reboots ==<br />
<br />
All UK weekend hours should be considered 'at risk' for unannounced reboots. There may be as little as 30 or even 15 minutes warning of a quick reboot, usually for a new kernel.<br />
<br />
In general such a reboot would take less than 5 minutes, that being the extent of the disruption to service. There is usually no reason for more than one such reboot in any given weekend.<br />
<br />
If planned work is intended to take longer than this then it will be announced at least 3 days in advance, preferably a week. Such announcement would be on the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist, as outlined below. Obviously if there is some pressing security concern work then quick reboots, or extended work, may be carried out at any time without warning.<br />
<br />
You may want to take note of details in the section [[Main_Page#Contacting Admins During Downtime|Contacting Admins During Downtime]] below in case of extended outage.<br />
<br />
== Notification of Service Changes ==<br />
If you wish to know about any changes to Fysh.Org services, which may include scheduled outages, you should be subscribed to the [http://www.fysh.org/mailman3/postorius/lists/fysh-announce.fysh.org/ Fysh-Announce] maillist. We will automatically subscribe any new accounts to this list. Subscriptions require confirmation from the given email address as well as our approval. Anyone may subscribe, but if we don't recognise the email address as being that of one of our users we may check who it is first, or just deny it, so it's best to also email the [mailto:postmaster@fysh.org Fyshy Postmaster] so we know who you are.<br />
<br />
'''NB: The gating of Fysh-Announce emails to the local fysh.announce newsgroup which is then used for the login announcements is disabled as of 2020-08-12. Any old announcements not yet seen will still be shown as outlined below, but for the time being no new announcements will be shown via this mechanism.'''<br />
<br />
By default when you login to a [[river.fysh.org]] [[Shell Accounts|shell account]] you will be shown any new announcements since your last login. If you wish to disable this simply issue this command at a shell prompt on [[river.fysh.org]]:<br />
<br />
touch $HOME/.no_announce<br />
<br />
and to enable it once more if needed:<br />
<br />
rm $HOME/.no_announce<br />
<br />
Note that the file $HOME/.last_announce is used to keep track of when you were last shown new announces. Removing this will cause you to be shown ''every'' existing announcement on next login! Which announces to show you are determined by comparing the 'last update' timestamp on this file against the timestamp on the files holding the announces. So if you change that timestamp on $HOME/.last_announce you may see more, or less, announces than you should.<br />
<br />
== Discussion of Fysh.Org Services ==<br />
If you wish to generally discuss Fysh.Org services you may use the [http://www.fysh.org/mailman/listinfo/Fysh-Discussion Fysh-Discussion] maillist, but note we do not automatically subscribe anyone to this which means you may have to subscribe yourself first and may find that not many people are reading the list itself.<br />
<br />
Alternatively any queries about the Services may be directed to the [mailto:root@fysh.org Fyshy Admin] or you may edit [[Fyshy Suggestions]] here on this wiki.<br />
<br />
== Contacting Admins During Downtime ==<br />
If Fysh.Org services are down and you need to get in touch with an admin then try the following email addresses. Note these aren't guaranteed to be up to date, or even checked routinely. But it's better than only having Fysh.Org-reliant contact details for us:<br />
<br />
{| border="1"<br />
|+ Alternative Fysh.Org Admin contact details<br />
|-<br />
! Admin<br />
! Contact Via<br />
|-<br />
| Athan<br />
| Email: Suisanahta (at) gmail (dot) com<br />
<br />
Twitter: AthanSpod (tag #fyshorg on relevant posts)<br />
|}</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2049Mail2020-08-10T16:41:17Z<p>Athan: /* Email Spam */ Add mention of honeypot address</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
=== Hey spammers ===<br />
I'd love some spam to [mailto:poohshoney@fysh.org poohshoney@fysh.org] !<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2048Mail2020-08-01T13:22:44Z<p>Athan: /* Reading Mail */ sanemail.fysh.org deprecated</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
NB: If you have been using "sanemail.fysh.org" as the server in any SMTP, POP3 or IMAP setup you no longer need to, just use "mail.fysh.org" as per the instructions below. The hostname has been redirected (and thus also gained IPv6 connectivity).<br />
<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2047Mail2019-11-04T14:31:07Z<p>Athan: /* Bogofilter */ Adjust examples to Maildir format</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
.spam/<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### .fysh/<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### $MAILDIR/<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
.spam/<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
.spam/<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2046Mail2019-11-04T14:29:51Z<p>Athan: /* Spamassassin */ Adjust example for Maildir style procmail destination</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
.spam/<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2045Mail2019-11-04T13:07:53Z<p>Athan: /* Greylisting */ No need for bold, and link to Wikipedia for what it is/was</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
As of 2019-10-17 ~12:53 BST we have turned off [https://en.wikipedia.org/wiki/Greylisting greylisting] entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2044Mail2019-11-04T13:06:43Z<p>Athan: /* Greylisting */ No more greylisting</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
''' As of 2019-10-17 ~12:53 BST we have turned off greylisting entirely. As no issues were reported or observed in the following 2 weeks this was then made permanent.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2043Mail2019-10-17T16:25:27Z<p>Athan: /* Webmail */ We've only used roundcube for a long time now</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [https://roundcube.net/ Roundcube]. The address for this service is [https://roundcube.fysh.org/ https://roundcube.fysh.org/].<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. For historical reasons it uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://roundcube.fysh.org/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
''' As of 2019-10-17 ~12:53 BST we have turned off greylisting entirely. This is an experiment to see how much more spam makes it through to users. We might turn it back on again.'''<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2042Mail2019-10-17T16:22:05Z<p>Athan: /* IMAP */ General cleanup for un-fucking of IMAP namespaces/config.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org provides our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| Leave blank to use the configuration we have set up for you. See the notes below.<br />
|-<br />
|}<br />
<br />
# If you don't use a prefix, and have previously been using IMAP with the default namespace such that it placed any additional folders in your home directory then you will continue to use that configuration for the time being. We will likely be in touch in order to migrate you to a saner configuration.<br />
# Where we know you store additional email folders within a sub-directory of your home directory we have overridden the default IMAP configuration to ensure your IMAP clients only see that sub-directory's contents, rather than your whole home directory. If you're not using Maildir for your INBOX then this will continue to be /var/mail/<user>.<br />
# If we have converted you to using Maildir format then IMAP access with no prefix will be using ~user/Maildir/ for both INBOX and additional folders.<br />
<br />
If you remember a symbolic link 'hack' in order to have INBOX show up properly with the old Maildir prefix, then don't worry, that is no longer necessary.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
''' As of 2019-10-17 ~12:53 BST we have turned off greylisting entirely. This is an experiment to see how much more spam makes it through to users. We might turn it back on again.'''<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2041Mail2019-10-17T16:13:10Z<p>Athan: /* Using Maildir instead of mbox */ Cleanups for Maildir default, sanemail not necessary.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ . Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX. However we have implemented per-user overrides to this to either force the user to ~user/Maildir, or something else that is appropriate where they are not actively already using IMAP with folders directly in their home directory.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes, or simply use IMAP, before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file. An example at /etc/skel/procmailrc-example should be available on [[river.fysh.org]]<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
* '''Whilst the historical mbox-folder setup uses `mail.fysh.org` it is highly recommended that all users utilise [[Mail#Using_Maildir_instead_of_mbox|Maildir format]] and `sanemail.fysh.org`.''' All the rest of the configuration below is still valid, just use `sanemail.fysh.org` for the Server Name. Ignore what it says about prefixes, leave it blank as there's only the one namespace on `sanemail.fysh.org`.<br />
<br />
mail.fysh.org is the historical hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
''' As of 2019-10-17 ~12:53 BST we have turned off greylisting entirely. This is an experiment to see how much more spam makes it through to users. We might turn it back on again.'''<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2040Mail2019-10-17T11:54:21Z<p>Athan: /* Greylisting */ Un-bold bottom copy of "you can opt out"</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ and using sanemail.fysh.org for IMAP or POP3 access. Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use `sanemail.fysh.org` as the IMAP/POP3 'Server Name', and **not** `mail.fysh.org`. If still using `mail.fysh.org` you should use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
* '''Whilst the historical mbox-folder setup uses `mail.fysh.org` it is highly recommended that all users utilise [[Mail#Using_Maildir_instead_of_mbox|Maildir format]] and `sanemail.fysh.org`.''' All the rest of the configuration below is still valid, just use `sanemail.fysh.org` for the Server Name. Ignore what it says about prefixes, leave it blank as there's only the one namespace on `sanemail.fysh.org`.<br />
<br />
mail.fysh.org is the historical hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
''' As of 2019-10-17 ~12:53 BST we have turned off greylisting entirely. This is an experiment to see how much more spam makes it through to users. We might turn it back on again.'''<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2039Mail2019-10-17T11:53:50Z<p>Athan: /* Greylisting */ Turning this off for now.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ and using sanemail.fysh.org for IMAP or POP3 access. Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use `sanemail.fysh.org` as the IMAP/POP3 'Server Name', and **not** `mail.fysh.org`. If still using `mail.fysh.org` you should use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
* '''Whilst the historical mbox-folder setup uses `mail.fysh.org` it is highly recommended that all users utilise [[Mail#Using_Maildir_instead_of_mbox|Maildir format]] and `sanemail.fysh.org`.''' All the rest of the configuration below is still valid, just use `sanemail.fysh.org` for the Server Name. Ignore what it says about prefixes, leave it blank as there's only the one namespace on `sanemail.fysh.org`.<br />
<br />
mail.fysh.org is the historical hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
''' As of 2019-10-17 ~12:53 BST we have turned off greylisting entirely. This is an experiment to see how much more spam makes it through to users. We might turn it back on again.'''<br />
<br />
''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2038Mail2019-10-16T14:15:00Z<p>Athan: /* Reading Mail */ POP3 has been TLS only for years now</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ and using sanemail.fysh.org for IMAP or POP3 access. Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use `sanemail.fysh.org` as the IMAP/POP3 'Server Name', and **not** `mail.fysh.org`. If still using `mail.fysh.org` you should use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
* '''Whilst the historical mbox-folder setup uses `mail.fysh.org` it is highly recommended that all users utilise [[Mail#Using_Maildir_instead_of_mbox|Maildir format]] and `sanemail.fysh.org`.''' All the rest of the configuration below is still valid, just use `sanemail.fysh.org` for the Server Name. Ignore what it says about prefixes, leave it blank as there's only the one namespace on `sanemail.fysh.org`.<br />
<br />
mail.fysh.org is the historical hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2037Mail2019-10-14T17:24:34Z<p>Athan: /* IMAP */ Emphasise sanemail.fysh.org more, link to Maildir instructions.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ and using sanemail.fysh.org for IMAP or POP3 access. Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use `sanemail.fysh.org` as the IMAP/POP3 'Server Name', and **not** `mail.fysh.org`. If still using `mail.fysh.org` you should use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
* '''Whilst the historical mbox-folder setup uses `mail.fysh.org` it is highly recommended that all users utilise [[Mail#Using_Maildir_instead_of_mbox|Maildir format]] and `sanemail.fysh.org`.''' All the rest of the configuration below is still valid, just use `sanemail.fysh.org` for the Server Name. Ignore what it says about prefixes, leave it blank as there's only the one namespace on `sanemail.fysh.org`.<br />
<br />
mail.fysh.org is the historical hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2036Mail2019-10-14T17:19:36Z<p>Athan: /* IMAP */ Please use sanemail.fysh.org</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ and using sanemail.fysh.org for IMAP or POP3 access. Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use `sanemail.fysh.org` as the IMAP/POP3 'Server Name', and **not** `mail.fysh.org`. If still using `mail.fysh.org` you should use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
* '''Whilst the historical mbox-folder setup uses `mail.fysh.org` it is highly recommended that all users utilise Maildir format and `sanemail.fysh.org`.'''<br />
<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2035Mail2019-10-14T17:17:11Z<p>Athan: /* Using Maildir instead of mbox */ Emphasise that we're migrating to Maildir</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
For historical reasons much of the configuration on Fysh.Org still targets the use of mbox style email folders, both over IMAP/POP3 and locally on [[river.fysh.org]]. However there are multiple reasons why we are migrating away from this as of October 2019.<br />
<br />
# Default IMAP namespace targets the user's home directory, and as such will attempt to treat every single file and folder inside of it as an IMAP folder. This then creates a .imap/ directory at the location of each found file or directory and populates it with dovecot-related files for keeping track of the index and UID values of any email items. This both makes a mess and also greatly slows down IMAP 'LIST' operations (which also can cause 'permissions denied' errors).<br />
# If a user often saves email to, or deletes email from, or even just updates the status of an email, in an mbox file this causes the entire file to then be included in the next incremental system backup. If someone has a 1GB INBOX and receives only 1 new email a day (or deletes on a day, or marks one a day as read when it was unread previously) then that entire 1GB goes into the next incremental backup. This is wasteful of space. Using Maildir format for this means only one small new file is written, or an old one deleted/updated.<br />
<br />
As such the only officially supported email storage scheme is now Maildir, storing in ~user/Maildir/ and using sanemail.fysh.org for IMAP or POP3 access. Whilst the migration is on-going the defaults for MTA/MDA final delivery, MUAs and mail.fysh.org IMAP/POP3 access will still be using mbox and /var/mail/user as INBOX.<br />
<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use `sanemail.fysh.org` as the IMAP/POP3 'Server Name', and **not** `mail.fysh.org`. If still using `mail.fysh.org` you should use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2034Mail2019-10-14T17:07:33Z<p>Athan: /* Using Maildir instead of mbox */ out -> our</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with our IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Databases&diff=2033Databases2018-10-16T09:36:29Z<p>Athan: /* Web Admin */ phpmyadmin disabled due to password dictionary attacks</p>
<hr />
<div>= Introduction =<br />
Fysh.Org has a number of database implementations available to any user upon request. These range from the simple use of file-based databases such as Berkeley DB through to full RDBMS like MySQL and PostgreSQL. In the case of file-based databases the most you may have to ask [mailto:root@fysh.org us] to do is install a development package for it so that you can link your own code against the libraries.<br />
<br />
In the case of MySQL and PostgreSQL we run whatever version is current in Debian 'stable'. Any user of [[river.fysh.org]] can request any number of databases be set up for them. Simply ask [mailto:root@fysh.org us] to create the database and grant access. Please specify preferred database name, username and password, and whether you'll need access from anything but localhost. By default access is only from localhost plus our web service and if you need it from other hosts we'll need to know their IP(s). If you intend to make use of the database from web pages then they'll need to use the hostname 'db.fysh.org' to access the database. Using this hostname, rather than the IP it resolves to, is preferred because it means you don't need to make any changes if we change IP for any reason.<br />
<br />
If you find that we don't have support for the RDBMS you requested in the development language you prefer to use just ask and we'll do our best to install it. If it's available as a Debian 'stable' package there will be no problem, however if it isn't then we can't guarantee being able to fulfil the request. In particular remember that we're now 64-bit.<br />
<br />
== Database Backups ==<br />
Note users of any RDBMS database on [[river.fysh.org]] should implement a backup schedule themselves. For MySQL a command like the following works well:<br />
<br />
mysqldump -u <database user> --lock-tables --opt --quick <database name> > <backup file name><br />
<br />
For PostgreSQL check the documentation for the pg_dump command. Please do ensure you keep backups around only for as long as necessary so as to not over-use disk space.<br />
<br />
Once a user has such backups implemented the files produced by such will of course form part of our normal backup sets.<br />
<br />
We do perform nightly database backups, but these will become over-written on the next day, so you can't rely on them to retrieve days-old data. Also it would be much quicker for you to restore from your own backups.<br />
<br />
== Web Admin ==<br />
<strike>We do have [https://www.fysh.org/phpmyadmin/ PhpMyAdmin] installed for administrating MySQL databases and it is preferred users make use of this central installation rather than installing their own copy. This way we can be sure that the copy used is up to date with respect to security patches.</strike> Due to lack of use, and targeting by password dictionary attacks, this is currently disabled. If you do need phpmyadmin access then a user/password can be arranged for accessing it.<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2032Mail2018-03-13T10:11:01Z<p>Athan: /* Spamassassin */ pond? replaced ages ago</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with out IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under river:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=SSH&diff=2031SSH2018-02-13T11:44:51Z<p>Athan: /* Fysh.Org SSH Host Key(s) */ ECDSA key no longer in service</p>
<hr />
<div>= Introduction =<br />
The primary means of logging into a Fysh.Org [[Shell Accounts|shell account]] is to use a '''[[wikipedia:Secure shell|SSH]]''' client. This has the advantages of providing an encrypted connection, so no passwords, email etc can be casually snooped by 3rd parties between you and us, and also provides a variety of authentication methods.<br />
<br />
We run the current version in Debian 'stable' of OpenSSH as the server software on [[river.fysh.org]]. Note that version 1 of the SSH protocol is disabled, so you'll need a client that supports version 2.<br />
<br />
We also have [[wikipedia:SSH file transfer protocol|SFTP]] enabled should you wish to securely copy files to or from [[river.fysh.org]].<br />
<br />
= ssh.fysh.org =<br />
Some [[river.fysh.org]] account holders find themselves behind firewalls that won't allow them to connect directly to external hosts on port 22 which is the default port on which we run SSH. Unfortunately due to the current set-up (only one public IPv4 address) we can't offer to also run SSH on port 443 (https), which offers one way around firewalls.<br />
<br />
However if you have full IPv6 access we ''do'' offer the SSH service on port 443, along with the default port 22 and additional port 2222.<br />
<br />
= Fysh.Org SSH Host Key(s) =<br />
<br />
For reference the fingerprint of keys for hosts you may connect to are:<br />
<br />
{| border="1"<br />
|+Fysh.Org SSH Host Keys<br />
|-<br />
!Host<br />
!Key Type<br />
!MD5 Key Fingerprint<br />
!SHA256 Key Fingerprint<br />
!Comment<br />
|-<br />
|river.fysh.org<br />
|ed25519<br />
|eb:cf:67:1c:c6:f5:6f:98:3a:65:49:18:e0:81:38:a6<br />
|uFhV9BxFo83RDGC/0P6aP7GO3yKHlMgQHnsrknVXck8<br />
|In service from Mon 12 Feb 09:44:29 GMT 2018<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|6c:c8:c7:f3:de:c8:01:c2:d5:5d:82:70:fc:2b:54:85<br />
|N2Q8yXLZWWloRiq1bsuPkpex0laLoXN/EEtZ2Nr3XYk<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to <br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|73:cb:e4:46:49:6f:c6:34:7b:4f:e0:8c:07:4a:4c:ff<br />
|1q2sm3WjvxgJ5ugwhRhcyNjQygq+hdP1UC4AXR7mCj0<br />
|NO LONGER IN SERVICE - as we no longer offer a matching HostKeyAlgorithm. Was new due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to 2018-02-13 ~11:20 UTC<br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|9c:f8:b9:07:a8:0e:db:90:21:7e:63:67:38:45:29:d9<br />
|c3/TQP4okhunjKyGtdBZ9ggqr9Yaf7lgID80cjr32Jg<br />
|NO LONGER IN SERVICE. Was new due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to Mon 12 Feb 09:40:51 GMT 2018 (when it was removed because this key format long since fell out of favour).<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|77:fe:45:57:27:33:b5:98:ce:9b:dd:74:4b:83:4d:9e<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|b4:6d:27:bc:35:10:48:e2:ea:2e:54:11:73:78:ec:77<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|c5:28:32:66:db:86:cd:1a:c3:f9:22:16:8c:12:01:14<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-}<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=SSH&diff=2030SSH2018-02-12T09:55:18Z<p>Athan: /* Fysh.Org SSH Host Key(s) */ Remove extraneous )</p>
<hr />
<div>= Introduction =<br />
The primary means of logging into a Fysh.Org [[Shell Accounts|shell account]] is to use a '''[[wikipedia:Secure shell|SSH]]''' client. This has the advantages of providing an encrypted connection, so no passwords, email etc can be casually snooped by 3rd parties between you and us, and also provides a variety of authentication methods.<br />
<br />
We run the current version in Debian 'stable' of OpenSSH as the server software on [[river.fysh.org]]. Note that version 1 of the SSH protocol is disabled, so you'll need a client that supports version 2.<br />
<br />
We also have [[wikipedia:SSH file transfer protocol|SFTP]] enabled should you wish to securely copy files to or from [[river.fysh.org]].<br />
<br />
= ssh.fysh.org =<br />
Some [[river.fysh.org]] account holders find themselves behind firewalls that won't allow them to connect directly to external hosts on port 22 which is the default port on which we run SSH. Unfortunately due to the current set-up (only one public IPv4 address) we can't offer to also run SSH on port 443 (https), which offers one way around firewalls.<br />
<br />
However if you have full IPv6 access we ''do'' offer the SSH service on port 443, along with the default port 22 and additional port 2222.<br />
<br />
= Fysh.Org SSH Host Key(s) =<br />
<br />
For reference the fingerprint of keys for hosts you may connect to are:<br />
<br />
{| border="1"<br />
|+Fysh.Org SSH Host Keys<br />
|-<br />
!Host<br />
!Key Type<br />
!MD5 Key Fingerprint<br />
!SHA256 Key Fingerprint<br />
!Comment<br />
|-<br />
|river.fysh.org<br />
|ed25519<br />
|eb:cf:67:1c:c6:f5:6f:98:3a:65:49:18:e0:81:38:a6<br />
|uFhV9BxFo83RDGC/0P6aP7GO3yKHlMgQHnsrknVXck8<br />
|In service from Mon 12 Feb 09:44:29 GMT 2018<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|73:cb:e4:46:49:6f:c6:34:7b:4f:e0:8c:07:4a:4c:ff<br />
|1q2sm3WjvxgJ5ugwhRhcyNjQygq+hdP1UC4AXR7mCj0<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|6c:c8:c7:f3:de:c8:01:c2:d5:5d:82:70:fc:2b:54:85<br />
|N2Q8yXLZWWloRiq1bsuPkpex0laLoXN/EEtZ2Nr3XYk<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to <br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|9c:f8:b9:07:a8:0e:db:90:21:7e:63:67:38:45:29:d9<br />
|c3/TQP4okhunjKyGtdBZ9ggqr9Yaf7lgID80cjr32Jg<br />
|NO LONGER IN SERVICE. Was new due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to Mon 12 Feb 09:40:51 GMT 2018 (when it was removed because this key format long since fell out of favour).<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|77:fe:45:57:27:33:b5:98:ce:9b:dd:74:4b:83:4d:9e<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|b4:6d:27:bc:35:10:48:e2:ea:2e:54:11:73:78:ec:77<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|c5:28:32:66:db:86:cd:1a:c3:f9:22:16:8c:12:01:14<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-}<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=SSH&diff=2029SSH2018-02-12T09:48:16Z<p>Athan: /* Fysh.Org SSH Host Key(s) */ Document new river ed25519 key</p>
<hr />
<div>= Introduction =<br />
The primary means of logging into a Fysh.Org [[Shell Accounts|shell account]] is to use a '''[[wikipedia:Secure shell|SSH]]''' client. This has the advantages of providing an encrypted connection, so no passwords, email etc can be casually snooped by 3rd parties between you and us, and also provides a variety of authentication methods.<br />
<br />
We run the current version in Debian 'stable' of OpenSSH as the server software on [[river.fysh.org]]. Note that version 1 of the SSH protocol is disabled, so you'll need a client that supports version 2.<br />
<br />
We also have [[wikipedia:SSH file transfer protocol|SFTP]] enabled should you wish to securely copy files to or from [[river.fysh.org]].<br />
<br />
= ssh.fysh.org =<br />
Some [[river.fysh.org]] account holders find themselves behind firewalls that won't allow them to connect directly to external hosts on port 22 which is the default port on which we run SSH. Unfortunately due to the current set-up (only one public IPv4 address) we can't offer to also run SSH on port 443 (https), which offers one way around firewalls.<br />
<br />
However if you have full IPv6 access we ''do'' offer the SSH service on port 443, along with the default port 22 and additional port 2222.<br />
<br />
= Fysh.Org SSH Host Key(s) =<br />
<br />
For reference the fingerprint of keys for hosts you may connect to are:<br />
<br />
{| border="1"<br />
|+Fysh.Org SSH Host Keys<br />
|-<br />
!Host<br />
!Key Type<br />
!MD5 Key Fingerprint<br />
!SHA256 Key Fingerprint<br />
!Comment<br />
|-<br />
|river.fysh.org<br />
|ed25519<br />
|eb:cf:67:1c:c6:f5:6f:98:3a:65:49:18:e0:81:38:a6<br />
|uFhV9BxFo83RDGC/0P6aP7GO3yKHlMgQHnsrknVXck8<br />
|In service from Mon 12 Feb 09:44:29 GMT 2018<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|73:cb:e4:46:49:6f:c6:34:7b:4f:e0:8c:07:4a:4c:ff<br />
|1q2sm3WjvxgJ5ugwhRhcyNjQygq+hdP1UC4AXR7mCj0<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|6c:c8:c7:f3:de:c8:01:c2:d5:5d:82:70:fc:2b:54:85<br />
|N2Q8yXLZWWloRiq1bsuPkpex0laLoXN/EEtZ2Nr3XYk<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to <br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|9c:f8:b9:07:a8:0e:db:90:21:7e:63:67:38:45:29:d9<br />
|c3/TQP4okhunjKyGtdBZ9ggqr9Yaf7lgID80cjr32Jg<br />
|NO LONGER IN SERVICE. Was new due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to Mon 12 Feb 09:40:51 GMT 2018 (when it was removed because this key format long since fell out of favour)).<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|77:fe:45:57:27:33:b5:98:ce:9b:dd:74:4b:83:4d:9e<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|b4:6d:27:bc:35:10:48:e2:ea:2e:54:11:73:78:ec:77<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|c5:28:32:66:db:86:cd:1a:c3:f9:22:16:8c:12:01:14<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-}<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=SSH&diff=2028SSH2018-02-12T09:42:06Z<p>Athan: /* Fysh.Org SSH Host Key(s) */ Add SHA256 host key fingerprints, and mark DSA one no longer in use.</p>
<hr />
<div>= Introduction =<br />
The primary means of logging into a Fysh.Org [[Shell Accounts|shell account]] is to use a '''[[wikipedia:Secure shell|SSH]]''' client. This has the advantages of providing an encrypted connection, so no passwords, email etc can be casually snooped by 3rd parties between you and us, and also provides a variety of authentication methods.<br />
<br />
We run the current version in Debian 'stable' of OpenSSH as the server software on [[river.fysh.org]]. Note that version 1 of the SSH protocol is disabled, so you'll need a client that supports version 2.<br />
<br />
We also have [[wikipedia:SSH file transfer protocol|SFTP]] enabled should you wish to securely copy files to or from [[river.fysh.org]].<br />
<br />
= ssh.fysh.org =<br />
Some [[river.fysh.org]] account holders find themselves behind firewalls that won't allow them to connect directly to external hosts on port 22 which is the default port on which we run SSH. Unfortunately due to the current set-up (only one public IPv4 address) we can't offer to also run SSH on port 443 (https), which offers one way around firewalls.<br />
<br />
However if you have full IPv6 access we ''do'' offer the SSH service on port 443, along with the default port 22 and additional port 2222.<br />
<br />
= Fysh.Org SSH Host Key(s) =<br />
<br />
For reference the fingerprint of keys for hosts you may connect to are:<br />
<br />
{| border="1"<br />
|+Fysh.Org SSH Host Keys<br />
|-<br />
!Host<br />
!Key Type<br />
!MD5 Key Fingerprint<br />
!SHA256 Key Fingerprint<br />
!Comment<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|6c:c8:c7:f3:de:c8:01:c2:d5:5d:82:70:fc:2b:54:85<br />
|N2Q8yXLZWWloRiq1bsuPkpex0laLoXN/EEtZ2Nr3XYk<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to <br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|9c:f8:b9:07:a8:0e:db:90:21:7e:63:67:38:45:29:d9<br />
|c3/TQP4okhunjKyGtdBZ9ggqr9Yaf7lgID80cjr32Jg<br />
|NO LONGER IN SERVICE. Was new due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST to Mon 12 Feb 09:40:51 GMT 2018 (when it was removed because this key format long since fell out of favour)).<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|73:cb:e4:46:49:6f:c6:34:7b:4f:e0:8c:07:4a:4c:ff<br />
|1q2sm3WjvxgJ5ugwhRhcyNjQygq+hdP1UC4AXR7mCj0<br />
|New due to 'Heartbleed' bug - In service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|RSA<br />
|77:fe:45:57:27:33:b5:98:ce:9b:dd:74:4b:83:4d:9e<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|DSA<br />
|b4:6d:27:bc:35:10:48:e2:ea:2e:54:11:73:78:ec:77<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-<br />
|river.fysh.org<br />
|ECDSA<br />
|c5:28:32:66:db:86:cd:1a:c3:f9:22:16:8c:12:01:14<br />
|<not relevant when in use><br />
|Pre-'Heartbleed' bug - NO LONGER in service from 2014-04-12 12:21:00 BST<br />
|-}<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Web&diff=2027Web2018-02-11T11:56:18Z<p>Athan: /* HTTPS and SSL Certificates */</p>
<hr />
<div>= Introduction =<br />
[[stickelback.fysh.org]] runs web servers under the primary hostname of [[www.fysh.org]]. We currently utilise [http://httpd.apache.org/ Apache] 2.4.x (as supplied in Debian 'stable') for this purpose, with a single instance running on both port 80 for http access and port 443 for https access.<br />
<br />
We may also run additional instances of apache, or other web server software, on other ports as needed.<br />
<br />
= Webmaster =<br />
All enquiries about [[stickelback.fysh.org]] and other web services should be directed to the [mailto:webmaster@fysh.org Fyshy Webmaster].<br />
<br />
= User Pages =<br />
Any user with an account on [[river.fysh.org]] can create a page on the [http://en.wikipedia.org/wiki/World_wide_web WWW] simply by placing the requisite files in the directory ''/var/www/user/username/public_html/'' (where 'username' is theirs). This will yield a URL of http://www.fysh.org/~username/ for example. If you find that this directory does not yet exist, or file permissions deny you access, please contact [mailto:webmaster@fysh.org The Fyshy Webmaster] to correct this.<br />
<br />
Please note that /var/www/user on [[river.fysh.org]] is NFS mounted on [[www.fysh.org]] '''read-only'''. Web scripts and CGIs ''cannot'' write to any files. Thus any web page that needs to store data will have to utilise a [[Databases|Database]]. In some circumstances we will set up a separate area for a user to place web pages that absolutely ''require'' file write access, but we would much rather they use a proper [[Databases|Database]] instead. '''Furthermore''' files located in /var/www/user on [[river.fysh.org]] will appear under /home on [[www.fysh.org]]. This is to satisfy the requirements of SuExec when CGIs are run from a /~username/ URL. Be sure to write any scripts/CGIs to take account of this different path! It is noted that at this time you can in fact use the same /var/www/user/... path on [[stickelback.fysh.org]] as on [[river.fysh.org]], as a side effect of river:/var/www being mounted onto stickelback:/var/www, but this should not be relied upon.<br />
<br />
= User Domains =<br />
Additionally we will host the web pages for any domain that a user owns or controls and can get the DNS changed to point to our web server. Please note that our preference is that you use a CNAME record pointing to "www.fysh.org." (the trailing dot is important, don't miss it out!), as this allows us to move your domain webhosting about if we need to. The one practical exception to this is where you use solely the domain name for your URL, i.e. if your domain was 'example.com' and you wanted to use http://example.com/ instead of, or as well as, http://www.example.com/. You can't have a CNAME record and any other records for the same name, and a domain itself requires at least an SOA record. Thus in this configuration you would have to use an A record pointing to the same IP as www.fysh.org.<br />
<br />
Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] to enquire about setting this up. Note that you should ensure that you have a valid and working webmaster email account under your own domain when this is set up.<br />
<br />
The same caveat as for User Pages about '''read-only''' file access will apply.<br />
<br />
If you wish URLs of the form http://www.example.com/~username/ (but for your domain) to be served from /var/www/user/username/public_html/ (as is set up for www.fysh.org URLs) please let us know and we'll adjust the configuration for your domain. By default we assume you don't want just any Fysh.Org user to be able to utilise such a URL. We can configure things such that only certain users have URLs of this form working for your domain, so let us know which users to allow. As always it's the [mailto:webmaster@fysh.org Fyshy Webmaster] you need to contact for this.<br />
<br />
=== HTTPS and SSL Certificates ===<br />
<br />
Since February 2018 we have offered use of [https://letsencrypt.org/ LetsEncrypt] SSL Certificates on the fysh.org web service. Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] if you wish us to set this up for any of your domains we host. You won't need to do anything else as the method we use to obtain LetsEncrypt certificates relies solely on control of the website as proof of legitimacy for the domain, which of course we have.<br />
<br />
Do note that a single IP will still be used, so this relies on clients making use of [http://en.wikipedia.org/wiki/Server_Name_Indication Server Name Indication] to indicate which domain they want to access (so that the correct certificate is used), which has some limitations (mostly to do with rather old browsers and operating systems), but this is becoming less and less of an issue.<br />
<br />
= Dynamic Content =<br />
Going beyond simple static content we support use of [http://en.wikipedia.org/wiki/Common_gateway_interface CGI]s, written using any of the installed development languages, and PHP scripts directly via an Apache module.<br />
<br />
== PHP Configuration ==<br />
Our default PHP configuration is somewhat paranoid and as a result you may find you need to adjust some PHP settings, either by getting [mailto:webmaster@fysh.org us] to change the central config files, or by you making use of a .htaccess or .user.ini file as appropriate.<br />
<br />
=== Current Version ===<br />
We currently primarily support version 7.0.x of PHP as supplied by Debian 'stable'. Files with extensions of .php3, .php4, .php7, .php, .pht or .phtml are all handled by this version for web use, and bare 'php' on the command-line will use 7.0.x.<br />
<br />
=== Legacy Version ===<br />
After the upgrade to Debian Stretch we currently still have version 5.6.x available, but these packages are from the older Debian Jessie and will be entirely removed no later than 11th May 2018. Please ensure any of your code is updated to use PHP 7.0 by this date. Currently to use 5.6.x you'll need to either rename files to have a .php5 extension (for web use), or call them with the php5 program if on the command-line.<br />
<br />
=== Obsolete (and Removed) Versions ===<br />
PHP4 hasn't been supported for a long time as the PHP developers themselves dropped all support for it as of the end of 2007[http://www.php.net/downloads.php#v4].<br />
<br />
== CGIs ==<br />
In the case of CGIs there are two ways to ensure the file will run. Either the filename will have to end with .cgi or .pl, or the file must be inside the correct directory. For any domain we host this is its /cgi-bin/ directory. For user pages this will be river:/var/www/user/username/public_html/cgi-bin/ for the URL http://www.fysh.org/~username/cgi-bin/filename. Files outside that directory whose name have no extension or other than .cgi or .pl will have their contents displayed instead of being run.<br />
<br />
'''NB: river:/var/www/user is mounted on [[stickelback.fysh.org]] as /home'''. This means that whilst you will want to create a /~username/cgi-bin/file.cgi as river:/var/www/user/username/public_html/cgi-bin/file.cgi it will appear on [[stickelback.fysh.org]] as /home/username/public_html/cgi-bin/file.cgi and any reference within it to itself or associated files should use that /home/username/... path, ''not'' the /var/www/user/... one!<br />
<br />
Things wouldn't be so complicated if Apache's SuExec didn't treat /~username/ URLs as a special case (as we had everything under /var/www and had hoped that would be enough, but instead SuExec insists on checking the CGI is under the user's home directory AND has 'public_html' in its path). We could compile a local copy of SuExec, but we have done so in the past for other reasons and it leads to maintainability problems when the Debian Apache2 package is updated.<br />
<br />
We will not simply mount river:/home as www:/home, even read-only, as this potentially allows web access to all of everyone's world-readable files, and if [[stickelback.fysh.org]], but not [[river.fysh.org]], is ever compromised the hacker would have full access (even if read-only).<br />
<br />
= Sending EMail from WWW Scripts =<br />
You may send email from scripts on the WWW host, but there are some caveats.<br />
<br />
The 'envelope from' address will always be www-data@stickelback.fysh.org, even if you set a 'From: ' header in the email itself. In April 2016 configuration was adjusted so this should no longer cause issues with remote MTAs trying to perform anti-spam checks.<br />
<br />
If any email sent in this way bounces or otherwise fails the notification will come to the [mailto:webmaster@fysh.org Fyshy Webmaster], not to any email address you set in the 'From: ' header.<br />
<br />
= References =<br />
<references/><br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Web&diff=2026Web2018-02-11T11:55:41Z<p>Athan: /* User Domains */ SSL is from LetsEncrypt now, and easy</p>
<hr />
<div>= Introduction =<br />
[[stickelback.fysh.org]] runs web servers under the primary hostname of [[www.fysh.org]]. We currently utilise [http://httpd.apache.org/ Apache] 2.4.x (as supplied in Debian 'stable') for this purpose, with a single instance running on both port 80 for http access and port 443 for https access.<br />
<br />
We may also run additional instances of apache, or other web server software, on other ports as needed.<br />
<br />
= Webmaster =<br />
All enquiries about [[stickelback.fysh.org]] and other web services should be directed to the [mailto:webmaster@fysh.org Fyshy Webmaster].<br />
<br />
= User Pages =<br />
Any user with an account on [[river.fysh.org]] can create a page on the [http://en.wikipedia.org/wiki/World_wide_web WWW] simply by placing the requisite files in the directory ''/var/www/user/username/public_html/'' (where 'username' is theirs). This will yield a URL of http://www.fysh.org/~username/ for example. If you find that this directory does not yet exist, or file permissions deny you access, please contact [mailto:webmaster@fysh.org The Fyshy Webmaster] to correct this.<br />
<br />
Please note that /var/www/user on [[river.fysh.org]] is NFS mounted on [[www.fysh.org]] '''read-only'''. Web scripts and CGIs ''cannot'' write to any files. Thus any web page that needs to store data will have to utilise a [[Databases|Database]]. In some circumstances we will set up a separate area for a user to place web pages that absolutely ''require'' file write access, but we would much rather they use a proper [[Databases|Database]] instead. '''Furthermore''' files located in /var/www/user on [[river.fysh.org]] will appear under /home on [[www.fysh.org]]. This is to satisfy the requirements of SuExec when CGIs are run from a /~username/ URL. Be sure to write any scripts/CGIs to take account of this different path! It is noted that at this time you can in fact use the same /var/www/user/... path on [[stickelback.fysh.org]] as on [[river.fysh.org]], as a side effect of river:/var/www being mounted onto stickelback:/var/www, but this should not be relied upon.<br />
<br />
= User Domains =<br />
Additionally we will host the web pages for any domain that a user owns or controls and can get the DNS changed to point to our web server. Please note that our preference is that you use a CNAME record pointing to "www.fysh.org." (the trailing dot is important, don't miss it out!), as this allows us to move your domain webhosting about if we need to. The one practical exception to this is where you use solely the domain name for your URL, i.e. if your domain was 'example.com' and you wanted to use http://example.com/ instead of, or as well as, http://www.example.com/. You can't have a CNAME record and any other records for the same name, and a domain itself requires at least an SOA record. Thus in this configuration you would have to use an A record pointing to the same IP as www.fysh.org.<br />
<br />
Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] to enquire about setting this up. Note that you should ensure that you have a valid and working webmaster email account under your own domain when this is set up.<br />
<br />
The same caveat as for User Pages about '''read-only''' file access will apply.<br />
<br />
If you wish URLs of the form http://www.example.com/~username/ (but for your domain) to be served from /var/www/user/username/public_html/ (as is set up for www.fysh.org URLs) please let us know and we'll adjust the configuration for your domain. By default we assume you don't want just any Fysh.Org user to be able to utilise such a URL. We can configure things such that only certain users have URLs of this form working for your domain, so let us know which users to allow. As always it's the [mailto:webmaster@fysh.org Fyshy Webmaster] you need to contact for this.<br />
<br />
== HTTPS and SSL Certificates ==<br />
<br />
Since February 2018 we have offered use of [https://letsencrypt.org/ LetsEncrypt] SSL Certificates on the fysh.org web service. Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] if you wish us to set this up for any of your domains we host. You won't need to do anything else as the method we use to obtain LetsEncrypt certificates relies solely on control of the website as proof of legitimacy for the domain, which of course we have.<br />
<br />
Do note that a single IP will still be used, so this relies on clients making use of [http://en.wikipedia.org/wiki/Server_Name_Indication Server Name Indication] to indicate which domain they want to access (so that the correct certificate is used), which has some limitations (mostly to do with rather old browsers and operating systems), but this is becoming less and less of an issue.<br />
<br />
= Dynamic Content =<br />
Going beyond simple static content we support use of [http://en.wikipedia.org/wiki/Common_gateway_interface CGI]s, written using any of the installed development languages, and PHP scripts directly via an Apache module.<br />
<br />
== PHP Configuration ==<br />
Our default PHP configuration is somewhat paranoid and as a result you may find you need to adjust some PHP settings, either by getting [mailto:webmaster@fysh.org us] to change the central config files, or by you making use of a .htaccess or .user.ini file as appropriate.<br />
<br />
=== Current Version ===<br />
We currently primarily support version 7.0.x of PHP as supplied by Debian 'stable'. Files with extensions of .php3, .php4, .php7, .php, .pht or .phtml are all handled by this version for web use, and bare 'php' on the command-line will use 7.0.x.<br />
<br />
=== Legacy Version ===<br />
After the upgrade to Debian Stretch we currently still have version 5.6.x available, but these packages are from the older Debian Jessie and will be entirely removed no later than 11th May 2018. Please ensure any of your code is updated to use PHP 7.0 by this date. Currently to use 5.6.x you'll need to either rename files to have a .php5 extension (for web use), or call them with the php5 program if on the command-line.<br />
<br />
=== Obsolete (and Removed) Versions ===<br />
PHP4 hasn't been supported for a long time as the PHP developers themselves dropped all support for it as of the end of 2007[http://www.php.net/downloads.php#v4].<br />
<br />
== CGIs ==<br />
In the case of CGIs there are two ways to ensure the file will run. Either the filename will have to end with .cgi or .pl, or the file must be inside the correct directory. For any domain we host this is its /cgi-bin/ directory. For user pages this will be river:/var/www/user/username/public_html/cgi-bin/ for the URL http://www.fysh.org/~username/cgi-bin/filename. Files outside that directory whose name have no extension or other than .cgi or .pl will have their contents displayed instead of being run.<br />
<br />
'''NB: river:/var/www/user is mounted on [[stickelback.fysh.org]] as /home'''. This means that whilst you will want to create a /~username/cgi-bin/file.cgi as river:/var/www/user/username/public_html/cgi-bin/file.cgi it will appear on [[stickelback.fysh.org]] as /home/username/public_html/cgi-bin/file.cgi and any reference within it to itself or associated files should use that /home/username/... path, ''not'' the /var/www/user/... one!<br />
<br />
Things wouldn't be so complicated if Apache's SuExec didn't treat /~username/ URLs as a special case (as we had everything under /var/www and had hoped that would be enough, but instead SuExec insists on checking the CGI is under the user's home directory AND has 'public_html' in its path). We could compile a local copy of SuExec, but we have done so in the past for other reasons and it leads to maintainability problems when the Debian Apache2 package is updated.<br />
<br />
We will not simply mount river:/home as www:/home, even read-only, as this potentially allows web access to all of everyone's world-readable files, and if [[stickelback.fysh.org]], but not [[river.fysh.org]], is ever compromised the hacker would have full access (even if read-only).<br />
<br />
= Sending EMail from WWW Scripts =<br />
You may send email from scripts on the WWW host, but there are some caveats.<br />
<br />
The 'envelope from' address will always be www-data@stickelback.fysh.org, even if you set a 'From: ' header in the email itself. In April 2016 configuration was adjusted so this should no longer cause issues with remote MTAs trying to perform anti-spam checks.<br />
<br />
If any email sent in this way bounces or otherwise fails the notification will come to the [mailto:webmaster@fysh.org Fyshy Webmaster], not to any email address you set in the 'From: ' header.<br />
<br />
= References =<br />
<references/><br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Web&diff=2025Web2018-02-11T11:47:05Z<p>Athan: /* PHP Configuration */ PHP 7.0.x now the current version, 5.6.x will be going away.</p>
<hr />
<div>= Introduction =<br />
[[stickelback.fysh.org]] runs web servers under the primary hostname of [[www.fysh.org]]. We currently utilise [http://httpd.apache.org/ Apache] 2.4.x (as supplied in Debian 'stable') for this purpose, with a single instance running on both port 80 for http access and port 443 for https access.<br />
<br />
We may also run additional instances of apache, or other web server software, on other ports as needed.<br />
<br />
= Webmaster =<br />
All enquiries about [[stickelback.fysh.org]] and other web services should be directed to the [mailto:webmaster@fysh.org Fyshy Webmaster].<br />
<br />
= User Pages =<br />
Any user with an account on [[river.fysh.org]] can create a page on the [http://en.wikipedia.org/wiki/World_wide_web WWW] simply by placing the requisite files in the directory ''/var/www/user/username/public_html/'' (where 'username' is theirs). This will yield a URL of http://www.fysh.org/~username/ for example. If you find that this directory does not yet exist, or file permissions deny you access, please contact [mailto:webmaster@fysh.org The Fyshy Webmaster] to correct this.<br />
<br />
Please note that /var/www/user on [[river.fysh.org]] is NFS mounted on [[www.fysh.org]] '''read-only'''. Web scripts and CGIs ''cannot'' write to any files. Thus any web page that needs to store data will have to utilise a [[Databases|Database]]. In some circumstances we will set up a separate area for a user to place web pages that absolutely ''require'' file write access, but we would much rather they use a proper [[Databases|Database]] instead. '''Furthermore''' files located in /var/www/user on [[river.fysh.org]] will appear under /home on [[www.fysh.org]]. This is to satisfy the requirements of SuExec when CGIs are run from a /~username/ URL. Be sure to write any scripts/CGIs to take account of this different path! It is noted that at this time you can in fact use the same /var/www/user/... path on [[stickelback.fysh.org]] as on [[river.fysh.org]], as a side effect of river:/var/www being mounted onto stickelback:/var/www, but this should not be relied upon.<br />
<br />
= User Domains =<br />
Additionally we will host the web pages for any domain that a user owns or controls and can get the DNS changed to point to our web server. Please note that our preference is that you use a CNAME record pointing to "www.fysh.org." (the trailing dot is important, don't miss it out!), as this allows us to move your domain webhosting about if we need to. The one practical exception to this is where you use solely the domain name for your URL, i.e. if your domain was 'example.com' and you wanted to use http://example.com/ instead of, or as well as, http://www.example.com/. You can't have a CNAME record and any other records for the same name, and a domain itself requires at least an SOA record. Thus in this configuration you would have to use an A record pointing to the same IP as www.fysh.org.<br />
<br />
Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] to enquire about setting this up. Note that you should ensure that you have a valid and working webmaster email account under your own domain when this is set up.<br />
<br />
The same caveat as for User Pages about '''read-only''' file access will apply.<br />
<br />
If you wish URLs of the form http://www.example.com/~username/ (but for your domain) to be served from /var/www/user/username/public_html/ (as is set up for www.fysh.org URLs) please let us know and we'll adjust the configuration for your domain. By default we assume you don't want just any Fysh.Org user to be able to utilise such a URL. We can configure things such that only certain users have URLs of this form working for your domain, so let us know which users to allow. As always it's the [mailto:webmaster@fysh.org Fyshy Webmaster] you need to contact for this.<br />
<br />
''NB: The following about HTTPS and need for additional IP per domain is no longer strictly true. See [http://en.wikipedia.org/wiki/Server_Name_Indication Server Name Indication] and its limitations (mostly to do with rather old browsers and operating systems).''<br />
<br />
Note that whilst we can offer HTTPS access to your domain it will be with the *.fysh.org wild-card certificate we use for www.fysh.org<ref>see [[Fysh SSL Certificates]]</ref>, and as such any user accessing your domain's web page via HTTPS will get a warning about the hostname mismatch. This is because an SSL certificate for HTTPS is tied to the hostname that the IP the server runs on resolves to. We only have the one IP to use for web services and thus only one possible HTTPS certificate.<br />
<br />
If you have a desperate need to run a domain with HTTPS on the www.fysh.org server under your own domain and need to have your own distinct certificate then we may be able to arrange for additional IP(s) as needed. Note however that we will have to pass on to you the charges of our hosting provider for this, which may entail you needing to pay such to us for an entire year up front.<br />
<br />
= Dynamic Content =<br />
Going beyond simple static content we support use of [http://en.wikipedia.org/wiki/Common_gateway_interface CGI]s, written using any of the installed development languages, and PHP scripts directly via an Apache module.<br />
<br />
== PHP Configuration ==<br />
Our default PHP configuration is somewhat paranoid and as a result you may find you need to adjust some PHP settings, either by getting [mailto:webmaster@fysh.org us] to change the central config files, or by you making use of a .htaccess or .user.ini file as appropriate.<br />
<br />
=== Current Version ===<br />
We currently primarily support version 7.0.x of PHP as supplied by Debian 'stable'. Files with extensions of .php3, .php4, .php7, .php, .pht or .phtml are all handled by this version for web use, and bare 'php' on the command-line will use 7.0.x.<br />
<br />
=== Legacy Version ===<br />
After the upgrade to Debian Stretch we currently still have version 5.6.x available, but these packages are from the older Debian Jessie and will be entirely removed no later than 11th May 2018. Please ensure any of your code is updated to use PHP 7.0 by this date. Currently to use 5.6.x you'll need to either rename files to have a .php5 extension (for web use), or call them with the php5 program if on the command-line.<br />
<br />
=== Obsolete (and Removed) Versions ===<br />
PHP4 hasn't been supported for a long time as the PHP developers themselves dropped all support for it as of the end of 2007[http://www.php.net/downloads.php#v4].<br />
<br />
== CGIs ==<br />
In the case of CGIs there are two ways to ensure the file will run. Either the filename will have to end with .cgi or .pl, or the file must be inside the correct directory. For any domain we host this is its /cgi-bin/ directory. For user pages this will be river:/var/www/user/username/public_html/cgi-bin/ for the URL http://www.fysh.org/~username/cgi-bin/filename. Files outside that directory whose name have no extension or other than .cgi or .pl will have their contents displayed instead of being run.<br />
<br />
'''NB: river:/var/www/user is mounted on [[stickelback.fysh.org]] as /home'''. This means that whilst you will want to create a /~username/cgi-bin/file.cgi as river:/var/www/user/username/public_html/cgi-bin/file.cgi it will appear on [[stickelback.fysh.org]] as /home/username/public_html/cgi-bin/file.cgi and any reference within it to itself or associated files should use that /home/username/... path, ''not'' the /var/www/user/... one!<br />
<br />
Things wouldn't be so complicated if Apache's SuExec didn't treat /~username/ URLs as a special case (as we had everything under /var/www and had hoped that would be enough, but instead SuExec insists on checking the CGI is under the user's home directory AND has 'public_html' in its path). We could compile a local copy of SuExec, but we have done so in the past for other reasons and it leads to maintainability problems when the Debian Apache2 package is updated.<br />
<br />
We will not simply mount river:/home as www:/home, even read-only, as this potentially allows web access to all of everyone's world-readable files, and if [[stickelback.fysh.org]], but not [[river.fysh.org]], is ever compromised the hacker would have full access (even if read-only).<br />
<br />
= Sending EMail from WWW Scripts =<br />
You may send email from scripts on the WWW host, but there are some caveats.<br />
<br />
The 'envelope from' address will always be www-data@stickelback.fysh.org, even if you set a 'From: ' header in the email itself. In April 2016 configuration was adjusted so this should no longer cause issues with remote MTAs trying to perform anti-spam checks.<br />
<br />
If any email sent in this way bounces or otherwise fails the notification will come to the [mailto:webmaster@fysh.org Fyshy Webmaster], not to any email address you set in the 'From: ' header.<br />
<br />
= References =<br />
<references/><br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2024Mail2018-02-11T11:38:51Z<p>Athan: /* Webmail */ Roundcube alternative now in testing</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with out IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. It uses the squirrelmail.fysh.org certificate listed there.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
<br />
----<br />
<br />
<br />
Since we upgraded to Debian Stretch in February 2018 we now also have an alternative available in 'roundcube'. As of 2018-02-11 this is an experimental installation to be sure it meets our needs. One advantage of this is that it can cater to both mail.fysh.org and sanemail.fysh.org use.<br />
<br />
{|border="1" cellspacing="0" cellpadding="5" align="center<br />
| URL<br />
| https://www.fysh.org/roundcube/<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
| Server<br />
| Select the entry containing either sanemail.fysh.org (if you know your setup supports it) or mail.fysh.org as appropriate (see [[Mail#IMAP]]).<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under pond:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=AUP&diff=2023AUP2017-03-09T00:29:44Z<p>Athan: /* Fysh.Org Acceptable Usage Policy */ Not hosted in the USA</p>
<hr />
<div>= Fysh.Org Acceptable Usage Policy =<br />
We're fairly easy going about use of Fysh.Org services, placing as few restrictions as possible upon our users.<br />
<br />
Having said that realise that we are required to comply with the relevant policies of our service provider, [https://www.ovh.co.uk/ OVH], and by extension any user of Fysh.Org services needs to ensure they also comply with those conditions.<br />
<br />
Also for the record we explicitly state that any use of Fysh.Org services should not constitute criminal activity under UK or EU law, or any laws of the state of residence of the user.<br />
<br />
If contacted by law enforcement services or other authorised bodies about activity associated with Fysh.Org services we will co-operate as fully as UK law requires and allows (i.e. we will comply with the UK Data Protection Act).<br />
<br />
As we're not currently hosted in the USA users should not have to worry about the US DMCA.<br />
<br />
== Contactability of Users ==<br />
All users should ensure that email sent to <your username>@fysh.org from at least root@fysh.org is deliverable and read at least once a week. This means ''not'' setting up anti-spam measures that will cause such email to be rejected or thrown away without being read.<br />
<br />
Ideally you will also have this address, or an alternate one, subscribed to the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist so as to receive important announcements about Fysh.Org services.<br />
<br />
== Spam ==<br />
We explicitly do NOT allow any use of Fysh.Org services to send spam, unsolicited commercial email, or other bulk email. If you feel you have a legitimate reason to carry out such activity through our services then please first discuss it with [mailto:postmaster@fysh.org us]. Any failure to do so prior to such activity will result in an account suspension upon discovery of the activity whilst we investigate and we reserve the right to terminate the offending account if deemed by us to be appropriate after such investigation.<br />
<br />
Note that this does also mean that care should be taken not to create a vulnerability where a 3rd party may cause spam to be sent via our services, i.e. be careful of installing 'mail form' scripts in web space.<br />
<br />
== Authorised Access ==<br />
The only person authorised by us to access any Fysh.Org account in any way, shape or form is the person who paid for the account (where applicable) or who was directly given the access details (where payment is not applicable) by us.<br />
<br />
Any unauthorised access may lead to account closure when it is discovered.<br />
<br />
In short, please don't give out your login details to anyone who 'just needs to borrow an account'. It is probably acceptable to share login details temporarily (i.e. change your password afterwards) with someone you know well and trust for the purpose of them helping you with a problem, but doing so is entirely at your own risk.</div>Athanhttps://www.fysh.org/wiki/index.php?title=AUP&diff=2022AUP2017-03-09T00:28:41Z<p>Athan: /* Fysh.Org Acceptable Usage Policy */ Provider is OVH, not Redstation</p>
<hr />
<div>= Fysh.Org Acceptable Usage Policy =<br />
We're fairly easy going about use of Fysh.Org services, placing as few restrictions as possible upon our users.<br />
<br />
Having said that realise that we are required to comply with the relevant policies of our service provider, [https://www.ovh.co.uk/ OVH], and by extension any user of Fysh.Org services needs to ensure they also comply with those conditions.<br />
<br />
Also for the record we explicitly state that any use of Fysh.Org services should not constitute criminal activity under UK or EU law, or any laws of the state of residence of the user.<br />
<br />
If contacted by law enforcement services or other authorised bodies about activity associated with Fysh.Org services we will co-operate as fully as UK law requires and allows (i.e. we will comply with the UK Data Protection Act).<br />
<br />
As we're once more hosted in the UK users should no longer have to worry about the US DMCA.<br />
<br />
== Contactability of Users ==<br />
All users should ensure that email sent to <your username>@fysh.org from at least root@fysh.org is deliverable and read at least once a week. This means ''not'' setting up anti-spam measures that will cause such email to be rejected or thrown away without being read.<br />
<br />
Ideally you will also have this address, or an alternate one, subscribed to the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist so as to receive important announcements about Fysh.Org services.<br />
<br />
== Spam ==<br />
We explicitly do NOT allow any use of Fysh.Org services to send spam, unsolicited commercial email, or other bulk email. If you feel you have a legitimate reason to carry out such activity through our services then please first discuss it with [mailto:postmaster@fysh.org us]. Any failure to do so prior to such activity will result in an account suspension upon discovery of the activity whilst we investigate and we reserve the right to terminate the offending account if deemed by us to be appropriate after such investigation.<br />
<br />
Note that this does also mean that care should be taken not to create a vulnerability where a 3rd party may cause spam to be sent via our services, i.e. be careful of installing 'mail form' scripts in web space.<br />
<br />
== Authorised Access ==<br />
The only person authorised by us to access any Fysh.Org account in any way, shape or form is the person who paid for the account (where applicable) or who was directly given the access details (where payment is not applicable) by us.<br />
<br />
Any unauthorised access may lead to account closure when it is discovered.<br />
<br />
In short, please don't give out your login details to anyone who 'just needs to borrow an account'. It is probably acceptable to share login details temporarily (i.e. change your password afterwards) with someone you know well and trust for the purpose of them helping you with a problem, but doing so is entirely at your own risk.</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2021Mail2016-11-05T17:17:26Z<p>Athan: /* SMTP */ Port 465 will do TLS from the start.</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this. If your client expects TLS right from the start of the connection use port 465. '')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with out IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. Note that this is '''web'''mail and thus uses the www.fysh.org certificate, not the mail.fysh.org certificate.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under pond:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Web&diff=2020Web2016-07-20T12:28:36Z<p>Athan: /* Introduction */ Apache 2.4.x now</p>
<hr />
<div>= Introduction =<br />
[[stickelback.fysh.org]] runs web servers under the primary hostname of [[www.fysh.org]]. We currently utilise [http://httpd.apache.org/ Apache] 2.4.x (as supplied in Debian 'stable') for this purpose, with a single instance running on both port 80 for http access and port 443 for https access.<br />
<br />
We may also run additional instances of apache, or other web server software, on other ports as needed.<br />
<br />
= Webmaster =<br />
All enquiries about [[stickelback.fysh.org]] and other web services should be directed to the [mailto:webmaster@fysh.org Fyshy Webmaster].<br />
<br />
= User Pages =<br />
Any user with an account on [[river.fysh.org]] can create a page on the [http://en.wikipedia.org/wiki/World_wide_web WWW] simply by placing the requisite files in the directory ''/var/www/user/username/public_html/'' (where 'username' is theirs). This will yield a URL of http://www.fysh.org/~username/ for example. If you find that this directory does not yet exist, or file permissions deny you access, please contact [mailto:webmaster@fysh.org The Fyshy Webmaster] to correct this.<br />
<br />
Please note that /var/www/user on [[river.fysh.org]] is NFS mounted on [[www.fysh.org]] '''read-only'''. Web scripts and CGIs ''cannot'' write to any files. Thus any web page that needs to store data will have to utilise a [[Databases|Database]]. In some circumstances we will set up a separate area for a user to place web pages that absolutely ''require'' file write access, but we would much rather they use a proper [[Databases|Database]] instead. '''Furthermore''' files located in /var/www/user on [[river.fysh.org]] will appear under /home on [[www.fysh.org]]. This is to satisfy the requirements of SuExec when CGIs are run from a /~username/ URL. Be sure to write any scripts/CGIs to take account of this different path! It is noted that at this time you can in fact use the same /var/www/user/... path on [[stickelback.fysh.org]] as on [[river.fysh.org]], as a side effect of river:/var/www being mounted onto stickelback:/var/www, but this should not be relied upon.<br />
<br />
= User Domains =<br />
Additionally we will host the web pages for any domain that a user owns or controls and can get the DNS changed to point to our web server. Please note that our preference is that you use a CNAME record pointing to "www.fysh.org." (the trailing dot is important, don't miss it out!), as this allows us to move your domain webhosting about if we need to. The one practical exception to this is where you use solely the domain name for your URL, i.e. if your domain was 'example.com' and you wanted to use http://example.com/ instead of, or as well as, http://www.example.com/. You can't have a CNAME record and any other records for the same name, and a domain itself requires at least an SOA record. Thus in this configuration you would have to use an A record pointing to the same IP as www.fysh.org.<br />
<br />
Contact the [mailto:webmaster@fysh.org Fyshy Webmaster] to enquire about setting this up. Note that you should ensure that you have a valid and working webmaster email account under your own domain when this is set up.<br />
<br />
The same caveat as for User Pages about '''read-only''' file access will apply.<br />
<br />
If you wish URLs of the form http://www.example.com/~username/ (but for your domain) to be served from /var/www/user/username/public_html/ (as is set up for www.fysh.org URLs) please let us know and we'll adjust the configuration for your domain. By default we assume you don't want just any Fysh.Org user to be able to utilise such a URL. We can configure things such that only certain users have URLs of this form working for your domain, so let us know which users to allow. As always it's the [mailto:webmaster@fysh.org Fyshy Webmaster] you need to contact for this.<br />
<br />
''NB: The following about HTTPS and need for additional IP per domain is no longer strictly true. See [http://en.wikipedia.org/wiki/Server_Name_Indication Server Name Indication] and its limitations (mostly to do with rather old browsers and operating systems).''<br />
<br />
Note that whilst we can offer HTTPS access to your domain it will be with the *.fysh.org wild-card certificate we use for www.fysh.org<ref>see [[Fysh SSL Certificates]]</ref>, and as such any user accessing your domain's web page via HTTPS will get a warning about the hostname mismatch. This is because an SSL certificate for HTTPS is tied to the hostname that the IP the server runs on resolves to. We only have the one IP to use for web services and thus only one possible HTTPS certificate.<br />
<br />
If you have a desperate need to run a domain with HTTPS on the www.fysh.org server under your own domain and need to have your own distinct certificate then we may be able to arrange for additional IP(s) as needed. Note however that we will have to pass on to you the charges of our hosting provider for this, which may entail you needing to pay such to us for an entire year up front.<br />
<br />
= Dynamic Content =<br />
Going beyond simple static content we support use of [http://en.wikipedia.org/wiki/Common_gateway_interface CGI]s, written using any of the installed development languages, and PHP scripts directly via an Apache module.<br />
<br />
== PHP Configuration ==<br />
We currently only provide and support version 5.4.4 of PHP, as supplied by Debian 'stable'. PHP4 is no longer supported as the PHP developers themselves dropped all support for it as of the end of 2007[http://www.php.net/downloads.php#v4].<br />
<br />
Our default PHP configuration is somewhat paranoid and as a result you may find you need to adjust some PHP settings, either by getting [mailto:webmaster@fysh.org us] to change the central config files, or by you making use of a .htaccess file as appropriate.<br />
<br />
== CGIs ==<br />
In the case of CGIs there are two ways to ensure the file will run. Either the filename will have to end with .cgi or .pl, or the file must be inside the correct directory. For any domain we host this is its /cgi-bin/ directory. For user pages this will be river:/var/www/user/username/public_html/cgi-bin/ for the URL http://www.fysh.org/~username/cgi-bin/filename. Files outside that directory whose name have no extension or other than .cgi or .pl will have their contents displayed instead of being run.<br />
<br />
'''NB: river:/var/www/user is mounted on [[stickelback.fysh.org]] as /home'''. This means that whilst you will want to create a /~username/cgi-bin/file.cgi as river:/var/www/user/username/public_html/cgi-bin/file.cgi it will appear on [[stickelback.fysh.org]] as /home/username/public_html/cgi-bin/file.cgi and any reference within it to itself or associated files should use that /home/username/... path, ''not'' the /var/www/user/... one!<br />
<br />
Things wouldn't be so complicated if Apache's SuExec didn't treat /~username/ URLs as a special case (as we had everything under /var/www and had hoped that would be enough, but instead SuExec insists on checking the CGI is under the user's home directory AND has 'public_html' in its path). We could compile a local copy of SuExec, but we have done so in the past for other reasons and it leads to maintainability problems when the Debian Apache2 package is updated.<br />
<br />
We will not simply mount river:/home as www:/home, even read-only, as this potentially allows web access to all of everyone's world-readable files, and if [[stickelback.fysh.org]], but not [[river.fysh.org]], is ever compromised the hacker would have full access (even if read-only).<br />
<br />
= Sending EMail from WWW Scripts =<br />
You may send email from scripts on the WWW host, but there are some caveats.<br />
<br />
The 'envelope from' address will always be www-data@stickelback.fysh.org, even if you set a 'From: ' header in the email itself. In April 2016 configuration was adjusted so this should no longer cause issues with remote MTAs trying to perform anti-spam checks.<br />
<br />
If any email sent in this way bounces or otherwise fails the notification will come to the [mailto:webmaster@fysh.org Fyshy Webmaster], not to any email address you set in the 'From: ' header.<br />
<br />
= References =<br />
<references/><br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Main_Page&diff=2019Main Page2016-04-30T12:34:52Z<p>Athan: /* Contacting Admins During Downtime */ Only list Athan OOB contact methods that will work.</p>
<hr />
<div>== Introduction ==<br />
Welcome to the Fyshy Wiki. The initial intent of this Wiki is to document the various services available on the Fysh.Org [[Main_Page#Servers|servers]].<br />
<br />
== Servers ==<br />
The principal server for Fysh.Org services is [[river.fysh.org]]. Amongst other things it provides [[Mail|email]] and [[Shell Accounts|shell accounts]] to all its users. [[DNS|DNS / Domains]] service also resides on [[river.fysh.org]].<br />
<br />
[[Web|Web hosting]] is on the separate machine [[stickelback.fysh.org]].<br />
<br />
There are also a few other machines that various [[Fysh]] use to provide peripheral services, such as [[SSH#ssh.fysh.org|ssh.fysh.org]].<br />
<br />
Fysh.Org services moved again on 29th June 2013. The new shell machine is [[river.fysh.org]]. The old machines [[lake.fysh.org]] and [[guppy.fysh.org]] ''no longer exist''.<br />
<br />
Note that the old UK2-hosted servers, [[bowl.fysh.org]] and [[www.fysh.org]] ''no longer exist''. We took them down on 17th April 2008, a few days before UK2 were due to turn them off, so as to securely erase all user data on them before we lost access.<br />
<br />
The panix-hosted servers [[pond.fysh.org]] and [[fyn.fysh.org]] ceased to exist mid-March 2010 after our move to Redstation.<br />
<br />
== Services ==<br />
A list of all our service is available [[:Category:Services|here]].<br />
<br />
== Weekends are At-Risk time for Reboots ==<br />
<br />
All UK weekend hours should be considered 'at risk' for unannounced reboots. There may be as little as 30 or even 15 minutes warning of a quick reboot, usually for a new kernel.<br />
<br />
In general such a reboot would take less than 5 minutes, that being the extent of the disruption to service. There is usually no reason for more than one such reboot in any given weekend.<br />
<br />
If planned work is intended to take longer than this then it will be announced at least 3 days in advance, preferably a week. Such announcement would be on the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist, as outlined below. Obviously if there is some pressing security concern work then quick reboots, or extended work, may be carried out at any time without warning.<br />
<br />
You may want to take note of details in the section [[Main_Page#Contacting Admins During Downtime|Contacting Admins During Downtime]] below in case of extended outage.<br />
<br />
== Notification of Service Changes ==<br />
If you wish to know about any changes to Fysh.Org services, which may include scheduled outages, you should be subscribed to the [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce] maillist. We will automatically subscribe any new accounts to this list. Subscriptions require confirmation from the given email address as well as our approval. Anyone may subscribe, but if we don't recognise the email address as being that of one of our users we may check who it is first, or just deny it, so it's best to also email the [mailto:postmaster@fysh.org Fyshy Postmaster] so we know who you are.<br />
<br />
By default when you login to a [[river.fysh.org]] [[Shell Accounts|shell account]] you will be shown any new announcements since your last login. If you wish to disable this simply issue this command at a shell prompt on [[river.fysh.org]]:<br />
<br />
touch $HOME/.no_announce<br />
<br />
and to enable it once more if needed:<br />
<br />
rm $HOME/.no_announce<br />
<br />
Note that the file $HOME/.last_announce is used to keep track of when you were last shown new announces. Removing this will cause you to be shown ''every'' existing announcement on next login! Which announces to show you are determined by comparing the 'last update' timestamp on this file against the timestamp on the files holding the announces. So if you change that timestamp on $HOME/.last_announce you may see more, or less, announces than you should.<br />
<br />
== Discussion of Fysh.Org Services ==<br />
If you wish to generally discuss Fysh.Org services you may use the [http://www.fysh.org/mailman/listinfo/Fysh-Discussion Fysh-Discussion] maillist, but note we do not automatically subscribe anyone to this which means you may have to subscribe yourself first and may find that not many people are reading the list itself.<br />
<br />
Alternatively any queries about the Services may be directed to the [mailto:root@fysh.org Fyshy Admin] or you may edit [[Fyshy Suggestions]] here on this wiki.<br />
<br />
== Contacting Admins During Downtime ==<br />
If Fysh.Org services are down and you need to get in touch with an admin then try the following email addresses. Note these aren't guaranteed to be up to date, or even checked routinely. But it's better than only having Fysh.Org-reliant contact details for us:<br />
<br />
{| border="1"<br />
|+ Alternative Fysh.Org Admin contact details<br />
|-<br />
! Admin<br />
! Contact Via<br />
|-<br />
| Athan<br />
| Email: Suisanahta (at) gmail (dot) com<br />
<br />
Twitter: AthanSpod (tag #fyshorg on relevant posts)<br />
|}</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2018Mail2016-04-29T10:15:49Z<p>Athan: /* Realtime Black Lists */ Linkify 'Zen'</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this.'')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with out IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. Note that this is '''web'''mail and thus uses the www.fysh.org certificate, not the mail.fysh.org certificate.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] [https://www.spamhaus.org/zen/ Zen] ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under pond:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athanhttps://www.fysh.org/wiki/index.php?title=Mail&diff=2017Mail2016-04-29T10:14:49Z<p>Athan: /* Realtime Black Lists */ Switched to zen.spamhaus.org, and had long since dropped ordb</p>
<hr />
<div>{{Accuracy Disclaimer}}<br />
<!-- Even making this just a H1 with style to center causes it to be a section heading, I don't want that!<br />
<h1 style="text-align: center;">'''Fysh.Org Mail Services'''</h1><br />
--><br />
<br />
== Introduction ==<br />
[[river.fysh.org]] hosts the primary Mail eXchanger (MX) for fysh.org itself and any other domain configured to deliver locally on it. In addition we have agreements with operators of other servers to provide secondary MX capability. If you have a domain that you would like us to host then please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with details for us to configure things correctly.<br />
<br />
We currently use [http://www.exim.org/ exim4] as the [[wikipedia:Mail transfer agent|MTA]] that handles all email. This is configured to allow relaying only for [[Mail#SMTP|authorised users]] and also employs a number of [[Mail#MTA_Counter-Measures|Anti-Spam Countermeasures]] to help protect our users from the deluge that is the norm on the Internet today.<br />
<br />
In addition we provide both POP3 and IMAP access for remote retrieval of email, both of which are accessible using SSL/TLS encryption. Users can opt to use a [[wikipedia:Mail user agent|MUA]] locally through a [[Services/Shell Accounts|shell login]] if they prefer.<br />
<br />
== SMTP ==<br />
mail.fysh.org is the official hostname for our SMTP service. Any domain expected to deliver locally should specify that hostname as its primary MX record. Likewise you should list that hostname in secondary MX records when we're configured to relay for a domain.<br />
<br />
Additionally users may relay outbound email via the same hostname from arbitrary hosts on the internet so long as they make use of SMTP AUTH. '''NB: To use SMTP AUTH you''' ''must'' '''be using an encrypted connection,''' this is an attempt to not have SMTP AUTH passwords sniffed on random WiFi connections and abused to send spam. To use SMTP AUTH you will first have to [https://www.fysh.org/utils/smtppassword.cgi set up a password] specifically for this.<br />
<br />
Configure your Email Client with the following details:<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Mail/SMTP Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 25 (''NB: You may also use the SMTP Submission Port, 587, for this.'')<br />
|-<br />
| Secure Connection<br />
| Enable 'STARTTLS' or 'TLS', preferably at least version 1.2 if given the option.<br />
|-<br />
| Authentication<br />
| Your [[river.fysh.org]] username and the password you [https://www.fysh.org/utils/smtppassword.cgi set up] specifically for this, ''not'' your [[Services/Shell Accounts|shell login]] password. Note that we currently support 'CRAM-MD5', 'PLAIN' and 'LOGIN' authentication against this extra password.<br />
|-<br />
|}<br />
<br />
== Using Maildir instead of mbox ==<br />
Using Maildir format for email is recommended over the default of mbox. You get a separate file per email making deletion/update operations much quicker and can easily setup sub folders for use with IMAP. There is a particular way of doing this that best interacts with out IMAP service. '''Be sure that any local email client you run on [[river.fysh.org]] can deal with Maildir format mailboxes before implementing the changes below'''.<br />
<br />
=== Use Procmail to save in Maildir format ===<br />
The first thing you need is to either create a minimal ~/.procmailrc file containing at least the following, or see [[Mail#Procmail]] for the location of a full example file:<br />
<br />
<pre>MAILDIR=$HOME/Maildir<br />
DEFAULT=$MAILDIR/<br />
</pre><br />
<br />
The key thing here is that you're saving email into ~/Maildir, and any mention of a folder ends with a slash (/). This tells Procmail to save in Maildir format instead of mbox.<br />
<br />
If you have email filtered into separate inboxes using Procmail then for optimal interaction with IMAP you'll want their names to start with a dot (.). So, for instance, you might have:<br />
<br />
<pre>######################################<br />
# The Fysh List<br />
######################################<br />
:0<br />
* ^TOfysh@fysh\.org<br />
.fyshlist/<br />
<br />
:0<br />
* ^FROMfysh-.*@fysh\.org<br />
.fyshlist/<br />
######################################</pre><br />
<br />
Note the leading dot and the trailing slash on the target of each rule. This leads to these folders being seen as sub-folders of INBOX in IMAP.<br />
<br />
=== IMAP Configuration ===<br />
Be sure to use the 'Maildir/' prefix when [[Mail#IMAP|configuring IMAP]].<br />
<br />
== Reading Mail==<br />
=== POP3 ===<br />
mail.fysh.org is the official hostname for our POP3 service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 110 <sup>1</sup><br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available (or SSL if your client only offers that).<br />
|-<br />
| Use Secure Authentication<br />
| OFF <sup>2</sup><br />
|-<br />
|}<br />
<br />
<sup>1</sup> - If port 110 is blocked from your site, then try port 995 instead, with TLS or SSL for 'Secure Connection'.<br />
<br />
<sup>2</sup> - APOP authentication is no longer supported.<br />
<br />
=== IMAP ===<br />
mail.fysh.org is the official hostname for our IMAP service. You should authenticate the connection with your account username and password. '''This service is only available over encrypted connections as of 2016-04-01 (and this is not an April Fool joke)''', so please be sure to enable any 'TLS' (preferred) or 'SSL' (if TLS is not available) options in your client. See [[Fysh SSL Certificates]] for details of the certificate we use.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
! Parameter<br />
! Value<br />
|-<br />
| Server Name<br />
| mail.fysh.org<br />
|-<br />
| Port<br />
| 993<br />
|-<br />
| User Name<br />
| Your [[river.fysh.org]] username <br />
|-<br />
| Password<br />
| Your [[Shell Accounts|shell login]] password, not the SMTP AUTH one.<br />
|-<br />
| Secure Connection<br />
| Use TLS if available, else SSL.<br />
|-<br />
| Use Secure Authentication<br />
| OFF (the server doesn't support this)<br />
|-<br />
| Prefix<br />
| * Leave blank to access your home directory (along with /var/mail/<username>)<br />
* "mail/" to access ~/mail/ (along with /var/mail/<username>)<br />
* "Maildir/" to access ~/Maildir/ (WITHOUT /var/mail/<username>)<br />
|-<br />
|}<br />
<br />
'''NB: Leaving the prefix blank so that IMAP looks for email in your home directory as a whole can lead to slow startup and downloads of email. Only use this if you already have mail stored this way and have some reason to not change this.''' The only reason this is the default on Fysh.Org is historical configuration.<br />
<br />
'''Maildir, IMAP and INBOX''': Due to the historical configuration mentioned above you have to trick IMAP into seeing the ~/Maildir/ as an Inbox. Login to your fysh shell account and do:<br />
<br />
<pre>mkdir ~/Maildir/.Inbox<br />
cd ~/Maildir/.Inbox<br />
ln -s ../cur<br />
ln -s ../new<br />
ln -s ../tmp</pre><br />
<br />
This creates an extra sub-folder called Inbox that links back to the same Maildir directories as INBOX would usually use. Some clients don't like this and you have to use .FakeInbox or the like instead of .Inbox.<br />
<br />
A new (2016-02-29) alternative is to use 'sanemail.fysh.org' instead of just 'mail.fysh.org' as the mail server name. This '''only offers Maildir support, with the inbox as ~/Maildir/, and .-prefixed sub-folders'''. Leave the 'Prefix' blank if using this.<br />
<br />
=== Webmail ===<br />
We offer access to email held on mail.fysh.org via HTTP using [http://www.squirrelmail.org/ Squirrelmail]. The address for this service is [https://squirrelmail.fysh.org/ https://squirrelmail.fysh.org/], or if for some reason you're unable to use encrypted HTTP (HTTPS) use [http://squirrelmail.fysh.org/ http://squirrelmail.fysh.org/] but please be aware that this will cause your username, password, and any email read to be transmitted in plain text over the Internet, allowing for easy interception by malicious third parties.<br />
<br />
See [[Fysh SSL Certificates]] for details of the certificate we use. Note that this is '''web'''mail and thus uses the www.fysh.org certificate, not the mail.fysh.org certificate.<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center"<br />
| URL<br />
| https://squirrelmail.fysh.org/ (preferred) or http://squirrelmail.fysh.org/ if that is unuseable from your location<br />
|-<br />
| Username/Password<br />
| Your [[river.fysh.org]] [[Shell Accounts|shell login]] details.<br />
|-<br />
|}<br />
<br />
=== Reading Mail Locally ===<br />
Many Fysh.Org users prefer to read their email by logging in to [[river.fysh.org]] and then using a Mail User Agent directly there.<br />
<br />
==== Mutt ====<br />
The main supported program for this is Mutt. This supports default mbox style mailboxes along with [[Mail#Using Maildir instead of mbox|Maildir]], and also has built-in IMAP and POP support.<br />
<br />
==== Emacs ====<br />
Users should be able to get Emacs to read mail as well. No configuration advice is offered here.<br />
<br />
==== Alpine (Pine) ====<br />
<br />
Pine has been replaced by Alpine (a symbolic link means if you run 'pine' you'll actually get alpine). Alpine is based on Pine, but is properly open source and doesn't have the "no binary redistribution" licensing problems that affected Pine. <br />
<br />
Alpine does not directly support [[Mail#Using Maildir instead of mbox|Maildir]] format, but you can simply use IMAP instead if needs be. Set 'Inbox Path' to something like (change 'athan' to your username):<br />
<br />
<pre>mail.fysh.org/user=athan</pre><br />
<br />
And the Folder to:<br />
<br />
<pre>Maildir/INBOX</pre><br />
<br />
==== ELM ====<br />
ELM is no longer installed; 'elm' will now run a script to convert your elm aliases and give you a ~/.muttrc that will work, and then run mutt.<br />
<br />
== Maillists ==<br />
fysh.org is available for hosting multi-user maillists using [http://www.gnu.org/software/mailman/index.html MailMan] as the software. There is a [http://www.fysh.org/mailman/listinfo list] of those hosted maillists that allow public viewing of details. Please contact [mailto:postmaster@fysh.org The Fyshy Postmaster] if you wish to request a new list be set up for you. Note that if you have a domain hosted to deliver on mail.fysh.org then we can setup the new maillist within that domain.<br />
<br />
<br />
== Email Spam ==<br />
If you're really unenlightened as to what it is then the [[wikipedia:Email spam|Wikipedia Email Spam]] page may prove useful.<br />
<br />
In short it's unsolicited email being sent either to promote some product or in an attempt to distribute [[wikipedia:Malware|malware]]. So, unsurprisingly, the majority of people using email do not wish to read such emails, and would prefer to never receive them so as to not even have to spend a minimal amount of time deleting them.<br />
<br />
<br />
=== MTA Counter-Measures ===<br />
mail.fysh.org employs some measures at the [[wikipedia:MTA|MTA]] level to reduce the amount of spam that ends up in our users' inboxes. At the time of writing these are applied in the order listed here.<br />
<br />
'''NB: None of these measures are employed if one of the following conditions is true:'''<br />
<br />
*The email is being submitted to us on port 587, the Mail Submission port which ''requires'' all senders to be [[Mail#SMTP|authenticated]].<br />
*The connection is otherwise [[Mail#SMTP|authenticated]] and thus trusted.<br />
*Email being sent by mail.fysh.org itself, i.e. locally generated email.<br />
*Any email addressed to either 'postmaster' or 'abuse' at the target domain. Whilst a lot of spam gets through due to this we feel it is more important to remain contactable on such addresses despite anti-spam measures.<br />
<br />
<br />
==== Mail From Dialup/DSL Hosts ====<br />
We use a list of [[wikipedia:Regular expression|regexes]] which match hostnames known to be used solely for Dialup or ADSL by home users, and block all attempts by such hosts to send email to mail.fysh.org. The premise is that such hosts should be able to send email perfectly well via their ISP's outbound mail server(s) and the most likely reason for them attempting to send email directly to us is that it's a spambot doing so. As tempting as it is we do ''not'' put such regexes as:<br />
<br />
:.*\.cn$<br />
<br />
in that file, i.e. to block all email originating in China, because we realise that our users may wish to correspond with people sending email from anywhere in the world and do not restrict it.<br />
<br />
Any email rejected due to the sending host matching one of the patterns we test on will receive the message 'rejected because we have blacklisted your host (tried using your ISP's email server to smarthost/relay?)' as part of the rejection code.<br />
<br />
''NB: Users with a fysh.org [[Shell Accounts|shell account]] login can still send email to/via mail.fysh.org from a dialup or ADSL connection, provided that they have [[Mail#SMTP|authenticated]] themselves first, which is necessary anyway to relay through mail.fysh.org.''<br />
<br />
==== Realtime Black Lists ====<br />
The single measure we find reduces locally-delivered spam the most is to make use of the [http://www.spamhaus.org/ Spamhaus] Zen ''[[wikipedia:DNSBL|RBL]]''. This is basically a list of hosts known to send email spam, plus all known "dialup" IPs (i.e. home dialup, DSL or Cable connections that should be sending email through their ISP's email server). Whilst there is some risk of false positives in such things in our experience this either never happens with the Zen DNSBL, or at so low a rate as to be unnoticeable.<br />
<br />
If you believe an email that should have been delivered on mail.fysh.org wasn't due to the sending host being erroneously included in an RBL we use please send an email to [mailto:postmaster@fysh.org The Fyshy Postmaster] with full details, preferably the full headers and relevant body of the original email and/or the bounce that was generated.<br />
<br />
==== Sender Verification ====<br />
Anyone sending email should be doing so from an address that email can be successfully sent to. The sole exception to this is the 'null envelope sender' address of <> which is used to avoid bounce loops. Thus we employ a check where we attempt to verify that the envelope-from of incoming email is an address we could send email back to. The results of these checks are cached so that we don't hammer another email server if a single sender occurs repeatedly.<br />
<br />
If the sender is genuine, and correctly configured, this check will cause no problems. If the sender is a spammer and happens to be utilising a from address that isn't valid (they sometimes seem to use a random local part in a valid domain, or simply an email address that is no longer valid) then the email will be rejected.<br />
<br />
We are aware of some of the downsides to this action and have chosen to continue it.<br />
<br />
==== Greylisting ====<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
The other primary anti-spam counter-measure that we employ is called Greylisting.<br />
<br />
Any time an email is sent to a domain delivered on mail.fysh.org it is checked against our [[wikipedia:Greylisting|Greylisting]] lists. The data that is checked is the tuple "<originating IP> <mail from address> <mail to address>" (NB: These are the ''Envelope'' To and From, which aren't necessarily the same as those in the headers of the email itself). If we have never seen email from the tuple before it is put in the ''greylist listing'', with the time and attempts so far recorded, and then a ''temporary'' failure is returned to the sending MTA.<br />
<br />
With well behaved MTAs this is no problem as they will simply retry delivery at intervals, usually shortly after the initial attempt and then at increasingly larger intervals until they finally give up if no successful delivery is achieved.<br />
So, if the remote MTA then retries the email, but we consider it too soon (within 5 minutes) we'll again return a ''temporary'' failure to the sending MTA, note the attempt, the email stays on the ''greylist listing'', bump the attempt count and update the time of last attempt.<br />
<br />
When the MTA subsequently retries at an interval of more than 5 minutes since the last attempt we will see that the item of email is on our ''greylist list'' and allow the email through. Additionally the tuple, delivery attempt count and last delivery time are moved to the ''whitelist list''.<br />
<br />
Alternatively if there is no attempt to retry delivery the tuple will be expired from the ''greylist list'' after 8 hours, removing the data held for the attempt. This is what happens to a great majority of spam attempts, due to them being sent by programs that are not full MTAs such that they never retry deliveries (they just move on to their next target address instead).<br />
<br />
If an email is received where the data matches a tuple already on the ''whitelist list'' we will accept the email immediately, once more updating the delivery count and time of last attempt. So long as an email passes through our system matching a given tuple that tuple will stay on the ''whitelist list'' for 60 days after the last successful delivery. Thus for any tuple that represents email that you regularly receive greylisting will only affect the first email, and probably only its first attempted delivery.<br />
<br />
The effectiveness of greylisting for email that goes through mail.fysh.org can be seen on [http://www.fysh.org/eximstats our email stats page]. Generally 85% or more of email that is initially greylisted is never retried. This is almost certainly spam, spam that we don't have to handle any further.<br />
<br />
'''Problems With Greylisting'''<br />
<br />
There are a few problems with greylisting. The primary one is to do with the way some maillists send out a copy to each list member. Instead of setting the 'mail from address' to the person who sent the email to the list they set this to a unique mail address per recipient, and indeed per email sent through the list. Such email addresses might look like:<br />
<br />
:bugtraq-return-28583-bugtraq=miggy.org @ securityfocus.com<br />
:bugtraq-return-28581-bugtraq=miggy.org @ securityfocus.com<br />
<br />
Obviously in this case the email with the second from address isn't going to match the same tuple as the first and thus will not be passed through due to listing in the ''whitelist list''. If you find you are on such a maillist and the delays caused by greylisting for every email from the list are causing you problems please [mailto:postmaster@fysh.org forward us] the full headers from at least 3 example emails that came from the list so that we can determine which sending host(s) to add to our hardcoded ''whitelisted hosts list''. From then on, unless the sending hosts for the maillist change, any email for that list will skip entirely over our greylisting checks.<br />
<br />
There are also some misconfigured or badly coded MTAs in use which don't retry email at all. In this case you'll again need to [mailto:postmaster@fysh.org inform us] of the IP(s) of the sending host(s) so we can add them to our ''whitelisted hosts list'' for greylisting.<br />
<br />
Of course the other problem is that if we've not seen a tuple for an incoming email yet it will be delayed at least 5 minutes, quite possibly more depending on the retry behaviour of the sending host (they might not retry, after the initial attempt, for something like 30 minutes).<br />
<br />
'''If you really feel this is too onerous for you to bear we can look into making your email skip the greylisting checks entirely. Please email [mailto:postmaster@fysh.org The Fyshy Postmaster] with any requests to have either your entire domain or just a set of full addresses skip greylisting. We can also offer you control of a file for your domain(s) with delivery on fysh.org so that you can control which local parts do not undergo greylist checks.'''<br />
<br />
=== User Counter-Measures ===<br />
There are several ways in which mail.fysh.org users can apply additional filtering to their email to further counter spam, in addition to those measures applied at the MTA level. All of these will require some familiarity with Linux/Unix systems as you'll need to login to your shell account and edit files at the very least.<br />
<br />
==== Procmail ====<br />
Procmail is a general filter for email. As well as deciding to put email matching given criteria into separate folders you can use it to safely forward email, CC it to another person, or return it as if undelivered to the sending address. You can also combine it with anti-spam filters such as Spamassassin and Bogofilter to cut down on the amount of spam you receive.<br />
<br />
Any email delivered to a user on [[river.fysh.org]] will be run through procmail if the user has a .procmailrc file in their home directory. Please see 'man procmail', 'man procmailrc' and 'man procmailex' once logged in for the documentation on how to use it.<br />
<br />
''NB: The procmail 'TO' directive, whilst matching things like To, Cc and even X-Envelope, does not match Exim's Envelope-To header''. This is a header added during final delivery on [[river.fysh.org]] denoting the specific email address to which the email was sent to cause that delivery. Please bear this in mind when constructing rules that need to match emails which you are only receiving due to being BCC'd on them. A prime example of this is any email from [http://www.fysh.org/mailman/listinfo/Fysh-Announce Fysh-Announce].<br />
<br />
<!-- Link to an example .procmailrc for specific fysh.org anti-spam measures--><br />
<!-- This example needs cleaning up and putting within this Wiki, or at least within the web area and linked here--><br />
There is an example usage of procmail available on [[river.fysh.org]] in /etc/skel/procmailrc-example . This should work as-is, so long as you make sure (as detailed in the file) to have ~/Maildir/ already exist as a directory. Simply run the command 'mkdir ~/Maildir/' if needs be. As of 2016-04-01 (and not an April Fool) that file implements:<br />
<br />
# filtering spam into a sub-folder using spamassassin<br />
# a fixup for a quirk to do with the 'From ' (no-colon) email header.<br />
# de-duplication of messages with the exact same Message-ID header (such as if you were CC'd on a message that was also sent to an email list you're on).<br />
# Ensuring proper headers are in place for PGP/GPG encrypted/signed messages.<br />
<br />
Furthermore it has examples for:<br />
<br />
# Temporarily disabling email delivery<br />
# Excepting email from even being run through spamassassin.<br />
# Discarding email without causing a bounce.<br />
# Filtering extra 'me' email addresses into your INBOX.<br />
<br />
==== Exim Filters ====<br />
As we use exim (version 4) for the MTA on mail.fysh.org you can make use of [http://www.exim.org/exim-html-4.50/doc/html/filter_toc.html Exim Filters] in your .forward file.<br />
<br />
==== Spamassassin ====<br />
Spamassassin is an automatable means of determining if an item of email is spam or not. It runs a whole battery oh tests on both the headers and body, including attachments, of a test email, with each test contributing, positively or negatively, to the final score. If the score is over the configured threshold then the email will be marked as spam. This method is useful because it is a lot more powerful than a simple black and white "if this email is in HTML then I'll bin it as spam" approach.<br />
<br />
We do not run Spamassassin at the MTA level on mail.fysh.org, but any user can make use of [[Mail#Procmail|procmail]] to filter their email via it themselves. A section similar to the following in ~/.procmailrc will enable spamassassin filtering of email into a separate folder:<br />
<br />
<pre><br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
#<br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
#<br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
# All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
spam<br />
</pre><br />
<br />
Please be sure to read the Spamassassin documentation ('''man spamassassin''' and the files under pond:/usr/share/doc/spamassassin) before putting a section like this in your ~/.procmailrc file.<br />
<br />
==== Bogofilter ====<br />
Bogofilter is another means of automatically determining if an email is spam. It uses something called [[wikipedia:Bayesian spam filtering|Bayesian filtering]] to attempt to determine if the text of an email is spam or not. [[Mail#Spamassassin|Spamassassin]] actually uses its own implementation of this method of spam determination, but some users find it better to run both Spamassassin ''and'' Bogofilter over all their email, as each will catch some spam emails that the other doesn't.<br />
<br />
Again you'll use a section of your ~/.procmailrc file to enable filtering with bogofilter. Do NOT enable this until you have first trained bogofilter on a good corpus of ham (emails that are not spam) and spam, else it will give effectively random results.<br />
<br />
<pre><br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a command line option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
spam<br />
</pre><br />
<br />
Note that if you do also use Spamassassin in your ~/.procmailrc then the sum of the configuration should be something like:<br />
<br />
<pre><br />
## Any rules to have email skip the spamassassin and bogofilter tests here:<br />
## i.e. uncomment these and edit to taste<br />
### ## Absolutely whitelisted email addresses<br />
### :0 H<br />
### * ^TO(abuse|postmaster)@(fysh\.org|miggy\.org|dsl\.miggy\.org|novusinitium\.org)<br />
### {<br />
### :0<br />
### * ^TO(abuse|postmaster)@fysh\.org<br />
### fysh<br />
### <br />
### :0 B<br />
### * !http://www\.bodybouncer\.com/<br />
### inbox<br />
### }<br />
<br />
## Run through bogofilter<br />
# -p, --passthrough - passthrough.<br />
# -e, --ham-true - in -p mode, exit with code 0 when the mail<br />
# is not spam.<br />
# -u, --update-as-scored - score message as spam or non-spam and register<br />
# accordingly.<br />
:0fw: bogofilter.lock<br />
| bogofilter -e -p -u<br />
<br />
# if bogofilter failed, return the mail to the queue, the MTA will<br />
# retry to deliver it later<br />
## NB: This causes more headaches than it is worth, when combined with<br />
## using 'formail -D' to de-dupe emails. If we return it to the queue<br />
## for later delivery and it hits the de-dupe it'll get dropped as a<br />
## dupe. The filter rule above DOES still pass the original through<br />
## unchanged if bogofilter failed for some reason (at least if it<br />
## doesn't recognise a commandline option), so we're safe to continue.<br />
## At worse we'll not filter some spam. That's as opposed to plain<br />
## losing email due to the bad interaction with de-duping! <br />
#:0e<br />
#{<br />
# EXITCODE=75<br />
#}<br />
<br />
# Pipe the mail through spamassassin (replace 'spamassassin' with<br />
# 'spamc' if you use the spamc/spamd combination)<br />
# <br />
# The condition line ensures that only messages smaller than 250 kB<br />
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam<br />
# isn't bigger than a few k and working with big messages can bring<br />
# SpamAssassin to its knees.<br />
# <br />
# The lock file ensures that only 1 spamassassin invocation happens<br />
# at 1 time, to keep the load down.<br />
#<br />
:0fw: spamassassin.lock<br />
* < 256000<br />
| spamassassin<br />
<br />
## If bogofilter thinks it's spam, save<br />
:0:<br />
* ^X-Bogosity: Yes, tests=bogofilter<br />
{<br />
## Make sure SpamAssassin's filtering learns it as spam<br />
:0c<br />
|formail -I "X-Bogosity" | sa-learn --spam<br />
<br />
## Something we don't filter/recognise, we'll look at it<br />
:0<br />
spam<br />
}<br />
<br />
## All mail tagged as spam (eg. with a score higher than the set<br />
# threshold) is moved to "spam".<br />
:0<br />
* ^X-Spam-Status: Yes<br />
{<br />
## Make sure Bogofilter learns it as spam<br />
:0c<br />
|spamassassin -d | bogofilter --register-spam<br />
<br />
:0<br />
spam<br />
}<br />
</pre><br />
<br />
This ensures that any email which bogofilter thinks is spam also gets fed to spamassassin's bayesian filtering to learn as spam.<br />
<br />
To train bogofilter initially you should, at the very least, teach it what your non-spam email looks like (''the -M option assumes the email folder is in mbox format''):<br />
<br />
bogofilter -M -n /var/mail/${USER}<br />
<br />
or adjust the name of the mailbox you feed it to suit where you've saved a significant amount of non-spam email. Likewise if you have some saved spam email you can train it on that:<br />
<br />
bogofilter -M -s ~/Mail/spam<br />
<br />
==== Correcting Spamassassin and/or Bogofilter Registration of Email ====<br />
If your ~/.procmailrc rules cause spamassassin and/or bogofilter to mistakenly flag an email as spam you should be sure you re-train them on it to decrease the likelihood of them making the same mistake again. An example of doing this for the MUA Mutt would be the following in ~/.muttrc:<br />
<br />
<pre><br />
## spam<br />
# Register as Spam, unregister as Ham (non-Spam)<br />
#<br />
macro index S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\<br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<untag-pattern>.*\n\<br />
<enter-command>set wait_key\n"<br />
macro pager S "<enter-command>unset wait_key\n\<br />
<tag-entry>\<br />
<tag-prefix>\<br />
<clear-flag>N\<br />
<tag-prefix>\<br />
<pipe-entry>/usr/bin/spamassassin -r\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-spam --unregister-nonspam -v\n\ <br />
<tag-prefix>\<br />
<save-message>=savedspam\n\<br />
<enter-command>set wait_key\n"<br />
# Register as Ham (non-spam), unregister as Spam<br />
macro index H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
macro pager H "<enter-command>unset wait_key\n\<br />
<pipe-entry>/usr/bin/sa-learn --ham --no-sync\n\<br />
<pipe-entry>spamassassin -d | bogofilter --register-ham --unregister-spam -v\n\<br />
<enter-command>set wait_key\n"<br />
</pre><br />
<br />
This allows you to press H (capital H, not lowercase h) to have mutt tell both spamassassin and bogofilter that the current email, in the mail index or pager, is Ham, not Spam. Likewise it also enables you, by pressing S (capital S), to re-train an actual Spam email that got past your filtering as Spam instead of Ham.<br />
<br />
==== Per-site email aliases ====<br />
<br />
Note by 'per-site' here we mean this is a convenient way to give a unique email address to any web sites you might want to sign up to (so you know who leaked your address to spammers if they do).<br />
<br />
For any email set to deliver on [[mail.fysh.org]] a user can utilise a feature that allows them to define almost limitless email addresses. Basically if your user account is 'foo' then you can use an email address of the form 'foo-bar@fysh.org' and it will be delivered as if to 'foo@fysh.org'. The 'bar' may be any legal text for the left-hand-side of an email address. '''NB: This only works for email directly to an @fysh.org address, not other domains delivered on mail.fysh.org. This is because it explicitly breaks any aliases that have a hyphen in them. We can control that for fysh.org, but users may be unaware of this for their own domains and it wouldn't be easy to figure out why things aren't working. They can, of course, just create whatever aliases they want for their own domain anyway.'''<br />
<br />
=== Sending Spam from mail.fysh.org ===<br />
It '''should''' go without saying, but we'll say it here anyway, that sending any form of email spam from mail.fysh.org is strictly prohibited. Upon discovery, and we ''will'' discover such activity, your account will be closed forthwith pending investigation.<br />
<br />
In the past we have even rejected requests to use our service to send out mass-marketing emails to opt-in lists, not having enough information to be 100% sure the list was opt-in and 100% up to date.<br />
<br />
Note that this does mean that care should be taken with any scripts you have for sending email out to multiple people, as well as any scripts you place in your [[Web|web space]] that could possibly be insecure and allow for sending of email to arbitrary addresses.<br />
<br />
<br />
== Known Problems & Workarounds ==<br />
=== 'procmail -d' introducing >From header ===<br />
There is a known bug<sup>1</sup> with the version of procmail in Debian/Etch as of 2007-08-29, specifically version 3.22-16. We '''no longer use''' 'procmail -d <user>' to deliver email to users locally on [[pond.fysh.org]], ''thus this bug will only affect users who are specifically delivering mail in this way for some reason''.<br />
<br />
Use of the -d flag adds an extra header of the form '>From <user> <Date and time>'. Unfortunately because this header has the '>' character at the start it can confuse other programs. Ironically this includes procmail itself!<br />
<br />
If you aren't using a ~/.procmailrc file at all then you shouldn't be affected, email will simply get delivered into your /var/mail/<user> file. If you ''are'' using a ~/.procmailrc then there is a simple workaround. Simply add this recipe as the very first one in your ~/.procmailrc:<br />
<br />
<pre><br />
## 'procmail -d' is buggy, it adds a '>From <user> ...' header which<br />
## can confuse other programs (due to it starting with >).<br />
## So, let's filter that<br />
:0fHw<br />
| sed -r -e 's/^>From (.*)/Old-From_: \1/;'<br />
</pre><br />
<br />
<sup>1</sup> - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=128160<br />
<br />
[[Category:Services]]</div>Athan