cmf – configure mail fetching

Provide a centralized point of fetching mail from a remote mail server
for all local LAN users with an external mail account

Copyright (C) 2004 DFG/M.Watermann, D-10247 Berlin, FRG
All rights reserved
EMail: Support@MWat.De

Table of Contents:

Overview  

Quite a lot of small LAN installations (e.g. at lawyers, physicians, SOHOs, travel agencies etc.) I've seen over the years where the users don't have an internal accesspoint for their emails incoming from the Internet; almost always a consequence of their (on demand) dialup internet connection. While there might exist an internal mailserver (for intranet mail), their emails from "outside" (i.e. somewhere in the Internet) end up somewhere else (their ISP, a freemail provider). To actually read their mails each of them has to setup her/his own email program to pick up the mails from whereever they happen to be, which in turn means, that the firewall / Internet gateway must be set up to allow POP/IMAP traffic from all intranet hosts to the outside world – which might turn out as a potential security risk at least as long as there are still some WinDOS hosts around.

A better way for both the users and the LAN admin would be, if the mail fetching could be centralized on one machine (possibly the gateway system). All users would pick up their mails at a host within the LAN. And the firewall could be set up in a more restrictive way. Although this problem seemed quite common to me I couldn't find a ready-to-use solution on the net (newsgroups, web-pages). So I came up with my own. – But please note that we're talking here on about the receiving part of email communications. This script has nothing to with the sending of emails; this is completely up to the LANs postmaster to setup so I won't discuss that here any more.

[up to Table of Contents]

How it works

From the users point of view the cmf script is just a little web-interface which allows them to enter, change or delete a mail fetching job. "Mail fetching" means, that "someone" fetches their incoming mails from their ISP (or freemailer or ...) and forwards them to a mailserver within their own LAN. Here they can read the mail locally with their preferred email-program.

The visual appearance of the scripts web-interface (i.e. the HTML pages and forms) is completely configurable: Both the design and the wording can be customized by the administrator, the latter be editing – or adding – a language file holding all text/strings used by the pages, the former by editing a CSS files to be used by the browser when rendering the pages.

Internally the cmf script uses a private database where it stores all the user-related data. Every time this database is changed by use of the web-interface the script checks whether there are active jobs. If this is the case the relevant database fields are converted to an other file called .fetchmailrc, the latter being a configuration file for the fetchmail utility by Eric S. Raymond, and a cronjob is set up to run that tool to look for new emails on a regular base and forwarding them to the respective intranet user accounts.

What I called a "private database" is in fact just a textfile which is read from and written to by help of the systems awk utility; so there's no need to install additional software to run this script. And since the users are changing their settings by means of a Web-form there (on the users machine) as well isn't any additional software needed apart from a web-browser (which may be a graphical one or a text-based). Actually, one of my goals was to keep the requirements as low as possible. – The flow of data would look like this:

        User "visible"           "internal" processing
       ----------------         -----------------------
  user (browser) ---> cmf (intranet-webserver)  <-->  database file
  ^  ^                |   |
  |  |                |   |                           + <--> ISP-1
  |  +--- web-form ---+   + ---> .fetchmailrc         | <--> ISP-2
  |                                    |              |   ...
  +-->  LAN mailhost  <---   fetchmail (crond)  <-->  + <--> ISP-n

So the user has to setup – once – her mail fetching using this scripts web-interface and can from then on read her incoming emails at the "LAN mailhost". The latter may be a POP/IMAP server within the LAN or just the host where the she happens to have her local account. It would be even possible to skip the users step of registering her mail fetching by delegating that job to the LANs administrator.

[up to Table of Contents]

Considerations

There are several issues to deal with when implementing and running this script, most of which related to privacy/security in one way or another:

To address these points I came up with a solution internally using two scripts, a small CGI stub (which is called by the users web-browser) and this script which is called by that CGI script using the systems sudo facility (see section Installation / Setup below). A separate account with no special privileges and no login shell at all is used as the owner of all files involved (see section Files below). Also on behalf of this account the fetchmail utility is run.

The database file used internally does not require any database management system (DBMS) to be installed. Generally it's designed as a flat ASCII file consisting of several lines each of which representing a data record holding a number of data fields. The GNU/Linux awk utility is used to read/write that file. – Surely such a solution would become kind of slow if there were tens of thousands of data records to handle. But a company or organization using this script for that many users should reconsider its point of view: forwarding mail in such a setup is not really a good idea. A central mail-server (a combination of SMTP and POP/IMAP servers) would be a better solution for all involved (users, admins and programs/scripts). This script, in contrast, may be used in LAN environments with some tens of users.

[up to Table of Contents]

The Web Interface

The web-interface (i.e. the web-pages sent to the user) consists of only some two or three forms. The first form requests the user to enter three values:

  1. The name of the account at the remote mail server that is to be used to read the incoming mails;
  2. the password at the remote server used to authenticate (along with the username);
  3. the name of the remote mail server that is the host where to pick up the incoming mail;
  4. Finally the user has to decide whether she wants to add a new mail fetching job, change an already existing one or delete one altogether.

The user input is then looked up by the script in its database. It's also checked whether an entry already exists that should be added or does not exist although the user selected to change it. The user then gets a second form (possibly with some err-texts pointing out missing or invalid fields) with three additional fields:

  1. A box to select a protocol to use when fetching the mail (such as POP or IMAP); this are the protocols supported by the fetchmail utility;
  2. the name of the local LAN account which is supposed to receive the fetched and forwarded emails;
  3. a checkbox indicating whether or not this fetching is active, where "inactive" means that the user data are still in the scripts database but not passed to fetchmail for use.

Below that is shown for informational purposes when and from what host the settings were last changed. Finally selecting the update button causes the script to update its database accordingly and – if necessary – rewrite the .fechmailrc as well and start or stop the cronjob which periodically runs the fetchmail utiltity. – The user finally receives a good-bye page listing her current settings.

That's it! Everthing else is done automatically by cron and the fetchmail utility.

[up to Table of Contents]

Installation / Setup

This script is not a "standalone" solution doing all and everthing but kind of fetchmail addon providing a web-interface for configuring a LAN-wide multiuser mail fetching and forwarding facility as discussed above. As such besides the fetchmail program a web-server (such as Apache) or -proxy (such as WWWoffle) is needed, configured to allow for delivery of CGI generated pages. While not required by this script as such, the localhost (i.e. the host where the script is supposed to run) needs an active SMTP server (such as PostFix) capable to handle all the mails picked up and forwarded by fetchmail. To sum up, you'll need a standard GNU/Linux installation (such as Debian, Fedora or Redhat) including

Presuming that your machine meets this requirements, to successfully run this script perform (as user root) the following steps:

  1. Create an user account with no privileges at all apart from using (i.e. reading/writing) its own home-directory; to be safe, that account should have a login shell of /bin/false and no valid password. Its only purpose is to provide a UID/GID for use by the fetchmail utility and this very script. – An easy way to create such a new account:
        #
        # useradd -s /bin/false cmf-user
        # _
    
    where cmf-user is the name you'd like to use; for more info refer to man useradd.
  2. Copy this script into that users home directory and decompress it performing the following commands:
        #
        # cd ~cmf-user/
        # mv /wherever/you/downloaded/the/script/cmf.gz .
        # gzip -d cmf.gz
        # chown cmf-user:cmf-user cmf
        # chmod 0700 cmf
        # _
    
    This step is quite important since the script uses its own directory to store its files and makes the owner of that directory the owner of those files. This is necessary not only for security reasons but also in order to run fetchmail successfully on behalf of that account.
  3. Let the script check whether all required files are available:
        #
        # cmf --test
        # _
    
    The output of the --test option should look like this:
        +++ "awk" found as "/bin/awk"
        +++ "cat" found as "/bin/cat"
        +++ "chmod" found as "/bin/chmod"
        +++ "chown" found as "/bin/chown"
        +++ "crontab" found as "/usr/bin/crontab"
        +++ "date" found as "/bin/date"
        +++ "fetchmail" found as "/usr/bin/fetchmail"
        +++ "hostname" found as "/bin/hostname"
        +++ "ls" found as "/bin/ls"
        +++ "mv" found as "/bin/mv"
        +++ "pwd" found as "/bin/pwd"
        +++ "ps" found as "/bin/ps"
        +++ "rm" found as "/bin/rm"
        +++ "sed" found as "/bin/sed"
        +++ "sudo" found as "/usr/bin/sudo"
        +++ "cmf" found as "/home/cmf-user/cmf"
        +++ "/home/cmf-user/.cmf.conf" created
        +++ "/home/cmf-user/.cmf.lang.de" created
        +++ "/home/cmf-user/.cmf.lang.en" created
        +++ no user crontab installed for "cmf-user"
        +++ file permissions/ownership OK
    
    If any needed files are missing, check what happened to them and install (see also the Files section below). As far I'm aware of all of them should be installed by default, except perhaps the sudo utility which might need installing it seperately. Also, if you see a message like
        --- there's already a user crontab installed for "cmf-user"!
    
    you know that you're running into trouble because we need this user completely for our mail fetching job (see also the last point, 7, below). – Repeat this step as long as there are any lines with leading "---" (instead of "+++") in the output.
  4. Review the cmf.conf file and check, whether all settings there seem reasonable to you. – Note, that the RC_URILOCAL entry will be automatically updated by the following step, so don't worry about it yet.
  5. Since this script is run by means of a CGI script, there should already exist another user account used only to run the web-server (or a CGI-capable proxy such as WWWOffle); figure out the full path of that web-servers CGI directory and call this script like this:
        #
        # cmf --cgi /path/to/httpd/cgi-bin /cgi-bin
        # _
    
    which creates the starter CGI script /path/to/httpd/cgi-bin/cmf.cgi (see also section Commandline Options below) and should give you a line like:
        +++ updated "/home/cmf-user/.cmf.conf"
    
  6. To make this work, you'll need an entry in the /etc/sudoers file (allowing the web-server/proxy user to call this script) like this one:
        wwwoffle  ALL= NOPASSWD: /home/cmf-user/cmf
    
    You might create such an entry be either using the visudo utility (which is part of the sudo package) or by calling this script:
        #
        # cmf --sudo httpd-account
        # _
    
    where httpd-account is the user account used by your web-server (wwwoffle in the example entry above). – For more information about sudo see man sudo.
  7. While not directly needed for this script to be useable, there remains the step to set up a cronjob for our user (created in step 1) to run the fetchmail utility periodically for actually fetching the mails. But don't worry: the whole magic of maintaining the users crontab is dealt with by this script. It dynamically removes the users crontab (if there are no active users to fetch mail for) and installs it again whenever needed (e.g. a user becomes added or switches her/his setting from disabled to enabled). – You can see the default cronjob definition in the ~/.cmf.config file, entry RC_CRON. – Please note, that it is possible, but not recommended, to use an already existing user account to run this script. If, however, that user happens to have its own cronjobs you'll run into problems: As mentioned above the crontab is dynamically enabled/disabled according to the current user settings. In consequence all other cronjobs of this user are affected (i.e. deleted) as well.

Now you should be able to test the setup by pointing your favourite web-browser (such as e.g. Lynx, Mozilla or Opera) to an address like http://www.your.webserver/cgi-bin/cmf.cgi. This should result in a web-form to show up in your browsers window requesting your authentification/login. Otherwise – i.e. in case of errors – you should check the web-servers logfiles for hints what might have gone wrong. You could try a su -l httpd-user at the commandline prompt and call the CGI stub /path/to/httpd/cgi-bin/cmf.cgi manually to see whether you get a HTML page this way. If that results in an error you probably missed the 6. step (sudo) above.

For information about how to setup fetchmail and your favourite web-server/-proxy please consult their respective documentation and man pages.

[up to Table of Contents]

Commandline Options

Usually – that is in CGI mode – there are no commandline options to worry about: all data to process is streamed via stdin to this script (for details refer to your web-servers CGI documentation). There are, however, some options to print out some information about this script and how to use it as well as two used for initial setup. Please note that intentionally there's no loop for commandline parsing, so you can always use only one option per call. They are in alphabetically order:

-c directory localpart or --cgi directory localpart
This option writes the CGI script cmf.cgi – which in turn calls this script – to the given directory. Depending on the httpd used you may or may not have to adjust the ownership of the created CGI script in order to have it run by the web-server. Consult your web-server manual for possible constraints when running CGI scripts. – The localpart refers to the URI the users are supposed to use: so if the full URI would look like e.g. http://your.proxy.domain:8080/cgi-bin/cmf.cgi then the localpart would be /cgi-bin, while the cmf.cgi will be appended automatically. – Please note that no attempt is made to check whether the directory and localpart arguments match since that would involve parsing your web-server or proxy configuration files as well, so just use your administrative knowledge to make sure, they're fitting. Please note as well, that this option modifys the scripts config file ~/.cmf.conf in order to store the localpart which is used/needed to generate all the forms the users get presented when they access this script.
-h or --help
Print out a short usage note and terminate without any further work. (See also the --html and --info options below.)
-H or --html
Print out this text in HTML format, suitable for reading with a HTML-viewer (aka web-browser), and terminate without any further work. (See also the --help and --info options.) [This switch produced what you're reading right now.]
-I or --info
Print out this text in ASCII format, suitable for printing, and terminate without any further work. (See also the --help and --html options above.)
-s account or --sudoers account
This option adds an entry to the /etc/sudoers file allowing the given account – which must be the one running the web-server – to call this script.
-t or --test
This switch checks whether all required files are available, printing out an error message in case one is missing (see also the section Installation / Setup above). Besides that it creates the initial config- and language-files (see section Files below as well).

[up to Table of Contents]

System tools used

Since this script might be running on an exposed host – the internet gateway/firewall – it requires no additional software to be installed apart from the usual system tools, neither JAVA, PERL or PHP etc. are needed, nor any database servers like MySQL; everything is done by GNU/Linux tools available with every recent distro I'm aware of. In detail the following utilities are used by this script and are assumed to be reachable in PATH:

    awk, bash, cat, chmod, chown, crontab, date,
    hostname, mv, pwd, ps, rm, sed, sudo

Although not needed by this script directly, Eric S. Raymond's fetchmail utility has to be installed on the system (see the original fetchmail homepage for details). To check whether all tools are available use the --test switch (see section Commandline Options above)

[up to Table of Contents]

Files

Apart from the system tools listed above the following files are used, all of which are stored in the selected unprivileged users home directory and owned by that user with no access rights for either group or others. Some of this files are used [i]nternally by this script, some are [o]ptional, some [m]andatory; all this files are to be owned and read-/writeable by the user you created for this script (see section Installation / Setup above):

~/.fetchmailrc [i]
The configuration file for the fetchmail utility. This file is automatically (over-)written whenever a user changes (adds/updates) her/his data, so do not modify!
~/cmf [m]
This is, umh, the script itself, obviously ...
~/.cmf.conf [o]
This is the configuration file for this script, it will be created automatically during processing the --test option (see section Commandline options above) and holds the default settings for a couple of options (mostly regarding the fetchmail utility). – If you happen to change the file ownership or permissions, or if this file is removed altogether it will be re-created with the default settings thus reverting all changes you might have done, including the --cgi setting you made during setup! So be carefull ;-)
You'll find the following entries in this file:
RC_CRON
The crontab entry used to run the fetchmail utility (see man crontab for details).
RC_DOMAINNAME
The domain name to be used by fetchmail's "--smtpaddress domain" option; it defaults to the domain of the local host (i.e. the one running this script).
RC_HOSTNAMES
The value to use for fetchmail's "--smtphost hosts" option i.e. the host where the fetched mail is forwarded to; it defaults to the name of the local host (i.e. the one running this script). fetchmail allows a comma delimited list of hosts here, see man fetchmail for details.
RC_LOGFILE
The logfile to be used by fetchmail's "--logfile filename" option. – Note that you must leave this variable empty or unset (by commenting out the respective line) if you want to use I/O-redirection with the fetchmail cronjob.
RC_POSTMASTER
The value to use for fetchmail's "--postmaster name" option; it defaults to postmaster, which must be a working alias on your local host (i.e. the one running this script); see man fetchmail for more information.
RC_URILOCAL
The local part of the URI to access this file via your web-server / -proxy. Please note: this line is updated/rewritten by the --cgi option (see section Commandline Options above).

All fetchmail related settings are written to its config file ~/.fetchmailrc which is automatically generated/maintained by this script.
~/.cmf.css [o]
~/.cmf.css.default [i]
The HTML pages generated by this script use CSS (Cascading Style Sheets) for visual design. The ~/.cmf.css.default is created (but not used otherwise) to show you the default CSS settings used for all HTML pages. In case you don't like these you may create the file ~/.cmf.css with your own favourite styles. If such a file exists and its filemode is 0600 (i.e. "-rw-------") and it is owned by "our" user (see section Installation / Setup above) it will be used instead of the builtin defaults for designing the HTML pages. – Please note that the styles in ~/.cmf.css will get inserted as is (in the HTML "head" section) not used as an external stylesheet; so just write styles to that file but no e.g. "media" selectors or other fancy stuff. – And if you don't have a clue what I'm talking about here, just forget this whole paragraph ;-)
~/.cmf.db [i]
This is the database file used internally to store all user related data. Deleting it will result in unrecoverable loss of all user settings regarding the fetching of their mail! The file is maintained (read/write) solely by this script and must not be changed otherwise; its contents is used to rebuild the ~/.fetchmailrc file.
~/.cmf.dec [i]
A sed script used to decode URL-encoded form-data submitted by the user. You should not modify it, since it's rather important (incl. its internal order) for dealing with the user data; it's automatically (re-)created if you deleted it by accident.
~/.cmf.lang.de [o]
~/.cmf.lang.en [o]
The default language used by all generated pages/forms is english; you can find the text/strings used in the file ~/.cmf.lang.en which is created automatically by the --test option (see the sections Installation / Setup and Commandline Options above). The same applies for the german language file ~/.cmf.lang.de. At runtime the script checks whether the user sent a preferred language setting. If that user setting matches one of the installed language files ("en" aka english and "de" aka german by default) the respective strings are used when generating the HMTL pages sent to the user. If no matching language file exists, the default builtin english strings are used.
You're free to add as many other language files as you like (each of which with its own filename extension specifying the respective language), but please note that – like all other files – this files have to be owned by the user you created in step 1 of the installation and the files must have mode 0600 ("-rw-------"); files not matching that conditions are ignored. – If you happen to succeed in creating additional language files I'd appreciate very much if you could send me a copy (or URL) by email.
~/.cmf.MenAtWork [i]
A temporary file used to avoid problems caused by concurrent access. You shouldn't ever see this file since it's automatically created/deleted only when needed (i.e. the script is reading/writing its files).

[up to Table of Contents]

ChangeLog

    $Log: cmf,v $
    Revision 1.6  2004/08/05 18:51:12  matthias
    * made private files "hidden" dot-files (again);
    * implemented error-page (for serious errors only);
    * completed string localization;
    * added more documentation;

    Revision 1.5  2004/07/31 21:59:09  matthias
    * changed name of script (and all depended files) to cmf to avoid
        confusion with esr's fetchmailconf;
    * form-data encoding is set to application/x-www-form-urlencoded
        explicitely to not rely on browser implementation;
    * renamed internal vars for binaries (OS tools) with a prefix of EXE;
    + implemented localization for all strings in HTML pages
        (files for de and en are created by --test option);

    Revision 1.4  2004/07/29 20:29:02  matthias
    + made CSS used for generated pages user configurable;
    + added handling (creating/removing) of user crontab;
    + added configuration file to allow users tweaking some settings;
    * added stronger permission/ownership tests for our files;

    Revision 1.3  2004/07/28 19:12:54  matthias
    * replaced various global variables by three arrays holding form-data,
        error-messages and missing-flags respectivily;
    + added --test switch, enhanced --cgi option;
    + added lock file handling for concurrent access;
    * added more internal checks for better state- and error-handling;
    * login form now allows for selecting Add/Change/Delete a record;
    * added explicit "tabindex" attribute for navigation to form fields and
        reworked overall HTML structure for use with no-CSS browsers;

    Revision 1.2  2004/07/25 20:25:39  matthias
    + added commandline options and a lot of documentation;
    * implemented error array and seterr() function;

    Revision 1.1  2004/07/17 19:19:33  matthias
    + initial CVS checkin;

[up to Table of Contents]


Disclaimer: No bits or bytes were harmed and no harddisk destroyed in order to create this page.
All letters and digits on this page are strictly virtual and
any resemblance to real letters or digits – monospaced, serif or sans-serif – is purely coincidental.