CheckPassword authentication in Dovecot: a practical example in Shell

In this post I’ll show how to set up authentication in Dovecot via custom CheckPassword script written in Shell. While this is not generally suitable for any kind of production deployment for performance reasons, employing CheckPassword authentication is great for prototyping custom authentication that cannot be done via other available standard drivers.

Disclaimer: This post will not contain any new useful information for Plesk users. Configuration described here is not and will not be used in Plesk. This post is primarily aimed for those who develop some kind of integration with Dovecot.

Dovecot as seen by developer

I thought this post will be a good opportunity to share some thoughts on Dovecot as a software product from developer point of view. The promised example will follow shortly.

After working with Dovecot for quite a while to integrate it into Plesk perfectly, I’ve come to understand that it is a really great piece of software. It is easy to build, easy to configure, and easy to troubleshoot. The source code is sleek and for the most part easy to read and track control flow. If I had to pick three design solutions I liked the most, they would be:

1. The overall architecture of Dovecot regarding process and privilege separation. There’s really not much to add here, you need to feel and experience it for yourself to see how flexible and useful it is. Oh, and separate processes’ (services in terms of Dovecot configuration) permissions are highly configurable — you may set up user and group(s) for them to run under, chroot them, restrict what other processes they are allowed to talk to (via unix socket permissions), limit maximum memory, number of instances and number of client connections. Usually each service is spawned from a separate executable. This also adds to code readability since it makes the code cleaner and less dependent on the code of other services.
2. Highly robust and easy to use memory pools. Dovecot is written in C language. The language requires explicitly allocating and freeing memory. Failure to do so properly usually causes a number of serious problems, e.g. your program may crash or continuously eat more and more memory as it runs (leak memory). Errors in manual memory management are usually relatively hard to pinpoint. Dovecot solves these and other problems I haven’t mentioned here by using memory allocations from custom memory pools pretty much throughout all of its code.
   First of all, while memory pools have a single interface to access them, they may implement and/or enforce different memory management policies. For example, while writing custom authentication module you’ll most likely use only two pools — temporary pool for temporary variables and structures and request pool for allocations within one authentication request. Once you’ve allocated memory from a memory pool, you don’t need to explicitly deallocate it somewhere. Memory will be deallocated at a later stage depending on the pool memory management policy. E.g., allocations from request pools are freed once the request is finished. Scope where allocations from temporary pool remain valid may be controlled by enclosing it into handy T_BEGIN {  and } T_END  macros pair. A number of widely used memory and string management functions (like realloc() , strdup() , and asprintf() ) are also reimplemented to use memory pools. This allows for clean and readable code not cluttered with manual memory management. Using memory pools also serves a security purpose — currently all allocations from memory pools are zeroed out, preventing unsafe use of uninitialized dynamically allocated memory.
   All in all, using memory pools is not new, but it is a great find. It’s not without its disadvantages, however it fits greatly in such software as Dovecot.
3. Error message generation and handling. One of the strengths of Dovecot is its comprehensive error reporting. If something went wrong, log messages would often contain not only enough information to locate the issue reason, but will also provide hints at its resolution. For example, if a certain configuration directive causes troubles, then log message would often contain both reason of failure and point to the exact configuration directive that caused it.
   In C programs you usually have only so much of the context to give a reasonably helpful error message. This can be partially explained by the lack of such commonly used mechanic as exceptions and partially by the desire to keep knowledge about parts of the source code (mostly) unrelated to a given one to a minimum. In Dovecot many functions have const char **error_r  as a last parameter. It is used exclusively to hold error message in case an error occurred. error_r  argument is usually passed to all child functions that accept such parameter as well. That way on each execution stack frame (or abstraction layer, if you will) the error message may be extended with useful information known only on a given frame (or layer). This is a slightly cumbersome approach, yet an extremely simple and a very powerful one.
   However it is relatively rarely seen in other C projects, most probably due to the constant need to either manage one more entity allocated from dynamic memory or bear with limitations of statically allocated storage for error messages. This is where memory pools mentioned above come into play. Thanks to them the code that manages error_r  values becomes very simple and straightforward.
   I think it’s also interesting to highlight the differences and similarities between this approach and classic exceptions approach (which would be used instead in C++, for example). The first important thing to note is that setting error_r  does not signify an error — instead error condition in Dovecot is always indicated by function return value. Once a function call fails, the error together with an error message is usually propagated upwards using proper function return codes until it hits a place where it can either be recovered or reported to user and/or to log. This is very similar to the way exceptions work unwinding the call stack, but is done in an explicit way. Exceptions also may pass arbitrary information from the point of error origin to the point of error handling. Usually this information includes the error message — exactly what is passed via error_r  variable in Dovecot. However since error handling in C is explicit and error_r  function parameter is there in the signature, it is much harder to forget to handle error or extend error message on a given abstraction layer. This arguably results in a better, more detailed and useful error reporting. Of course you need to pay the price of explicit error handling throughout your project for that.

Certainly there are more interesting and useful design decisions and tricks to be gleaned from Dovecot source code. It’s definitely a good read for a C programmer. Be warned though: development and small part of end user documentation over at wiki2.dovecot.org is outdated up to the point when it’s plain wrong. Nec aspera terrent: you may always ask the Dovecot community to help you figure things out.

OK, enough of my ramblings. Let’s get to the example 🙂

Example of CheckPassword authentication via Shell scipt

A practical authentication problem

Let’s say /etc/dovecot/users  has the following content:

[email protected]:1mail
[email protected]:2mail

It will be our authentication database. Each line corresponds to a separate mail account and has the following format: <full username>:<plaintext password> .

All mail accounts will be virtual. That is all of them will be mapped to a single system user named popuser . Mail storage will be in Maildir++ format with maildirs located at /var/qmail/mailnames/<domain>/<mailname>/Maildir , e.g. /var/qmail/mailnames/test2.a10-52-181-149.qa.plesk.ru/mail2/Maildir . Both <domain>  and <mailname>  must be in lowercase, though logins with mixed case user names are allowed. For example logging in as [email protected]  will be treated as logging in as [email protected] . This is roughly similar to the way mail subsystem is set up in Plesk with the exception of authentication database, which is not a text file.

Solution via passwd-file driver

Normally one would use passwd-file passdb driver in Dovecot configuration to authenticate against /etc/passwd -like file as in our case. So the relevant part of the configuration would look like:

mail_home = /var/qmail/mailnames/%Ld/%Ln
mail_location = maildir:~/Maildir

passdb {
  driver = passwd-file
  args = scheme=PLAIN username_format=%Lu /etc/dovecot/users
}

userdb {
  driver = static
  args = uid=popuser gid=popuser
}

Solution via checkpassword driver with custom Shell script

Now let’s solve the same problem via checkpassword passdb driver. To use custom CheckPassword script located at /etc/dovecot/checkpassword.sh  as passdb and run it with superuser privileges, following Dovecot configuration should be used:

mail_location = maildir:~/Maildir

# This (rather than service auth-worker) applies to checkpassword 
# since it effectively runs in blocking=no mode.
service auth {
  # For simplicity run authentication under root user
  # instead of $default_internal_user which is usually 'dovecot'.
  user =
  group =
}

passdb {
  driver = checkpassword
  args = /etc/dovecot/checkpassword.sh
}

userdb {
  # Static userdb will be enough for our case, 
  # however the checkpassword script may as well be used for userdb.
  driver = static
  args = uid=popuser gid=popuser
}

And here is example of the /etc/dovecot/checkpassword.sh  script itself. It is rather well-commented to be easily understandable.

#!/bin/bash

# Example Dovecot checkpassword script that may be used as both passdb or userdb.
#
# Originally written by Nikolay Vizovitin, 2013.

# Assumes authentication DB is in /etc/dovecot/users, each line has '<user>:<password>' format.
# Implementation guidelines at http://wiki2.dovecot.org/AuthDatabase/CheckPassword

# The first and only argument is path to checkpassword-reply binary.
# It should be executed at the end if authentication succeeds.
CHECKPASSWORD_REPLY_BINARY="$1"

# Messages to stderr will end up in mail log (prefixed with "dovecot: auth: Error:")
LOG=/dev/stderr

# User and password will be supplied on file descriptor 3.
INPUT_FD=3

# Error return codes.
ERR_PERMFAIL=1
ERR_NOUSER=3
ERR_TEMPFAIL=111

# Make testing this script easy. To check it just run:
#   printf '%s\x0%s\x0' <user> <password> | ./checkpassword.sh test; echo "$?"
if [ "$CHECKPASSWORD_REPLY_BINARY" = "test" ]; then
    CHECKPASSWORD_REPLY_BINARY=/bin/true
    INPUT_FD=0
fi

# Credentials lookup function. Given a user name it should output 'user:password' if such
# account exists or nothing if it does not. Return non-zero code in case of error.
credentials_lookup()
{
    local db="$1"
    local user="$2"

    awk -F ':' -v USER="$user" '($1 == USER) {print}' "$db" 2>>$LOG
}

# Credentials verification function. Given a user name and password it should output non-empty
# string (this implementation outputs 'user:password') in case supplied credentials are valid
# or nothing if they are not. Return non-zero code in case of error.
credentials_verify()
{
    local db="$1"
    local user="$2"
    local pass="$3"

    awk -F ':' -v USER="$user" -v PASS="$pass" '($1 == USER && $2 == PASS) {print}' "$db" 2>>$LOG
}

# Just a simple logging helper.
log_result()
{
    echo "$*; Input: $USER:$PASS; Home: $HOME; Reply binary: $CHECKPASSWORD_REPLY_BINARY" >> $LOG
}

# Read input data. Password may be empty if not available (i.e. if doing credentials lookup).
read -d $'\x0' -r -u $INPUT_FD USER
read -d $'\x0' -r -u $INPUT_FD PASS

# Both mailbox and domain directories should be in lowercase on file system.
# So let's convert login user name to lowercase and tell Dovecot 'user' and 'home' (which overrides
# 'mail_home' global parameter) values should be updated.
# Of course, conversion to lowercase may be done in Dovecot configuration as well.
export USER="`echo \"$USER\" | tr 'A-Z' 'a-z'`"
mail_name="`echo \"$USER\" | awk -F '@' '{ print $1 }'`"
domain_name="`echo \"$USER\" | awk -F '@' '{ print $2 }'`"
export HOME="/var/qmail/mailnames/$domain_name/$mail_name/"

# CREDENTIALS_LOOKUP=1 environment is set when doing non-plaintext authentication.
if [ "$CREDENTIALS_LOOKUP" = 1 ]; then
    action=credentials_lookup
else
    action=credentials_verify
fi

# Perform credentials lookup/verification.
lookup_result=`$action "/etc/dovecot/users" "$USER" "$PASS"` || {
    # If it failed, consider it an internal temporary error.
    log_result "internal error (ran as `id`)"
    exit $ERR_TEMPFAIL
}

if [ -n "$lookup_result" ]; then
    # Dovecot calls the script with AUTHORIZED=1 environment set when performing a userdb lookup.
    # The script must acknowledge this by changing the environment to AUTHORIZED=2,
    # otherwise the lookup fails.
    [ "$AUTHORIZED" != 1 ] || export AUTHORIZED=2

    # And here's how to return extra fields from userdb/passdb lookup, e.g. 'uid' and 'gid'.
    # All virtual mail users in Plesk actually run under 'popuser'.
    # See also:
    #   http://wiki2.dovecot.org/PasswordDatabase/ExtraFields
    #   http://wiki2.dovecot.org/UserDatabase/ExtraFields
    #   http://wiki2.dovecot.org/VirtualUsers
    export userdb_uid=popuser
    export userdb_gid=popuser
    export EXTRA="userdb_uid userdb_gid $EXTRA"

    if [ "$CREDENTIALS_LOOKUP" = 1 ]; then
        # If this is a credentials lookup, return password together with its scheme.
        # The password scheme that Dovecot wants is available in SCHEME environment variable
        # (e.g. SCHEME=CRAM-MD5), however 'PLAIN' scheme can be converted to anything internally
        # by Dovecot, so we'll just return 'PLAIN' password.
        found_password="`echo \"$lookup_result\" | awk -F ':' '{ print $2 }'`"
        export password="{PLAIN}$found_password"
        export EXTRA="password $EXTRA"
        log_result "credentials lookup result: '$password' [SCHEME='$SCHEME', EXTRA='$EXTRA']"
    else
        log_result "lookup result: '$lookup_result'"
    fi

    # At the end of successful authentication execute checkpassword-reply binary.
    exec $CHECKPASSWORD_REPLY_BINARY
else
    # If matching credentials were not found, return proper error code depending on lookup mode.
    if [ "$AUTHORIZED" = 1 -a "$CREDENTIALS_LOOKUP" = 1 ]; then
        log_result "lookup failed (user not found)"
        exit $ERR_NOUSER
    else
        log_result "lookup failed (credentials are invalid)"
        exit $ERR_PERMFAIL
    fi
fi

Due to WordPress syntax highlighter plug-in quirkiness I had to slightly modify the source. Original version can be found here.

You may naturally ask “how is this better than the much shorter passwd-file example given earlier?” The answer is you may have virtually any authentication behavior without writing a full-blown custom authentication module. In the simplest case you would only need to adjust the lines highlighted in the script above to specify your own custom credentials lookup and/or verification procedures. For example, you may generate account passwords based on their user names, or fetch credentials from a storage which is not directly supported by Dovecot.

The example CheckPassword script above was written for Bash shell, however it should work in Dash shell as well. It should also give a good idea how to write similar scripts in other programming languages. Basically, the input data is passed on the file descriptor #3 (user name and password separated by a zero byte) and through environment variables. The script should set up environment (export  statements in the script above) in a specific way to indicate output data and call the checkpassword-reply  binary in case of success, path to which is passed to the script as the first and only argument when Dovecot calls it.

Testing custom authentication

Personally, I prefer testing authentication changes manually via telnet. This gives a certain degree of control and understanding of how things really work. This is especially simple when using plaintext authentication in POP3 or IMAP. There are many examples on the Internet that show how to authenticate manually.

Another useful tool is imtest utility from the Cyrus suite. It provides a really easy way to test any IMAP authentication mechanism, including shared secret mechanisms like CRAM-MD5 — just specify the one via -m  switch. The tool is usually available from one of the cyrus clients packages in your OS of choice.

If you have unexpected authentication problems, look into system mail log (BTW, it’s /var/log/maillog  since Plesk 12.0). Both entries logged by Dovecot and CheckPassword script would end up there. You may also alter the CheckPassword script log destination by modifying the LOG  variable value in the script example posted above.

The CheckPassword script posted above also has a “test” mode that allows you to check its operation outside of Dovecot. Just run  /etc/dovecot/checkpassword.sh test  to use it, e.g.:

$ printf '%s\x0%s\x0' "[email protected]" "" | 
> env CREDENTIALS_LOOKUP=1 /etc/dovecot/checkpassword.sh test
credentials lookup result: '{PLAIN}2mail' [SCHEME='', EXTRA='password userdb_uid userdb_gid ']; Input: [email protected]:; Home: /var/qmail/mailnames/test2.a10-52-181-149.qa.plesk.ru/mail2/; Reply binary: /bin/true
$ echo $?
0
$ printf '%s\x0%s\x0' "[email protected]" "12345" | 
> /etc/dovecot/checkpassword.sh test
lookup failed (credentials are invalid); Input: [email protected]:12345; Home: /var/qmail/mailnames/test2.a10-52-181-149.qa.plesk.ru/mail2/; Reply binary: /bin/true
$ echo $?
1

Conclusion

Examples in this post are available from my Bitbucket repository. Feel free to use them in any way you see fit.

Please take care when using CheckPassword authentication.  It is not the best choice in terms of security or performance, but great to quickly get a custom authentication up and running. Dovecot also has quite a lot of other passdb drivers available fit for various cases. Consider using one of them instead — it might as well end up being easier to configure and more efficient to use.