One possible use case for the integration endpoint is a SMTP server that forwards all local mail to docspell. This way there is no periodic polling involved and documents (e-mails) get into docspell without delay.
The tools/exim
folder contains a docker file and a sample
exim.conf
to help start with this setup. Note that these files
provide a minimal setup, you might want to add tls and spam protection
when opening it to the public.
You need to own a domain and add the appropriate MX records to point
to your server. In this document, the domain test.org
is used.
You need to enable the integration endpoint in the docspell configuration.
Exim is a popular smtp server (message transfer agent). It is used here only because of previous knowledge, but same can be achieved with other MTAs.
Here is the example config file for exim:
## Provide certificates to enable StartTLS
# tls_certificate = /var/lib/acme/test.org/fullchain.pem
# tls_privatekey = /var/lib/acme/test.org/key.pem
tls_advertise_hosts =
primary_hostname = test.org
domainlist local_domains = test.org
timeout_frozen_after = 1m
acl_smtp_rcpt = acl_check_rcpt
acl_smtp_data = acl_check_data
never_users = root
host_lookup = *
daemon_smtp_ports = 25
message_size_limit = 30m
keep_environment = DS_HEADER : DS_URL
begin acl
acl_check_rcpt:
require
domains = +local_domains
require
message = Sender verification failed
verify = sender
require
message = Receiver verification failed
verify = recipient
require
message = Recipient unknown
condition = ${run{/usr/bin/curl --out /dev/null --silent --fail -H "Docspell-Integration: ${env{DS_HEADER}{$value} fail}" "${env{DS_URL}{$value} fail}/api/v1/open/integration/item/$local_part"}{yes}{no}}
warn
message = Reverse lookup failed
!verify = reverse_host_lookup
accept
acl_check_data:
deny
message = Sender verification failed
!verify = header_sender
accept
begin routers
local_users:
driver = accept
transport = docspell
begin transports
docspell:
driver = pipe
command = /usr/bin/curl --out /dev/null --silent --fail -H "Docspell-Integration: ${env{DS_HEADER}{$value} fail}" -F "file=@-;filename=\"$h_subject:\"" "${env{DS_URL}{$value} fail}/api/v1/open/integration/item/$local_part"
return_fail_output
user = nobody
delivery_date_add
envelope_to_add
return_path_add
log_output
Exim has good documentation, look there for more info. The following is only a quick summary of the file above.
The domainlist local_domains
should list your domain. Only mails to
this domain are allowed, as specified in the first rule in
acl_check_rcpt
. So mails to name@test.org
are ok, but
name@someother.org
not.
Another rule in acl_check_rcpt
executes a GET
request against the
integration endpoint. If that fails, the recipient is wrong (or the
endpoint disabled) and the mail is rejected right away.
Then the routers
define how a mail is handled. There is only one
router that accepts all mails (that have not been rejected by a rule
in acls) and uses the docspell
transport to deliver it. The
transport specifies a command via the pipe
driver that is run with
the mail. The mail itself is provided via stdin. So a simple curl
command can upload it to the integration endpoint. Here are some quick
notes about the used options (see man curl
):
--silent
and --out /dev/null
don't print upload progress
information and no output to stdout--fail
return non-zero if http status code is not success-F
use a multipart/form-data request (defaults to a POST request)"file=@-;filename=\"$_subject:\""
add one part with name file
and take the data from stdin (@-
). Since there is no filename, we
use the subject of the mail. This is supported by
exim
by expanding the subject mail header via $h_subject:
(the colon is
required).$local_part
this is expanded by exim to the recipient address,
only the part until the @
sign.${env{DS_HEADER}{$value} fail}
looks up an environment variable by
key DS_HEADER
. This is usually defined in docker-compose.yml
.
The value must be the "secret" header value as defined in docspell's
configuration file.${env{DS_URL}{$value} fail}
the url to docspell. It is looked up
from the environment with key DS_URL
, which is usually defined in
docker-compose.yml
. Adding the $local_part
at the end means that
mails to somename@test.org
are uploaded to the collective
somename
.Go into the tools/exim
directory and build the docker image:
docker build -t ds-exim:latest -f exim.dockerfile .
Then start docspell somewhere and configure the integration endpoint to use http-header protection; i.e. set this in the config file:
docspell.server {
integration-endpoint {
enabled = true
http-header = {
enabled = true
header-value = "test123"
}
}
}
Then edit the docker-compose.yml
and change the environment
variables as needed.
Finally start the container:
docker-compose up
Now it is possible to send mails to this MTA which will be immediatly
uploaded to docspell for the collective corresponding to the
$local_part
of the recipients address. Here is a quick telnet
session (the collective is named family
):
~> telnet localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 test.org ESMTP Exim 4.93 Sun, 14 Jun 2020 19:03:51 +0000
ehlo localhost
250-test.org Hello localhost [::1]
250-SIZE 31457280
250-8BITMIME
250-PIPELINING
250-CHUNKING
250 HELP
mail from:<me@test.org>
250 OK
rcpt to:<family@test.org>
250 Accepted
data
354 Enter message, ending with "." on a line by itself
From: me@test.org
To: family@test.org
Subject: This is a test
Test,
this is just a test mail.
.
250 OK id=1jkXwf-000007-0d
quit
221 test.org closing connection
Connection closed by foreign host.
~>
The mail is processed and results in an item:
However, if a mail is to an unknown collective or not to the configured local domain, the server rejects it immediately:
~> telnet localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 test.org ESMTP Exim 4.93 Sun, 14 Jun 2020 19:07:04 +0000
ehlo localhost
250-test.org Hello localhost [::1]
250-SIZE 31457280
250-8BITMIME
250-PIPELINING
250-CHUNKING
250 HELP
mail from:<me@test.org>
250 OK
rcpt to:<family22@test.org>
550 Recipient unknown
rcpt to:<family@gmail.com>
550 Administrative prohibition
quit
221 test.org closing connection
Connection closed by foreign host.
~>