Docspell CLI🔗

Introduction🔗

The docspell client, short dsc, is a tool to use docspell through the command line. It is also aims to be useful for your own scripts and programs.

It is a work in progress; eventually most of the api will be covered.

Usage🔗

Download the binary for your architecture from the release page and rename it to dsc. Then run dsc help to see an overview of all commands. The help of each command is available via dsc help [command] or dsc [command] --help.

There are docker images at dockerhub, but it's usually easier to just download the binary. They should work on most systems without additional setups.

Below are some quick infos to get started, please see the project page for more info.

Configuration🔗

A configuration file can be used to have some predefined settings, for example the docspell url, the admin secret etc. They can be overriden by specifying them as options.

The config looks like this:

docspell_url = "http://localhost:7880"
default_format = "Tabular"
admin_secret = "admin123"
default_account = "demo"
pdf_viewer = ["zathura", "{}"]
#pass_entry = "my/entry"

For linuxes, the default location is ~/.config/dsc/config.toml. You can give a config file explicitly via an option or the environment variable DSC_CONFIG.

If you use the pass password manager, you can add your password entry to the config file as well.

Output format🔗

The "output format" defines how the information is printed on screen. The default output format is Tabular which prints a simple table. This table can also be formatted as CSV using csv as output format. These two modes are intended for humans and they may not present all information available.

Alternatively, there is json and lisp as output format. These are intended for machine consumption. They always contain all information. If you look for some detail, use for example json to get all data in a structured form. On the shell you can use the awesome tool jq to get exactly what you want.

Login🔗

Many tasks require to be logged in. This can be done via the login subcommand. You can specify account and password or fallback to the values in the config file.

Once logged in, the session token will be saved to the filesystem (next to the config file) and is used for subsequent commands. It is renewed if expiry is near. If you don't issue any commands for a while you need to login again.

Demo🔗

Use Cases / Examples🔗

These are some examples. Each command has a good help explaining all the options. Run dsc [subcommand] --help to see it.

Uploads files🔗

The upload subcommand can upload files to docspell. This is the replacement for the ds.sh shell script.

You can specify a list of files that are all being uploaded. This command doesn't require to be logged in, it can also upload via a source id or via the integration endpoint.

A source id can be given in the config file, then there are no additional options required. The simplest form is this:

❯ dsc upload *.pdf
File already in Docspell: article-velo.pdf
Adding to request: test-ocr.pdf
Sending request …
┌─────────┬──────────────────┐
│ success │ message          │
├─────────┼──────────────────┤
│ true    │ Files submitted. │
└─────────┴──────────────────┘

By default, duplicate files are detected and not uploaded. This uploads all files in one single request. By default, each file results in one item. Using --single-item all files can be put into one item.

It is possible to specify certain metadata, like tags or a folder, that is then attached to the resulting item.

Upload by traversing a directory🔗

The above use case was about uploading files. Using the upload subcommand with the --traverse option, you can traverse directories and upload all files in them. In this mode, each file will be uploaded in a separate request, so the --single-item option cannot be used.

There are options to exclude/include files based on a glob pattern.

❯ dsc upload --traverse .
File already in Docspell: article-velo.pdf
File already in Docspell: demo/dirc/scan.21-03-12.15-50-54.pdf
File already in Docspell: demo/dirc/test-stamp.pdf
File already in Docspell: demo/letter-de.pdf
Uploading eike/keywords.pdf
File already in Docspell: eike/large-file.pdf
Uploading eike/letter-en.pdf
File already in Docspell: test-ocr.pdf
┌─────────┬────────────┐
│ success │ message    │
├─────────┼────────────┤
│ true    │ Uploaded 2 │
└─────────┴────────────┘

The --poll option allows to periodically traverse and upload directories.

Watch a directory🔗

The watch subcommand can be used to watch one or more directories and upload files when they arrive. It uses the upload command under the hood and therefore most options are also available here. You can upload via a source url, the integration endpoint or a valid session (requires to login).

It detects file creations and skips a rename within a watched folder. The flag -r or --recursive is required to recursively watch a directory.

❯ dsc watch -r .
Watching directory (Recursive): .
Press Ctrl-C to quit.
------------------------------------------------------------------------------
Got: /home/eike/workspace/projects/dsc/local/files/./demo/letter-de.pdf
Adding to request: /home/eike/workspace/projects/dsc/local/files/./demo/letter-de.pdf
Sending request …
Server: Files submitted.

The --matches option allows to define a pattern for files to include.

If watching a directory is not possible due to system constraints (e.g. when using NFS or SAMBA shares), a less efficient option is to use the upload subcommand with --poll option which periodically traverses a directory.

When using the integration endpoint, it requires to specify -i and potentially a secret if the endpoint is protected with a secret.

Download files🔗

The download command allows to download files that match a given query. It is possible to download them all flat into some directory or directly into a zip file. For example, download all files that are tagged with todo into a zip file:

❯ dsc download --zip 'tag:todo'
Zipping 2 attachments into docspell-files.zip
Downloading DOC-20191223-155707.jpg.pdf …
Downloading DOC-20200803-174448.jpg.pdf …

It downloads the converted PDF files by default, which can be changed via some options.

❯ dsc download --zip --original 'tag:todo'
Zipping original files of 2 attachments into docspell-files.zip
Downloading DOC-20191223-155707.jpg …
Downloading DOC-20200803-174448.jpg …

Export data🔗

The export command allows to download all items with metadata and their files. It downloads all items by default, but a query is also supported.

In contrast to the download command, this is intended for getting everything out of docspell in some independent format. Files are downloaded (only original files) and the items metadata is also stored. So you don't loose the tags and correspondents that are carefully maintained with each item.

It expects one directory where it will create a specfific directory structure as follows:

exports/
├── by_date
│   ├── 2019-07
│   ├── 2020-08
|       ├── BV2po65mAFU-…-bqUiwjz8f2W -> ../../items/BV/BV2po65mAFU-…-bqUiwjz8f2W
|       └── FTUnhZ3AE1H-…-RQ9KhtRi486 -> ../../items/FT/FTUnhZ3AE1H-…-RQ9KhtRi486
│   ├── …
│   └── 2021-07
├── by_folder
│   ├── Finance
|   │   ├── 455h3cQNdna-…-t6dF7NjAuDm -> ../../items/45/455h3cQNdna-…-t6dF7NjAuDm
|   │   ├── 5yQ95tQ4khY-…-S9KrxcbRkZR -> ../../items/5y/5yQ95tQ4khY-…-S9KrxcbRkZR
|   │   ├── Bank
|   │   │   ├── 7xoiE4XdwgD-…-Eb2S3BCSd38 -> ../../items/7x/7xoiE4XdwgD-…-Eb2S3BCSd38
|   │   │   ├── 93npVoA73Cx-…-BnxYNsf4Qvi -> ../../items/93/93npVoA73Cx-…-BnxYNsf4Qvi
├── by_tag
│   ├── Contract
│   ├── Important
│   ├── Invoice
|   │   ├── 455h3cQNdna-…-t6dF7NjAuDm -> ../../items/45/455h3cQNdna-…-t6dF7NjAuDm
|   │   ├── 5yQ95tQ4khY-…-S9KrxcbRkZR -> ../../items/5y/5yQ95tQ4khY-…-S9KrxcbRkZR
|   │   ├── 7xoiE4XdwgD-…-Eb2S3BCSd38 -> ../../items/7x/7xoiE4XdwgD-…-Eb2S3BCSd38
|   │   └── 93npVoA73Cx-…-BnxYNsf4Qvi -> ../../items/93/93npVoA73Cx-…-BnxYNsf4Qvi
│   ├── …
│   └── Todo
└── items
    ├── 45
    |   └── 455h3cQNdna-w8oTEw9wtpE-G7bCJbVpZPw-t6dF7NjAuDm
    |       ├── files
    |       │   └── DOC-20200803-174448.jpg
    |       └── metadata.json
    ├── …
    └── Hb

All your items are stored below the items directory. It contains subdirectories that are created from the first two characters of the item id. Inside this folder, a folder with the complete item id is created and there all the data to the item is stored: the metadata in metadata.json and all files below files/.

The options --date-links, --folder-links, and --tag-links create the other two folders: by_tag, by_folder and by_date. In there you'll find symlinks into the items folder that are organized by some metadata, namely tags, folder, and the item's date.

With the --link-naming option you can control the filename used for the links created in the by_* folders. The default is the items' id, which is not very self-speaking and thus might not fit your needs. Using --link-naming name you can configure it to use the item's name instead.

If you use path separators in your folder names, such as the "/" in Finance/Institute, you can configure export to unwrap your folder names into recursive paths (as shown in the example output above) using the --folder-delimiter "/" option. Otherwise, the name would be sanitized resulting in Finance-Institute.

Example run (producing output shown above):

❯ dsc export --all --date-links --folder-links --folder-delimiter "/" --tag-links --target exports
Exported item: test3.zip
Exported item: README.md
Exported item: LICENSE.txt
Exported item: TestRMarkdown.pdf
Exported item: DOC-20191223-155729.jpg
Exported item: DOC-20191223-155707.jpg
Exported item: DOC-20200808-154204.jpg
Exported item: DOC-20200807-115654.jpg
Exported item: DOC-20200803-174448.jpg
Exported item: DOC-20200803-174448.jpg
Exported item: DOC-20200803-174448.jpg
Exported item: DOC-20200803-174448.jpg
Exported item: DOC-20200804-132305.jpg
Exported item: DOC-20191223-155707.jpg
Exported item: keyweb.eml
Exported 15 items.

With output format json or lisp each item is printed instead in full detail.

Admin commands🔗

These are a set of commands that simply call a route at the server to submit a maintenance task or to reset the password of some user. These commands require the admin secret either in the config file or as an argument.

Reset user password🔗

❯ dsc admin reset-password --account demo
┌─────────┬──────────────┬──────────────────┐
│ success │ new password │ message          │
├─────────┼──────────────┼──────────────────┤
│ true    │ 2q2UeCVvMYg  │ Password updated │
└─────────┴──────────────┴──────────────────┘

Recreate fulltext index🔗

❯ dsc admin --admin-secret admin123 recreate-index
┌─────────┬─────────────────────────────────────┐
│ success │ message                             │
├─────────┼─────────────────────────────────────┤
│ true    │ Full-text index will be re-created. │
└─────────┴─────────────────────────────────────┘

Convert all files to PDF🔗

❯ dsc admin --admin-secret admin123 convert-all-pdf
┌─────────┬─────────────────────────────────┐
│ success │ message                         │
├─────────┼─────────────────────────────────┤
│ true    │ Convert all PDFs task submitted │
└─────────┴─────────────────────────────────┘

This may be necessary if you disabled pdf conversion before and are enabling it now.

Regenerate preview images🔗

❯ dsc admin --admin-secret admin123 convert-all-pdf
┌─────────┬───────────────────────────────────────┐
│ success │ message                               │
├─────────┼───────────────────────────────────────┤
│ true    │ Generate all previews task submitted. │
└─────────┴───────────────────────────────────────┘

This submits tasks to (re)generate preview images of all files. This is necessary if you changed the preview.dpi setting in joex' config.

Search for items🔗

The search command takes a query and prints the results.

❯ dsc search 'corr:*'
┌──────────┬────────────────────────────┬───────────┬────────────┬─────┬───────────────────────────┬───────────────┬────────┬─────────────┬────────────┬───────┐
│ id       │ name                       │ state     │ date       │ due │ correspondent             │ concerning    │ folder │ tags        │ fields     │ files │
├──────────┼────────────────────────────┼───────────┼────────────┼─────┼───────────────────────────┼───────────────┼────────┼─────────────┼────────────┼───────┤
│ HVK7JuCF │ test-ocr.pdf               │ created   │ 2021-07-18 │     │ Pancake Company           │               │        │ Certificate │            │ 1     │
│ 3odNawKE │ letter-en.pdf              │ confirmed │ 2021-07-18 │     │ Pancake Company           │               │        │ invoice     │            │ 1     │
│ 3MA5NdhS │ large-file.pdf             │ confirmed │ 2021-07-18 │     │ Axa                       │               │        │ Certificate │            │ 1     │
│ HDumXkRm │ keywords.pdf               │ confirmed │ 2021-07-18 │     │ Pancake Company           │               │        │ invoice     │            │ 1     │
│ 733gM656 │ test-stamp.pdf             │ created   │ 2021-07-18 │     │ Pancake Company           │               │        │ Contract    │            │ 1     │
│ 8LiciitB │ scan.21-03-12.15-50-54.pdf │ confirmed │ 2021-07-18 │     │ Supermarket               │               │        │ Receipt     │ CHF 89.44  │ 1     │
│ 8nFt2z7T │ article-velo.pdf           │ confirmed │ 2021-07-18 │     │ Supermarket/Rudolf Müller │ Rudolf Müller │        │ invoice     │ CHF 123.11 │ 1     │
│ kfugGdXU │ letter-de.pdf              │ created   │ 2021-07-18 │     │ Axa                       │ Rudolf Müller │        │ invoice     │            │ 1     │
└──────────┴────────────────────────────┴───────────┴────────────┴─────┴───────────────────────────┴───────────────┴────────┴─────────────┴────────────┴───────┘

The same can be formatted as json and, for example, only print the ids:

❯ dsc -f json search 'corr:*' | jq '.groups[].items[].id'
"HVK7JuCFt4W-qxkcwq1cWCV-dvpGo4DpZzU-Q16Xoujojas"
"3odNawKE1Ek-YJrWfPzekAq-47cjt14sexd-GK35JAEAanx"
"3MA5NdhSrbx-3JkjEpqHiyU-XyVNb15tioh-SUVjMLi1aoV"
"HDumXkRmDea-dNryjtRjk3V-ysdJmJNQGQS-UFb4DWNZJ3F"
"733gM656S4T-d4HmEgdAV6Z-9zuHAd3biKM-mBwNriZpqMB"
"8LiciitBVTi-DTmgiEUdqAJ-xXPckMvFHMc-JSiJMYvLaWh"
"8nFt2z7T9go-1qaCTTgodub-592n6gpmdNR-VRcyYAyT7qj"
"kfugGdXUGUc-mReaUnJxyUL-R44Lf7yH6RK-2JbZ1bv7dw"

Docker🔗

The provided docker-compose setup runs this script to watch a single directory, ./docs in current directory, for new files. If a new file is detected, it is pushed to docspell.

This utilizes the integration endpoint, which is enabled in the config file, to allow uploading documents for all collectives. A subfolder must be created for each registered collective. The docker containers are configured to use http-header protection for the integration endpoint. This requires you to provide a secret, that is shared between the rest-server and the dsc tool. This can be done by defining an environment variable which gets picked up by the containers defined in docker-compose.yml:

docker-compose up -d

Now you can create a folder ./docs/<collective-name> and place all files in there that you want to import. Once dropped in this folder the consumedir container will push it to docspell.