Replace cronjobs with systemd timers

Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/756

Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/737

I feel like timers are somewhat more complicated and dirty (compared to
cronjobs), but they come with these benefits:

- log output goes to journald
- on newer systemd distros, you can see when the timer fired, when it
will fire, etc.
- we don't need to rely on cron (reducing our dependencies to just
systemd + Docker)

Cronjobs work well, but it's one more dependency that needs to be
installed. We were even asking people to install it manually
(in `docs/prerequisites.md`), which could have gone unnoticed.

Once in a while someone says "my SSL certificates didn't renew"
and it's likely because they forgot to install a cron daemon.

Switching to systemd timers means that installation is simpler
and more unified.

CHANGELOG.md

@@ -1,3 +1,14 @@
+# 2021-01-14
+
+## Moving from cronjobs to systemd timers
+
+We no longer use cronjobs for Let's Encrypt SSL renewal and `matrix-nginx-proxy`/`matrix-coturn` reloading. Instead, we've switched to systemd timers.
+
+The largest benefit of this is that we no longer require you to install a cron daemon, thus simplifying our install procedure.
+
+The playbook will migrate you from cronjobs to systemd timers automatically. This is just a heads up.
+
+
 # 2021-01-08
 
 ## (Breaking Change) New SSL configuration

docs/prerequisites.md

@@ -16,8 +16,6 @@ If your distro runs within an [LXC container](https://linuxcontainers.org/), you
 
 - [Python](https://www.python.org/) being installed on the server. Most distributions install Python by default, but some don't (e.g. Ubuntu 18.04) and require manual installation (something like `apt-get install python3`). On some distros, Ansible may incorrectly [detect the Python version](https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html) (2 vs 3) and you may need to explicitly specify the interpreter path in `inventory/hosts` during installation (e.g. `ansible_python_interpreter=/usr/bin/python3`)
 
-- A `cron`-like tool installed on the server such as `cron` or `anacron` to automatically schedule the Let's Encrypt SSL certificates's renewal. *This can be ignored if you use your own SSL certificates.*
-
 - The [Ansible](http://ansible.com/) program being installed on your own computer. It's used to run this playbook and configures your server for you. Take a look at [our guide about Ansible](ansible.md) for more information, as well as [version requirements](ansible.md#supported-ansible-versions) and alternative ways to run Ansible.
 
 - Either the `dig` tool or `python-dns` installed on your own computer. Used later on, by the playbook's [services check](maintenance-checking-services.md) feature.

docs/uninstalling.md

@@ -23,15 +23,13 @@ If you prefer to uninstall manually, run these commands (most are meant to be ex
 
 - ensure all Matrix services are stopped: `ansible-playbook -i inventory/hosts setup.yml --tags=stop` (if you can't get Ansible working to run this command, you can run `systemctl stop 'matrix*'` manually on the server)
 
-- delete the Matrix-related systemd `.service` files (`rm -f /etc/systemd/system/matrix*.service`) and reload systemd (`systemctl daemon-reload`)
-
-- delete all Matrix-related cronjobs (`rm -f /etc/cron.d/matrix*`)
+- delete the Matrix-related systemd `.service` and `.timer` files (`rm -f /etc/systemd/system/matrix*.{service,timer}`) and reload systemd (`systemctl daemon-reload`)
 
 - delete some helper scripts (`rm -f /usr/local/bin/matrix*`)
 
 - delete some cached Docker images (`docker system prune -a`) or just delete them all (`docker rmi $(docker images -aq)`)
 
-- delete the Docker network: `docker network rm matrix` (might have been deleted already if you ran the `docker system prune` command)
+- delete the Docker networks: `docker network rm matrix matrix-coturn` (might have been deleted already if you ran the `docker system prune` command)
 
 - uninstall Docker itself, if necessary
 

roles/matrix-base/defaults/main.yml

@@ -48,7 +48,11 @@ matrix_base_data_path_mode: "750"
 
 matrix_static_files_base_path: "{{ matrix_base_data_path }}/static-files"
 matrix_systemd_path: "/etc/systemd/system"
+
+# This is now unused. We keep it so that cleanup tasks can use it.
+# To be removed in the future.
 matrix_cron_path: "/etc/cron.d"
+
 matrix_local_bin_path: "/usr/local/bin"
 
 matrix_host_command_docker: "/usr/bin/env docker"

roles/matrix-base/templates/usr-local-bin/matrix-remove-all.j2

@@ -20,8 +20,6 @@ else
 		rm -f {{ matrix_systemd_path }}/$s
 	done
 	systemctl daemon-reload
-	echo "Remove matrix cronjobs"
-	find /etc/cron.d/ -name "matrix-*" -delete
 	echo "Remove matrix scripts"
 	find {{ matrix_local_bin_path }}/ -name "matrix-*" -delete
 	echo "Remove unused Docker images and resources"

roles/matrix-common-after/tasks/start.yml

@@ -1,6 +1,6 @@
 ---
 
-- name: Deterimne whether we should make services autostart
+- name: Determine whether we should make services autostart
   set_fact:
     matrix_services_autostart_enabled_bool: "{{ true if matrix_services_autostart_enabled|default('') == '' else matrix_services_autostart_enabled|bool }}"
 
@@ -46,7 +46,7 @@
         Try running `systemctl status {{ item }}` and `journalctl -fu {{ item }}` on the server to investigate.
     with_items: "{{ matrix_systemd_services_list }}"
     when:
-      - "ansible_facts.services[item]|default(none) is none or ansible_facts.services[item].state != 'running'"
+      - "item.endswith('.service') and (ansible_facts.services[item]|default(none) is none or ansible_facts.services[item].state != 'running')"
   when: " ansible_distribution != 'Archlinux'"
 
 - block:

roles/matrix-coturn/tasks/init.yml

@@ -2,6 +2,10 @@
     matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-coturn.service'] }}"
   when: matrix_coturn_enabled|bool
 
+- set_fact:
+    matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-coturn-reload.timer'] }}"
+  when: "matrix_coturn_enabled|bool and matrix_coturn_tls_enabled|bool"
+
 # ansible lower than 2.8, does not support docker_image build parameters
 # for self buildig it is explicitly needed, so we rather fail here
 - name: Fail if running on Ansible lower than 2.8 and trying self building

roles/matrix-coturn/tasks/setup_install.yml

@@ -1,5 +1,11 @@
 ---
 
+# This is a cleanup/migration task. It can be removed some time in the future.
+- name: (Migration) Remove deprecated cronjob
+  file:
+    path: "{{ matrix_cron_path }}/matrix-coturn-ssl-reload"
+    state: absent
+
 - name: Ensure Matrix Coturn path exists
   file:
     path: "{{ item.path }}"
@@ -19,24 +25,24 @@
     force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_coturn_docker_image_force_pull }}"
   when: "not matrix_coturn_container_image_self_build|bool"
 
-- name: Ensure Coturn repository is present on self-build
-  git:
-    repo: "{{ matrix_coturn_container_image_self_build_repo }}"
-    dest: "{{ matrix_coturn_docker_src_files_path }}"
-    version: "{{ matrix_coturn_docker_image.split(':')[1] }}"
-    force: "yes"
-  register: matrix_coturn_git_pull_results
-  when: "matrix_coturn_container_image_self_build|bool"
+- block:
+    - name: Ensure Coturn repository is present on self-build
+      git:
+        repo: "{{ matrix_coturn_container_image_self_build_repo }}"
+        dest: "{{ matrix_coturn_docker_src_files_path }}"
+        version: "{{ matrix_coturn_docker_image.split(':')[1] }}"
+        force: "yes"
+      register: matrix_coturn_git_pull_results
 
-- name: Ensure Coturn Docker image is built
-  docker_image:
-    name: "{{ matrix_coturn_docker_image }}"
-    source: build
-    force_source: "{{ matrix_coturn_git_pull_results.changed }}"
-    build:
-      dockerfile: Dockerfile
-      path: "{{ matrix_coturn_docker_src_files_path }}"
-      pull: yes
+    - name: Ensure Coturn Docker image is built
+      docker_image:
+        name: "{{ matrix_coturn_docker_image }}"
+        source: build
+        force_source: "{{ matrix_coturn_git_pull_results.changed }}"
+        build:
+          dockerfile: Dockerfile
+          path: "{{ matrix_coturn_docker_src_files_path }}"
+          pull: yes
   when: "matrix_coturn_container_image_self_build|bool"
 
 - name: Ensure Coturn configuration path exists
@@ -52,6 +58,8 @@
     src: "{{ role_path }}/templates/turnserver.conf.j2"
     dest: "{{ matrix_coturn_config_path }}"
     mode: 0644
+    owner: "{{ matrix_user_username }}"
+    group: "{{ matrix_user_groupname }}"
 
 - name: Ensure Coturn network is created in Docker
   docker_network:
@@ -63,26 +71,34 @@
     src: "{{ role_path }}/templates/systemd/matrix-coturn.service.j2"
     dest: "{{ matrix_systemd_path }}/matrix-coturn.service"
     mode: 0644
-  register: matrix_coturn_systemd_service_result
-
-- name: Ensure systemd reloaded after matrix-coturn.service installation
-  service:
-    daemon_reload: yes
-  when: "matrix_coturn_systemd_service_result.changed"
+  register: matrix_coturn_systemd_service_change_results
 
 # This may be unnecessary when more long-lived certificates are used.
 # We optimize for the common use-case though (short-lived Let's Encrypt certificates).
 # Reloading doesn't hurt anyway, so there's no need to make this more flexible.
-- name: Ensure periodic reloading of matrix-coturn is configured for SSL renewal (matrix-coturn-reload)
+- name: Ensure reloading systemd units installed, if necessary
   template:
-    src: "{{ role_path }}/templates/cron.d/matrix-coturn-ssl-reload.j2"
-    dest: /etc/cron.d/matrix-coturn-ssl-reload
+    src: "{{ role_path }}/templates/systemd/{{ item }}.j2"
+    dest: "{{ matrix_systemd_path }}/{{ item }}"
     mode: 0644
+  register: "matrix_coturn_systemd_service_change_results"
   when: "matrix_coturn_tls_enabled|bool"
+  with_items:
+    - matrix-coturn-reload.service
+    - matrix-coturn-reload.timer
 
 # A similar task exists in `setup_uninstall.yml`
-- name: Ensure matrix-coturn-ssl-reload cronjob removed
+- name: Ensure reloading systemd units uninstalled, if unnecessary
   file:
-    path: /etc/cron.d/matrix-coturn-ssl-reload
+    path: "{{ item }}"
     state: absent
+  register: "matrix_coturn_systemd_service_change_results"
   when: "not matrix_coturn_tls_enabled|bool"
+  with_items:
+    - matrix-coturn-reload.service
+    - matrix-coturn-reload.timer
+
+- name: Ensure systemd reloaded if systemd units changed
+  service:
+    daemon_reload: yes
+  when: "matrix_coturn_systemd_service_change_results.changed"

roles/matrix-coturn/tasks/setup_uninstall.yml

@@ -1,11 +1,5 @@
 ---
 
-# A similar task exists in `setup_install.yml`
-- name: Ensure matrix-coturn-ssl-reload cronjob removed
-  file:
-    path: /etc/cron.d/matrix-coturn-ssl-reload
-    state: absent
-
 - name: Check existence of matrix-coturn service
   stat:
     path: "{{ matrix_systemd_path }}/matrix-coturn.service"
@@ -17,28 +11,37 @@
     name: matrix-coturn
     state: stopped
     daemon_reload: yes
-  register: stopping_result
-  when: "not matrix_coturn_enabled|bool and matrix_coturn_service_stat.stat.exists"
+  when: "matrix_coturn_service_stat.stat.exists|bool"
 
-- name: Ensure matrix-coturn.service doesn't exist
+- name: Ensure matrix-coturn-reload.timer is stopped
+  service:
+    name: matrix-coturn
+    state: stopped
+    daemon_reload: yes
+  failed_when: false
+  when: "matrix_coturn_service_stat.stat.exists|bool"
+
+- name: Ensure systemd units don't exist
   file:
-    path: "{{ matrix_systemd_path }}/matrix-coturn.service"
+    path: "{{ matrix_systemd_path }}/{{ item }}"
     state: absent
-  when: "not matrix_coturn_enabled|bool and matrix_coturn_service_stat.stat.exists"
+  register: matrix_coturn_systemd_unit_uninstallation_result
+  with_items:
+    - matrix-coturn.service
+    - matrix-coturn-reload.service
+    - matrix-coturn-reload.timer
 
-- name: Ensure systemd reloaded after matrix-coturn.service removal
+- name: Ensure systemd reloaded after unit removal
   service:
     daemon_reload: yes
-  when: "not matrix_coturn_enabled|bool and matrix_coturn_service_stat.stat.exists"
+  when: "matrix_coturn_systemd_unit_uninstallation_result.changed|bool"
 
 - name: Ensure Matrix coturn paths don't exist
   file:
     path: "{{ matrix_coturn_base_path }}"
     state: absent
-  when: "not matrix_coturn_enabled|bool"
 
 - name: Ensure coturn Docker image doesn't exist
   docker_image:
     name: "{{ matrix_coturn_docker_image }}"
     state: absent
-  when: "not matrix_coturn_enabled|bool"

roles/matrix-coturn/templates/cron.d/matrix-coturn-ssl-reload.j2

@@ -1 +0,0 @@
-20 4 */5 * * root {{ matrix_host_command_systemctl }} reload matrix-coturn.service

roles/matrix-coturn/templates/systemd/matrix-coturn-reload.service.j2

@@ -0,0 +1,6 @@
+[Unit]
+Description=Reloads matrix-coturn so that new SSL certificates can kick in
+
+[Service]
+Type=oneshot
+ExecStart={{ matrix_host_command_systemctl }} reload matrix-coturn.service

roles/matrix-coturn/templates/systemd/matrix-coturn-reload.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Reloads matrix-coturn periodically so that new SSL certificates can kick in
+
+[Timer]
+Unit=matrix-coturn-reload.service
+OnCalendar=Sunday *-*-* 13:00:00
+RandomizedDelaySec=3h
+
+[Install]
+WantedBy=timers.target

roles/matrix-nginx-proxy/tasks/init.yml

@@ -1,3 +1,8 @@
 - set_fact:
     matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-nginx-proxy.service'] }}"
   when: matrix_nginx_proxy_enabled|bool
+
+- set_fact:
+    matrix_systemd_services_list: "{{ matrix_systemd_services_list + [item.name] }}"
+  when: "item.applicable|bool and item.enableable|bool"
+  with_items: "{{ matrix_ssl_renewal_systemd_units_list }}"

roles/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt.yml

@@ -10,53 +10,56 @@
     - "{{ matrix_local_bin_path }}/matrix-ssl-certificates-renew"
     - "{{ matrix_cron_path }}/matrix-ssl-certificate-renewal"
     - "{{ matrix_cron_path }}/matrix-nginx-proxy-periodic-restarter"
+    - "/etc/cron.d/matrix-ssl-lets-encrypt"
+    - "{{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew"
 
 #
 # Tasks related to setting up Let's Encrypt's management of certificates
 #
 
-- name: Ensure certbot Docker image is pulled
-  docker_image:
-    name: "{{ matrix_ssl_lets_encrypt_certbot_docker_image }}"
-    source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
-    force_source: "{{ matrix_ssl_lets_encrypt_certbot_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
-    force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_ssl_lets_encrypt_certbot_docker_image_force_pull }}"
-  when: "matrix_ssl_retrieval_method == 'lets-encrypt'"
+- block:
+    - name: Ensure certbot Docker image is pulled
+      docker_image:
+        name: "{{ matrix_ssl_lets_encrypt_certbot_docker_image }}"
+        source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
+        force_source: "{{ matrix_ssl_lets_encrypt_certbot_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
+        force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_ssl_lets_encrypt_certbot_docker_image_force_pull }}"
 
-- name: Obtain Let's Encrypt certificates
-  include_tasks: "{{ role_path }}/tasks/ssl/setup_ssl_lets_encrypt_obtain_for_domain.yml"
-  with_items: "{{ matrix_ssl_domains_to_obtain_certificates_for }}"
-  loop_control:
-    loop_var: domain_name
-  when: "matrix_ssl_retrieval_method == 'lets-encrypt'"
+    - name: Obtain Let's Encrypt certificates
+      include_tasks: "{{ role_path }}/tasks/ssl/setup_ssl_lets_encrypt_obtain_for_domain.yml"
+      with_items: "{{ matrix_ssl_domains_to_obtain_certificates_for }}"
+      loop_control:
+        loop_var: domain_name
 
-- name: Ensure Let's Encrypt SSL renewal script installed
-  template:
-    src: "{{ role_path }}/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2"
-    dest: "{{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew"
-    mode: 0750
-  when: "matrix_ssl_retrieval_method == 'lets-encrypt'"
+    - name: Ensure Let's Encrypt SSL renewal script installed
+      template:
+        src: "{{ role_path }}/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2"
+        dest: "{{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew"
+        mode: 0750
 
-- name: Ensure periodic SSL renewal cronjob configured
-  template:
-    src: "{{ role_path }}/templates/cron.d/matrix-ssl-lets-encrypt.j2"
-    dest: /etc/cron.d/matrix-ssl-lets-encrypt
-    mode: 0644
+    - name: Ensure SSL renewal systemd units installed
+      template:
+        src: "{{ role_path }}/templates/systemd/{{ item.name }}.j2"
+        dest: "{{ matrix_systemd_path }}/{{ item.name }}"
+        mode: 0644
+      when: "item.applicable|bool"
+      with_items: "{{ matrix_ssl_renewal_systemd_units_list }}"
   when: "matrix_ssl_retrieval_method == 'lets-encrypt'"
 
-
 #
 # Tasks related to getting rid of Let's Encrypt's management of certificates
 #
 
-- name: Ensure matrix-ssl-lets-encrypt-renew cronjob removed
-  file:
-    path: /etc/cron.d/matrix-ssl-lets-encrypt
-    state: absent
-  when: "matrix_ssl_retrieval_method != 'lets-encrypt'"
+- block:
+    - name: Ensure matrix-ssl-lets-encrypt-renew cronjob removed
+      file:
+        path: "{{ matrix_systemd_path }}/{{ item.name }}"
+        state: absent
+      when: "{{ not item.applicable }}"
+      with_items: "{{ matrix_ssl_renewal_systemd_units_list }}"
 
-- name: Ensure Let's Encrypt SSL renewal script removed
-  file:
-    path: "{{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew"
-    state: absent
+    - name: Ensure Let's Encrypt SSL renewal script removed
+      file:
+        path: "{{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew"
+        state: absent
   when: "matrix_ssl_retrieval_method != 'lets-encrypt'"

roles/matrix-nginx-proxy/templates/cron.d/matrix-ssl-lets-encrypt.j2

@@ -1,5 +0,0 @@
-MAILTO="{{ matrix_ssl_lets_encrypt_support_email }}"
-15 4 * * * root {{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew
-{% if matrix_nginx_proxy_enabled %}
-20 5 * * * root {{ matrix_host_command_systemctl }} reload matrix-nginx-proxy.service
-{% endif %}

roles/matrix-nginx-proxy/templates/systemd/matrix-ssl-lets-encrypt-certificates-renew.service.j2

@@ -0,0 +1,6 @@
+[Unit]
+Description=Renews Let's Encrypt SSL certificates
+
+[Service]
+Type=oneshot
+ExecStart={{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew

roles/matrix-nginx-proxy/templates/systemd/matrix-ssl-lets-encrypt-certificates-renew.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Renews Let's Encrypt SSL certificates periodically
+
+[Timer]
+Unit=matrix-ssl-lets-encrypt-certificates-renew.service
+OnCalendar=Sunday *-*-* 05:00:00
+RandomizedDelaySec=3h
+
+[Install]
+WantedBy=timers.target

roles/matrix-nginx-proxy/templates/systemd/matrix-ssl-nginx-proxy-reload.service.j2

@@ -0,0 +1,6 @@
+[Unit]
+Description=Reloads matrix-nginx-proxy so that new SSL certificates can kick in
+
+[Service]
+Type=oneshot
+ExecStart={{ matrix_host_command_systemctl }} reload matrix-nginx-proxy.service

roles/matrix-nginx-proxy/templates/systemd/matrix-ssl-nginx-proxy-reload.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Reloads matrix-nginx-proxy periodically so that new SSL certificates can kick in
+
+[Timer]
+Unit=matrix-ssl-nginx-proxy-reload.service
+OnCalendar=Sunday *-*-* 13:00:00
+RandomizedDelaySec=3h
+
+[Install]
+WantedBy=timers.target

roles/matrix-nginx-proxy/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2

@@ -24,7 +24,6 @@ docker run \
 		{% if matrix_ssl_lets_encrypt_staging %}
 			--staging \
 		{% endif %}
-		--quiet \
 		--standalone \
 		--preferred-challenges http \
 		--agree-tos \

roles/matrix-nginx-proxy/vars/main.yml

@@ -1,4 +1,18 @@
 ---
 
 # Tells whether this role had executed or not. Toggled to `true` during runtime.
-matrix_nginx_proxy_role_executed: false
+matrix_nginx_proxy_role_executed: false
+
+matrix_ssl_renewal_systemd_units_list:
+  - name: matrix-ssl-lets-encrypt-certificates-renew.service
+    applicable: "{{ matrix_ssl_retrieval_method == 'lets-encrypt' }}"
+    enableable: false
+  - name: matrix-ssl-lets-encrypt-certificates-renew.timer
+    applicable: "{{ matrix_ssl_retrieval_method == 'lets-encrypt' }}"
+    enableable: true
+  - name: matrix-ssl-nginx-proxy-reload.service
+    applicable: "{{ matrix_ssl_retrieval_method == 'lets-encrypt' and matrix_nginx_proxy_enabled|bool }}"
+    enableable: false
+  - name: matrix-ssl-nginx-proxy-reload.timer
+    applicable: "{{ matrix_ssl_retrieval_method == 'lets-encrypt' and matrix_nginx_proxy_enabled|bool }}"
+    enableable: true