Update faq.md and maintenance-and-troubleshooting.md:pull/4029/head
| @@ -403,46 +403,9 @@ It can perform a local connection instead. Just set `ansible_connection=local` a | |||||
| If you're running Ansible from within a container (one of the possibilities we list on our [dedicated Ansible documentation page](ansible.md)), then using `ansible_connection=local` is not possible. | If you're running Ansible from within a container (one of the possibilities we list on our [dedicated Ansible documentation page](ansible.md)), then using `ansible_connection=local` is not possible. | ||||
| ## Troubleshooting | |||||
| ## Maintenance and Troubleshooting | |||||
| ### I get "Error response from daemon: configured logging driver does not support reading" when I do `docker logs matrix-synapse`. | |||||
| See [How can I see the logs?](#how-can-i-see-the-logs). | |||||
| ### How can I see the logs? | |||||
| We utilize [systemd/journald](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html#Description) for logging. | |||||
| To see logs for Synapse, run `journalctl -fu matrix-synapse.service`. You may wish to see the [manual page for journalctl](https://www.commandlinux.com/man-page/man1/journalctl.1.html). | |||||
| Available service names can be seen by doing `ls /etc/systemd/system/matrix*.service` on the server. | |||||
| Some services also log to files in `/matrix/*/data/..`, but we're slowly moving away from that. | |||||
| We also disable Docker logging, so you can't use `docker logs matrix-*` either. We do this to prevent useless double (or even triple) logging and to avoid having to rotate log files. | |||||
| We just simply delegate logging to journald and it takes care of persistence and expiring old data. | |||||
| Also see: [How long do systemd/journald logs persist for?](#how-long-do-systemdjournald-logs-persist-for) | |||||
| ### How long do systemd/journald logs persist for? | |||||
| On some distros, the journald logs are just in-memory and not persisted to disk. | |||||
| Consult (and feel free to adjust) your distro's journald logging configuration in `/etc/systemd/journald.conf`. | |||||
| To enable persistence and put some limits on how large the journal log files can become, adjust your configuration like this: | |||||
| ```ini | |||||
| [Journal] | |||||
| RuntimeMaxUse=200M | |||||
| SystemMaxUse=1G | |||||
| RateLimitInterval=0 | |||||
| RateLimitBurst=0 | |||||
| Storage=persistent | |||||
| ``` | |||||
| ## Maintenance | |||||
| 💡 Also see this page for generic information about maintaining the services and troubleshooting: [Maintenance and Troubleshooting](maintenance-and-troubleshooting.md) | |||||
| ### Do I need to do anything to keep my Matrix server updated? | ### Do I need to do anything to keep my Matrix server updated? | ||||
| @@ -456,34 +419,17 @@ If you have an existing installation done using this Ansible playbook, you can e | |||||
| If your previous installation is done in some other way (not using this Ansible playbook), see [I installed Synapse some other way. Can I migrate such a setup to the playbook?](#i-installed-synapse-some-other-way-can-i-migrate-such-a-setup-to-the-playbook). | If your previous installation is done in some other way (not using this Ansible playbook), see [I installed Synapse some other way. Can I migrate such a setup to the playbook?](#i-installed-synapse-some-other-way-can-i-migrate-such-a-setup-to-the-playbook). | ||||
| ### How do I back up the data on my server? | |||||
| We haven't documented this properly yet, but the general advice is to: | |||||
| - back up Postgres by making a database dump. See [Backing up PostgreSQL](maintenance-postgres.md#backing-up-postgresql) | |||||
| - back up all `/matrix` files, except for `/matrix/postgres/data` (you already have a dump) and `/matrix/postgres/data-auto-upgrade-backup` (this directory may exist and contain your old data if you've [performed a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql)). | |||||
| You can later restore these by: | |||||
| - Restoring the `/matrix` directory and files on the new server manually | |||||
| - Following the instruction described on [Installing a server into which you'll import old data](installing.md#installing-a-server-into-which-youll-import-old-data) | |||||
| If your server's IP address has changed, you may need to [set up DNS](configuring-dns.md) again. | |||||
| ### What is this `/matrix/postgres/data-auto-upgrade-backup` directory that is taking up so much space? | ### What is this `/matrix/postgres/data-auto-upgrade-backup` directory that is taking up so much space? | ||||
| When you [perform a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql), we save the the old data files in `/matrix/postgres/data-auto-upgrade-backup`, just so you could easily restore them should something have gone wrong. | When you [perform a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql), we save the the old data files in `/matrix/postgres/data-auto-upgrade-backup`, just so you could easily restore them should something have gone wrong. | ||||
| After verifying that everything still works after the Postgres upgrade, you can safely delete `/matrix/postgres/data-auto-upgrade-backup` | After verifying that everything still works after the Postgres upgrade, you can safely delete `/matrix/postgres/data-auto-upgrade-backup` | ||||
| ### How do I debug or force SSL certificate renewal? | |||||
| SSL certificates are managed automatically by the [Traefik](https://doc.traefik.io/traefik/) reverse-proxy server. | |||||
| ### I get "Error response from daemon: configured logging driver does not support reading" when I run `docker logs matrix-synapse`. Why? | |||||
| If you're having trouble with SSL certificate renewal, check the Traefik logs (`journalctl -fu matrix-traefik`). | |||||
| To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you cannot view logs using `docker logs matrix-*`. | |||||
| If you're [using your own webserver](configuring-playbook-own-webserver.md) instead of the integrated one (Traefik), you should investigate in another way. | |||||
| See [this section](maintenance-and-troubleshooting.md#how-to-see-the-logs) on the page for maintenance and troubleshooting for more details to see the logs. | |||||
| ## Miscellaneous | ## Miscellaneous | ||||
| @@ -1,6 +1,43 @@ | |||||
| # Maintenance and Troubleshooting | # Maintenance and Troubleshooting | ||||
| ## How to see the current status of your services | |||||
| ## Maintenance | |||||
| ### How to back up the data on your server | |||||
| We haven't documented this properly yet, but the general advice is to: | |||||
| - back up Postgres by making a database dump. See [Backing up PostgreSQL](maintenance-postgres.md#backing-up-postgresql) | |||||
| - back up all `/matrix` files, except for `/matrix/postgres/data` (you already have a dump) and `/matrix/postgres/data-auto-upgrade-backup` (this directory may exist and contain your old data if you've [performed a major Postgres upgrade](maintenance-postgres.md#upgrading-postgresql)). | |||||
| You can later restore these by: | |||||
| - Restoring the `/matrix` directory and files on the new server manually | |||||
| - Following the instruction described on [Installing a server into which you'll import old data](installing.md#installing-a-server-into-which-youll-import-old-data) | |||||
| If your server's IP address has changed, you may need to [set up DNS](configuring-dns.md) again. | |||||
| ### Remove unused Docker data | |||||
| You can free some disk space from Docker by removing its unused data. See [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. | |||||
| ```sh | |||||
| ansible-playbook -i inventory/hosts setup.yml --tags=run-docker-prune | |||||
| ``` | |||||
| The shortcut command with `just` program is also available: `just run-tags run-docker-prune` | |||||
| ### Postgres | |||||
| See the dedicated [PostgreSQL maintenance](maintenance-postgres.md) documentation page. | |||||
| ### Synapse | |||||
| See the dedicated [Synapse maintenance](maintenance-synapse.md) documentation page. | |||||
| ## Troubleshooting | |||||
| ### How to see the current status of your services | |||||
| You can check the status of your services by using `systemctl status`. Example: | You can check the status of your services by using `systemctl status`. Example: | ||||
| @@ -12,17 +49,38 @@ sudo systemctl status matrix-synapse | |||||
| Active: active (running) since Sun 2024-01-14 09:13:06 UTC; 1h 31min ago | Active: active (running) since Sun 2024-01-14 09:13:06 UTC; 1h 31min ago | ||||
| ``` | ``` | ||||
| Docker containers that the playbook configures are supervised by [systemd](https://wiki.archlinux.org/title/Systemd) and their logs are configured to go to [systemd-journald](https://wiki.archlinux.org/title/Systemd/Journal). | |||||
| ### How to see the logs | |||||
| To prevent double-logging, Docker logging is disabled by explicitly passing `--log-driver=none` to all containers. Due to this, you **cannot** view logs using `docker logs`. | |||||
| Docker containers that the playbook configures are supervised by [systemd](https://wiki.archlinux.org/title/Systemd) and their logs are configured to go to [systemd-journald](https://wiki.archlinux.org/title/Systemd/Journal). | |||||
| To view systemd-journald logs using [journalctl](https://man.archlinux.org/man/journalctl.1), run a command like this: | |||||
| For example, you can find the logs of `matrix-synapse` in `systemd-journald` by logging in to the server with SSH and running the command as below: | |||||
| ```sh | ```sh | ||||
| sudo journalctl -fu matrix-synapse | sudo journalctl -fu matrix-synapse | ||||
| ``` | ``` | ||||
| ## How to check if services work | |||||
| Available service names can be seen by doing `ls /etc/systemd/system/matrix*.service` on the server. Some services also log to files in `/matrix/*/data/..`, but we're slowly moving away from that. | |||||
| We just simply delegate logging to journald and it takes care of persistence and expiring old data. | |||||
| #### Enable systemd/journald logs persistence | |||||
| On some distros, the journald logs are just in-memory and not persisted to disk. | |||||
| Consult (and feel free to adjust) your distro's journald logging configuration in `/etc/systemd/journald.conf`. | |||||
| To enable persistence and put some limits on how large the journal log files can become, adjust your configuration like this: | |||||
| ```ini | |||||
| [Journal] | |||||
| RuntimeMaxUse=200M | |||||
| SystemMaxUse=1G | |||||
| RateLimitInterval=0 | |||||
| RateLimitBurst=0 | |||||
| Storage=persistent | |||||
| ``` | |||||
| ### How to check if services work | |||||
| The playbook can perform a check to ensure that you've configured things correctly and that services are running. | The playbook can perform a check to ensure that you've configured things correctly and that services are running. | ||||
| @@ -38,16 +96,10 @@ If it's all green, everything is probably running correctly. | |||||
| Besides this self-check, you can also check whether your server federates with the Matrix network by using the [Federation Tester](https://federationtester.matrix.org/) against your base domain (`example.com`), not the `matrix.example.com` subdomain. | Besides this self-check, you can also check whether your server federates with the Matrix network by using the [Federation Tester](https://federationtester.matrix.org/) against your base domain (`example.com`), not the `matrix.example.com` subdomain. | ||||
| ## Remove unused Docker data | |||||
| ### How to debug or force SSL certificate renewal | |||||
| You can free some disk space from Docker, see [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/) for more information. | |||||
| ```sh | |||||
| ansible-playbook -i inventory/hosts setup.yml --tags=run-docker-prune | |||||
| ``` | |||||
| The shortcut command with `just` program is also available: `just run-tags run-docker-prune` | |||||
| SSL certificates are managed automatically by the [Traefik](https://doc.traefik.io/traefik/) reverse-proxy server. | |||||
| ## Postgres | |||||
| If you're having trouble with SSL certificate renewal, check the Traefik logs (`journalctl -fu matrix-traefik`). | |||||
| See the dedicated [PostgreSQL Maintenance](maintenance-postgres.md) documentation page. | |||||
| If you're [using your own webserver](configuring-playbook-own-webserver.md) instead of the integrated one (Traefik), you should investigate in another way. | |||||