c-l/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2024-12-28 22:51:59 +00:00
Code Releases Activity

feat(docs): microservices be gone (#9558)

Browse source 
* env vars

* docs

* more info on workers

* fix

* typo

* fix bash

* typo

* service -> contianer

* worker -> workers

* updating jobs and workers

* split workers

* redis

* fix conflict

* node -> immich

* add NO_COLOR

* Update docs/docs/administration/jobs-workers.md

* chore: fix broken links now jobs page is gone

* Update environment-variables.md

* Update environment-variables.md

---------

Co-authored-by: Zack Pollard <zackpollard@ymail.com>
This commit is contained in:
Matthew Momjian 2024-05-30 09:25:27 -04:00 committed by GitHub
parent 53562e8439
commit 5ef2553bca
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 131 additions and 90 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/docs/FAQ.mdx
View file

@ -67,7 +67,7 @@ Yes, with an [External Library](/docs/features/libraries.md).
### What happens to existing files after I choose a new [Storage Template](/docs/administration/storage-template.mdx)?
Template changes will only apply to _new_ assets. To retroactively apply the template to previously uploaded assets, run the Storage Migration Job, available on the [Jobs](/docs/administration/jobs.md) page.
Template changes will only apply to _new_ assets. To retroactively apply the template to previously uploaded assets, run the Storage Migration Job, available on the [Jobs](/docs/administration/jobs-workers/#jobs) page.
### Why are only photos and not videos being uploaded to Immich?

55
docs/docs/administration/jobs-workers.md Normal file
View file

@ -0,0 +1,55 @@
# Jobs and Workers
## Workers
### Architecture
The `immich-server` container contains multiple workers:
- `api`: responds to API requests for data and files for the web and mobile app.
- `microservices`: handles most other work, such as thumbnail generation and video encoding, in the form of _jobs_. Simply put, a job is a request to process data in the background.
## Split workers
If you prefer to throttle or distribute the workers, you can do this using the [environment variables](/docs/install/environment-variables) to specify which container should pick up which tasks.
For example, for a simple setup with one container for the Web/API and one for all other microservices, you can do the following:
Copy the entire `immich-server` block as a new service and make the following changes to the **copy**:
```diff
- immich-server:
- container_name: immich_server
...
- ports:
- - 2283:3001
+ immich-microservices:
+ container_name: immich_microservices
```
Once you have two copies of the immich-server service, make the following chnages to each one. This will allow one container to only serve the web UI and API, and the other one to handle all other tasks.
```diff
services:
immich-server:
...
+ environment:
+ IMMICH_WORKERS_INCLUDE: 'api'
immich-microservices:
...
+ environment:
+ IMMICH_WORKERS_EXCLUDE: 'api'
```
## Jobs
When a new asset is uploaded it kicks off a series of jobs, which include metadata extraction, thumbnail generation, machine learning tasks, and storage template migration, if enabled. To view the status of a job navigate to the Administration -> Jobs page.
Additionally, some jobs run on a schedule, which is every night at midnight. This schedule, with the exception of [External Libraries](/docs/features/libraries) scanning, cannot be changed.
:::info
Storage Migration job can be run after changing the [Storage Template](/docs/administration/storage-template.mdx), in order to apply the change to the existing library.
:::
<img src={require('./img/admin-jobs.png').default} width="80%" title="Admin jobs" />

13
docs/docs/administration/jobs.md
View file

@ -1,13 +0,0 @@
# Jobs
The `immich-server` responds to API requests for data and files for the web and mobile app. To do this quickly and reliably, it offloads most other work to `immich-microservices` in the form of _jobs_. Simply put, a job is a request to process data in the background. Jobs are picked up automatically by microservices containers.
When a new asset is uploaded it kicks off a series of jobs, which include metadata extraction, thumbnail generation, machine learning tasks, and storage template migration, if enabled. To view the status of a job navigate to the Administration -> Jobs page.
Additionally, some jobs run on a schedule, which is every night at midnight. This schedule, with the exception of [External Libraries](/docs/features/libraries) scanning, cannot be changed.
:::info
Storage Migration job can be run after changing the [Storage Template](/docs/administration/storage-template.mdx), in order to apply the change to the existing library.
:::
<img src={require('./img/admin-jobs.png').default} width="80%" title="Admin jobs" />

2
docs/docs/developer/architecture.mdx
View file

@ -80,7 +80,7 @@ The Immich Microservices image uses the same `Dockerfile` as the Immich Server,
- Background jobs (file deletion, user deletion)
:::info
This list closely matches what is available on the [Administration > Jobs](/docs/administration/jobs.md) page, which provides some remote queue management capabilities.
This list closely matches what is available on the [Administration > Jobs](/docs/administration/jobs-workers/#jobs) page, which provides some remote queue management capabilities.
:::
### Machine Learning

17
docs/docs/features/libraries.md
View file

@ -44,16 +44,16 @@ If the import paths are edited in a way that an external file is no longer in an
### Troubleshooting
Sometimes, an external library will not scan correctly. This can happen if immich_server or immich_microservices can't access the files. Here are some things to check:
Sometimes, an external library will not scan correctly. This can happen if Immich can't access the files. Here are some things to check:
- In the docker-compose file, are the volumes mounted correctly?
- Are the volumes identical between the `server` and `microservices` container?
- Are the volumes also mounted to any worker containers?
- Are the import paths set correctly, and do they match the path set in docker-compose file?
- Make sure you don't use symlinks in your import libraries, and that you aren't linking across docker mounts.
- Are the permissions set correctly?
- Make sure you are using forward slashes (`/`) and not backward slashes.
To validate that Immich can reach your external library, start a shell inside the container. Run `docker exec -it immich_microservices /bin/bash` to a bash shell. If your import path is `/data/import/photos`, check it with `ls /data/import/photos`. Do the same check for the `immich_server` container. If you cannot access this directory in both the `microservices` and `server` containers, Immich won't be able to import files.
To validate that Immich can reach your external library, start a shell inside the container. Run `docker exec -it immich_server bash` to a bash shell. If your import path is `/data/import/photos`, check it with `ls /data/import/photos`. Do the same check for the same in any microservices containers.
### Exclusion Patterns
@ -98,7 +98,7 @@ First, we need to plan how we want to organize the libraries. The christmas trip
### Mount Docker Volumes
`immich-server` and `immich-microservices` containers will need access to the gallery. Modify your docker compose file as follows
The `immich-server` container will need access to the gallery. Modify your docker compose file as follows
```diff title="docker-compose.yml"
immich-server:
@ -107,15 +107,6 @@ First, we need to plan how we want to organize the libraries. The christmas trip
+ - /mnt/nas/christmas-trip:/mnt/media/christmas-trip:ro
+ - /home/user/old-pics:/mnt/media/old-pics:ro
+ - /mnt/media/videos:/mnt/media/videos:ro
+ - "C:/Users/user_name/Desktop/my media:/mnt/media/my-media:ro" # import path in Windows system.
immich-microservices:
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
+ - /mnt/nas/christmas-trip:/mnt/media/christmas-trip:ro
+ - /home/user/old-pics:/mnt/media/old-pics:ro
+ - /mnt/media/videos:/mnt/media/videos:ro
+ - "C:/Users/user_name/Desktop/my media:/mnt/media/my-media:ro" # import path in Windows system.
```

2
docs/docs/guides/custom-locations.md
View file

@ -9,7 +9,7 @@ It is important to remember to update the backup settings after following the gu
In our `.env` file, we will define variables that will help us in the future when we want to move to a more advanced server in the future
```diff title=".env"
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
# You can find documentation for all the supported env variables [here](/docs/install/environment-variables)
# Custom location where your uploaded, thumbnails, and transcoded video files are stored
- UPLOAD_LOCATION=./library

6
docs/docs/guides/docker-help.md
View file

@ -11,9 +11,8 @@ docker ps -a # see a list of running and stopped containers
```bash
docker exec -it <id or name> <command> # attach to a container with a command
docker exec -it immich_server sh
docker exec -it immich_microservices sh
docker exec -it immich_machine_learning sh
docker exec -it immich_server bash
docker exec -it immich_machine_learning bash
```
## Logs
@ -22,7 +21,6 @@ docker exec -it immich_machine_learning sh
docker logs <id or name> # see the logs for a specific container (by id or name)
docker logs immich_server
docker logs immich_microservices
docker logs immich_machine_learning
```

8
docs/docs/guides/external-library.md
View file

@ -6,20 +6,14 @@ in a directory on the same machine.
# Mount the directory into the containers.
Edit `docker-compose.yml` to add two new mount points in the sections `immich-server:` and `immich-microservices:` under `volumes:`
Edit `docker-compose.yml` to add two new mount points in the section `immich-server:` under `volumes:`
```diff
immich-server:
volumes:
+ - ${EXTERNAL_PATH}:/usr/src/app/external
immich-microservices:
volumes:
+ - ${EXTERNAL_PATH}:/usr/src/app/external
```
Be sure to add exactly the same path to both services.
Edit `.env` to define `EXTERNAL_PATH`, substituting in the correct path for your computer:
```

2
docs/docs/guides/remote-machine-learning.md
View file

@ -7,7 +7,7 @@ To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-imm
- Start the container by running `docker compose up -d`.
:::info
Starting with version v1.93.0 face detection work and face recognize were split. From now on face detection is done in the immich_machine_learning service, but facial recognition is done in the immich_microservices service.
Starting with version v1.93.0 face detection work and face recognize were split. From now on face detection is done in the immich_machine_learning container, but facial recognition is done in the `microservices` worker.
:::
:::note

114
docs/docs/install/environment-variables.md
View file

@ -17,11 +17,11 @@ If this should not work, try running `docker compose up -d --force-recreate`.
## Docker Compose
| Variable | Description | Default | Services |
| :----------------- | :------------------------------ | :-------: | :-------------------------------------- |
| `IMMICH_VERSION` | Image tags | `release` | server, microservices, machine learning |
| `UPLOAD_LOCATION` | Host Path for uploads | | server, microservices |
| `DB_DATA_LOCATION` | Host Path for Postgres database | | database |
| Variable | Description | Default | Containers |
| :----------------- | :------------------------------ | :-------: | :----------------------- |
| `IMMICH_VERSION` | Image tags | `release` | server, machine learning |
| `UPLOAD_LOCATION` | Host Path for uploads | | server |
| `DB_DATA_LOCATION` | Host Path for Postgres database | | database |
:::tip
These environment variables are used by the `docker-compose.yml` file and do **NOT** affect the containers directly.
@ -38,15 +38,16 @@ Regardless of filesystem, it is not recommended to use a network share for your
## General
| Variable | Description | Default | Services |
| :------------------------------ | :------------------------------------------- | :----------------------: | :-------------------------------------- |
| `TZ` | Timezone | | microservices |
| `IMMICH_ENV` | Environment (production, development) | `production` | server, microservices, machine learning |
| `IMMICH_LOG_LEVEL` | Log Level (verbose, debug, log, warn, error) | `log` | server, microservices, machine learning |
| `IMMICH_MEDIA_LOCATION` | Media Location | `./upload`<sup>\*1</sup> | server, microservices |
| `IMMICH_CONFIG_FILE` | Path to config file | | server, microservices |
| `IMMICH_WEB_ROOT` | Path of root index.html | `/usr/src/app/www` | server |
| `IMMICH_REVERSE_GEOCODING_ROOT` | Path of reverse geocoding dump directory | `/usr/src/resources` | microservices |
| Variable | Description | Default | Containers | Workers |
| :------------------------------ | :---------------------------------------------- | :----------------------: | :----------------------- | :----------------- |
| `TZ` | Timezone | | server | microservices |
| `IMMICH_ENV` | Environment (production, development) | `production` | server, machine learning | api, microservices |
| `IMMICH_LOG_LEVEL` | Log Level (verbose, debug, log, warn, error) | `log` | server, machine learning | api, microservices |
| `IMMICH_MEDIA_LOCATION` | Media Location | `./upload`<sup>\*1</sup> | server | api, microservices |
| `IMMICH_CONFIG_FILE` | Path to config file | | server | api, microservices |
| `IMMICH_WEB_ROOT` | Path of root index.html | `/usr/src/app/www` | server | api |
| `IMMICH_REVERSE_GEOCODING_ROOT` | Path of reverse geocoding dump directory | `/usr/src/resources` | server | microservices |
| `NO_COLOR` | Set to `true` to disable color-coded log output | `false` | server, machine learning | |
\*1: With the default `WORKDIR` of `/usr/src/app`, this path will resolve to `/usr/src/app/upload`.
It only need to be set if the Immich deployment method is changing.
@ -54,28 +55,39 @@ It only need to be set if the Immich deployment method is changing.
:::tip
`TZ` should be set to a `TZ identifier` from [this list][tz-list]. For example, `TZ="Etc/UTC"`.
`TZ` is only used by `exiftool`, which is present in the microservices container, as a fallback in case the timezone cannot be determined from the image metadata.
`TZ` is used by `exiftool` as a fallback in case the timezone cannot be determined from the image metadata. It is also used for logfile timestamps and cron job execution.
:::
## Workers
| Variable | Description | Default | Containers |
| :----------------------- | :--------------------------------------------------------------------------------------------------- | :-----: | :--------- |
| `IMMICH_WORKERS_INCLUDE` | Only run these workers. | | server |
| `IMMICH_WORKERS_EXCLUDE` | Do not run these workers. Matches against default workers, or `IMMICH_WORKERS_INCLUDE` if specified. | | server |
:::info
Information on the current workers can be found [here](/docs/administration/jobs-workers).
:::
## Ports
| Variable | Description | Default |
| :------------ | :------------- | :------------------------------------: |
| `IMMICH_HOST` | Listening host | `0.0.0.0` |
| `IMMICH_PORT` | Listening port | 3001 (server), 3003 (machine learning) |
| Variable | Description | Default |
| :------------ | :------------- | :----------------------------------------: |
| `IMMICH_HOST` | Listening host | `0.0.0.0` |
| `IMMICH_PORT` | Listening port | `3001` (server), `3003` (machine learning) |
## Database
| Variable | Description | Default | Services |
| :---------------------------------- | :----------------------------------------------------------------------- | :----------: | :-------------------------------------------- |
| `DB_URL` | Database URL | | server, microservices |
| `DB_HOSTNAME` | Database Host | `database` | server, microservices |
| `DB_PORT` | Database Port | `5432` | server, microservices |
| `DB_USERNAME` | Database User | `postgres` | server, microservices, database<sup>\*1</sup> |
| `DB_PASSWORD` | Database Password | `postgres` | server, microservices, database<sup>\*1</sup> |
| `DB_DATABASE_NAME` | Database Name | `immich` | server, microservices, database<sup>\*1</sup> |
| `DB_VECTOR_EXTENSION`<sup>\*2</sup> | Database Vector Extension (one of [`pgvector`, `pgvecto.rs`]) | `pgvecto.rs` | server, microservices |
| `DB_SKIP_MIGRATIONS` | Whether to skip running migrations on startup (one of [`true`, `false`]) | `false` | server, microservices |
| Variable | Description | Default | Containers |
| :---------------------------------- | :----------------------------------------------------------------------- | :----------: | :----------------------------- |
| `DB_URL` | Database URL | | server |
| `DB_HOSTNAME` | Database Host | `database` | server |
| `DB_PORT` | Database Port | `5432` | server |
| `DB_USERNAME` | Database User | `postgres` | server, database<sup>\*1</sup> |
| `DB_PASSWORD` | Database Password | `postgres` | server, database<sup>\*1</sup> |
| `DB_DATABASE_NAME` | Database Name | `immich` | server, database<sup>\*1</sup> |
| `DB_VECTOR_EXTENSION`<sup>\*2</sup> | Database Vector Extension (one of [`pgvector`, `pgvecto.rs`]) | `pgvecto.rs` | server |
| `DB_SKIP_MIGRATIONS` | Whether to skip running migrations on startup (one of [`true`, `false`]) | `false` | server |
\*1: The values of `DB_USERNAME`, `DB_PASSWORD`, and `DB_DATABASE_NAME` are passed to the Postgres container as the variables `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` in `docker-compose.yml`.
@ -83,30 +95,34 @@ It only need to be set if the Immich deployment method is changing.
:::info
All `DB_` variables must be provided to all Immich workers, including `api` and `microservices`.
`DB_URL` must be in the format `postgresql://immichdbusername:immichdbpassword@postgreshost:postgresport/immichdatabasename`.
You can require SSL by adding `?sslmode=require` to the end of the `DB_URL` string, or require SSL and skip certificate verification by adding `?sslmode=require&sslmode=no-verify`.
When `DB_URL` is defined, the `DB_HOSTNAME`, `DB_PORT`, `DB_USERNAME`, `DB_PASSWORD` and `DB_DATABASE_NAME` database variables are ignored.
:::
## Redis
| Variable | Description | Default | Services |
| :--------------- | :------------- | :-----: | :-------------------- |
| `REDIS_URL` | Redis URL | | server, microservices |
| `REDIS_HOSTNAME` | Redis Host | `redis` | server, microservices |
| `REDIS_PORT` | Redis Port | `6379` | server, microservices |
| `REDIS_DBINDEX` | Redis DB Index | `0` | server, microservices |
| `REDIS_USERNAME` | Redis Username | | server, microservices |
| `REDIS_PASSWORD` | Redis Password | | server, microservices |
| `REDIS_SOCKET` | Redis Socket | | server, microservices |
| Variable | Description | Default | Containers |
| :--------------- | :------------- | :-----: | :--------- |
| `REDIS_URL` | Redis URL | | server |
| `REDIS_SOCKET` | Redis Socket | | server |
| `REDIS_HOSTNAME` | Redis Host | `redis` | server |
| `REDIS_PORT` | Redis Port | `6379` | server |
| `REDIS_USERNAME` | Redis Username | | server |
| `REDIS_PASSWORD` | Redis Password | | server |
| `REDIS_DBINDEX` | Redis DB Index | `0` | server |
:::info
All `REDIS_` variables must be provided to all Immich workers, including `api` and `microservices`.
`REDIS_URL` must start with `ioredis://` and then include a `base64` encoded JSON string for the configuration.
More info can be found in the upstream [ioredis][redis-api] documentation.
- When `REDIS_URL` is defined, the other redis (`REDIS_*`) variables are ignored.
- When `REDIS_SOCKET` is defined, the other redis (`REDIS_*`) variables are ignored.
When `REDIS_URL` or `REDIS_SOCKET` are defined, the `REDIS_HOSTNAME`, `REDIS_PORT`, `REDIS_USERNAME`, `REDIS_PASSWORD`, and `REDIS_DBINDEX` variables are ignored.
:::
Redis (Sentinel) URL example JSON before encoding:
@ -138,7 +154,7 @@ Redis (Sentinel) URL example JSON before encoding:
## Machine Learning
| Variable | Description | Default | Services |
| Variable | Description | Default | Containers |
| :----------------------------------------------- | :------------------------------------------------------------------- | :-----------------: | :--------------- |
| `MACHINE_LEARNING_MODEL_TTL` | Inactivity time (s) before a model is unloaded (disabled if \<= 0) | `300` | machine learning |
| `MACHINE_LEARNING_MODEL_TTL_POLL_S` | Interval (s) between checks for the model TTL (disabled if \<= 0) | `10` | machine learning |
@ -163,13 +179,13 @@ Other machine learning parameters can be tuned from the admin UI.
## Prometheus
| Variable | Description | Default | Services |
| :----------------------------- | :-------------------------------------------------------------------------------------------- | :-----: | :-------------------- |
| `IMMICH_METRICS`<sup>\*1</sup> | Toggle all metrics (one of [`true`, `false`]) | | server, microservices |
| `IMMICH_API_METRICS` | Toggle metrics for endpoints and response times (one of [`true`, `false`]) | | server, microservices |
| `IMMICH_HOST_METRICS` | Toggle metrics for CPU and memory utilization for host and process (one of [`true`, `false`]) | | server, microservices |
| `IMMICH_IO_METRICS` | Toggle metrics for database queries, image processing, etc. (one of [`true`, `false`]) | | server, microservices |
| `IMMICH_JOB_METRICS` | Toggle metrics for jobs and queues (one of [`true`, `false`]) | | server, microservices |
| Variable | Description | Default | Containers | Workers |
| :----------------------------- | :-------------------------------------------------------------------------------------------- | :-----: | :--------- | :----------------- |
| `IMMICH_METRICS`<sup>\*1</sup> | Toggle all metrics (one of [`true`, `false`]) | | server | api, microservices |
| `IMMICH_API_METRICS` | Toggle metrics for endpoints and response times (one of [`true`, `false`]) | | server | api, microservices |
| `IMMICH_HOST_METRICS` | Toggle metrics for CPU and memory utilization for host and process (one of [`true`, `false`]) | | server | api, microservices |
| `IMMICH_IO_METRICS` | Toggle metrics for database queries, image processing, etc. (one of [`true`, `false`]) | | server | api, microservices |
| `IMMICH_JOB_METRICS` | Toggle metrics for jobs and queues (one of [`true`, `false`]) | | server | api, microservices |
\*1: Overridden for a metric group when its corresponding environmental variable is set.