diff --git a/docs/docs/FAQ.mdx b/docs/docs/FAQ.mdx index 5458a22620..590094817e 100644 --- a/docs/docs/FAQ.mdx +++ b/docs/docs/FAQ.mdx @@ -67,7 +67,7 @@ Yes, with an [External Library](/docs/features/libraries.md). ### What happens to existing files after I choose a new [Storage Template](/docs/administration/storage-template.mdx)? -Template changes will only apply to _new_ assets. To retroactively apply the template to previously uploaded assets, run the Storage Migration Job, available on the [Jobs](/docs/administration/jobs.md) page. +Template changes will only apply to _new_ assets. To retroactively apply the template to previously uploaded assets, run the Storage Migration Job, available on the [Jobs](/docs/administration/jobs-workers/#jobs) page. ### Why are only photos and not videos being uploaded to Immich? diff --git a/docs/docs/administration/jobs-workers.md b/docs/docs/administration/jobs-workers.md new file mode 100644 index 0000000000..ed07e1bcc9 --- /dev/null +++ b/docs/docs/administration/jobs-workers.md @@ -0,0 +1,55 @@ +# Jobs and Workers + +## Workers + +### Architecture + +The `immich-server` container contains multiple workers: + +- `api`: responds to API requests for data and files for the web and mobile app. +- `microservices`: handles most other work, such as thumbnail generation and video encoding, in the form of _jobs_. Simply put, a job is a request to process data in the background. + +## Split workers + +If you prefer to throttle or distribute the workers, you can do this using the [environment variables](/docs/install/environment-variables) to specify which container should pick up which tasks. + +For example, for a simple setup with one container for the Web/API and one for all other microservices, you can do the following: + +Copy the entire `immich-server` block as a new service and make the following changes to the **copy**: + +```diff +- immich-server: +- container_name: immich_server +... +- ports: +- - 2283:3001 ++ immich-microservices: ++ container_name: immich_microservices +``` + +Once you have two copies of the immich-server service, make the following chnages to each one. This will allow one container to only serve the web UI and API, and the other one to handle all other tasks. + +```diff +services: + immich-server: + ... ++ environment: ++ IMMICH_WORKERS_INCLUDE: 'api' + + immich-microservices: + ... ++ environment: ++ IMMICH_WORKERS_EXCLUDE: 'api' +``` + +## Jobs + +When a new asset is uploaded it kicks off a series of jobs, which include metadata extraction, thumbnail generation, machine learning tasks, and storage template migration, if enabled. To view the status of a job navigate to the Administration -> Jobs page. + +Additionally, some jobs run on a schedule, which is every night at midnight. This schedule, with the exception of [External Libraries](/docs/features/libraries) scanning, cannot be changed. + +:::info +Storage Migration job can be run after changing the [Storage Template](/docs/administration/storage-template.mdx), in order to apply the change to the existing library. +::: + + diff --git a/docs/docs/administration/jobs.md b/docs/docs/administration/jobs.md deleted file mode 100644 index 9c8536cfa7..0000000000 --- a/docs/docs/administration/jobs.md +++ /dev/null @@ -1,13 +0,0 @@ -# Jobs - -The `immich-server` responds to API requests for data and files for the web and mobile app. To do this quickly and reliably, it offloads most other work to `immich-microservices` in the form of _jobs_. Simply put, a job is a request to process data in the background. Jobs are picked up automatically by microservices containers. - -When a new asset is uploaded it kicks off a series of jobs, which include metadata extraction, thumbnail generation, machine learning tasks, and storage template migration, if enabled. To view the status of a job navigate to the Administration -> Jobs page. - -Additionally, some jobs run on a schedule, which is every night at midnight. This schedule, with the exception of [External Libraries](/docs/features/libraries) scanning, cannot be changed. - -:::info -Storage Migration job can be run after changing the [Storage Template](/docs/administration/storage-template.mdx), in order to apply the change to the existing library. -::: - - diff --git a/docs/docs/developer/architecture.mdx b/docs/docs/developer/architecture.mdx index e6fa232e24..cf004a1119 100644 --- a/docs/docs/developer/architecture.mdx +++ b/docs/docs/developer/architecture.mdx @@ -80,7 +80,7 @@ The Immich Microservices image uses the same `Dockerfile` as the Immich Server, - Background jobs (file deletion, user deletion) :::info -This list closely matches what is available on the [Administration > Jobs](/docs/administration/jobs.md) page, which provides some remote queue management capabilities. +This list closely matches what is available on the [Administration > Jobs](/docs/administration/jobs-workers/#jobs) page, which provides some remote queue management capabilities. ::: ### Machine Learning diff --git a/docs/docs/features/libraries.md b/docs/docs/features/libraries.md index 782dfdac14..680067c8db 100644 --- a/docs/docs/features/libraries.md +++ b/docs/docs/features/libraries.md @@ -44,16 +44,16 @@ If the import paths are edited in a way that an external file is no longer in an ### Troubleshooting -Sometimes, an external library will not scan correctly. This can happen if immich_server or immich_microservices can't access the files. Here are some things to check: +Sometimes, an external library will not scan correctly. This can happen if Immich can't access the files. Here are some things to check: - In the docker-compose file, are the volumes mounted correctly? -- Are the volumes identical between the `server` and `microservices` container? +- Are the volumes also mounted to any worker containers? - Are the import paths set correctly, and do they match the path set in docker-compose file? - Make sure you don't use symlinks in your import libraries, and that you aren't linking across docker mounts. - Are the permissions set correctly? - Make sure you are using forward slashes (`/`) and not backward slashes. -To validate that Immich can reach your external library, start a shell inside the container. Run `docker exec -it immich_microservices /bin/bash` to a bash shell. If your import path is `/data/import/photos`, check it with `ls /data/import/photos`. Do the same check for the `immich_server` container. If you cannot access this directory in both the `microservices` and `server` containers, Immich won't be able to import files. +To validate that Immich can reach your external library, start a shell inside the container. Run `docker exec -it immich_server bash` to a bash shell. If your import path is `/data/import/photos`, check it with `ls /data/import/photos`. Do the same check for the same in any microservices containers. ### Exclusion Patterns @@ -98,7 +98,7 @@ First, we need to plan how we want to organize the libraries. The christmas trip ### Mount Docker Volumes -`immich-server` and `immich-microservices` containers will need access to the gallery. Modify your docker compose file as follows +The `immich-server` container will need access to the gallery. Modify your docker compose file as follows ```diff title="docker-compose.yml" immich-server: @@ -107,15 +107,6 @@ First, we need to plan how we want to organize the libraries. The christmas trip + - /mnt/nas/christmas-trip:/mnt/media/christmas-trip:ro + - /home/user/old-pics:/mnt/media/old-pics:ro + - /mnt/media/videos:/mnt/media/videos:ro -+ - "C:/Users/user_name/Desktop/my media:/mnt/media/my-media:ro" # import path in Windows system. - - - immich-microservices: - volumes: - - ${UPLOAD_LOCATION}:/usr/src/app/upload -+ - /mnt/nas/christmas-trip:/mnt/media/christmas-trip:ro -+ - /home/user/old-pics:/mnt/media/old-pics:ro -+ - /mnt/media/videos:/mnt/media/videos:ro + - "C:/Users/user_name/Desktop/my media:/mnt/media/my-media:ro" # import path in Windows system. ``` diff --git a/docs/docs/guides/custom-locations.md b/docs/docs/guides/custom-locations.md index 3e70345caa..69844f3dba 100644 --- a/docs/docs/guides/custom-locations.md +++ b/docs/docs/guides/custom-locations.md @@ -9,7 +9,7 @@ It is important to remember to update the backup settings after following the gu In our `.env` file, we will define variables that will help us in the future when we want to move to a more advanced server in the future ```diff title=".env" -# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables +# You can find documentation for all the supported env variables [here](/docs/install/environment-variables) # Custom location where your uploaded, thumbnails, and transcoded video files are stored - UPLOAD_LOCATION=./library diff --git a/docs/docs/guides/docker-help.md b/docs/docs/guides/docker-help.md index 37fdf88cba..5f03de2d04 100644 --- a/docs/docs/guides/docker-help.md +++ b/docs/docs/guides/docker-help.md @@ -11,9 +11,8 @@ docker ps -a # see a list of running and stopped containers ```bash docker exec -it # attach to a container with a command -docker exec -it immich_server sh -docker exec -it immich_microservices sh -docker exec -it immich_machine_learning sh +docker exec -it immich_server bash +docker exec -it immich_machine_learning bash ``` ## Logs @@ -22,7 +21,6 @@ docker exec -it immich_machine_learning sh docker logs # see the logs for a specific container (by id or name) docker logs immich_server -docker logs immich_microservices docker logs immich_machine_learning ``` diff --git a/docs/docs/guides/external-library.md b/docs/docs/guides/external-library.md index 042f3900cc..5f5d0f168d 100644 --- a/docs/docs/guides/external-library.md +++ b/docs/docs/guides/external-library.md @@ -6,20 +6,14 @@ in a directory on the same machine. # Mount the directory into the containers. -Edit `docker-compose.yml` to add two new mount points in the sections `immich-server:` and `immich-microservices:` under `volumes:` +Edit `docker-compose.yml` to add two new mount points in the section `immich-server:` under `volumes:` ```diff immich-server: volumes: + - ${EXTERNAL_PATH}:/usr/src/app/external - -immich-microservices: - volumes: -+ - ${EXTERNAL_PATH}:/usr/src/app/external ``` -Be sure to add exactly the same path to both services. - Edit `.env` to define `EXTERNAL_PATH`, substituting in the correct path for your computer: ``` diff --git a/docs/docs/guides/remote-machine-learning.md b/docs/docs/guides/remote-machine-learning.md index 6caef05b3a..30249cd9d1 100644 --- a/docs/docs/guides/remote-machine-learning.md +++ b/docs/docs/guides/remote-machine-learning.md @@ -7,7 +7,7 @@ To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-imm - Start the container by running `docker compose up -d`. :::info -Starting with version v1.93.0 face detection work and face recognize were split. From now on face detection is done in the immich_machine_learning service, but facial recognition is done in the immich_microservices service. +Starting with version v1.93.0 face detection work and face recognize were split. From now on face detection is done in the immich_machine_learning container, but facial recognition is done in the `microservices` worker. ::: :::note diff --git a/docs/docs/install/environment-variables.md b/docs/docs/install/environment-variables.md index a3c08f2c70..e078c1ee72 100644 --- a/docs/docs/install/environment-variables.md +++ b/docs/docs/install/environment-variables.md @@ -17,11 +17,11 @@ If this should not work, try running `docker compose up -d --force-recreate`. ## Docker Compose -| Variable | Description | Default | Services | -| :----------------- | :------------------------------ | :-------: | :-------------------------------------- | -| `IMMICH_VERSION` | Image tags | `release` | server, microservices, machine learning | -| `UPLOAD_LOCATION` | Host Path for uploads | | server, microservices | -| `DB_DATA_LOCATION` | Host Path for Postgres database | | database | +| Variable | Description | Default | Containers | +| :----------------- | :------------------------------ | :-------: | :----------------------- | +| `IMMICH_VERSION` | Image tags | `release` | server, machine learning | +| `UPLOAD_LOCATION` | Host Path for uploads | | server | +| `DB_DATA_LOCATION` | Host Path for Postgres database | | database | :::tip These environment variables are used by the `docker-compose.yml` file and do **NOT** affect the containers directly. @@ -38,15 +38,16 @@ Regardless of filesystem, it is not recommended to use a network share for your ## General -| Variable | Description | Default | Services | -| :------------------------------ | :------------------------------------------- | :----------------------: | :-------------------------------------- | -| `TZ` | Timezone | | microservices | -| `IMMICH_ENV` | Environment (production, development) | `production` | server, microservices, machine learning | -| `IMMICH_LOG_LEVEL` | Log Level (verbose, debug, log, warn, error) | `log` | server, microservices, machine learning | -| `IMMICH_MEDIA_LOCATION` | Media Location | `./upload`\*1 | server, microservices | -| `IMMICH_CONFIG_FILE` | Path to config file | | server, microservices | -| `IMMICH_WEB_ROOT` | Path of root index.html | `/usr/src/app/www` | server | -| `IMMICH_REVERSE_GEOCODING_ROOT` | Path of reverse geocoding dump directory | `/usr/src/resources` | microservices | +| Variable | Description | Default | Containers | Workers | +| :------------------------------ | :---------------------------------------------- | :----------------------: | :----------------------- | :----------------- | +| `TZ` | Timezone | | server | microservices | +| `IMMICH_ENV` | Environment (production, development) | `production` | server, machine learning | api, microservices | +| `IMMICH_LOG_LEVEL` | Log Level (verbose, debug, log, warn, error) | `log` | server, machine learning | api, microservices | +| `IMMICH_MEDIA_LOCATION` | Media Location | `./upload`\*1 | server | api, microservices | +| `IMMICH_CONFIG_FILE` | Path to config file | | server | api, microservices | +| `IMMICH_WEB_ROOT` | Path of root index.html | `/usr/src/app/www` | server | api | +| `IMMICH_REVERSE_GEOCODING_ROOT` | Path of reverse geocoding dump directory | `/usr/src/resources` | server | microservices | +| `NO_COLOR` | Set to `true` to disable color-coded log output | `false` | server, machine learning | | \*1: With the default `WORKDIR` of `/usr/src/app`, this path will resolve to `/usr/src/app/upload`. It only need to be set if the Immich deployment method is changing. @@ -54,28 +55,39 @@ It only need to be set if the Immich deployment method is changing. :::tip `TZ` should be set to a `TZ identifier` from [this list][tz-list]. For example, `TZ="Etc/UTC"`. -`TZ` is only used by `exiftool`, which is present in the microservices container, as a fallback in case the timezone cannot be determined from the image metadata. +`TZ` is used by `exiftool` as a fallback in case the timezone cannot be determined from the image metadata. It is also used for logfile timestamps and cron job execution. +::: + +## Workers + +| Variable | Description | Default | Containers | +| :----------------------- | :--------------------------------------------------------------------------------------------------- | :-----: | :--------- | +| `IMMICH_WORKERS_INCLUDE` | Only run these workers. | | server | +| `IMMICH_WORKERS_EXCLUDE` | Do not run these workers. Matches against default workers, or `IMMICH_WORKERS_INCLUDE` if specified. | | server | + +:::info +Information on the current workers can be found [here](/docs/administration/jobs-workers). ::: ## Ports -| Variable | Description | Default | -| :------------ | :------------- | :------------------------------------: | -| `IMMICH_HOST` | Listening host | `0.0.0.0` | -| `IMMICH_PORT` | Listening port | 3001 (server), 3003 (machine learning) | +| Variable | Description | Default | +| :------------ | :------------- | :----------------------------------------: | +| `IMMICH_HOST` | Listening host | `0.0.0.0` | +| `IMMICH_PORT` | Listening port | `3001` (server), `3003` (machine learning) | ## Database -| Variable | Description | Default | Services | -| :---------------------------------- | :----------------------------------------------------------------------- | :----------: | :-------------------------------------------- | -| `DB_URL` | Database URL | | server, microservices | -| `DB_HOSTNAME` | Database Host | `database` | server, microservices | -| `DB_PORT` | Database Port | `5432` | server, microservices | -| `DB_USERNAME` | Database User | `postgres` | server, microservices, database\*1 | -| `DB_PASSWORD` | Database Password | `postgres` | server, microservices, database\*1 | -| `DB_DATABASE_NAME` | Database Name | `immich` | server, microservices, database\*1 | -| `DB_VECTOR_EXTENSION`\*2 | Database Vector Extension (one of [`pgvector`, `pgvecto.rs`]) | `pgvecto.rs` | server, microservices | -| `DB_SKIP_MIGRATIONS` | Whether to skip running migrations on startup (one of [`true`, `false`]) | `false` | server, microservices | +| Variable | Description | Default | Containers | +| :---------------------------------- | :----------------------------------------------------------------------- | :----------: | :----------------------------- | +| `DB_URL` | Database URL | | server | +| `DB_HOSTNAME` | Database Host | `database` | server | +| `DB_PORT` | Database Port | `5432` | server | +| `DB_USERNAME` | Database User | `postgres` | server, database\*1 | +| `DB_PASSWORD` | Database Password | `postgres` | server, database\*1 | +| `DB_DATABASE_NAME` | Database Name | `immich` | server, database\*1 | +| `DB_VECTOR_EXTENSION`\*2 | Database Vector Extension (one of [`pgvector`, `pgvecto.rs`]) | `pgvecto.rs` | server | +| `DB_SKIP_MIGRATIONS` | Whether to skip running migrations on startup (one of [`true`, `false`]) | `false` | server | \*1: The values of `DB_USERNAME`, `DB_PASSWORD`, and `DB_DATABASE_NAME` are passed to the Postgres container as the variables `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` in `docker-compose.yml`. @@ -83,30 +95,34 @@ It only need to be set if the Immich deployment method is changing. :::info +All `DB_` variables must be provided to all Immich workers, including `api` and `microservices`. + +`DB_URL` must be in the format `postgresql://immichdbusername:immichdbpassword@postgreshost:postgresport/immichdatabasename`. +You can require SSL by adding `?sslmode=require` to the end of the `DB_URL` string, or require SSL and skip certificate verification by adding `?sslmode=require&sslmode=no-verify`. + When `DB_URL` is defined, the `DB_HOSTNAME`, `DB_PORT`, `DB_USERNAME`, `DB_PASSWORD` and `DB_DATABASE_NAME` database variables are ignored. ::: ## Redis -| Variable | Description | Default | Services | -| :--------------- | :------------- | :-----: | :-------------------- | -| `REDIS_URL` | Redis URL | | server, microservices | -| `REDIS_HOSTNAME` | Redis Host | `redis` | server, microservices | -| `REDIS_PORT` | Redis Port | `6379` | server, microservices | -| `REDIS_DBINDEX` | Redis DB Index | `0` | server, microservices | -| `REDIS_USERNAME` | Redis Username | | server, microservices | -| `REDIS_PASSWORD` | Redis Password | | server, microservices | -| `REDIS_SOCKET` | Redis Socket | | server, microservices | +| Variable | Description | Default | Containers | +| :--------------- | :------------- | :-----: | :--------- | +| `REDIS_URL` | Redis URL | | server | +| `REDIS_SOCKET` | Redis Socket | | server | +| `REDIS_HOSTNAME` | Redis Host | `redis` | server | +| `REDIS_PORT` | Redis Port | `6379` | server | +| `REDIS_USERNAME` | Redis Username | | server | +| `REDIS_PASSWORD` | Redis Password | | server | +| `REDIS_DBINDEX` | Redis DB Index | `0` | server | :::info +All `REDIS_` variables must be provided to all Immich workers, including `api` and `microservices`. `REDIS_URL` must start with `ioredis://` and then include a `base64` encoded JSON string for the configuration. More info can be found in the upstream [ioredis][redis-api] documentation. -- When `REDIS_URL` is defined, the other redis (`REDIS_*`) variables are ignored. -- When `REDIS_SOCKET` is defined, the other redis (`REDIS_*`) variables are ignored. - +When `REDIS_URL` or `REDIS_SOCKET` are defined, the `REDIS_HOSTNAME`, `REDIS_PORT`, `REDIS_USERNAME`, `REDIS_PASSWORD`, and `REDIS_DBINDEX` variables are ignored. ::: Redis (Sentinel) URL example JSON before encoding: @@ -138,7 +154,7 @@ Redis (Sentinel) URL example JSON before encoding: ## Machine Learning -| Variable | Description | Default | Services | +| Variable | Description | Default | Containers | | :----------------------------------------------- | :------------------------------------------------------------------- | :-----------------: | :--------------- | | `MACHINE_LEARNING_MODEL_TTL` | Inactivity time (s) before a model is unloaded (disabled if \<= 0) | `300` | machine learning | | `MACHINE_LEARNING_MODEL_TTL_POLL_S` | Interval (s) between checks for the model TTL (disabled if \<= 0) | `10` | machine learning | @@ -163,13 +179,13 @@ Other machine learning parameters can be tuned from the admin UI. ## Prometheus -| Variable | Description | Default | Services | -| :----------------------------- | :-------------------------------------------------------------------------------------------- | :-----: | :-------------------- | -| `IMMICH_METRICS`\*1 | Toggle all metrics (one of [`true`, `false`]) | | server, microservices | -| `IMMICH_API_METRICS` | Toggle metrics for endpoints and response times (one of [`true`, `false`]) | | server, microservices | -| `IMMICH_HOST_METRICS` | Toggle metrics for CPU and memory utilization for host and process (one of [`true`, `false`]) | | server, microservices | -| `IMMICH_IO_METRICS` | Toggle metrics for database queries, image processing, etc. (one of [`true`, `false`]) | | server, microservices | -| `IMMICH_JOB_METRICS` | Toggle metrics for jobs and queues (one of [`true`, `false`]) | | server, microservices | +| Variable | Description | Default | Containers | Workers | +| :----------------------------- | :-------------------------------------------------------------------------------------------- | :-----: | :--------- | :----------------- | +| `IMMICH_METRICS`\*1 | Toggle all metrics (one of [`true`, `false`]) | | server | api, microservices | +| `IMMICH_API_METRICS` | Toggle metrics for endpoints and response times (one of [`true`, `false`]) | | server | api, microservices | +| `IMMICH_HOST_METRICS` | Toggle metrics for CPU and memory utilization for host and process (one of [`true`, `false`]) | | server | api, microservices | +| `IMMICH_IO_METRICS` | Toggle metrics for database queries, image processing, etc. (one of [`true`, `false`]) | | server | api, microservices | +| `IMMICH_JOB_METRICS` | Toggle metrics for jobs and queues (one of [`true`, `false`]) | | server | api, microservices | \*1: Overridden for a metric group when its corresponding environmental variable is set.