* fix(server): Split database queries based on PostgreSQL bound params limit
PostgreSQL uses a 16-bit integer to indicate the number of bound
parameters.
This means that the maximum number of parameters for any query is 65535.
Any query that tries to bind more than that (e.g. searching by a list of
IDs) requires splitting the query into multiple chunks.
This change includes refactoring every Repository that runs queries
using a list of ids, and either flattening or merging results.
Fixes#5788, #5997.
Also, potentially a fix for #4648 (at least based on
[this comment](https://github.com/immich-app/immich/issues/4648#issuecomment-1826134027)).
References:
* https://github.com/typeorm/typeorm/issues/7565
* [PostgreSQL message format - Bind](https://www.postgresql.org/docs/15/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-BIND)
* misc: Create Chunked decorator to simplify implementation
* feat: Add ChunkedArray/ChunkedSet decorators
* feat(server): Throw error when PostgreSQL version is not within the supported versions
The pgvecto.rs extension, though not distributed, can be built for PostgreSQL 12 and 13.
An installation of PostgreSQL 12 with the pgvecto.rs extensions installed will not be caught by immich.
This causes immich to attempt to run the database migrations without having a proper environment.
With assertPostgresql the server will throw an error if the PostgreSQL version is not within the supported range.
* Replaced assertion with lesser than comparison
As requested by @zackpollard
* Changed the comparison to use the minPostgresVersion variable.
If we define one we might as well use it. makes changing the versioning later easier
* Added two new tests, modified two existing tests
`should return if minimum supported PostgreSQL and vectors version are installed`:
Check if init returns properly and that getPostgresVersion is called twice
`should thrown an error if PostgreSQL version is below minimum supported version`:
Checks if the init function correctly returns an error
`should suggest image with postgres ${major} if database is ${major}`:
Modified to set MockResolvedValue instead of MockResolvedValueOnce. With the new check we get the PostgreSQL version twice. So it needs to be set during the entire test.
`should not suggest image if postgres version is not in 14, 15 or 16`:
Modified the bounds to [14, 18]. Because values below 14 now will not get called.
Also Modified to call `getPostgresVersion.MockResolvedValueOnce` for twice, because it gets called twice.
* Fixed two mistakes in the jest functions from previous commit #2abcb60
`should thrown an error if PostgreSQL version is below minimum supported version`:
The regex function I wrote mistakingly used the negate function which check that the error *did not* contain the phrase "PostgreSQL". Which is the opposite
`should not suggest image if postgres version is not in 14, 15 or 16`:
confused bounds for a normal javascript array. Changed the test to only check for values above 16. As values below 14 will get thrown out by test `should return if minimum supported PostgreSQL and vectors version are installed`
I apologise for the mistakes in my previous commit.
* Format fix
---------
Co-authored-by: max <wak@vanling.net>
* feat(web): onboarding
* feat: openapi
* feat: modulization
* feat: page advancing
* Animation
* Add storage templaete settings
* sql
* more style
* Theme
* information and styling
* hide/show table
* Styling
* Update user property
* fix test
* fix test:
* fix e2e
* test
* Update web/src/lib/components/onboarding-page/onboarding-hello.svelte
Co-authored-by: bo0tzz <git@bo0tzz.me>
* naming
* use System Metadata
* better return type
* onboarding using server metadata
* revert previous changes in user entity
* sql
* test web
* fix test server
* server/web test
* more test
* consolidate color theme change logic
* consolidate save button to storage template
* merge main
* fix web
---------
Co-authored-by: bo0tzz <git@bo0tzz.me>
* feat(server): Enqueue jobs in bulk
The Job Repository now has a `queueAll` method, that enqueues messages
in bulk (using BullMQ's
[`addBulk`](https://docs.bullmq.io/guide/queues/adding-bulks)),
improving performance when many jobs must be enqueued within the same
operation.
Primary change is in `src/domain/job/job.service.ts`, and other services
have been refactored to use `queueAll` when useful.
As a simple local benchmark, triggering a full thumbnail generation
process over a library of ~1,200 assets and ~350 faces went from
**~600ms** to **~250ms**.
* fix: Review feedback
* fix: locks need to be acquired and released using the same session
* feat: only allow a single storage template operation at a single time
this has been added to avoid possible rare race conditions where two files are moved at once to the same path with the same name, causing our duplicate iterator to not detect them and therefore both files will have the same name and overwrite eachother
* fix: pgvecto.rs extension breaks typeorm schema:drop command
* fix: parse postgres bigints to javascript number types when selecting data
* feat: verify file size is the same as original asset after copying file for storage template job
* feat: allow disabling of storage template job, defaults to disabled for new instances
* fix: don't allow setting concurrency for storage template migration, can cause race conditions above 1
* feat: add checksum verification when file is copied for storage template job
* fix: extract metadata for assets that aren't visible on timeline
* run migrations after checks
* optional migrations
* only run checks in server and e2e
* re-add migrations for microservices
* refactor
* move e2e init
* remove assert from migration
* update providers
* update microservices app service
* fixed logging
* refactored version check, added unit tests
* more version tests
* don't use mocks for sut
* refactor tests
* suggest image only if postgres is 14, 15 or 16
* review suggestions
* fixed regexp escape
* fix typing
* update migration
* Hide the person age if it is negative
* Add validation to prevent future birth dates
* Add comment
* Add test, Add birth date validation and update birth date modal
* Add birthDate validation in PersonService and SetBirthDateModal
* Running npm run format:fix
* Generating the migration file propoerly, and Make the birthdate form logic simpler
* Make birthDate type only string
* Adding useLocationPin back
Modify Access repository, to evaluate `asset` permissions in bulk.
This is the last set of permission changes, to migrate all of them to
run in bulk!
Queries have been validated to match what they currently generate for single ids.
Queries:
* `activity` owner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "activity" "ActivityEntity"
WHERE
"ActivityEntity"."id" = $1
AND "ActivityEntity"."userId" = $2
)
LIMIT 1
-- After
SELECT "ActivityEntity"."id" AS "ActivityEntity_id"
FROM "activity" "ActivityEntity"
WHERE
"ActivityEntity"."id" IN ($1)
AND "ActivityEntity"."userId" = $2
```
* `activity` album owner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "activity" "ActivityEntity"
LEFT JOIN "albums" "ActivityEntity__ActivityEntity_album"
ON "ActivityEntity__ActivityEntity_album"."id"="ActivityEntity"."albumId"
AND "ActivityEntity__ActivityEntity_album"."deletedAt" IS NULL
WHERE
"ActivityEntity"."id" = $1
AND "ActivityEntity__ActivityEntity_album"."ownerId" = $2
)
LIMIT 1
-- After
SELECT "ActivityEntity"."id" AS "ActivityEntity_id"
FROM "activity" "ActivityEntity"
LEFT JOIN "albums" "ActivityEntity__ActivityEntity_album"
ON "ActivityEntity__ActivityEntity_album"."id"="ActivityEntity"."albumId"
AND "ActivityEntity__ActivityEntity_album"."deletedAt" IS NULL
WHERE
"ActivityEntity"."id" IN ($1)
AND "ActivityEntity__ActivityEntity_album"."ownerId" = $2
```
* `activity` create access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "albums" "AlbumEntity"
LEFT JOIN "albums_shared_users_users" "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."albumsId"="AlbumEntity"."id"
LEFT JOIN "users" "AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity__AlbumEntity_sharedUsers"."id"="AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."usersId"
AND "AlbumEntity__AlbumEntity_sharedUsers"."deletedAt" IS NULL
WHERE
(
(
"AlbumEntity"."id" = $1
AND "AlbumEntity"."isActivityEnabled" = $2
AND "AlbumEntity__AlbumEntity_sharedUsers"."id" = $3
)
OR (
"AlbumEntity"."id" = $4
AND "AlbumEntity"."isActivityEnabled" = $5
AND "AlbumEntity"."ownerId" = $6
)
)
AND "AlbumEntity"."deletedAt" IS NULL
)
LIMIT 1
-- After
SELECT "AlbumEntity"."id" AS "AlbumEntity_id"
FROM "albums" "AlbumEntity"
LEFT JOIN "albums_shared_users_users" "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."albumsId"="AlbumEntity"."id"
LEFT JOIN "users" "AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity__AlbumEntity_sharedUsers"."id"="AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."usersId"
AND "AlbumEntity__AlbumEntity_sharedUsers"."deletedAt" IS NULL
WHERE
(
(
"AlbumEntity"."id" IN ($1)
AND "AlbumEntity"."isActivityEnabled" = $2
AND "AlbumEntity__AlbumEntity_sharedUsers"."id" = $3
)
OR (
"AlbumEntity"."id" IN ($4)
AND "AlbumEntity"."isActivityEnabled" = $5
AND "AlbumEntity"."ownerId" = $6
)
)
AND "AlbumEntity"."deletedAt" IS NULL
```
* fix: don't limit merge face selector to 10 people
* fix: don't use class to hide people in detail-panel
* fix: map faces and person in asset response
* fix(mobile): album thumbnail list tile overflow on large album title
* fix: notify clients about live photo linked event
* refactor: notify clients during meta extraction
---------
Co-authored-by: shalong-tanwen <139912620+shalong-tanwen@users.noreply.github.com>
* feat: unassign person faces
* multiple improvements
* chore: regenerate api
* feat: improve face interactions in photos
* fix: tests
* fix: tests
* optimize
* fix: wrong assignment on complex-multiple re-assignments
* fix: thumbnails with large photos
* fix: complex reassign
* fix: don't send people with faces
* fix: person thumbnail generation
* chore: regenerate api
* add tess
* feat: face box even when zoomed
* fix: change feature photo
* feat: make the blue icon hoverable
* chore: regenerate api
* feat: use websocket
* fix: loading spinner when clicking on the done button
* fix: use the svelte way
* fix: tests
* simplify
* fix: unused vars
* fix: remove unused code
* fix: add migration
* chore: regenerate api
* ci: add unit tests
* chore: regenerate api
* feat: if a new person is created for a face and the server takes more than 15 seconds to generate the person thumbnail, don't wait for it
* reorganize
* chore: regenerate api
* feat: global edit
* pr feedback
* pr feedback
* simplify
* revert test
* fix: face generation
* fix: tests
* fix: face generation
* fix merge
* feat: search names in unmerge face selector modal
* fix: merge face selector
* simplify feature photo generation
* fix: change endpoint
* pr feedback
* chore: fix merge
* chore: fix merge
* fix: tests
* fix: edit & hide buttons
* fix: tests
* feat: show if person is hidden
* feat: rename face to person
* feat: split in new panel
* copy-paste-error
* pr feedback
* fix: feature photo
* do not leak faces
* fix: unmerge modal
* fix: merge modal event
* feat(server): remove duplicates
* fix: title for image thumbnails
* fix: disable side panel when there's no face until next PR
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
Modify Access repository, to evaluate `asset` permissions in bulk.
Queries have been validated to match what they currently generate for single ids.
Queries:
* `asset` album access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "albums" "AlbumEntity"
LEFT JOIN "albums_assets_assets" "AlbumEntity_AlbumEntity__AlbumEntity_assets"
ON "AlbumEntity_AlbumEntity__AlbumEntity_assets"."albumsId"="AlbumEntity"."id"
LEFT JOIN "assets" "AlbumEntity__AlbumEntity_assets"
ON "AlbumEntity__AlbumEntity_assets"."id"="AlbumEntity_AlbumEntity__AlbumEntity_assets"."assetsId"
AND "AlbumEntity__AlbumEntity_assets"."deletedAt" IS NULL
LEFT JOIN "albums_shared_users_users" "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."albumsId"="AlbumEntity"."id"
LEFT JOIN "users" "AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity__AlbumEntity_sharedUsers"."id"="AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."usersId"
AND "AlbumEntity__AlbumEntity_sharedUsers"."deletedAt" IS NULL
WHERE
(
("AlbumEntity"."ownerId" = $1 AND "AlbumEntity__AlbumEntity_assets"."id" = $2)
OR ("AlbumEntity__AlbumEntity_sharedUsers"."id" = $3 AND "AlbumEntity__AlbumEntity_assets"."id" = $4)
OR ("AlbumEntity"."ownerId" = $5 AND "AlbumEntity__AlbumEntity_assets"."livePhotoVideoId" = $6)
OR ("AlbumEntity__AlbumEntity_sharedUsers"."id" = $7 AND "AlbumEntity__AlbumEntity_assets"."livePhotoVideoId" = $8)
)
AND "AlbumEntity"."deletedAt" IS NULL
)
LIMIT 1
-- After
SELECT
"asset"."id" AS "assetId",
"asset"."livePhotoVideoId" AS "livePhotoVideoId"
FROM "albums" "album"
INNER JOIN "albums_assets_assets" "album_asset"
ON "album_asset"."albumsId"="album"."id"
INNER JOIN "assets" "asset"
ON "asset"."id"="album_asset"."assetsId"
AND "asset"."deletedAt" IS NULL
LEFT JOIN "albums_shared_users_users" "album_sharedUsers"
ON "album_sharedUsers"."albumsId"="album"."id"
LEFT JOIN "users" "sharedUsers"
ON "sharedUsers"."id"="album_sharedUsers"."usersId"
AND "sharedUsers"."deletedAt" IS NULL
WHERE
(
"album"."ownerId" = $1
OR "sharedUsers"."id" = $2
)
AND (
"asset"."id" IN ($3, $4)
OR "asset"."livePhotoVideoId" IN ($5, $6)
)
AND "album"."deletedAt" IS NULL
```
* `asset` owner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "assets" "AssetEntity"
WHERE
"AssetEntity"."id" = $1
AND "AssetEntity"."ownerId" = $2
)
LIMIT 1
-- After
SELECT
"AssetEntity"."id" AS "AssetEntity_id"
FROM "assets" "AssetEntity"
WHERE
"AssetEntity"."id" IN ($1, $2)
AND "AssetEntity"."ownerId" = $3
```
* `asset` partner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "partners" "PartnerEntity"
LEFT JOIN "users" "PartnerEntity__PartnerEntity_sharedWith"
ON "PartnerEntity__PartnerEntity_sharedWith"."id"="PartnerEntity"."sharedWithId"
AND "PartnerEntity__PartnerEntity_sharedWith"."deletedAt" IS NULL
LEFT JOIN "users" "PartnerEntity__PartnerEntity_sharedBy"
ON "PartnerEntity__PartnerEntity_sharedBy"."id"="PartnerEntity"."sharedById"
AND "PartnerEntity__PartnerEntity_sharedBy"."deletedAt" IS NULL
LEFT JOIN "assets" "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"
ON "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"."ownerId"="PartnerEntity__PartnerEntity_sharedBy"."id"
AND "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"."deletedAt" IS NULL
LEFT JOIN "users" "PartnerEntity__sharedBy"
ON "PartnerEntity__sharedBy"."id"="PartnerEntity"."sharedById"
AND "PartnerEntity__sharedBy"."deletedAt" IS NULL
LEFT JOIN "users" "PartnerEntity__sharedWith"
ON "PartnerEntity__sharedWith"."id"="PartnerEntity"."sharedWithId"
AND "PartnerEntity__sharedWith"."deletedAt" IS NULL
WHERE
"PartnerEntity__PartnerEntity_sharedWith"."id" = $1
AND "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"."id" = $2
)
LIMIT 1
-- After
SELECT
"asset"."id" AS "assetId"
FROM "partners" "partner"
INNER JOIN "users" "sharedBy"
ON "sharedBy"."id"="partner"."sharedById"
AND "sharedBy"."deletedAt" IS NULL
INNER JOIN "assets" "asset"
ON "asset"."ownerId"="sharedBy"."id"
AND "asset"."deletedAt" IS NULL
WHERE
"partner"."sharedWithId" = $1
AND "asset"."id" IN ($2, $3)
```
* `asset` shared link access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "shared_links" "SharedLinkEntity"
LEFT JOIN "albums" "SharedLinkEntity__SharedLinkEntity_album"
ON "SharedLinkEntity__SharedLinkEntity_album"."id"="SharedLinkEntity"."albumId"
AND "SharedLinkEntity__SharedLinkEntity_album"."deletedAt" IS NULL
LEFT JOIN "albums_assets_assets" "760f12c00d97bdcec1ce224d1e3bf449859942b6"
ON "760f12c00d97bdcec1ce224d1e3bf449859942b6"."albumsId"="SharedLinkEntity__SharedLinkEntity_album"."id"
LEFT JOIN "assets" "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"
ON "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."id"="760f12c00d97bdcec1ce224d1e3bf449859942b6"."assetsId"
AND "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."deletedAt" IS NULL
LEFT JOIN "shared_link__asset" "SharedLinkEntity__SharedLinkEntity_assets_SharedLinkEntity"
ON "SharedLinkEntity__SharedLinkEntity_assets_SharedLinkEntity"."sharedLinksId"="SharedLinkEntity"."id"
LEFT JOIN "assets" "SharedLinkEntity__SharedLinkEntity_assets"
ON "SharedLinkEntity__SharedLinkEntity_assets"."id"="SharedLinkEntity__SharedLinkEntity_assets_SharedLinkEntity"."assetsId"
AND "SharedLinkEntity__SharedLinkEntity_assets"."deletedAt" IS NULL
WHERE (
("SharedLinkEntity"."id" = $1 AND "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."id" = $2)
OR ("SharedLinkEntity"."id" = $3 AND "SharedLinkEntity__SharedLinkEntity_assets"."id" = $4)
OR ("SharedLinkEntity"."id" = $5 AND "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."livePhotoVideoId" = $6)
OR ("SharedLinkEntity"."id" = $7 AND "SharedLinkEntity__SharedLinkEntity_assets"."livePhotoVideoId" = $8)
)
)
LIMIT 1
-- After
SELECT
"assets"."id" AS "assetId",
"assets"."livePhotoVideoId" AS "assetLivePhotoVideoId",
"albumAssets"."id" AS "albumAssetId",
"albumAssets"."livePhotoVideoId" AS "albumAssetLivePhotoVideoId"
FROM "shared_links" "sharedLink"
LEFT JOIN "albums" "album"
ON "album"."id"="sharedLink"."albumId"
AND "album"."deletedAt" IS NULL
LEFT JOIN "shared_link__asset" "assets_sharedLink"
ON "assets_sharedLink"."sharedLinksId"="sharedLink"."id"
LEFT JOIN "assets" "assets"
ON "assets"."id"="assets_sharedLink"."assetsId"
AND "assets"."deletedAt" IS NULL
LEFT JOIN "albums_assets_assets" "album_albumAssets"
ON "album_albumAssets"."albumsId"="album"."id"
LEFT JOIN "assets" "albumAssets"
ON "albumAssets"."id"="album_albumAssets"."assetsId"
AND "albumAssets"."deletedAt" IS NULL
WHERE
"sharedLink"."id" = $1
AND (
"assets"."id" IN ($2, $3)
OR "albumAssets"."id" IN ($4, $5)
OR "assets"."livePhotoVideoId" IN ($6, $7)
OR "albumAssets"."livePhotoVideoId" IN ($8, $9)
)
```
