This change simplifies the Access repository methods, by using the
`ChunkedSet` decorator. As the methods expect sets, the `chunks` util
needed to be fixed, so it returns chunks of the same type it received.
Now `chunks` is overloaded, to have proper typing based on the input
parameter.
* add unicorn to eslint
* fix lint errors for cli
* fix merge
* fix album name extraction
* Update cli/src/commands/upload.command.ts
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
* es2k23
* use lowercase os
* return undefined album name
* fix bug in asset response dto
* auto fix issues
* fix server code style
* es2022 and formatting
* fix compilation error
* fix test
* fix config load
* fix last lint errors
* set string type
* bump ts
* start work on web
* web formatting
* Fix UUIDParamDto as UUIDParamDto
* fix library service lint
* fix web errors
* fix errors
* formatting
* wip
* lints fixed
* web can now start
* alphabetical package json
* rename error
* chore: clean up
---------
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* fix(server): Split database queries based on PostgreSQL bound params limit
PostgreSQL uses a 16-bit integer to indicate the number of bound
parameters.
This means that the maximum number of parameters for any query is 65535.
Any query that tries to bind more than that (e.g. searching by a list of
IDs) requires splitting the query into multiple chunks.
This change includes refactoring every Repository that runs queries
using a list of ids, and either flattening or merging results.
Fixes#5788, #5997.
Also, potentially a fix for #4648 (at least based on
[this comment](https://github.com/immich-app/immich/issues/4648#issuecomment-1826134027)).
References:
* https://github.com/typeorm/typeorm/issues/7565
* [PostgreSQL message format - Bind](https://www.postgresql.org/docs/15/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-BIND)
* misc: Create Chunked decorator to simplify implementation
* feat: Add ChunkedArray/ChunkedSet decorators
Modify Access repository, to evaluate `asset` permissions in bulk.
Queries have been validated to match what they currently generate for single ids.
Queries:
* `asset` album access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "albums" "AlbumEntity"
LEFT JOIN "albums_assets_assets" "AlbumEntity_AlbumEntity__AlbumEntity_assets"
ON "AlbumEntity_AlbumEntity__AlbumEntity_assets"."albumsId"="AlbumEntity"."id"
LEFT JOIN "assets" "AlbumEntity__AlbumEntity_assets"
ON "AlbumEntity__AlbumEntity_assets"."id"="AlbumEntity_AlbumEntity__AlbumEntity_assets"."assetsId"
AND "AlbumEntity__AlbumEntity_assets"."deletedAt" IS NULL
LEFT JOIN "albums_shared_users_users" "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."albumsId"="AlbumEntity"."id"
LEFT JOIN "users" "AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity__AlbumEntity_sharedUsers"."id"="AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."usersId"
AND "AlbumEntity__AlbumEntity_sharedUsers"."deletedAt" IS NULL
WHERE
(
("AlbumEntity"."ownerId" = $1 AND "AlbumEntity__AlbumEntity_assets"."id" = $2)
OR ("AlbumEntity__AlbumEntity_sharedUsers"."id" = $3 AND "AlbumEntity__AlbumEntity_assets"."id" = $4)
OR ("AlbumEntity"."ownerId" = $5 AND "AlbumEntity__AlbumEntity_assets"."livePhotoVideoId" = $6)
OR ("AlbumEntity__AlbumEntity_sharedUsers"."id" = $7 AND "AlbumEntity__AlbumEntity_assets"."livePhotoVideoId" = $8)
)
AND "AlbumEntity"."deletedAt" IS NULL
)
LIMIT 1
-- After
SELECT
"asset"."id" AS "assetId",
"asset"."livePhotoVideoId" AS "livePhotoVideoId"
FROM "albums" "album"
INNER JOIN "albums_assets_assets" "album_asset"
ON "album_asset"."albumsId"="album"."id"
INNER JOIN "assets" "asset"
ON "asset"."id"="album_asset"."assetsId"
AND "asset"."deletedAt" IS NULL
LEFT JOIN "albums_shared_users_users" "album_sharedUsers"
ON "album_sharedUsers"."albumsId"="album"."id"
LEFT JOIN "users" "sharedUsers"
ON "sharedUsers"."id"="album_sharedUsers"."usersId"
AND "sharedUsers"."deletedAt" IS NULL
WHERE
(
"album"."ownerId" = $1
OR "sharedUsers"."id" = $2
)
AND (
"asset"."id" IN ($3, $4)
OR "asset"."livePhotoVideoId" IN ($5, $6)
)
AND "album"."deletedAt" IS NULL
```
* `asset` owner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "assets" "AssetEntity"
WHERE
"AssetEntity"."id" = $1
AND "AssetEntity"."ownerId" = $2
)
LIMIT 1
-- After
SELECT
"AssetEntity"."id" AS "AssetEntity_id"
FROM "assets" "AssetEntity"
WHERE
"AssetEntity"."id" IN ($1, $2)
AND "AssetEntity"."ownerId" = $3
```
* `asset` partner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "partners" "PartnerEntity"
LEFT JOIN "users" "PartnerEntity__PartnerEntity_sharedWith"
ON "PartnerEntity__PartnerEntity_sharedWith"."id"="PartnerEntity"."sharedWithId"
AND "PartnerEntity__PartnerEntity_sharedWith"."deletedAt" IS NULL
LEFT JOIN "users" "PartnerEntity__PartnerEntity_sharedBy"
ON "PartnerEntity__PartnerEntity_sharedBy"."id"="PartnerEntity"."sharedById"
AND "PartnerEntity__PartnerEntity_sharedBy"."deletedAt" IS NULL
LEFT JOIN "assets" "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"
ON "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"."ownerId"="PartnerEntity__PartnerEntity_sharedBy"."id"
AND "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"."deletedAt" IS NULL
LEFT JOIN "users" "PartnerEntity__sharedBy"
ON "PartnerEntity__sharedBy"."id"="PartnerEntity"."sharedById"
AND "PartnerEntity__sharedBy"."deletedAt" IS NULL
LEFT JOIN "users" "PartnerEntity__sharedWith"
ON "PartnerEntity__sharedWith"."id"="PartnerEntity"."sharedWithId"
AND "PartnerEntity__sharedWith"."deletedAt" IS NULL
WHERE
"PartnerEntity__PartnerEntity_sharedWith"."id" = $1
AND "0aabe9f4a62b794e2c24a074297e534f51a4ac6c"."id" = $2
)
LIMIT 1
-- After
SELECT
"asset"."id" AS "assetId"
FROM "partners" "partner"
INNER JOIN "users" "sharedBy"
ON "sharedBy"."id"="partner"."sharedById"
AND "sharedBy"."deletedAt" IS NULL
INNER JOIN "assets" "asset"
ON "asset"."ownerId"="sharedBy"."id"
AND "asset"."deletedAt" IS NULL
WHERE
"partner"."sharedWithId" = $1
AND "asset"."id" IN ($2, $3)
```
* `asset` shared link access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "shared_links" "SharedLinkEntity"
LEFT JOIN "albums" "SharedLinkEntity__SharedLinkEntity_album"
ON "SharedLinkEntity__SharedLinkEntity_album"."id"="SharedLinkEntity"."albumId"
AND "SharedLinkEntity__SharedLinkEntity_album"."deletedAt" IS NULL
LEFT JOIN "albums_assets_assets" "760f12c00d97bdcec1ce224d1e3bf449859942b6"
ON "760f12c00d97bdcec1ce224d1e3bf449859942b6"."albumsId"="SharedLinkEntity__SharedLinkEntity_album"."id"
LEFT JOIN "assets" "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"
ON "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."id"="760f12c00d97bdcec1ce224d1e3bf449859942b6"."assetsId"
AND "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."deletedAt" IS NULL
LEFT JOIN "shared_link__asset" "SharedLinkEntity__SharedLinkEntity_assets_SharedLinkEntity"
ON "SharedLinkEntity__SharedLinkEntity_assets_SharedLinkEntity"."sharedLinksId"="SharedLinkEntity"."id"
LEFT JOIN "assets" "SharedLinkEntity__SharedLinkEntity_assets"
ON "SharedLinkEntity__SharedLinkEntity_assets"."id"="SharedLinkEntity__SharedLinkEntity_assets_SharedLinkEntity"."assetsId"
AND "SharedLinkEntity__SharedLinkEntity_assets"."deletedAt" IS NULL
WHERE (
("SharedLinkEntity"."id" = $1 AND "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."id" = $2)
OR ("SharedLinkEntity"."id" = $3 AND "SharedLinkEntity__SharedLinkEntity_assets"."id" = $4)
OR ("SharedLinkEntity"."id" = $5 AND "4a35f463ae8c5544ede95c4b6d9ce8c686b6bfe6"."livePhotoVideoId" = $6)
OR ("SharedLinkEntity"."id" = $7 AND "SharedLinkEntity__SharedLinkEntity_assets"."livePhotoVideoId" = $8)
)
)
LIMIT 1
-- After
SELECT
"assets"."id" AS "assetId",
"assets"."livePhotoVideoId" AS "assetLivePhotoVideoId",
"albumAssets"."id" AS "albumAssetId",
"albumAssets"."livePhotoVideoId" AS "albumAssetLivePhotoVideoId"
FROM "shared_links" "sharedLink"
LEFT JOIN "albums" "album"
ON "album"."id"="sharedLink"."albumId"
AND "album"."deletedAt" IS NULL
LEFT JOIN "shared_link__asset" "assets_sharedLink"
ON "assets_sharedLink"."sharedLinksId"="sharedLink"."id"
LEFT JOIN "assets" "assets"
ON "assets"."id"="assets_sharedLink"."assetsId"
AND "assets"."deletedAt" IS NULL
LEFT JOIN "albums_assets_assets" "album_albumAssets"
ON "album_albumAssets"."albumsId"="album"."id"
LEFT JOIN "assets" "albumAssets"
ON "albumAssets"."id"="album_albumAssets"."assetsId"
AND "albumAssets"."deletedAt" IS NULL
WHERE
"sharedLink"."id" = $1
AND (
"assets"."id" IN ($2, $3)
OR "albumAssets"."id" IN ($4, $5)
OR "assets"."livePhotoVideoId" IN ($6, $7)
OR "albumAssets"."livePhotoVideoId" IN ($8, $9)
)
```
* chore(server): Check album permissions in bulk
Modify Access repository, to evaluate `album` permissions in bulk.
Queries have been validated to match what they currently generate for
single ids.
Queries:
* Owner access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "albums" "AlbumEntity"
WHERE
"AlbumEntity"."id" = $1
AND "AlbumEntity"."ownerId" = $2
AND "AlbumEntity"."deletedAt" IS NULL
)
LIMIT 1
-- After
SELECT
"AlbumEntity"."id" AS "AlbumEntity_id"
FROM "albums" "AlbumEntity"
WHERE
"AlbumEntity"."id" IN ($1, $2)
AND "AlbumEntity"."ownerId" = $3
AND "AlbumEntity"."deletedAt" IS NULL
```
* Shared link access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "shared_links" "SharedLinkEntity"
WHERE
"SharedLinkEntity"."id" = $1
AND "SharedLinkEntity"."albumId" = $2
)
LIMIT 1
-- After
SELECT
"SharedLinkEntity"."albumId" AS "SharedLinkEntity_albumId",
"SharedLinkEntity"."id" AS "SharedLinkEntity_id"
FROM "shared_links" "SharedLinkEntity"
WHERE
"SharedLinkEntity"."id" = $1
AND "SharedLinkEntity"."albumId" IN ($2, $3)
```
* Shared album access:
```sql
-- Before
SELECT 1 AS "row_exists" FROM (SELECT 1 AS dummy_column) "dummy_table" WHERE EXISTS (
SELECT 1
FROM "albums" "AlbumEntity"
LEFT JOIN "albums_shared_users_users" "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."albumsId"="AlbumEntity"."id"
LEFT JOIN "users" "AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity__AlbumEntity_sharedUsers"."id"="AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."usersId"
AND "AlbumEntity__AlbumEntity_sharedUsers"."deletedAt" IS NULL
WHERE
"AlbumEntity"."id" = $1
AND "AlbumEntity__AlbumEntity_sharedUsers"."id" = $2
AND "AlbumEntity"."deletedAt" IS NULL
)
LIMIT 1
-- After
SELECT
"AlbumEntity"."id" AS "AlbumEntity_id"
FROM "albums" "AlbumEntity"
LEFT JOIN "albums_shared_users_users" "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."albumsId"="AlbumEntity"."id"
LEFT JOIN "users" "AlbumEntity__AlbumEntity_sharedUsers"
ON "AlbumEntity__AlbumEntity_sharedUsers"."id"="AlbumEntity_AlbumEntity__AlbumEntity_sharedUsers"."usersId"
AND "AlbumEntity__AlbumEntity_sharedUsers"."deletedAt" IS NULL
WHERE
"AlbumEntity"."id" IN ($1, $2)
AND "AlbumEntity__AlbumEntity_sharedUsers"."id" = $3
AND "AlbumEntity"."deletedAt" IS NULL
```
* chore(server): Add set utils, avoid double queries for same ids
* chore(server): Review feedback
* feat(server): GET /assets endpoint
* chore: open api
* chore: use dumb name
* feat: search by make, model, lens, city, state, country
* chore: open api
* chore: pagination validation and tests
* chore: pr feedback
* add automatic library scan config options
* add validation
* open api
* use CronJob instead of cron-validator
* fix tests
* catch potential error of the library scan initialization
* better description for input field
* move library scan job initialization to server app service
* fix tests
* add comments to all parameters of cronjob contructor
* make scan a child of a more general library object
* open api
* chore: cleanup
* move cronjob handling to job repoistory
* web: select for common cron expressions
* fix open api
* fix tests
* put scanning settings in nested accordion
* fix system config validation
* refactor, tests
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* custom `IsOptional`
* added link to source
* formatting
* Update server/src/domain/domain.util.ts
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
* nullable birth date endpoint
* made `nullable` a property
* formatting
* removed unused dto
* updated decorator arg
* fixed album e2e tests
* add null tests for auth e2e
* add null test for person e2e
* fixed tests
* added null test for user e2e
* removed unusued import
* log key in test name
* chore: add note about mobile not being able to use the endpoint
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
