Daniel Dietzler
ef4a492cb1
chore(server): move services ( #8133 )
...
move services
2024-03-20 18:07:30 -05:00
Daniel Dietzler
6d9e7694b1
chore(server): move dtos ( #8131 )
...
move dtos
2024-03-20 23:53:07 +01:00
Jason Rasmussen
c1402eee8e
chore: migrate database files ( #8126 )
2024-03-20 21:02:51 +00:00
Daniel Dietzler
84f7ca855a
chore(server): move domain interfaces ( #8124 )
...
move domain interfaces
2024-03-20 16:42:58 -04:00
Daniel Dietzler
4b29bccc7c
chore(server): move cores ( #8120 )
...
move cores
2024-03-20 20:20:38 +00:00
Jason Rasmussen
81f0265095
chore: organize config, validation, decorators ( #8118 )
...
* refactor: validation
* refactor: utilities
* refactor: config
2024-03-20 16:04:03 -04:00
Daniel Dietzler
30f499cf2e
chore(server): use absolute import paths ( #8080 )
...
update server to use absolute import paths
2024-03-20 14:32:04 -04:00
Jonathan Jogenfors
e6f2bb9f89
fix(server): use extension in originalFileName for libraries ( #8083 )
...
* use file base
* fix: test
* fix: e2e-job tests
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-03-19 23:40:28 -05:00
Mert
2daed747cd
chore(server): change save -> update in asset repository ( #8055 )
...
* `save` -> `update`
* change return type
* include relations
* fix tests
* remove when mocks
* fix
* stricter typing
* simpler type
2024-03-19 22:42:10 -04:00
Jason Rasmussen
40262c30cb
refactor(server): library service ( #8050 )
...
* refactor: library service
* chore: open api
* fix: checks
2024-03-18 16:59:53 -04:00
Daniel Dietzler
148428a564
feat(server): use nestjs events to validate config ( #7986 )
...
* use events for config validation
* chore: better types
* add unit tests
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-03-17 20:16:02 +01:00
Mert
a9438a9c2d
fix(server): prevent feedback loop during library scan ( #7944 )
...
* prevent feedback loop
* add nesting
* made nesting less ugly
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-03-15 22:01:58 +00:00
Daniel Dietzler
a46366d336
chore(server): introduce proper job status ( #7932 )
...
* introduce proper job status
* fix condition for onDone jobs
* fix tests
2024-03-15 09:16:54 -04:00
Jonathan Jogenfors
ba38713fbc
fix(server): queue library asset refresh in batches ( #7914 )
...
* add debug logs
* scan assets in batches
* Cleanup
* don't normalize
* Removing extra log
* remove unneeded code
* change log levels
2024-03-14 14:43:05 +01:00
Mert
d67cc00e4e
feat(server): lower library scan memory usage ( #7939 )
...
* use trie
* update tests
* formatting
* pr feedback
* linting
2024-03-14 01:52:30 -04:00
Mert
5bd597f14b
fix(server): external library sync not working for large libraries ( #7759 )
2024-03-10 22:30:57 -04:00
Jonathan Jogenfors
4cb0f37918
chore(server): Move library watcher to microservices ( #7533 )
...
* move watcher init to micro
* document watcher recovery
* chore: fix lint
* add try lock
* use global library watch lock
* fix: ensure lock stays on
* fix: mocks
* unit test for library watch lock
* move statement to correct test
* fix: correct return type of try lock
* fix: tests
* add library teardown
* add chokidar error handler
* make event strings an enum
* wait for event refactor
* refactor event type mocks
* expect correct error
* don't release lock in teardown
* chore: lint
* use enum
* fix mock
* fix lint
* fix watcher await
* remove await
* simplify typing
* remove async
* Revert "remove async"
This reverts commit 84ab5abac4
2024-03-07 11:36:53 -06:00
Mert
1ec5d612fa
perf(server): use queries to refresh library assets ( #7685 )
...
* use queries instead of js
* missing await
* add mock methods
* fix test
* update sql
* linting
2024-03-06 21:23:10 -06:00
Jonathan Jogenfors
5d377e5b0f
chore(server): eslint await-thenable ( #7545 )
...
* await-thenable
* fix library watchers
* moar eslint
* fix test
* fix typo
* try to remove check void return
* fix checksVoidReturn
* move to domain utils
* remove eslint ignores
* chore: cleanup types
* chore: use logger
* fix: e2e
---------
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-03-05 17:23:06 -05:00
Jonathan Jogenfors
efa6efd200
feat(server,web): remove external path nonsense and make libraries admin-only ( #7237 )
...
* remove external path
* open-api
* make sql
* move library settings to admin panel
* Add documentation
* show external libraries only
* fix library list
* make user library settings look good
* fix test
* fix tests
* fix tests
* can pick user for library
* fix tests
* fix e2e
* chore: make sql
* Use unauth exception
* delete user library list
* cleanup
* fix e2e
* fix await lint
* chore: remove unused code
* chore: cleanup
* revert docs
* fix: is admin stuff
* table alignment
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-02-29 12:35:37 -06:00
Jonathan Jogenfors
e4f32a045d
chore: remove watcher polling option ( #7480 )
...
* remove watcher polling
* fix lint
* add db migration
2024-02-28 21:20:10 +01:00
Jonathan Jogenfors
b3c7bebbd4
feat(server,web) Semantic import path validation ( #7076 )
...
* add library validation api
* chore: open api
* show warning i UI
* add flex row
* fix e2e
* tests
* fix tests
* enforce path validation
* enforce validation on refresh
* return 400 on bad import path
* add limits to import paths
* set response code to 200
* fix e2e
* fix lint
* fix test
* restore e2e folder
* fix import
* use startsWith
* icon color
* notify user of failed validation
* add parent div to validation
* add docs to the import validation
* improve library troubleshooting docs
* fix button alignment
---------
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-02-20 09:53:12 -06:00
Jason Rasmussen
b648025e2f
refactor: library watching ( #7071 )
2024-02-13 08:48:47 -05:00
Jonathan Jogenfors
f44fa45aa0
chore(server,cli,web): housekeeping and stricter code style ( #6751 )
...
* add unicorn to eslint
* fix lint errors for cli
* fix merge
* fix album name extraction
* Update cli/src/commands/upload.command.ts
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
* es2k23
* use lowercase os
* return undefined album name
* fix bug in asset response dto
* auto fix issues
* fix server code style
* es2022 and formatting
* fix compilation error
* fix test
* fix config load
* fix last lint errors
* set string type
* bump ts
* start work on web
* web formatting
* Fix UUIDParamDto as UUIDParamDto
* fix library service lint
* fix web errors
* fix errors
* formatting
* wip
* lints fixed
* web can now start
* alphabetical package json
* rename error
* chore: clean up
---------
Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-02-01 22:18:00 -05:00
Jason Rasmussen
b0d7434e7b
fix: library watching ( #6802 )
2024-01-31 11:26:51 -05:00
Jonathan Jogenfors
068e703e88
feat(server): Automatic watching of library folders ( #6192 )
...
* feat: initial watch support
* allow offline files
* chore: ignore query errors when resetting e2e db
* revert db query
* add savepoint
* guard the user query
* chore: openapi and db migration
* wip
* support multiple libraries
* fix tests
* wip
* can now cleanup chokidar watchers
* fix unit tests
* add library watch queue
* add missing init from merge
* wip
* can now filter file extensions
* remove watch api from non job client
* Fix e2e test
* watch library with updated import path and exclusion pattern
* add library watch frontend ui
* case sensitive watching extensions
* can auto watch libraries
* move watcher e2e tests to separate file
* don't watch libraries from a queue
* use event emitters
* shorten e2e test timeout
* refactor chokidar code to filesystem provider
* expose chokidar parameters to config file
* fix storage mock
* set default config for library watching
* add fs provider mocks
* cleanup
* add more unit tests for watcher
* chore: fix format + sql
* add more tests
* move unwatch feature back to library service
* add file event unit tests
* chore: formatting
* add documentation
* fix e2e tests
* chore: fix e2e tests
* fix library updating
* test cleanup
* fix typo
* cleanup
* fixing as per pr comments
* reduce library watch config file
* update storage config and mocks
* move negative event tests to unit tests
* fix library watcher e2e
* make watch configuration global
* remove the feature flag
* refactor watcher teardown
* fix microservices init
* centralize asset scan job queue
* improve docs
* add more tests
* chore: open api
* initialize app service
* fix docs
* fix library watch feature flag
* Update docs/docs/features/libraries.md
Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
* fix: import right app service
* don't be truthy
* fix test speling
* stricter library update tests
* move fs watcher mock to external file
* subscribe to config changes
* docker does not need polling
* make library watch() private
* feat: add configuration ui
---------
Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-31 08:15:54 +00:00
Michael Manganiello
4a5b8c3770
feat(server): Enqueue jobs in bulk ( #5974 )
...
* feat(server): Enqueue jobs in bulk
The Job Repository now has a `queueAll` method, that enqueues messages
in bulk (using BullMQ's
[`addBulk`](https://docs.bullmq.io/guide/queues/adding-bulks )),
improving performance when many jobs must be enqueued within the same
operation.
Primary change is in `src/domain/job/job.service.ts`, and other services
have been refactored to use `queueAll` when useful.
As a simple local benchmark, triggering a full thumbnail generation
process over a library of ~1,200 assets and ~350 faces went from
**~600ms** to **~250ms**.
* fix: Review feedback
2024-01-01 15:45:42 -05:00
Jason Rasmussen
9768931275
feat(web,server)!: runtime log level ( #5672 )
...
* feat: change log level at runtime
* chore: open api
* chore: prefer env over runtime
* chore: remove default env value
2023-12-14 16:55:40 +00:00
Jason Rasmussen
33529d1d9b
refactor(server): auth dto ( #5593 )
...
* refactor: AuthUserDto => AuthDto
* refactor: reorganize auth-dto
* refactor: AuthUser() => Auth()
2023-12-09 23:34:12 -05:00
Daniel Dietzler
cd375a976e
feat(server): custom library scanning interval ( #4390 )
...
* add automatic library scan config options
* add validation
* open api
* use CronJob instead of cron-validator
* fix tests
* catch potential error of the library scan initialization
* better description for input field
* move library scan job initialization to server app service
* fix tests
* add comments to all parameters of cronjob contructor
* make scan a child of a more general library object
* open api
* chore: cleanup
* move cronjob handling to job repoistory
* web: select for common cron expressions
* fix open api
* fix tests
* put scanning settings in nested accordion
* fix system config validation
* refactor, tests
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-10-31 15:19:12 -05:00
Jason Rasmussen
088d5addf2
refactor(server): user core ( #4733 )
2023-10-31 11:01:32 -04:00
Daniel Dietzler
50bc92aac0
refactor(server): make access core singleton ( #4609 )
2023-10-23 08:37:51 -04:00
Jonathan Jogenfors
56eb7bf0fc
fix(server): improve library scan queuing performance ( #4418 )
...
* fix: inline mark asset as offline
* fix: improve log message
* chore: lint
* fix: offline asset algorithm
* fix: use set comparison to check what to import
* fix: only mark new offline files as offline
* fix: compare the correct array
* fix: set default library concurrency to 5
* fix: remove one db call when scanning new files
* chore: remove unused import
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-10-10 18:59:13 -04:00
Jonathan Jogenfors
83b63ca12e
fix: only check external path once ( #4419 )
2023-10-10 13:44:43 +00:00
Jason Rasmussen
2ea080cacd
refactor: domain repositories ( #4403 )
2023-10-09 14:25:03 +00:00
Jason Rasmussen
9070a361bc
chore: organize library imports ( #4396 )
2023-10-08 22:52:12 -05:00
Jason Rasmussen
d8e66acd02
chore: use force instead of forceRefresh ( #4394 )
2023-10-08 23:16:13 -04:00
Daniel Dietzler
66ccf298ba
fix library deletion actually deleting assets ( #4386 )
2023-10-07 16:42:08 -04:00
Jason Rasmussen
982dcd7b8d
refactor(server): library asset deletion ( #4366 )
2023-10-07 13:44:10 -04:00
Jason Rasmussen
8d5bf93360
test(server): full backend end-to-end testing with microservices ( #4225 )
...
* feat: asset e2e with job option
* feat: checkout test assets
* feat: library e2e tests
* fix: use node 21 in e2e
* fix: tests
* fix: use normalized external path
* feat: more external path tests
* chore: use parametrized tests
* chore: remove unused test code
* chore: refactor test asset path
* feat: centralize test app creation
* fix: correct error message for missing assets
* feat: test file formats
* fix: don't compare checksum
* feat: build libvips
* fix: install meson
* fix: use immich test asset repo
* feat: test nikon raw files
* fix: set Z timezone
* feat: test offline library files
* feat: richer metadata tests
* feat: e2e tests in docker
* feat: e2e test with arm64 docker
* fix: manual docker compose run
* fix: remove metadata processor import
* fix: run e2e tests in test.yml
* fix: checkout e2e assets
* fix: typo
* fix: checkout files in app directory
* fix: increase e2e memory
* fix: rm submodules
* fix: revert action name
* test: mark file offline when external path changes
* feat: rename env var to TEST_ENV
* docs: new test procedures
* feat: can run docker e2e tests manually if needed
* chore: use new node 20.8 for e2e
* chore: bump exiftool-vendored
* feat: simplify test launching
* fix: rename env vars to use immich_ prefix
* feat: asset folder is submodule
* chore: cleanup after 20.8 upgrade
* fix: don't log postgres in e2e
* fix: better warning about not running all tests
---------
Co-authored-by: Jonathan Jogenfors <jonathan@jogenfors.se>
2023-10-06 23:32:28 +02:00
Jonathan Jogenfors
4dffae3f39
fix(server): normalize external path ( #4239 )
...
* fix: use normalized external path
* fix: move normalization to user core
2023-10-06 20:47:38 +00:00
shenlong
4a8887f37b
feat(server): trash asset ( #4015 )
...
* refactor(server): delete assets endpoint
* fix: formatting
* chore: cleanup
* chore: open api
* chore(mobile): replace DeleteAssetDTO with BulkIdsDTOs
* feat: trash an asset
* chore(server): formatting
* chore: open api
* chore: wording
* chore: open-api
* feat(server): add withDeleted to getAssets queries
* WIP: mobile-recycle-bin
* feat(server): recycle-bin to system config
* feat(web): use recycle-bin system config
* chore(server): domain assetcore removed
* chore(server): rename recycle-bin to trash
* chore(web): rename recycle-bin to trash
* chore(server): always send soft deleted assets for getAllByUserId
* chore(web): formatting
* feat(server): permanent delete assets older than trashed period
* feat(web): trash empty placeholder image
* feat(server): empty trash
* feat(web): empty trash
* WIP: mobile-recycle-bin
* refactor(server): empty / restore trash to separate endpoint
* test(server): handle failures
* test(server): fix e2e server-info test
* test(server): deletion test refactor
* feat(mobile): use map settings from server-config to enable / disable map
* feat(mobile): trash asset
* fix(server): operations on assets in trash
* feat(web): show trash statistics
* fix(web): handle trash enabled
* fix(mobile): restore updates from trash
* fix(server): ignore trashed assets for person
* fix(server): add / remove search index when trashed / restored
* chore(web): format
* fix(server): asset service test
* fix(server): include trashed assts for duplicates from uploads
* feat(mobile): no dialog for trash, always dialog for permanent delete
* refactor(mobile): use isar where instead of dart filter
* refactor(mobile): asset provide - handle deletes in single db txn
* chore(mobile): review changes
* feat(web): confirmation before empty trash
* server: review changes
* fix(server): handle library changes
* fix: filter external assets from getting trashed / deleted
* fix(server): empty-bin
* feat: broadcast config update events through ws
* change order of trash button on mobile
* styling
* fix(mobile): do not show trashed toast for local only assets
---------
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-10-06 02:01:14 -05:00
Jason Rasmussen
192e950567
fix: use local time for time buckets and improve memories ( #4072 )
...
* fix: timezone bucket timezones
* chore: open api
* fix: interpret local time in utc
* fix: tests
* fix: refactor memory lane
* fix(web): use local date in memory viewer
* chore: set localDateTime non-null
* fix: filter out memories from the current year
* wip: move localDateTime to asset
* fix: correct sorting from db
* fix: migration
* fix: web typo
* fix: formatting
* fix: e2e
* chore: localDateTime is non-null
* chore: more non-nulliness
* fix: asset stub
* fix: tests
* fix: use extract and index for day of year
* fix: don't show memories before today
* fix: cleanup
* fix: tests
* fix: only use localtime for tz
* fix: display memories in client timezone
* fix: tests
* fix: svelte tests
* fix: bugs
* chore: open api
---------
Co-authored-by: Jonathan Jogenfors <jonathan@jogenfors.se>
2023-10-04 22:11:11 +00:00
Daniele Ricci
25d1b3e1b1
fix(server): use mtime instead of ctime for fileCreatedAt ( #4191 )
2023-09-24 20:14:25 +07:00
Jonathan Jogenfors
acdc66413c
feat(server,web): libraries ( #3124 )
...
* feat: libraries
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-09-20 13:16:33 +02:00
