1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2025-03-01 15:11:21 +01:00

refactor(server): job repository (#1382)

* refactor(server): job repository

* refactor: job repository

* chore: generate open-api

* fix: job panel

* Remove incorrect subtitle

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
This commit is contained in:
Jason Rasmussen 2023-01-21 23:13:36 -05:00 committed by GitHub
parent f4c90426a5
commit 4cfac47674
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
34 changed files with 418 additions and 1124 deletions

View file

@ -53,7 +53,6 @@ doc/JobCommand.md
doc/JobCommandDto.md doc/JobCommandDto.md
doc/JobCounts.md doc/JobCounts.md
doc/JobId.md doc/JobId.md
doc/JobStatusResponseDto.md
doc/LoginCredentialDto.md doc/LoginCredentialDto.md
doc/LoginResponseDto.md doc/LoginResponseDto.md
doc/LogoutResponseDto.md doc/LogoutResponseDto.md
@ -162,7 +161,6 @@ lib/model/job_command.dart
lib/model/job_command_dto.dart lib/model/job_command_dto.dart
lib/model/job_counts.dart lib/model/job_counts.dart
lib/model/job_id.dart lib/model/job_id.dart
lib/model/job_status_response_dto.dart
lib/model/login_credential_dto.dart lib/model/login_credential_dto.dart
lib/model/login_response_dto.dart lib/model/login_response_dto.dart
lib/model/logout_response_dto.dart lib/model/logout_response_dto.dart
@ -250,7 +248,6 @@ test/job_command_dto_test.dart
test/job_command_test.dart test/job_command_test.dart
test/job_counts_test.dart test/job_counts_test.dart
test/job_id_test.dart test/job_id_test.dart
test/job_status_response_dto_test.dart
test/login_credential_dto_test.dart test/login_credential_dto_test.dart
test/login_response_dto_test.dart test/login_response_dto_test.dart
test/logout_response_dto_test.dart test/logout_response_dto_test.dart

View file

@ -106,7 +106,6 @@ Class | Method | HTTP request | Description
*DeviceInfoApi* | [**updateDeviceInfo**](doc//DeviceInfoApi.md#updatedeviceinfo) | **PATCH** /device-info | *DeviceInfoApi* | [**updateDeviceInfo**](doc//DeviceInfoApi.md#updatedeviceinfo) | **PATCH** /device-info |
*DeviceInfoApi* | [**upsertDeviceInfo**](doc//DeviceInfoApi.md#upsertdeviceinfo) | **PUT** /device-info | *DeviceInfoApi* | [**upsertDeviceInfo**](doc//DeviceInfoApi.md#upsertdeviceinfo) | **PUT** /device-info |
*JobApi* | [**getAllJobsStatus**](doc//JobApi.md#getalljobsstatus) | **GET** /jobs | *JobApi* | [**getAllJobsStatus**](doc//JobApi.md#getalljobsstatus) | **GET** /jobs |
*JobApi* | [**getJobStatus**](doc//JobApi.md#getjobstatus) | **GET** /jobs/{jobId} |
*JobApi* | [**sendJobCommand**](doc//JobApi.md#sendjobcommand) | **PUT** /jobs/{jobId} | *JobApi* | [**sendJobCommand**](doc//JobApi.md#sendjobcommand) | **PUT** /jobs/{jobId} |
*OAuthApi* | [**callback**](doc//OAuthApi.md#callback) | **POST** /oauth/callback | *OAuthApi* | [**callback**](doc//OAuthApi.md#callback) | **POST** /oauth/callback |
*OAuthApi* | [**generateConfig**](doc//OAuthApi.md#generateconfig) | **POST** /oauth/config | *OAuthApi* | [**generateConfig**](doc//OAuthApi.md#generateconfig) | **POST** /oauth/config |
@ -189,7 +188,6 @@ Class | Method | HTTP request | Description
- [JobCommandDto](doc//JobCommandDto.md) - [JobCommandDto](doc//JobCommandDto.md)
- [JobCounts](doc//JobCounts.md) - [JobCounts](doc//JobCounts.md)
- [JobId](doc//JobId.md) - [JobId](doc//JobId.md)
- [JobStatusResponseDto](doc//JobStatusResponseDto.md)
- [LoginCredentialDto](doc//LoginCredentialDto.md) - [LoginCredentialDto](doc//LoginCredentialDto.md)
- [LoginResponseDto](doc//LoginResponseDto.md) - [LoginResponseDto](doc//LoginResponseDto.md)
- [LogoutResponseDto](doc//LogoutResponseDto.md) - [LogoutResponseDto](doc//LogoutResponseDto.md)

View file

@ -8,16 +8,11 @@ import 'package:openapi/api.dart';
## Properties ## Properties
Name | Type | Description | Notes Name | Type | Description | Notes
------------ | ------------- | ------------- | ------------- ------------ | ------------- | ------------- | -------------
**thumbnailGenerationQueueCount** | [**JobCounts**](JobCounts.md) | | **thumbnailGeneration** | [**JobCounts**](JobCounts.md) | |
**metadataExtractionQueueCount** | [**JobCounts**](JobCounts.md) | | **metadataExtraction** | [**JobCounts**](JobCounts.md) | |
**videoConversionQueueCount** | [**JobCounts**](JobCounts.md) | | **videoConversion** | [**JobCounts**](JobCounts.md) | |
**machineLearningQueueCount** | [**JobCounts**](JobCounts.md) | | **machineLearning** | [**JobCounts**](JobCounts.md) | |
**storageMigrationQueueCount** | [**JobCounts**](JobCounts.md) | | **storageTemplateMigration** | [**JobCounts**](JobCounts.md) | |
**isThumbnailGenerationActive** | **bool** | |
**isMetadataExtractionActive** | **bool** | |
**isVideoConversionActive** | **bool** | |
**isMachineLearningActive** | **bool** | |
**isStorageMigrationActive** | **bool** | |
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)

View file

@ -10,7 +10,6 @@ All URIs are relative to */api*
Method | HTTP request | Description Method | HTTP request | Description
------------- | ------------- | ------------- ------------- | ------------- | -------------
[**getAllJobsStatus**](JobApi.md#getalljobsstatus) | **GET** /jobs | [**getAllJobsStatus**](JobApi.md#getalljobsstatus) | **GET** /jobs |
[**getJobStatus**](JobApi.md#getjobstatus) | **GET** /jobs/{jobId} |
[**sendJobCommand**](JobApi.md#sendjobcommand) | **PUT** /jobs/{jobId} | [**sendJobCommand**](JobApi.md#sendjobcommand) | **PUT** /jobs/{jobId} |
@ -59,55 +58,6 @@ This endpoint does not need any parameter.
[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md) [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
# **getJobStatus**
> JobStatusResponseDto getJobStatus(jobId)
### Example
```dart
import 'package:openapi/api.dart';
// TODO Configure HTTP Bearer authorization: bearer
// Case 1. Use String Token
//defaultApiClient.getAuthentication<HttpBearerAuth>('bearer').setAccessToken('YOUR_ACCESS_TOKEN');
// Case 2. Use Function which generate token.
// String yourTokenGeneratorFunction() { ... }
//defaultApiClient.getAuthentication<HttpBearerAuth>('bearer').setAccessToken(yourTokenGeneratorFunction);
final api_instance = JobApi();
final jobId = ; // JobId |
try {
final result = api_instance.getJobStatus(jobId);
print(result);
} catch (e) {
print('Exception when calling JobApi->getJobStatus: $e\n');
}
```
### Parameters
Name | Type | Description | Notes
------------- | ------------- | ------------- | -------------
**jobId** | [**JobId**](.md)| |
### Return type
[**JobStatusResponseDto**](JobStatusResponseDto.md)
### Authorization
[bearer](../README.md#bearer)
### HTTP request headers
- **Content-Type**: Not defined
- **Accept**: application/json
[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
# **sendJobCommand** # **sendJobCommand**
> num sendJobCommand(jobId, jobCommandDto) > num sendJobCommand(jobId, jobCommandDto)

View file

@ -1,16 +0,0 @@
# openapi.model.JobStatusResponseDto
## Load the model package
```dart
import 'package:openapi/api.dart';
```
## Properties
Name | Type | Description | Notes
------------ | ------------- | ------------- | -------------
**isActive** | **bool** | |
**queueCount** | [**Object**](.md) | |
[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)

View file

@ -84,7 +84,6 @@ part 'model/job_command.dart';
part 'model/job_command_dto.dart'; part 'model/job_command_dto.dart';
part 'model/job_counts.dart'; part 'model/job_counts.dart';
part 'model/job_id.dart'; part 'model/job_id.dart';
part 'model/job_status_response_dto.dart';
part 'model/login_credential_dto.dart'; part 'model/login_credential_dto.dart';
part 'model/login_response_dto.dart'; part 'model/login_response_dto.dart';
part 'model/logout_response_dto.dart'; part 'model/logout_response_dto.dart';

View file

@ -60,59 +60,6 @@ class JobApi {
return null; return null;
} }
///
///
/// Note: This method returns the HTTP [Response].
///
/// Parameters:
///
/// * [JobId] jobId (required):
Future<Response> getJobStatusWithHttpInfo(JobId jobId,) async {
// ignore: prefer_const_declarations
final path = r'/jobs/{jobId}'
.replaceAll('{jobId}', jobId.toString());
// ignore: prefer_final_locals
Object? postBody;
final queryParams = <QueryParam>[];
final headerParams = <String, String>{};
final formParams = <String, String>{};
const contentTypes = <String>[];
return apiClient.invokeAPI(
path,
'GET',
queryParams,
postBody,
headerParams,
formParams,
contentTypes.isEmpty ? null : contentTypes.first,
);
}
///
///
/// Parameters:
///
/// * [JobId] jobId (required):
Future<JobStatusResponseDto?> getJobStatus(JobId jobId,) async {
final response = await getJobStatusWithHttpInfo(jobId,);
if (response.statusCode >= HttpStatus.badRequest) {
throw ApiException(response.statusCode, await _decodeBodyBytes(response));
}
// When a remote server returns no body with a status of 204, we shall not decode it.
// At the time of writing this, `dart:convert` will throw an "Unexpected end of input"
// FormatException when trying to decode an empty string.
if (response.body.isNotEmpty && response.statusCode != HttpStatus.noContent) {
return await apiClient.deserializeAsync(await _decodeBodyBytes(response), 'JobStatusResponseDto',) as JobStatusResponseDto;
}
return null;
}
/// ///
/// ///
/// Note: This method returns the HTTP [Response]. /// Note: This method returns the HTTP [Response].

View file

@ -280,8 +280,6 @@ class ApiClient {
return JobCounts.fromJson(value); return JobCounts.fromJson(value);
case 'JobId': case 'JobId':
return JobIdTypeTransformer().decode(value); return JobIdTypeTransformer().decode(value);
case 'JobStatusResponseDto':
return JobStatusResponseDto.fromJson(value);
case 'LoginCredentialDto': case 'LoginCredentialDto':
return LoginCredentialDto.fromJson(value); return LoginCredentialDto.fromJson(value);
case 'LoginResponseDto': case 'LoginResponseDto':

View file

@ -13,80 +13,50 @@ part of openapi.api;
class AllJobStatusResponseDto { class AllJobStatusResponseDto {
/// Returns a new [AllJobStatusResponseDto] instance. /// Returns a new [AllJobStatusResponseDto] instance.
AllJobStatusResponseDto({ AllJobStatusResponseDto({
required this.thumbnailGenerationQueueCount, required this.thumbnailGeneration,
required this.metadataExtractionQueueCount, required this.metadataExtraction,
required this.videoConversionQueueCount, required this.videoConversion,
required this.machineLearningQueueCount, required this.machineLearning,
required this.storageMigrationQueueCount, required this.storageTemplateMigration,
required this.isThumbnailGenerationActive,
required this.isMetadataExtractionActive,
required this.isVideoConversionActive,
required this.isMachineLearningActive,
required this.isStorageMigrationActive,
}); });
JobCounts thumbnailGenerationQueueCount; JobCounts thumbnailGeneration;
JobCounts metadataExtractionQueueCount; JobCounts metadataExtraction;
JobCounts videoConversionQueueCount; JobCounts videoConversion;
JobCounts machineLearningQueueCount; JobCounts machineLearning;
JobCounts storageMigrationQueueCount; JobCounts storageTemplateMigration;
bool isThumbnailGenerationActive;
bool isMetadataExtractionActive;
bool isVideoConversionActive;
bool isMachineLearningActive;
bool isStorageMigrationActive;
@override @override
bool operator ==(Object other) => identical(this, other) || other is AllJobStatusResponseDto && bool operator ==(Object other) => identical(this, other) || other is AllJobStatusResponseDto &&
other.thumbnailGenerationQueueCount == thumbnailGenerationQueueCount && other.thumbnailGeneration == thumbnailGeneration &&
other.metadataExtractionQueueCount == metadataExtractionQueueCount && other.metadataExtraction == metadataExtraction &&
other.videoConversionQueueCount == videoConversionQueueCount && other.videoConversion == videoConversion &&
other.machineLearningQueueCount == machineLearningQueueCount && other.machineLearning == machineLearning &&
other.storageMigrationQueueCount == storageMigrationQueueCount && other.storageTemplateMigration == storageTemplateMigration;
other.isThumbnailGenerationActive == isThumbnailGenerationActive &&
other.isMetadataExtractionActive == isMetadataExtractionActive &&
other.isVideoConversionActive == isVideoConversionActive &&
other.isMachineLearningActive == isMachineLearningActive &&
other.isStorageMigrationActive == isStorageMigrationActive;
@override @override
int get hashCode => int get hashCode =>
// ignore: unnecessary_parenthesis // ignore: unnecessary_parenthesis
(thumbnailGenerationQueueCount.hashCode) + (thumbnailGeneration.hashCode) +
(metadataExtractionQueueCount.hashCode) + (metadataExtraction.hashCode) +
(videoConversionQueueCount.hashCode) + (videoConversion.hashCode) +
(machineLearningQueueCount.hashCode) + (machineLearning.hashCode) +
(storageMigrationQueueCount.hashCode) + (storageTemplateMigration.hashCode);
(isThumbnailGenerationActive.hashCode) +
(isMetadataExtractionActive.hashCode) +
(isVideoConversionActive.hashCode) +
(isMachineLearningActive.hashCode) +
(isStorageMigrationActive.hashCode);
@override @override
String toString() => 'AllJobStatusResponseDto[thumbnailGenerationQueueCount=$thumbnailGenerationQueueCount, metadataExtractionQueueCount=$metadataExtractionQueueCount, videoConversionQueueCount=$videoConversionQueueCount, machineLearningQueueCount=$machineLearningQueueCount, storageMigrationQueueCount=$storageMigrationQueueCount, isThumbnailGenerationActive=$isThumbnailGenerationActive, isMetadataExtractionActive=$isMetadataExtractionActive, isVideoConversionActive=$isVideoConversionActive, isMachineLearningActive=$isMachineLearningActive, isStorageMigrationActive=$isStorageMigrationActive]'; String toString() => 'AllJobStatusResponseDto[thumbnailGeneration=$thumbnailGeneration, metadataExtraction=$metadataExtraction, videoConversion=$videoConversion, machineLearning=$machineLearning, storageTemplateMigration=$storageTemplateMigration]';
Map<String, dynamic> toJson() { Map<String, dynamic> toJson() {
final json = <String, dynamic>{}; final json = <String, dynamic>{};
json[r'thumbnailGenerationQueueCount'] = this.thumbnailGenerationQueueCount; json[r'thumbnail-generation'] = this.thumbnailGeneration;
json[r'metadataExtractionQueueCount'] = this.metadataExtractionQueueCount; json[r'metadata-extraction'] = this.metadataExtraction;
json[r'videoConversionQueueCount'] = this.videoConversionQueueCount; json[r'video-conversion'] = this.videoConversion;
json[r'machineLearningQueueCount'] = this.machineLearningQueueCount; json[r'machine-learning'] = this.machineLearning;
json[r'storageMigrationQueueCount'] = this.storageMigrationQueueCount; json[r'storage-template-migration'] = this.storageTemplateMigration;
json[r'isThumbnailGenerationActive'] = this.isThumbnailGenerationActive;
json[r'isMetadataExtractionActive'] = this.isMetadataExtractionActive;
json[r'isVideoConversionActive'] = this.isVideoConversionActive;
json[r'isMachineLearningActive'] = this.isMachineLearningActive;
json[r'isStorageMigrationActive'] = this.isStorageMigrationActive;
return json; return json;
} }
@ -109,16 +79,11 @@ class AllJobStatusResponseDto {
}()); }());
return AllJobStatusResponseDto( return AllJobStatusResponseDto(
thumbnailGenerationQueueCount: JobCounts.fromJson(json[r'thumbnailGenerationQueueCount'])!, thumbnailGeneration: JobCounts.fromJson(json[r'thumbnail-generation'])!,
metadataExtractionQueueCount: JobCounts.fromJson(json[r'metadataExtractionQueueCount'])!, metadataExtraction: JobCounts.fromJson(json[r'metadata-extraction'])!,
videoConversionQueueCount: JobCounts.fromJson(json[r'videoConversionQueueCount'])!, videoConversion: JobCounts.fromJson(json[r'video-conversion'])!,
machineLearningQueueCount: JobCounts.fromJson(json[r'machineLearningQueueCount'])!, machineLearning: JobCounts.fromJson(json[r'machine-learning'])!,
storageMigrationQueueCount: JobCounts.fromJson(json[r'storageMigrationQueueCount'])!, storageTemplateMigration: JobCounts.fromJson(json[r'storage-template-migration'])!,
isThumbnailGenerationActive: mapValueOfType<bool>(json, r'isThumbnailGenerationActive')!,
isMetadataExtractionActive: mapValueOfType<bool>(json, r'isMetadataExtractionActive')!,
isVideoConversionActive: mapValueOfType<bool>(json, r'isVideoConversionActive')!,
isMachineLearningActive: mapValueOfType<bool>(json, r'isMachineLearningActive')!,
isStorageMigrationActive: mapValueOfType<bool>(json, r'isStorageMigrationActive')!,
); );
} }
return null; return null;
@ -168,16 +133,11 @@ class AllJobStatusResponseDto {
/// The list of required keys that must be present in a JSON. /// The list of required keys that must be present in a JSON.
static const requiredKeys = <String>{ static const requiredKeys = <String>{
'thumbnailGenerationQueueCount', 'thumbnail-generation',
'metadataExtractionQueueCount', 'metadata-extraction',
'videoConversionQueueCount', 'video-conversion',
'machineLearningQueueCount', 'machine-learning',
'storageMigrationQueueCount', 'storage-template-migration',
'isThumbnailGenerationActive',
'isMetadataExtractionActive',
'isVideoConversionActive',
'isMachineLearningActive',
'isStorageMigrationActive',
}; };
} }

View file

@ -1,119 +0,0 @@
//
// AUTO-GENERATED FILE, DO NOT MODIFY!
//
// @dart=2.12
// ignore_for_file: unused_element, unused_import
// ignore_for_file: always_put_required_named_parameters_first
// ignore_for_file: constant_identifier_names
// ignore_for_file: lines_longer_than_80_chars
part of openapi.api;
class JobStatusResponseDto {
/// Returns a new [JobStatusResponseDto] instance.
JobStatusResponseDto({
required this.isActive,
required this.queueCount,
});
bool isActive;
Object queueCount;
@override
bool operator ==(Object other) => identical(this, other) || other is JobStatusResponseDto &&
other.isActive == isActive &&
other.queueCount == queueCount;
@override
int get hashCode =>
// ignore: unnecessary_parenthesis
(isActive.hashCode) +
(queueCount.hashCode);
@override
String toString() => 'JobStatusResponseDto[isActive=$isActive, queueCount=$queueCount]';
Map<String, dynamic> toJson() {
final json = <String, dynamic>{};
json[r'isActive'] = this.isActive;
json[r'queueCount'] = this.queueCount;
return json;
}
/// Returns a new [JobStatusResponseDto] instance and imports its values from
/// [value] if it's a [Map], null otherwise.
// ignore: prefer_constructors_over_static_methods
static JobStatusResponseDto? fromJson(dynamic value) {
if (value is Map) {
final json = value.cast<String, dynamic>();
// Ensure that the map contains the required keys.
// Note 1: the values aren't checked for validity beyond being non-null.
// Note 2: this code is stripped in release mode!
assert(() {
requiredKeys.forEach((key) {
assert(json.containsKey(key), 'Required key "JobStatusResponseDto[$key]" is missing from JSON.');
assert(json[key] != null, 'Required key "JobStatusResponseDto[$key]" has a null value in JSON.');
});
return true;
}());
return JobStatusResponseDto(
isActive: mapValueOfType<bool>(json, r'isActive')!,
queueCount: mapValueOfType<Object>(json, r'queueCount')!,
);
}
return null;
}
static List<JobStatusResponseDto>? listFromJson(dynamic json, {bool growable = false,}) {
final result = <JobStatusResponseDto>[];
if (json is List && json.isNotEmpty) {
for (final row in json) {
final value = JobStatusResponseDto.fromJson(row);
if (value != null) {
result.add(value);
}
}
}
return result.toList(growable: growable);
}
static Map<String, JobStatusResponseDto> mapFromJson(dynamic json) {
final map = <String, JobStatusResponseDto>{};
if (json is Map && json.isNotEmpty) {
json = json.cast<String, dynamic>(); // ignore: parameter_assignments
for (final entry in json.entries) {
final value = JobStatusResponseDto.fromJson(entry.value);
if (value != null) {
map[entry.key] = value;
}
}
}
return map;
}
// maps a json object with a list of JobStatusResponseDto-objects as value to a dart map
static Map<String, List<JobStatusResponseDto>> mapListFromJson(dynamic json, {bool growable = false,}) {
final map = <String, List<JobStatusResponseDto>>{};
if (json is Map && json.isNotEmpty) {
json = json.cast<String, dynamic>(); // ignore: parameter_assignments
for (final entry in json.entries) {
final value = JobStatusResponseDto.listFromJson(entry.value, growable: growable,);
if (value != null) {
map[entry.key] = value;
}
}
}
return map;
}
/// The list of required keys that must be present in a JSON.
static const requiredKeys = <String>{
'isActive',
'queueCount',
};
}

View file

@ -16,53 +16,28 @@ void main() {
// final instance = AllJobStatusResponseDto(); // final instance = AllJobStatusResponseDto();
group('test AllJobStatusResponseDto', () { group('test AllJobStatusResponseDto', () {
// JobCounts thumbnailGenerationQueueCount // JobCounts thumbnailGeneration
test('to test the property `thumbnailGenerationQueueCount`', () async { test('to test the property `thumbnailGeneration`', () async {
// TODO // TODO
}); });
// JobCounts metadataExtractionQueueCount // JobCounts metadataExtraction
test('to test the property `metadataExtractionQueueCount`', () async { test('to test the property `metadataExtraction`', () async {
// TODO // TODO
}); });
// JobCounts videoConversionQueueCount // JobCounts videoConversion
test('to test the property `videoConversionQueueCount`', () async { test('to test the property `videoConversion`', () async {
// TODO // TODO
}); });
// JobCounts machineLearningQueueCount // JobCounts machineLearning
test('to test the property `machineLearningQueueCount`', () async { test('to test the property `machineLearning`', () async {
// TODO // TODO
}); });
// JobCounts storageMigrationQueueCount // JobCounts storageTemplateMigration
test('to test the property `storageMigrationQueueCount`', () async { test('to test the property `storageTemplateMigration`', () async {
// TODO
});
// bool isThumbnailGenerationActive
test('to test the property `isThumbnailGenerationActive`', () async {
// TODO
});
// bool isMetadataExtractionActive
test('to test the property `isMetadataExtractionActive`', () async {
// TODO
});
// bool isVideoConversionActive
test('to test the property `isVideoConversionActive`', () async {
// TODO
});
// bool isMachineLearningActive
test('to test the property `isMachineLearningActive`', () async {
// TODO
});
// bool isStorageMigrationActive
test('to test the property `isStorageMigrationActive`', () async {
// TODO // TODO
}); });

View file

@ -24,13 +24,6 @@ void main() {
// TODO // TODO
}); });
//
//
//Future<JobStatusResponseDto> getJobStatus(JobId jobId) async
test('test getJobStatus', () async {
// TODO
});
// //
// //
//Future<num> sendJobCommand(JobId jobId, JobCommandDto jobCommandDto) async //Future<num> sendJobCommand(JobId jobId, JobCommandDto jobCommandDto) async

View file

@ -1,32 +0,0 @@
//
// AUTO-GENERATED FILE, DO NOT MODIFY!
//
// @dart=2.12
// ignore_for_file: unused_element, unused_import
// ignore_for_file: always_put_required_named_parameters_first
// ignore_for_file: constant_identifier_names
// ignore_for_file: lines_longer_than_80_chars
import 'package:openapi/api.dart';
import 'package:test/test.dart';
// tests for JobStatusResponseDto
void main() {
// final instance = JobStatusResponseDto();
group('test JobStatusResponseDto', () {
// bool isActive
test('to test the property `isActive`', () async {
// TODO
});
// Object queueCount
test('to test the property `queueCount`', () async {
// TODO
});
});
}

View file

@ -295,7 +295,7 @@ export class AssetController {
deleteAssetList.filter((a) => a.id == res.id && res.status == DeleteAssetStatusEnum.SUCCESS); deleteAssetList.filter((a) => a.id == res.id && res.status == DeleteAssetStatusEnum.SUCCESS);
}); });
await this.backgroundTaskService.deleteFileOnDisk(deleteAssetList); await this.backgroundTaskService.deleteFileOnDisk(deleteAssetList as any[]);
return result; return result;
} }

View file

@ -9,11 +9,11 @@ import { TimeGroupEnum } from './dto/get-asset-count-by-time-bucket.dto';
import { AssetCountByUserIdResponseDto } from './response-dto/asset-count-by-user-id-response.dto'; import { AssetCountByUserIdResponseDto } from './response-dto/asset-count-by-user-id-response.dto';
import { DownloadService } from '../../modules/download/download.service'; import { DownloadService } from '../../modules/download/download.service';
import { BackgroundTaskService } from '../../modules/background-task/background-task.service'; import { BackgroundTaskService } from '../../modules/background-task/background-task.service';
import { IAssetUploadedJob, IVideoTranscodeJob } from '@app/domain';
import { Queue } from 'bull';
import { IAlbumRepository } from '../album/album-repository'; import { IAlbumRepository } from '../album/album-repository';
import { StorageService } from '@app/storage'; import { StorageService } from '@app/storage';
import { ISharedLinkRepository } from '../share/shared-link.repository'; import { ISharedLinkRepository } from '../share/shared-link.repository';
import { IJobRepository } from '@app/domain';
import { newJobRepositoryMock } from '@app/domain/../test';
describe('AssetService', () => { describe('AssetService', () => {
let sui: AssetService; let sui: AssetService;
@ -22,10 +22,9 @@ describe('AssetService', () => {
let albumRepositoryMock: jest.Mocked<IAlbumRepository>; let albumRepositoryMock: jest.Mocked<IAlbumRepository>;
let downloadServiceMock: jest.Mocked<Partial<DownloadService>>; let downloadServiceMock: jest.Mocked<Partial<DownloadService>>;
let backgroundTaskServiceMock: jest.Mocked<BackgroundTaskService>; let backgroundTaskServiceMock: jest.Mocked<BackgroundTaskService>;
let assetUploadedQueueMock: jest.Mocked<Queue<IAssetUploadedJob>>;
let videoConversionQueueMock: jest.Mocked<Queue<IVideoTranscodeJob>>;
let storageSeriveMock: jest.Mocked<StorageService>; let storageSeriveMock: jest.Mocked<StorageService>;
let sharedLinkRepositoryMock: jest.Mocked<ISharedLinkRepository>; let sharedLinkRepositoryMock: jest.Mocked<ISharedLinkRepository>;
let jobMock: jest.Mocked<IJobRepository>;
const authUser: AuthUserDto = Object.freeze({ const authUser: AuthUserDto = Object.freeze({
id: 'user_id_1', id: 'user_id_1',
email: 'auth@test.com', email: 'auth@test.com',
@ -148,16 +147,17 @@ describe('AssetService', () => {
getByIdAndUserId: jest.fn(), getByIdAndUserId: jest.fn(),
}; };
jobMock = newJobRepositoryMock();
sui = new AssetService( sui = new AssetService(
assetRepositoryMock, assetRepositoryMock,
albumRepositoryMock, albumRepositoryMock,
a, a,
backgroundTaskServiceMock, backgroundTaskServiceMock,
assetUploadedQueueMock,
videoConversionQueueMock,
downloadServiceMock as DownloadService, downloadServiceMock as DownloadService,
storageSeriveMock, storageSeriveMock,
sharedLinkRepositoryMock, sharedLinkRepositoryMock,
jobMock,
); );
}); });

View file

@ -43,9 +43,7 @@ import { CheckExistingAssetsResponseDto } from './response-dto/check-existing-as
import { UpdateAssetDto } from './dto/update-asset.dto'; import { UpdateAssetDto } from './dto/update-asset.dto';
import { AssetFileUploadResponseDto } from './response-dto/asset-file-upload-response.dto'; import { AssetFileUploadResponseDto } from './response-dto/asset-file-upload-response.dto';
import { BackgroundTaskService } from '../../modules/background-task/background-task.service'; import { BackgroundTaskService } from '../../modules/background-task/background-task.service';
import { IAssetUploadedJob, IVideoTranscodeJob, JobName, QueueName } from '@app/domain'; import { IJobRepository, JobName } from '@app/domain';
import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';
import { DownloadService } from '../../modules/download/download.service'; import { DownloadService } from '../../modules/download/download.service';
import { DownloadDto } from './dto/download-library.dto'; import { DownloadDto } from './dto/download-library.dto';
import { IAlbumRepository } from '../album/album-repository'; import { IAlbumRepository } from '../album/album-repository';
@ -66,24 +64,14 @@ export class AssetService {
constructor( constructor(
@Inject(IAssetRepository) private _assetRepository: IAssetRepository, @Inject(IAssetRepository) private _assetRepository: IAssetRepository,
@Inject(IAlbumRepository) private _albumRepository: IAlbumRepository, @Inject(IAlbumRepository) private _albumRepository: IAlbumRepository,
@InjectRepository(AssetEntity) @InjectRepository(AssetEntity)
private assetRepository: Repository<AssetEntity>, private assetRepository: Repository<AssetEntity>,
private backgroundTaskService: BackgroundTaskService, private backgroundTaskService: BackgroundTaskService,
@InjectQueue(QueueName.ASSET_UPLOADED)
private assetUploadedQueue: Queue<IAssetUploadedJob>,
@InjectQueue(QueueName.VIDEO_CONVERSION)
private videoConversionQueue: Queue<IVideoTranscodeJob>,
private downloadService: DownloadService, private downloadService: DownloadService,
private storageService: StorageService, private storageService: StorageService,
@Inject(ISharedLinkRepository) sharedLinkRepository: ISharedLinkRepository, @Inject(ISharedLinkRepository) sharedLinkRepository: ISharedLinkRepository,
@Inject(IJobRepository) private jobRepository: IJobRepository,
) { ) {
this.shareCore = new ShareCore(sharedLinkRepository); this.shareCore = new ShareCore(sharedLinkRepository);
} }
@ -122,7 +110,7 @@ export class AssetService {
await this.storageService.moveAsset(livePhotoAssetEntity, originalAssetData.originalname); await this.storageService.moveAsset(livePhotoAssetEntity, originalAssetData.originalname);
await this.videoConversionQueue.add(JobName.VIDEO_CONVERSION, { asset: livePhotoAssetEntity }); await this.jobRepository.add({ name: JobName.VIDEO_CONVERSION, data: { asset: livePhotoAssetEntity } });
} }
const assetEntity = await this.createUserAsset( const assetEntity = await this.createUserAsset(
@ -146,11 +134,10 @@ export class AssetService {
const movedAsset = await this.storageService.moveAsset(assetEntity, originalAssetData.originalname); const movedAsset = await this.storageService.moveAsset(assetEntity, originalAssetData.originalname);
await this.assetUploadedQueue.add( await this.jobRepository.add({
JobName.ASSET_UPLOADED, name: JobName.ASSET_UPLOADED,
{ asset: movedAsset, fileName: originalAssetData.originalname }, data: { asset: movedAsset, fileName: originalAssetData.originalname },
{ jobId: movedAsset.id }, });
);
return new AssetFileUploadResponseDto(movedAsset.id); return new AssetFileUploadResponseDto(movedAsset.id);
} catch (err) { } catch (err) {

View file

@ -1,11 +1,9 @@
import { Controller, Get, Body, ValidationPipe, Put, Param } from '@nestjs/common'; import { Body, Controller, Get, Param, Put, ValidationPipe } from '@nestjs/common';
import { JobService } from './job.service';
import { ApiBearerAuth, ApiTags } from '@nestjs/swagger'; import { ApiBearerAuth, ApiTags } from '@nestjs/swagger';
import { Authenticated } from '../../decorators/authenticated.decorator'; import { Authenticated } from '../../decorators/authenticated.decorator';
import { AllJobStatusResponseDto } from './response-dto/all-job-status-response.dto'; import { AllJobStatusResponseDto } from './response-dto/all-job-status-response.dto';
import { GetJobDto } from './dto/get-job.dto'; import { GetJobDto } from './dto/get-job.dto';
import { JobStatusResponseDto } from './response-dto/job-status-response.dto'; import { JobService } from './job.service';
import { JobCommandDto } from './dto/job-command.dto'; import { JobCommandDto } from './dto/job-command.dto';
@Authenticated({ admin: true }) @Authenticated({ admin: true })
@ -20,21 +18,16 @@ export class JobController {
return this.jobService.getAllJobsStatus(); return this.jobService.getAllJobsStatus();
} }
@Get('/:jobId')
getJobStatus(@Param(ValidationPipe) params: GetJobDto): Promise<JobStatusResponseDto> {
return this.jobService.getJobStatus(params);
}
@Put('/:jobId') @Put('/:jobId')
async sendJobCommand( async sendJobCommand(
@Param(ValidationPipe) params: GetJobDto, @Param(ValidationPipe) params: GetJobDto,
@Body(ValidationPipe) body: JobCommandDto, @Body(ValidationPipe) body: JobCommandDto,
): Promise<number> { ): Promise<number> {
if (body.command === 'start') { if (body.command === 'start') {
return await this.jobService.startJob(params); return await this.jobService.start(params.jobId);
} }
if (body.command === 'stop') { if (body.command === 'stop') {
return await this.jobService.stopJob(params); return await this.jobService.stop(params.jobId);
} }
return 0; return 0;
} }

View file

@ -1,217 +1,118 @@
import { import { JobName, IJobRepository, QueueName } from '@app/domain';
IMachineLearningJob,
IMetadataExtractionJob,
IThumbnailGenerationJob,
IVideoTranscodeJob,
JobName,
QueueName,
} from '@app/domain';
import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';
import { BadRequestException, Inject, Injectable } from '@nestjs/common'; import { BadRequestException, Inject, Injectable } from '@nestjs/common';
import { AllJobStatusResponseDto } from './response-dto/all-job-status-response.dto'; import { AllJobStatusResponseDto } from './response-dto/all-job-status-response.dto';
import { IAssetRepository } from '../asset/asset-repository'; import { IAssetRepository } from '../asset/asset-repository';
import { AssetType } from '@app/infra'; import { AssetType } from '@app/infra';
import { GetJobDto, JobId } from './dto/get-job.dto'; import { JobId } from './dto/get-job.dto';
import { JobStatusResponseDto } from './response-dto/job-status-response.dto';
import { StorageService } from '@app/storage';
import { MACHINE_LEARNING_ENABLED } from '@app/common'; import { MACHINE_LEARNING_ENABLED } from '@app/common';
const jobIds = Object.values(JobId) as JobId[];
@Injectable() @Injectable()
export class JobService { export class JobService {
constructor( constructor(
@InjectQueue(QueueName.THUMBNAIL_GENERATION) @Inject(IAssetRepository) private _assetRepository: IAssetRepository,
private thumbnailGeneratorQueue: Queue<IThumbnailGenerationJob>, @Inject(IJobRepository) private jobRepository: IJobRepository,
@InjectQueue(QueueName.METADATA_EXTRACTION)
private metadataExtractionQueue: Queue<IMetadataExtractionJob>,
@InjectQueue(QueueName.VIDEO_CONVERSION)
private videoConversionQueue: Queue<IVideoTranscodeJob>,
@InjectQueue(QueueName.MACHINE_LEARNING)
private machineLearningQueue: Queue<IMachineLearningJob>,
@InjectQueue(QueueName.CONFIG)
private configQueue: Queue,
@Inject(IAssetRepository)
private _assetRepository: IAssetRepository,
private storageService: StorageService,
) { ) {
this.thumbnailGeneratorQueue.empty(); for (const jobId of jobIds) {
this.metadataExtractionQueue.empty(); this.jobRepository.empty(this.asQueueName(jobId));
this.videoConversionQueue.empty(); }
this.configQueue.empty();
} }
async startJob(jobDto: GetJobDto): Promise<number> { start(jobId: JobId): Promise<number> {
switch (jobDto.jobId) { return this.run(this.asQueueName(jobId));
case JobId.THUMBNAIL_GENERATION: }
return this.runThumbnailGenerationJob();
case JobId.METADATA_EXTRACTION: async stop(jobId: JobId): Promise<number> {
return this.runMetadataExtractionJob(); await this.jobRepository.empty(this.asQueueName(jobId));
case JobId.VIDEO_CONVERSION: return 0;
return this.runVideoConversionJob();
case JobId.MACHINE_LEARNING:
return this.runMachineLearningPipeline();
case JobId.STORAGE_TEMPLATE_MIGRATION:
return this.runStorageMigration();
default:
throw new BadRequestException('Invalid job id');
}
} }
async getAllJobsStatus(): Promise<AllJobStatusResponseDto> { async getAllJobsStatus(): Promise<AllJobStatusResponseDto> {
const thumbnailGeneratorJobCount = await this.thumbnailGeneratorQueue.getJobCounts();
const metadataExtractionJobCount = await this.metadataExtractionQueue.getJobCounts();
const videoConversionJobCount = await this.videoConversionQueue.getJobCounts();
const machineLearningJobCount = await this.machineLearningQueue.getJobCounts();
const storageMigrationJobCount = await this.configQueue.getJobCounts();
const response = new AllJobStatusResponseDto(); const response = new AllJobStatusResponseDto();
response.isThumbnailGenerationActive = Boolean(thumbnailGeneratorJobCount.waiting); for (const jobId of jobIds) {
response.thumbnailGenerationQueueCount = thumbnailGeneratorJobCount; response[jobId] = await this.jobRepository.getJobCounts(this.asQueueName(jobId));
response.isMetadataExtractionActive = Boolean(metadataExtractionJobCount.waiting); }
response.metadataExtractionQueueCount = metadataExtractionJobCount;
response.isVideoConversionActive = Boolean(videoConversionJobCount.waiting);
response.videoConversionQueueCount = videoConversionJobCount;
response.isMachineLearningActive = Boolean(machineLearningJobCount.waiting);
response.machineLearningQueueCount = machineLearningJobCount;
response.isStorageMigrationActive = Boolean(storageMigrationJobCount.active);
response.storageMigrationQueueCount = storageMigrationJobCount;
return response; return response;
} }
async getJobStatus(query: GetJobDto): Promise<JobStatusResponseDto> { private async run(name: QueueName): Promise<number> {
const response = new JobStatusResponseDto(); const isActive = await this.jobRepository.isActive(name);
if (query.jobId === JobId.THUMBNAIL_GENERATION) { if (isActive) {
response.isActive = Boolean((await this.thumbnailGeneratorQueue.getJobCounts()).waiting); throw new BadRequestException(`Job is already running`);
response.queueCount = await this.thumbnailGeneratorQueue.getJobCounts();
} }
if (query.jobId === JobId.METADATA_EXTRACTION) { switch (name) {
response.isActive = Boolean((await this.metadataExtractionQueue.getJobCounts()).waiting); case QueueName.VIDEO_CONVERSION: {
response.queueCount = await this.metadataExtractionQueue.getJobCounts(); const assets = await this._assetRepository.getAssetWithNoEncodedVideo();
} for (const asset of assets) {
await this.jobRepository.add({ name: JobName.VIDEO_CONVERSION, data: { asset } });
}
if (query.jobId === JobId.VIDEO_CONVERSION) { return assets.length;
response.isActive = Boolean((await this.videoConversionQueue.getJobCounts()).waiting);
response.queueCount = await this.videoConversionQueue.getJobCounts();
}
if (query.jobId === JobId.STORAGE_TEMPLATE_MIGRATION) {
response.isActive = Boolean((await this.configQueue.getJobCounts()).waiting);
response.queueCount = await this.configQueue.getJobCounts();
}
return response;
}
async stopJob(query: GetJobDto): Promise<number> {
switch (query.jobId) {
case JobId.THUMBNAIL_GENERATION:
this.thumbnailGeneratorQueue.empty();
return 0;
case JobId.METADATA_EXTRACTION:
this.metadataExtractionQueue.empty();
return 0;
case JobId.VIDEO_CONVERSION:
this.videoConversionQueue.empty();
return 0;
case JobId.MACHINE_LEARNING:
this.machineLearningQueue.empty();
return 0;
case JobId.STORAGE_TEMPLATE_MIGRATION:
this.configQueue.empty();
return 0;
default:
throw new BadRequestException('Invalid job id');
}
}
private async runThumbnailGenerationJob(): Promise<number> {
const jobCount = await this.thumbnailGeneratorQueue.getJobCounts();
if (jobCount.waiting > 0) {
throw new BadRequestException('Thumbnail generation job is already running');
}
const assetsWithNoThumbnail = await this._assetRepository.getAssetWithNoThumbnail();
for (const asset of assetsWithNoThumbnail) {
await this.thumbnailGeneratorQueue.add(JobName.GENERATE_JPEG_THUMBNAIL, { asset });
}
return assetsWithNoThumbnail.length;
}
private async runMetadataExtractionJob(): Promise<number> {
const jobCount = await this.metadataExtractionQueue.getJobCounts();
if (jobCount.waiting > 0) {
throw new BadRequestException('Metadata extraction job is already running');
}
const assetsWithNoExif = await this._assetRepository.getAssetWithNoEXIF();
for (const asset of assetsWithNoExif) {
if (asset.type === AssetType.VIDEO) {
await this.metadataExtractionQueue.add(JobName.EXTRACT_VIDEO_METADATA, { asset, fileName: asset.id });
} else {
await this.metadataExtractionQueue.add(JobName.EXIF_EXTRACTION, { asset, fileName: asset.id });
} }
case QueueName.CONFIG:
await this.jobRepository.add({ name: JobName.TEMPLATE_MIGRATION });
return 1;
case QueueName.MACHINE_LEARNING: {
if (!MACHINE_LEARNING_ENABLED) {
throw new BadRequestException('Machine learning is not enabled.');
}
const assets = await this._assetRepository.getAssetWithNoSmartInfo();
for (const asset of assets) {
await this.jobRepository.add({ name: JobName.IMAGE_TAGGING, data: { asset } });
await this.jobRepository.add({ name: JobName.OBJECT_DETECTION, data: { asset } });
}
return assets.length;
}
case QueueName.METADATA_EXTRACTION: {
const assets = await this._assetRepository.getAssetWithNoEXIF();
for (const asset of assets) {
if (asset.type === AssetType.VIDEO) {
await this.jobRepository.add({ name: JobName.EXTRACT_VIDEO_METADATA, data: { asset, fileName: asset.id } });
} else {
await this.jobRepository.add({ name: JobName.EXIF_EXTRACTION, data: { asset, fileName: asset.id } });
}
}
return assets.length;
}
case QueueName.THUMBNAIL_GENERATION: {
const assets = await this._assetRepository.getAssetWithNoThumbnail();
for (const asset of assets) {
await this.jobRepository.add({ name: JobName.GENERATE_JPEG_THUMBNAIL, data: { asset } });
}
return assets.length;
}
default:
return 0;
} }
return assetsWithNoExif.length;
} }
private async runMachineLearningPipeline(): Promise<number> { private asQueueName(jobId: JobId) {
if (!MACHINE_LEARNING_ENABLED) { switch (jobId) {
throw new BadRequestException('Machine learning is not enabled.'); case JobId.THUMBNAIL_GENERATION:
return QueueName.THUMBNAIL_GENERATION;
case JobId.METADATA_EXTRACTION:
return QueueName.METADATA_EXTRACTION;
case JobId.VIDEO_CONVERSION:
return QueueName.VIDEO_CONVERSION;
case JobId.STORAGE_TEMPLATE_MIGRATION:
return QueueName.CONFIG;
case JobId.MACHINE_LEARNING:
return QueueName.MACHINE_LEARNING;
default:
throw new BadRequestException(`Invalid job id: ${jobId}`);
} }
const jobCount = await this.machineLearningQueue.getJobCounts();
if (jobCount.waiting > 0) {
throw new BadRequestException('Metadata extraction job is already running');
}
const assetWithNoSmartInfo = await this._assetRepository.getAssetWithNoSmartInfo();
for (const asset of assetWithNoSmartInfo) {
await this.machineLearningQueue.add(JobName.IMAGE_TAGGING, { asset });
await this.machineLearningQueue.add(JobName.OBJECT_DETECTION, { asset });
}
return assetWithNoSmartInfo.length;
}
private async runVideoConversionJob(): Promise<number> {
const jobCount = await this.videoConversionQueue.getJobCounts();
if (jobCount.waiting > 0) {
throw new BadRequestException('Video conversion job is already running');
}
const assetsWithNoConvertedVideo = await this._assetRepository.getAssetWithNoEncodedVideo();
for (const asset of assetsWithNoConvertedVideo) {
await this.videoConversionQueue.add(JobName.VIDEO_CONVERSION, { asset });
}
return assetsWithNoConvertedVideo.length;
}
async runStorageMigration() {
const jobCount = await this.configQueue.getJobCounts();
if (jobCount.active > 0) {
throw new BadRequestException('Storage migration job is already running');
}
await this.configQueue.add(JobName.TEMPLATE_MIGRATION, {});
return 1;
} }
} }

View file

@ -1,4 +1,5 @@
import { ApiProperty } from '@nestjs/swagger'; import { ApiProperty } from '@nestjs/swagger';
import { JobId } from '../dto/get-job.dto';
export class JobCounts { export class JobCounts {
@ApiProperty({ type: 'integer' }) @ApiProperty({ type: 'integer' })
@ -12,35 +13,20 @@ export class JobCounts {
@ApiProperty({ type: 'integer' }) @ApiProperty({ type: 'integer' })
waiting!: number; waiting!: number;
} }
export class AllJobStatusResponseDto { export class AllJobStatusResponseDto {
isThumbnailGenerationActive!: boolean; @ApiProperty({ type: JobCounts })
isMetadataExtractionActive!: boolean; [JobId.THUMBNAIL_GENERATION]!: JobCounts;
isVideoConversionActive!: boolean;
isMachineLearningActive!: boolean;
isStorageMigrationActive!: boolean;
@ApiProperty({ @ApiProperty({ type: JobCounts })
type: JobCounts, [JobId.METADATA_EXTRACTION]!: JobCounts;
})
thumbnailGenerationQueueCount!: JobCounts;
@ApiProperty({ @ApiProperty({ type: JobCounts })
type: JobCounts, [JobId.VIDEO_CONVERSION]!: JobCounts;
})
metadataExtractionQueueCount!: JobCounts;
@ApiProperty({ @ApiProperty({ type: JobCounts })
type: JobCounts, [JobId.MACHINE_LEARNING]!: JobCounts;
})
videoConversionQueueCount!: JobCounts;
@ApiProperty({ @ApiProperty({ type: JobCounts })
type: JobCounts, [JobId.STORAGE_TEMPLATE_MIGRATION]!: JobCounts;
})
machineLearningQueueCount!: JobCounts;
@ApiProperty({
type: JobCounts,
})
storageMigrationQueueCount!: JobCounts;
} }

View file

@ -1,6 +0,0 @@
import Bull from 'bull';
export class JobStatusResponseDto {
isActive!: boolean;
queueCount!: Bull.JobCounts;
}

View file

@ -1,12 +1,9 @@
import { BullModule } from '@nestjs/bull';
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { QueueName } from '@app/domain';
import { BackgroundTaskProcessor } from './background-task.processor'; import { BackgroundTaskProcessor } from './background-task.processor';
import { BackgroundTaskService } from './background-task.service'; import { BackgroundTaskService } from './background-task.service';
@Module({ @Module({
imports: [BullModule.registerQueue({ name: QueueName.BACKGROUND_TASK })],
providers: [BackgroundTaskService, BackgroundTaskProcessor], providers: [BackgroundTaskService, BackgroundTaskProcessor],
exports: [BackgroundTaskService, BullModule], exports: [BackgroundTaskService],
}) })
export class BackgroundTaskModule {} export class BackgroundTaskModule {}

View file

@ -2,12 +2,12 @@ import { assetUtils } from '@app/common/utils';
import { Process, Processor } from '@nestjs/bull'; import { Process, Processor } from '@nestjs/bull';
import { Job } from 'bull'; import { Job } from 'bull';
import { JobName, QueueName } from '@app/domain'; import { JobName, QueueName } from '@app/domain';
import { AssetResponseDto } from '../../api-v1/asset/response-dto/asset-response.dto'; import { AssetEntity } from '@app/infra';
@Processor(QueueName.BACKGROUND_TASK) @Processor(QueueName.BACKGROUND_TASK)
export class BackgroundTaskProcessor { export class BackgroundTaskProcessor {
@Process(JobName.DELETE_FILE_ON_DISK) @Process(JobName.DELETE_FILE_ON_DISK)
async deleteFileOnDisk(job: Job<{ assets: AssetResponseDto[] }>) { async deleteFileOnDisk(job: Job<{ assets: AssetEntity[] }>) {
const { assets } = job.data; const { assets } = job.data;
for (const asset of assets) { for (const asset of assets) {

View file

@ -1,17 +1,12 @@
import { InjectQueue } from '@nestjs/bull/dist/decorators'; import { IJobRepository, JobName } from '@app/domain';
import { Injectable } from '@nestjs/common'; import { AssetEntity } from '@app/infra';
import { Queue } from 'bull'; import { Inject, Injectable } from '@nestjs/common';
import { JobName, QueueName } from '@app/domain';
import { AssetResponseDto } from '../../api-v1/asset/response-dto/asset-response.dto';
@Injectable() @Injectable()
export class BackgroundTaskService { export class BackgroundTaskService {
constructor( constructor(@Inject(IJobRepository) private jobRepository: IJobRepository) {}
@InjectQueue(QueueName.BACKGROUND_TASK)
private backgroundTaskQueue: Queue,
) {}
async deleteFileOnDisk(assets: AssetResponseDto[]) { async deleteFileOnDisk(assets: AssetEntity[]) {
await this.backgroundTaskQueue.add(JobName.DELETE_FILE_ON_DISK, { assets }); await this.jobRepository.add({ name: JobName.DELETE_FILE_ON_DISK, data: { assets } });
} }
} }

View file

@ -1,14 +1,11 @@
import { Injectable, Logger } from '@nestjs/common'; import { Inject, Injectable, Logger } from '@nestjs/common';
import { Cron, CronExpression } from '@nestjs/schedule'; import { Cron, CronExpression } from '@nestjs/schedule';
import { InjectRepository } from '@nestjs/typeorm'; import { InjectRepository } from '@nestjs/typeorm';
import { IsNull, Not, Repository } from 'typeorm'; import { IsNull, Not, Repository } from 'typeorm';
import { AssetEntity, AssetType, ExifEntity, UserEntity } from '@app/infra'; import { AssetEntity, AssetType, ExifEntity, UserEntity } from '@app/infra';
import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';
import { IMetadataExtractionJob, IVideoTranscodeJob, QueueName, JobName } from '@app/domain';
import { ConfigService } from '@nestjs/config'; import { ConfigService } from '@nestjs/config';
import { IUserDeletionJob } from '@app/domain';
import { userUtils } from '@app/common'; import { userUtils } from '@app/common';
import { IJobRepository, JobName } from '@app/domain';
@Injectable() @Injectable()
export class ScheduleTasksService { export class ScheduleTasksService {
@ -22,17 +19,7 @@ export class ScheduleTasksService {
@InjectRepository(ExifEntity) @InjectRepository(ExifEntity)
private exifRepository: Repository<ExifEntity>, private exifRepository: Repository<ExifEntity>,
@InjectQueue(QueueName.THUMBNAIL_GENERATION) @Inject(IJobRepository) private jobRepository: IJobRepository,
private thumbnailGeneratorQueue: Queue,
@InjectQueue(QueueName.VIDEO_CONVERSION)
private videoConversionQueue: Queue<IVideoTranscodeJob>,
@InjectQueue(QueueName.METADATA_EXTRACTION)
private metadataExtractionQueue: Queue<IMetadataExtractionJob>,
@InjectQueue(QueueName.USER_DELETION)
private userDeletionQueue: Queue<IUserDeletionJob>,
private configService: ConfigService, private configService: ConfigService,
) {} ) {}
@ -51,7 +38,7 @@ export class ScheduleTasksService {
} }
for (const asset of assets) { for (const asset of assets) {
await this.thumbnailGeneratorQueue.add(JobName.GENERATE_WEBP_THUMBNAIL, { asset: asset }); await this.jobRepository.add({ name: JobName.GENERATE_WEBP_THUMBNAIL, data: { asset } });
} }
} }
@ -69,7 +56,7 @@ export class ScheduleTasksService {
}); });
for (const asset of assets) { for (const asset of assets) {
await this.videoConversionQueue.add(JobName.VIDEO_CONVERSION, { asset }); await this.jobRepository.add({ name: JobName.VIDEO_CONVERSION, data: { asset } });
} }
} }
@ -87,11 +74,11 @@ export class ScheduleTasksService {
}); });
for (const exif of exifInfo) { for (const exif of exifInfo) {
await this.metadataExtractionQueue.add( await this.jobRepository.add({
JobName.REVERSE_GEOCODING, name: JobName.REVERSE_GEOCODING,
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
{ exifId: exif.id, latitude: exif.latitude!, longitude: exif.longitude! }, data: { exifId: exif.id, latitude: exif.latitude!, longitude: exif.longitude! },
); });
} }
} }
} }
@ -106,9 +93,9 @@ export class ScheduleTasksService {
for (const asset of exifAssets) { for (const asset of exifAssets) {
if (asset.type === AssetType.VIDEO) { if (asset.type === AssetType.VIDEO) {
await this.metadataExtractionQueue.add(JobName.EXTRACT_VIDEO_METADATA, { asset, fileName: asset.id }); await this.jobRepository.add({ name: JobName.EXTRACT_VIDEO_METADATA, data: { asset, fileName: asset.id } });
} else { } else {
await this.metadataExtractionQueue.add(JobName.EXIF_EXTRACTION, { asset, fileName: asset.id }); await this.jobRepository.add({ name: JobName.EXIF_EXTRACTION, data: { asset, fileName: asset.id } });
} }
} }
} }
@ -118,7 +105,7 @@ export class ScheduleTasksService {
const usersToDelete = await this.userRepository.find({ withDeleted: true, where: { deletedAt: Not(IsNull()) } }); const usersToDelete = await this.userRepository.find({ withDeleted: true, where: { deletedAt: Not(IsNull()) } });
for (const user of usersToDelete) { for (const user of usersToDelete) {
if (userUtils.isReadyForDeletion(user)) { if (userUtils.isReadyForDeletion(user)) {
await this.userDeletionQueue.add(JobName.USER_DELETION, { user }); await this.jobRepository.add({ name: JobName.USER_DELETION, data: { user } });
} }
} }
} }

View file

@ -1,17 +1,16 @@
import { QueueName } from '@app/domain'; import { Inject, Injectable, OnModuleInit } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bull'; import { IJobRepository, JobName } from '@app/domain';
import { Injectable, OnModuleInit } from '@nestjs/common';
import { Queue } from 'bull'; const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(() => resolve(), ms));
@Injectable() @Injectable()
export class MicroservicesService implements OnModuleInit { export class MicroservicesService implements OnModuleInit {
constructor( constructor(@Inject(IJobRepository) private jobRepository: IJobRepository) {}
@InjectQueue(QueueName.CHECKSUM_GENERATION)
private generateChecksumQueue: Queue,
) {}
async onModuleInit() { async onModuleInit() {
// wait for migration // wait for migration
await this.generateChecksumQueue.add({}, { delay: 10000 }); await sleep(10_000);
await this.jobRepository.add({ name: JobName.CHECKSUM_GENERATION });
} }
} }

View file

@ -1,5 +1,5 @@
import { AssetEntity } from '@app/infra'; import { AssetEntity } from '@app/infra';
import { QueueName } from '@app/domain'; import { JobName, QueueName } from '@app/domain';
import { Process, Processor } from '@nestjs/bull'; import { Process, Processor } from '@nestjs/bull';
import { Logger } from '@nestjs/common'; import { Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm'; import { InjectRepository } from '@nestjs/typeorm';
@ -15,7 +15,7 @@ export class GenerateChecksumProcessor {
private assetRepository: Repository<AssetEntity>, private assetRepository: Repository<AssetEntity>,
) {} ) {}
@Process() @Process(JobName.CHECKSUM_GENERATION)
async generateChecksum() { async generateChecksum() {
const pageSize = 200; const pageSize = 200;
let hasNext = true; let hasNext = true;

View file

@ -2721,40 +2721,6 @@
} }
}, },
"/jobs/{jobId}": { "/jobs/{jobId}": {
"get": {
"operationId": "getJobStatus",
"description": "",
"parameters": [
{
"name": "jobId",
"required": true,
"in": "path",
"schema": {
"$ref": "#/components/schemas/JobId"
}
}
],
"responses": {
"200": {
"description": "",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/JobStatusResponseDto"
}
}
}
}
},
"tags": [
"Job"
],
"security": [
{
"bearer": []
}
]
},
"put": { "put": {
"operationId": "sendJobCommand", "operationId": "sendJobCommand",
"description": "", "description": "",
@ -4569,48 +4535,28 @@
"AllJobStatusResponseDto": { "AllJobStatusResponseDto": {
"type": "object", "type": "object",
"properties": { "properties": {
"thumbnailGenerationQueueCount": { "thumbnail-generation": {
"$ref": "#/components/schemas/JobCounts" "$ref": "#/components/schemas/JobCounts"
}, },
"metadataExtractionQueueCount": { "metadata-extraction": {
"$ref": "#/components/schemas/JobCounts" "$ref": "#/components/schemas/JobCounts"
}, },
"videoConversionQueueCount": { "video-conversion": {
"$ref": "#/components/schemas/JobCounts" "$ref": "#/components/schemas/JobCounts"
}, },
"machineLearningQueueCount": { "machine-learning": {
"$ref": "#/components/schemas/JobCounts" "$ref": "#/components/schemas/JobCounts"
}, },
"storageMigrationQueueCount": { "storage-template-migration": {
"$ref": "#/components/schemas/JobCounts" "$ref": "#/components/schemas/JobCounts"
},
"isThumbnailGenerationActive": {
"type": "boolean"
},
"isMetadataExtractionActive": {
"type": "boolean"
},
"isVideoConversionActive": {
"type": "boolean"
},
"isMachineLearningActive": {
"type": "boolean"
},
"isStorageMigrationActive": {
"type": "boolean"
} }
}, },
"required": [ "required": [
"thumbnailGenerationQueueCount", "thumbnail-generation",
"metadataExtractionQueueCount", "metadata-extraction",
"videoConversionQueueCount", "video-conversion",
"machineLearningQueueCount", "machine-learning",
"storageMigrationQueueCount", "storage-template-migration"
"isThumbnailGenerationActive",
"isMetadataExtractionActive",
"isVideoConversionActive",
"isMachineLearningActive",
"isStorageMigrationActive"
] ]
}, },
"JobId": { "JobId": {
@ -4623,21 +4569,6 @@
"storage-template-migration" "storage-template-migration"
] ]
}, },
"JobStatusResponseDto": {
"type": "object",
"properties": {
"isActive": {
"type": "boolean"
},
"queueCount": {
"type": "object"
}
},
"required": [
"isActive",
"queueCount"
]
},
"JobCommand": { "JobCommand": {
"type": "string", "type": "string",
"enum": [ "enum": [

View file

@ -24,4 +24,5 @@ export enum JobName {
OBJECT_DETECTION = 'detect-object', OBJECT_DETECTION = 'detect-object',
IMAGE_TAGGING = 'tag-image', IMAGE_TAGGING = 'tag-image',
DELETE_FILE_ON_DISK = 'delete-file-on-disk', DELETE_FILE_ON_DISK = 'delete-file-on-disk',
CHECKSUM_GENERATION = 'checksum-generation',
} }

View file

@ -6,10 +6,19 @@ import {
IVideoConversionProcessor, IVideoConversionProcessor,
IReverseGeocodingProcessor, IReverseGeocodingProcessor,
IUserDeletionJob, IUserDeletionJob,
IVideoLengthExtractionProcessor,
JpegGeneratorProcessor, JpegGeneratorProcessor,
WebpGeneratorProcessor, WebpGeneratorProcessor,
} from './interfaces'; } from './interfaces';
import { JobName } from './job.constants'; import { JobName, QueueName } from './job.constants';
export interface JobCounts {
active: number;
completed: number;
failed: number;
delayed: number;
waiting: number;
}
export type JobItem = export type JobItem =
| { name: JobName.ASSET_UPLOADED; data: IAssetUploadedJob } | { name: JobName.ASSET_UPLOADED; data: IAssetUploadedJob }
@ -21,6 +30,8 @@ export type JobItem =
| { name: JobName.USER_DELETION; data: IUserDeletionJob } | { name: JobName.USER_DELETION; data: IUserDeletionJob }
| { name: JobName.TEMPLATE_MIGRATION } | { name: JobName.TEMPLATE_MIGRATION }
| { name: JobName.CONFIG_CHANGE } | { name: JobName.CONFIG_CHANGE }
| { name: JobName.CHECKSUM_GENERATION }
| { name: JobName.EXTRACT_VIDEO_METADATA; data: IVideoLengthExtractionProcessor }
| { name: JobName.OBJECT_DETECTION; data: IMachineLearningJob } | { name: JobName.OBJECT_DETECTION; data: IMachineLearningJob }
| { name: JobName.IMAGE_TAGGING; data: IMachineLearningJob } | { name: JobName.IMAGE_TAGGING; data: IMachineLearningJob }
| { name: JobName.DELETE_FILE_ON_DISK; data: IDeleteFileOnDiskJob }; | { name: JobName.DELETE_FILE_ON_DISK; data: IDeleteFileOnDiskJob };
@ -28,5 +39,8 @@ export type JobItem =
export const IJobRepository = 'IJobRepository'; export const IJobRepository = 'IJobRepository';
export interface IJobRepository { export interface IJobRepository {
empty(name: QueueName): Promise<void>;
add(item: JobItem): Promise<void>; add(item: JobItem): Promise<void>;
isActive(name: QueueName): Promise<boolean>;
getJobCounts(name: QueueName): Promise<JobCounts>;
} }

View file

@ -2,6 +2,9 @@ import { IJobRepository } from '../src';
export const newJobRepositoryMock = (): jest.Mocked<IJobRepository> => { export const newJobRepositoryMock = (): jest.Mocked<IJobRepository> => {
return { return {
empty: jest.fn(),
add: jest.fn().mockImplementation(() => Promise.resolve()), add: jest.fn().mockImplementation(() => Promise.resolve()),
isActive: jest.fn(),
getJobCounts: jest.fn(),
}; };
}; };

View file

@ -1,21 +1,110 @@
import { IJobRepository, JobItem, JobName, QueueName } from '@app/domain'; import {
IAssetUploadedJob,
IJobRepository,
IMachineLearningJob,
IMetadataExtractionJob,
IUserDeletionJob,
IVideoTranscodeJob,
JobCounts,
JobItem,
JobName,
QueueName,
} from '@app/domain';
import { InjectQueue } from '@nestjs/bull'; import { InjectQueue } from '@nestjs/bull';
import { Logger } from '@nestjs/common'; import { BadRequestException, Logger } from '@nestjs/common';
import { Queue } from 'bull'; import { Queue } from 'bull';
export class JobRepository implements IJobRepository { export class JobRepository implements IJobRepository {
private logger = new Logger(JobRepository.name); private logger = new Logger(JobRepository.name);
constructor(@InjectQueue(QueueName.CONFIG) private configQueue: Queue) {} constructor(
@InjectQueue(QueueName.ASSET_UPLOADED) private assetUploaded: Queue<IAssetUploadedJob>,
@InjectQueue(QueueName.BACKGROUND_TASK) private backgroundTask: Queue,
@InjectQueue(QueueName.CHECKSUM_GENERATION) private generateChecksum: Queue,
@InjectQueue(QueueName.MACHINE_LEARNING) private machineLearning: Queue<IMachineLearningJob>,
@InjectQueue(QueueName.METADATA_EXTRACTION) private metadataExtraction: Queue<IMetadataExtractionJob>,
@InjectQueue(QueueName.CONFIG) private storageMigration: Queue,
@InjectQueue(QueueName.THUMBNAIL_GENERATION) private thumbnail: Queue,
@InjectQueue(QueueName.USER_DELETION) private userDeletion: Queue<IUserDeletionJob>,
@InjectQueue(QueueName.VIDEO_CONVERSION) private videoTranscode: Queue<IVideoTranscodeJob>,
) {}
async isActive(name: QueueName): Promise<boolean> {
const counts = await this.getJobCounts(name);
return !!counts.active;
}
empty(name: QueueName) {
return this.getQueue(name).empty();
}
getJobCounts(name: QueueName): Promise<JobCounts> {
return this.getQueue(name).getJobCounts();
}
async add(item: JobItem): Promise<void> { async add(item: JobItem): Promise<void> {
switch (item.name) { switch (item.name) {
case JobName.CONFIG_CHANGE: case JobName.ASSET_UPLOADED:
await this.configQueue.add(JobName.CONFIG_CHANGE, {}); await this.assetUploaded.add(item.name, item.data, { jobId: item.data.asset.id });
break; break;
case JobName.DELETE_FILE_ON_DISK:
await this.backgroundTask.add(item.name, item.data);
break;
case JobName.CHECKSUM_GENERATION:
await this.generateChecksum.add(item.name, {});
break;
case JobName.OBJECT_DETECTION:
case JobName.IMAGE_TAGGING:
await this.machineLearning.add(item.name, item.data);
break;
case JobName.EXIF_EXTRACTION:
case JobName.EXTRACT_VIDEO_METADATA:
case JobName.REVERSE_GEOCODING:
await this.metadataExtraction.add(item.name, item.data);
break;
case JobName.TEMPLATE_MIGRATION:
case JobName.CONFIG_CHANGE:
await this.storageMigration.add(item.name, {});
break;
case JobName.GENERATE_JPEG_THUMBNAIL:
case JobName.GENERATE_WEBP_THUMBNAIL:
await this.thumbnail.add(item.name, item.data);
break;
case JobName.USER_DELETION:
await this.userDeletion.add(item.name, item.data);
break;
case JobName.VIDEO_CONVERSION:
await this.videoTranscode.add(item.name, item.data);
break;
default: default:
// TODO inject remaining queues and map job to queue // TODO inject remaining queues and map job to queue
this.logger.error('Invalid job', item); this.logger.error('Invalid job', item);
} }
} }
private getQueue(name: QueueName) {
switch (name) {
case QueueName.THUMBNAIL_GENERATION:
return this.thumbnail;
case QueueName.METADATA_EXTRACTION:
return this.metadataExtraction;
case QueueName.VIDEO_CONVERSION:
return this.videoTranscode;
case QueueName.CONFIG:
return this.storageMigration;
case QueueName.MACHINE_LEARNING:
return this.machineLearning;
default:
throw new BadRequestException('Invalid job name');
}
}
} }

View file

@ -13,24 +13,13 @@
*/ */
import {Configuration} from './configuration'; import { Configuration } from './configuration';
import globalAxios, {AxiosInstance, AxiosPromise, AxiosRequestConfig} from 'axios'; import globalAxios, { AxiosPromise, AxiosInstance, AxiosRequestConfig } from 'axios';
// Some imports not used depending on template conditions // Some imports not used depending on template conditions
// @ts-ignore // @ts-ignore
import { import { DUMMY_BASE_URL, assertParamExists, setApiKeyToObject, setBasicAuthToObject, setBearerAuthToObject, setOAuthToObject, setSearchParams, serializeDataIfNeeded, toPathString, createRequestFunction } from './common';
assertParamExists,
createRequestFunction,
DUMMY_BASE_URL,
serializeDataIfNeeded,
setApiKeyToObject,
setBasicAuthToObject,
setBearerAuthToObject,
setOAuthToObject,
setSearchParams,
toPathString
} from './common';
// @ts-ignore // @ts-ignore
import {BASE_PATH, BaseAPI, COLLECTION_FORMATS, RequestArgs, RequiredError} from './base'; import { BASE_PATH, COLLECTION_FORMATS, RequestArgs, BaseAPI, RequiredError } from './base';
/** /**
* *
@ -293,61 +282,31 @@ export interface AllJobStatusResponseDto {
* @type {JobCounts} * @type {JobCounts}
* @memberof AllJobStatusResponseDto * @memberof AllJobStatusResponseDto
*/ */
'thumbnailGenerationQueueCount': JobCounts; 'thumbnail-generation': JobCounts;
/** /**
* *
* @type {JobCounts} * @type {JobCounts}
* @memberof AllJobStatusResponseDto * @memberof AllJobStatusResponseDto
*/ */
'metadataExtractionQueueCount': JobCounts; 'metadata-extraction': JobCounts;
/** /**
* *
* @type {JobCounts} * @type {JobCounts}
* @memberof AllJobStatusResponseDto * @memberof AllJobStatusResponseDto
*/ */
'videoConversionQueueCount': JobCounts; 'video-conversion': JobCounts;
/** /**
* *
* @type {JobCounts} * @type {JobCounts}
* @memberof AllJobStatusResponseDto * @memberof AllJobStatusResponseDto
*/ */
'machineLearningQueueCount': JobCounts; 'machine-learning': JobCounts;
/** /**
* *
* @type {JobCounts} * @type {JobCounts}
* @memberof AllJobStatusResponseDto * @memberof AllJobStatusResponseDto
*/ */
'storageMigrationQueueCount': JobCounts; 'storage-template-migration': JobCounts;
/**
*
* @type {boolean}
* @memberof AllJobStatusResponseDto
*/
'isThumbnailGenerationActive': boolean;
/**
*
* @type {boolean}
* @memberof AllJobStatusResponseDto
*/
'isMetadataExtractionActive': boolean;
/**
*
* @type {boolean}
* @memberof AllJobStatusResponseDto
*/
'isVideoConversionActive': boolean;
/**
*
* @type {boolean}
* @memberof AllJobStatusResponseDto
*/
'isMachineLearningActive': boolean;
/**
*
* @type {boolean}
* @memberof AllJobStatusResponseDto
*/
'isStorageMigrationActive': boolean;
} }
/** /**
* *
@ -1269,25 +1228,6 @@ export const JobId = {
export type JobId = typeof JobId[keyof typeof JobId]; export type JobId = typeof JobId[keyof typeof JobId];
/**
*
* @export
* @interface JobStatusResponseDto
*/
export interface JobStatusResponseDto {
/**
*
* @type {boolean}
* @memberof JobStatusResponseDto
*/
'isActive': boolean;
/**
*
* @type {object}
* @memberof JobStatusResponseDto
*/
'queueCount': object;
}
/** /**
* *
* @export * @export
@ -5772,43 +5712,6 @@ export const JobApiAxiosParamCreator = function (configuration?: Configuration)
setSearchParams(localVarUrlObj, localVarQueryParameter);
let headersFromBaseOptions = baseOptions && baseOptions.headers ? baseOptions.headers : {};
localVarRequestOptions.headers = {...localVarHeaderParameter, ...headersFromBaseOptions, ...options.headers};
return {
url: toPathString(localVarUrlObj),
options: localVarRequestOptions,
};
},
/**
*
* @param {JobId} jobId
* @param {*} [options] Override http request option.
* @throws {RequiredError}
*/
getJobStatus: async (jobId: JobId, options: AxiosRequestConfig = {}): Promise<RequestArgs> => {
// verify required parameter 'jobId' is not null or undefined
assertParamExists('getJobStatus', 'jobId', jobId)
const localVarPath = `/jobs/{jobId}`
.replace(`{${"jobId"}}`, encodeURIComponent(String(jobId)));
// use dummy base URL string because the URL constructor only accepts absolute URLs.
const localVarUrlObj = new URL(localVarPath, DUMMY_BASE_URL);
let baseOptions;
if (configuration) {
baseOptions = configuration.baseOptions;
}
const localVarRequestOptions = { method: 'GET', ...baseOptions, ...options};
const localVarHeaderParameter = {} as any;
const localVarQueryParameter = {} as any;
// authentication bearer required
// http bearer authentication required
await setBearerAuthToObject(localVarHeaderParameter, configuration)
setSearchParams(localVarUrlObj, localVarQueryParameter); setSearchParams(localVarUrlObj, localVarQueryParameter);
let headersFromBaseOptions = baseOptions && baseOptions.headers ? baseOptions.headers : {}; let headersFromBaseOptions = baseOptions && baseOptions.headers ? baseOptions.headers : {};
localVarRequestOptions.headers = {...localVarHeaderParameter, ...headersFromBaseOptions, ...options.headers}; localVarRequestOptions.headers = {...localVarHeaderParameter, ...headersFromBaseOptions, ...options.headers};
@ -5880,16 +5783,6 @@ export const JobApiFp = function(configuration?: Configuration) {
const localVarAxiosArgs = await localVarAxiosParamCreator.getAllJobsStatus(options); const localVarAxiosArgs = await localVarAxiosParamCreator.getAllJobsStatus(options);
return createRequestFunction(localVarAxiosArgs, globalAxios, BASE_PATH, configuration); return createRequestFunction(localVarAxiosArgs, globalAxios, BASE_PATH, configuration);
}, },
/**
*
* @param {JobId} jobId
* @param {*} [options] Override http request option.
* @throws {RequiredError}
*/
async getJobStatus(jobId: JobId, options?: AxiosRequestConfig): Promise<(axios?: AxiosInstance, basePath?: string) => AxiosPromise<JobStatusResponseDto>> {
const localVarAxiosArgs = await localVarAxiosParamCreator.getJobStatus(jobId, options);
return createRequestFunction(localVarAxiosArgs, globalAxios, BASE_PATH, configuration);
},
/** /**
* *
* @param {JobId} jobId * @param {JobId} jobId
@ -5919,15 +5812,6 @@ export const JobApiFactory = function (configuration?: Configuration, basePath?:
getAllJobsStatus(options?: any): AxiosPromise<AllJobStatusResponseDto> { getAllJobsStatus(options?: any): AxiosPromise<AllJobStatusResponseDto> {
return localVarFp.getAllJobsStatus(options).then((request) => request(axios, basePath)); return localVarFp.getAllJobsStatus(options).then((request) => request(axios, basePath));
}, },
/**
*
* @param {JobId} jobId
* @param {*} [options] Override http request option.
* @throws {RequiredError}
*/
getJobStatus(jobId: JobId, options?: any): AxiosPromise<JobStatusResponseDto> {
return localVarFp.getJobStatus(jobId, options).then((request) => request(axios, basePath));
},
/** /**
* *
* @param {JobId} jobId * @param {JobId} jobId
@ -5958,17 +5842,6 @@ export class JobApi extends BaseAPI {
return JobApiFp(this.configuration).getAllJobsStatus(options).then((request) => request(this.axios, this.basePath)); return JobApiFp(this.configuration).getAllJobsStatus(options).then((request) => request(this.axios, this.basePath));
} }
/**
*
* @param {JobId} jobId
* @param {*} [options] Override http request option.
* @throws {RequiredError}
* @memberof JobApi
*/
public getJobStatus(jobId: JobId, options?: AxiosRequestConfig) {
return JobApiFp(this.configuration).getJobStatus(jobId, options).then((request) => request(this.axios, this.basePath));
}
/** /**
* *
* @param {JobId} jobId * @param {JobId} jobId

View file

@ -1,13 +1,12 @@
<script lang="ts"> <script lang="ts">
import LoadingSpinner from '$lib/components/shared-components/loading-spinner.svelte'; import LoadingSpinner from '$lib/components/shared-components/loading-spinner.svelte';
import { createEventDispatcher } from 'svelte'; import { createEventDispatcher } from 'svelte';
import { JobCounts } from '@api';
export let title: string; export let title: string;
export let subtitle: string; export let subtitle: string;
export let buttonTitle = 'Run'; export let buttonTitle = 'Run';
export let jobStatus: boolean; export let jobCounts: JobCounts;
export let waitingJobCount: number;
export let activeJobCount: number;
const dispatch = createEventDispatcher(); const dispatch = createEventDispatcher();
</script> </script>
@ -36,17 +35,23 @@
class="overflow-y-auto rounded-md w-full max-h-[320px] block border bg-white dark:border-immich-dark-gray dark:bg-immich-dark-gray/75 dark:text-immich-dark-fg" class="overflow-y-auto rounded-md w-full max-h-[320px] block border bg-white dark:border-immich-dark-gray dark:bg-immich-dark-gray/75 dark:text-immich-dark-fg"
> >
<tr class="text-center flex place-items-center w-full h-[60px]"> <tr class="text-center flex place-items-center w-full h-[60px]">
<td class="text-sm px-2 w-1/3 text-ellipsis">{jobStatus ? 'Active' : 'Idle'}</td> <td class="text-sm px-2 w-1/3 text-ellipsis">
<td class="flex justify-center text-sm px-2 w-1/3 text-ellipsis"> {#if jobCounts}
{#if activeJobCount !== undefined} <span>{jobCounts.active > 0 || jobCounts.waiting > 0 ? 'Active' : 'Idle'}</span>
{activeJobCount}
{:else} {:else}
<LoadingSpinner /> <LoadingSpinner />
{/if} {/if}
</td> </td>
<td class="flex justify-center text-sm px-2 w-1/3 text-ellipsis"> <td class="flex justify-center text-sm px-2 w-1/3 text-ellipsis">
{#if waitingJobCount !== undefined} {#if jobCounts.active !== undefined}
{waitingJobCount} {jobCounts.active}
{:else}
<LoadingSpinner />
{/if}
</td>
<td class="flex justify-center text-sm px-2 w-1/3 text-ellipsis">
{#if jobCounts.waiting !== undefined}
{jobCounts.waiting}
{:else} {:else}
<LoadingSpinner /> <LoadingSpinner />
{/if} {/if}
@ -59,9 +64,9 @@
<button <button
on:click={() => dispatch('click')} on:click={() => dispatch('click')}
class="px-6 py-3 text-sm bg-immich-primary dark:bg-immich-dark-primary font-medium rounded-2xl hover:bg-immich-primary/50 transition-all hover:cursor-pointer disabled:cursor-not-allowed shadow-sm text-immich-bg dark:text-immich-dark-gray" class="px-6 py-3 text-sm bg-immich-primary dark:bg-immich-dark-primary font-medium rounded-2xl hover:bg-immich-primary/50 transition-all hover:cursor-pointer disabled:cursor-not-allowed shadow-sm text-immich-bg dark:text-immich-dark-gray"
disabled={jobStatus} disabled={jobCounts.active > 0 && jobCounts.waiting > 0}
> >
{#if jobStatus} {#if jobCounts.active > 0 || jobCounts.waiting > 0}
<LoadingSpinner /> <LoadingSpinner />
{:else} {:else}
{buttonTitle} {buttonTitle}

View file

@ -8,203 +8,97 @@
import { onDestroy, onMount } from 'svelte'; import { onDestroy, onMount } from 'svelte';
import JobTile from './job-tile.svelte'; import JobTile from './job-tile.svelte';
let allJobsStatus: AllJobStatusResponseDto; let jobs: AllJobStatusResponseDto;
let setIntervalHandler: NodeJS.Timer; let timer: NodeJS.Timer;
const load = async () => {
const { data } = await api.jobApi.getAllJobsStatus();
jobs = data;
};
onMount(async () => { onMount(async () => {
const { data } = await api.jobApi.getAllJobsStatus(); await load();
allJobsStatus = data; timer = setInterval(async () => await load(), 5_000);
setIntervalHandler = setInterval(async () => {
const { data } = await api.jobApi.getAllJobsStatus();
allJobsStatus = data;
}, 1000);
}); });
onDestroy(() => { onDestroy(() => {
clearInterval(setIntervalHandler); clearInterval(timer);
}); });
const runThumbnailGeneration = async () => { const run = async (jobId: JobId, jobName: string, emptyMessage: string) => {
try { try {
const { data } = await api.jobApi.sendJobCommand(JobId.ThumbnailGeneration, { const { data } = await api.jobApi.sendJobCommand(jobId, { command: JobCommand.Start });
command: JobCommand.Start
});
if (data) { if (data) {
notificationController.show({ notificationController.show({
message: `Thumbnail generation job started for ${data} assets`, message: `Started ${jobName}`,
type: NotificationType.Info type: NotificationType.Info
}); });
} else { } else {
notificationController.show({ notificationController.show({ message: emptyMessage, type: NotificationType.Info });
message: `No missing thumbnails found`,
type: NotificationType.Info
});
}
} catch (e) {
console.log('[ERROR] runThumbnailGeneration', e);
notificationController.show({
message: `Error running thumbnail generation job, check console for more detail`,
type: NotificationType.Error
});
}
};
const runExtractEXIF = async () => {
try {
const { data } = await api.jobApi.sendJobCommand(JobId.MetadataExtraction, {
command: JobCommand.Start
});
if (data) {
notificationController.show({
message: `Extract EXIF job started for ${data} assets`,
type: NotificationType.Info
});
} else {
notificationController.show({
message: `No missing EXIF found`,
type: NotificationType.Info
});
}
} catch (e) {
console.log('[ERROR] runExtractEXIF', e);
notificationController.show({
message: `Error running extract EXIF job, check console for more detail`,
type: NotificationType.Error
});
}
};
const runMachineLearning = async () => {
try {
const { data } = await api.jobApi.sendJobCommand(JobId.MachineLearning, {
command: JobCommand.Start
});
if (data) {
notificationController.show({
message: `Object detection job started for ${data} assets`,
type: NotificationType.Info
});
} else {
notificationController.show({
message: `No missing object detection found`,
type: NotificationType.Info
});
} }
} catch (error) { } catch (error) {
handleError(error, `Error running machine learning job, check console for more detail`); handleError(error, `Unable to start ${jobName}`);
}
};
const runVideoConversion = async () => {
try {
const { data } = await api.jobApi.sendJobCommand(JobId.VideoConversion, {
command: JobCommand.Start
});
if (data) {
notificationController.show({
message: `Video conversion job started for ${data} assets`,
type: NotificationType.Info
});
} else {
notificationController.show({
message: `No videos without an encoded version found`,
type: NotificationType.Info
});
}
} catch (error) {
handleError(error, `Error running video conversion job, check console for more detail`);
}
};
const runTemplateMigration = async () => {
try {
const { data } = await api.jobApi.sendJobCommand(JobId.StorageTemplateMigration, {
command: JobCommand.Start
});
if (data) {
notificationController.show({
message: `Storage migration started`,
type: NotificationType.Info
});
} else {
notificationController.show({
message: `All files have been migrated to the new storage template`,
type: NotificationType.Info
});
}
} catch (e) {
console.log('[ERROR] runTemplateMigration', e);
notificationController.show({
message: `Error running template migration job, check console for more detail`,
type: NotificationType.Error
});
} }
}; };
</script> </script>
<div class="flex flex-col gap-10"> <div class="flex flex-col gap-10">
<JobTile {#if jobs}
title={'Generate thumbnails'} <JobTile
subtitle={'Regenerate missing thumbnail (JPEG, WEBP)'} title={'Generate thumbnails'}
on:click={runThumbnailGeneration} subtitle={'Regenerate missing thumbnail (JPEG, WEBP)'}
jobStatus={allJobsStatus?.isThumbnailGenerationActive} on:click={() =>
waitingJobCount={allJobsStatus?.thumbnailGenerationQueueCount.waiting} run(JobId.ThumbnailGeneration, 'thumbnail generation', 'No missing thumbnails found')}
activeJobCount={allJobsStatus?.thumbnailGenerationQueueCount.active} jobCounts={jobs[JobId.ThumbnailGeneration]}
/> />
<JobTile <JobTile
title={'Extract EXIF'} title={'Extract EXIF'}
subtitle={'Extract missing EXIF information'} subtitle={'Extract missing EXIF information'}
on:click={runExtractEXIF} on:click={() => run(JobId.MetadataExtraction, 'extract EXIF', 'No missing EXIF found')}
jobStatus={allJobsStatus?.isMetadataExtractionActive} jobCounts={jobs[JobId.MetadataExtraction]}
waitingJobCount={allJobsStatus?.metadataExtractionQueueCount.waiting} />
activeJobCount={allJobsStatus?.metadataExtractionQueueCount.active}
/>
<JobTile <JobTile
title={'Detect objects'} title={'Detect objects'}
subtitle={'Run machine learning process to detect and classify objects'} subtitle={'Run machine learning process to detect and classify objects'}
on:click={runMachineLearning} on:click={() =>
jobStatus={allJobsStatus?.isMachineLearningActive} run(JobId.MachineLearning, 'object detection', 'No missing object detection found')}
waitingJobCount={allJobsStatus?.machineLearningQueueCount.waiting} jobCounts={jobs[JobId.MachineLearning]}
activeJobCount={allJobsStatus?.machineLearningQueueCount.active}
>
Note that some assets may not have any objects detected, this is normal.
</JobTile>
<JobTile
title={'Video transcoding'}
subtitle={'Run video transcoding process to transcode videos not in the desired format'}
on:click={runVideoConversion}
jobStatus={allJobsStatus?.isVideoConversionActive}
waitingJobCount={allJobsStatus?.videoConversionQueueCount.waiting}
activeJobCount={allJobsStatus?.videoConversionQueueCount.active}
>
Note that some videos won't require transcoding, this is normal.
</JobTile>
<JobTile
title={'Storage migration'}
subtitle={''}
on:click={runTemplateMigration}
jobStatus={allJobsStatus?.isStorageMigrationActive}
waitingJobCount={allJobsStatus?.storageMigrationQueueCount.waiting}
activeJobCount={allJobsStatus?.storageMigrationQueueCount.active}
>
Apply the current
<a
href="/admin/system-settings?open=storage-template"
class="text-immich-primary dark:text-immich-dark-primary">Storage template</a
> >
to previously uploaded assets Note that some assets may not have any objects detected, this is normal.
</JobTile> </JobTile>
<JobTile
title={'Video transcoding'}
subtitle={'Run video transcoding process to transcode videos not in the desired format'}
on:click={() =>
run(
JobId.VideoConversion,
'video conversion',
'No videos without an encoded version found'
)}
jobCounts={jobs[JobId.MachineLearning]}
/>
<JobTile
title={'Storage migration'}
subtitle={''}
on:click={() =>
run(
JobId.StorageTemplateMigration,
'storage template migration',
'All files have been migrated to the new storage template'
)}
jobCounts={jobs[JobId.StorageTemplateMigration]}
>
Apply the current
<a
href="/admin/system-settings?open=storage-template"
class="text-immich-primary dark:text-immich-dark-primary">Storage template</a
>
to previously uploaded assets
</JobTile>
{/if}
</div> </div>