mirror of
https://github.com/kikootwo/ReadMeABook.git
synced 2026-06-02 20:30:10 +00:00
Add find_missing_ebooks scheduled job
Introduce a safety-net scheduled job that scans completed audiobooks and auto-triggers ebook fetches for missing companions. Changes include: - New Prisma migration + schema field: requests.ebook_auto_retry_count (nullable) to track lifetime auto-retries. - New processor: src/lib/processors/find-missing-ebooks.processor.ts implementing the scan (limit 50), gating by ebook_auto_grab_enabled and source flags, creating ebook child requests or retrying failed ones up to a cap of 5, using transactions for race-safety and rolling back the counter if enqueue fails. - Job queue integration: add job type, payload, processor registration, and addFindMissingEbooksJob helper. - Scheduler integration: register the scheduled job (daily midnight) and trigger path. - Documentation updates: backend scheduler and ebook-sidecar docs describing behavior and limits. - Tests: add comprehensive unit tests for the processor and update scheduler tests and job-queue test helper. This implements automated recovery for missing ebook companions while keeping the retry counter processor-private and ensuring safe concurrency handling.
This commit is contained in:
@@ -20,8 +20,9 @@ Manages recurring/scheduled jobs providing automated tasks (Plex scans, Audible
|
||||
3. **audible_refresh** - Default: daily midnight, fetches 200 popular + 200 new releases, stores with rankings, disabled by default
|
||||
4. **retry_missing_torrents** - Default: daily midnight, processes union of `awaiting_search` ∪ `awaiting_release` (limit 50), handles both audiobook and ebook requests. Bidirectional transitions: `awaiting_search` → `awaiting_release` when release date is future + `indexer.skip_unreleased` ON; `awaiting_release` → `awaiting_search` + run search when release date has passed or setting OFF. Sole owner of these transitions. Enabled by default.
|
||||
5. **retry_failed_imports** - Default: every 6 hours, re-attempts 'awaiting_import' status (limit 50), enabled by default
|
||||
6. **cleanup_seeded_torrents** - Default: every 30 mins, deletes torrents after seeding requirements met, respects `seeding_time_minutes` config (0 = never), enabled by default
|
||||
7. **monitor_rss_feeds** - Default: every 15 mins, checks RSS feeds from enabled indexers, matches against `awaiting_search` requests (audiobook and ebook, limit 100). Query is unchanged — release-date gate is applied AFTER a match is found: if matched book is unreleased + `indexer.skip_unreleased` ON, the match is skipped and request status is NOT mutated (retry job owns transitions). Enabled by default.
|
||||
6. **find_missing_ebooks** - Default: daily midnight, scans `downloaded` ∪ `available` audiobook requests (limit 50) for missing ebook companions and triggers the existing ebook fetch flow (`addSearchEbookJob`). Gated by `ebook_auto_grab_enabled` AND at least one ebook source enabled (`ebook_annas_archive_enabled` or `ebook_indexer_search_enabled`; legacy `ebook_sidecar_enabled` accepted as Anna's fallback). Skips ebook children in-flight (`pending`, `awaiting_approval`, `searching`, `downloading`, `processing`, `awaiting_search`, `awaiting_release`) or `cancelled`. Retries `failed`/`warn` children up to **5 lifetime auto-retries** per audiobook, tracked in `Request.ebookAutoRetryCount` (nullable; processor-private — manual "Fetch Ebook" never reads/writes it). Per-candidate writes are wrapped in `prisma.$transaction` for race-safety with concurrent auto-grab; counter rolls back if `addSearchEbookJob` throws. Enabled by default. Returns `{ scanned, gapsFound, triggered, created, retried, skippedInFlight, skippedCancelled, skippedCapHit }`.
|
||||
7. **cleanup_seeded_torrents** - Default: every 30 mins, deletes torrents after seeding requirements met, respects `seeding_time_minutes` config (0 = never), enabled by default
|
||||
8. **monitor_rss_feeds** - Default: every 15 mins, checks RSS feeds from enabled indexers, matches against `awaiting_search` requests (audiobook and ebook, limit 100). Query is unchanged — release-date gate is applied AFTER a match is found: if matched book is unreleased + `indexer.skip_unreleased` ON, the match is skipped and request status is NOT mutated (retry job owns transitions). Enabled by default.
|
||||
|
||||
## Architecture: Bull + Cron
|
||||
|
||||
|
||||
@@ -72,6 +72,10 @@ Ebooks are first-class citizens in RMAB, with their own request type, tracking,
|
||||
- *Auto-grab is automatically disabled if no ebook sources are enabled. Manual fetch via admin buttons still works.*
|
||||
- *Kindle fix toggle only visible when preferred format is EPUB.*
|
||||
|
||||
### Safety-Net: Find Missing Ebooks Job
|
||||
|
||||
A scheduled `find_missing_ebooks` job (daily midnight, enabled by default) backstops the auto-grab path for cases where it silently misses books (race conditions, transient indexer failures, requests created before sources were configured, books from Goodreads/Hardcover sync). Per run it scans up to 50 audiobook requests in `downloaded`/`available` status and triggers the existing ebook fetch flow for any audiobook missing a successful ebook companion. **Lifetime auto-retry cap: 5 per audiobook** — after 5 failed auto-attempts the job stops retrying that audiobook (admin Manual "Fetch Ebook" remains available). Counter is tracked in `Request.ebookAutoRetryCount` and is **processor-private**: manual Fetch Ebook routes never read, write, or reset it. Gated by `ebook_auto_grab_enabled` AND at least one source enabled; logs no-op runs honestly. See `documentation/backend/services/scheduler.md` for full details.
|
||||
|
||||
### Kindle EPUB Fix
|
||||
|
||||
**Purpose:** Apply compatibility fixes to EPUB files before organizing, ensuring successful Kindle import.
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
-- Add lifetime auto-retry counter for the find_missing_ebooks scheduled job.
|
||||
-- Nullable: NULL distinguishes "never touched by this job" from 0.
|
||||
-- Only the find-missing-ebooks processor reads/writes/increments this column.
|
||||
-- Manual Fetch Ebook routes do not touch it (counter is sacred per engineering brief).
|
||||
ALTER TABLE "requests" ADD COLUMN "ebook_auto_retry_count" INTEGER;
|
||||
@@ -234,6 +234,7 @@ model Request {
|
||||
downloadAttempts Int @default(0) @map("download_attempts")
|
||||
importAttempts Int @default(0) @map("import_attempts")
|
||||
maxImportRetries Int @default(5) @map("max_import_retries")
|
||||
ebookAutoRetryCount Int? @map("ebook_auto_retry_count")
|
||||
lastSearchAt DateTime? @map("last_search_at")
|
||||
customSearchTerms String? @map("custom_search_terms") @db.Text
|
||||
lastImportAt DateTime? @map("last_import_at")
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
/**
|
||||
* Component: Find Missing Ebooks Processor
|
||||
* Documentation: documentation/backend/services/scheduler.md
|
||||
*
|
||||
* Safety-net scheduled job for issue #191. Scans completed audiobook requests
|
||||
* (downloaded | available) and triggers the existing ebook fetch flow for any
|
||||
* audiobook whose ebook companion is missing, failed, or warned out.
|
||||
*
|
||||
* Gated by ebook_auto_grab_enabled AND at least one ebook source enabled.
|
||||
* Per-run scan cap = 50. Per-audiobook lifetime auto-retry cap = 5
|
||||
* (tracked in Request.ebookAutoRetryCount; counter is processor-private —
|
||||
* manual Fetch Ebook never touches it).
|
||||
*/
|
||||
|
||||
import { prisma } from '../db';
|
||||
import { RMABLogger } from '../utils/logger';
|
||||
import { getJobQueueService } from '../services/job-queue.service';
|
||||
import { getConfigService } from '../services/config.service';
|
||||
|
||||
export interface FindMissingEbooksPayload {
|
||||
jobId?: string;
|
||||
scheduledJobId?: string;
|
||||
}
|
||||
|
||||
interface CandidateRow {
|
||||
parent_request_id: string;
|
||||
user_id: string;
|
||||
audiobook_id: string;
|
||||
custom_search_terms: string | null;
|
||||
audiobook_title: string;
|
||||
audiobook_author: string;
|
||||
audible_asin: string | null;
|
||||
ebook_request_id: string | null;
|
||||
ebook_status: string | null;
|
||||
ebook_auto_retry_count: number | null;
|
||||
}
|
||||
|
||||
// Statuses indicating an in-flight ebook request that must not be duplicated
|
||||
// or re-triggered. `awaiting_release` is included per engineering brief's
|
||||
// "include awaiting_release in the in-flight skip set" directive.
|
||||
const IN_FLIGHT_STATUSES = new Set([
|
||||
'pending',
|
||||
'awaiting_approval',
|
||||
'searching',
|
||||
'downloading',
|
||||
'processing',
|
||||
'awaiting_search',
|
||||
'awaiting_release',
|
||||
]);
|
||||
|
||||
const AUTO_RETRY_CAP = 5;
|
||||
const PER_RUN_LIMIT = 50;
|
||||
|
||||
export async function processFindMissingEbooks(payload: FindMissingEbooksPayload): Promise<any> {
|
||||
const { jobId } = payload;
|
||||
const logger = RMABLogger.forJob(jobId, 'FindMissingEbooks');
|
||||
|
||||
logger.info('Starting find_missing_ebooks pass');
|
||||
|
||||
const zeroResult = (message: string, action: 'skipped-auto-grab-off' | 'skipped-no-source') => {
|
||||
logger.info(message, { action });
|
||||
return {
|
||||
success: true,
|
||||
message,
|
||||
scanned: 0,
|
||||
gapsFound: 0,
|
||||
triggered: 0,
|
||||
created: 0,
|
||||
retried: 0,
|
||||
skippedInFlight: 0,
|
||||
skippedCancelled: 0,
|
||||
skippedCapHit: 0,
|
||||
};
|
||||
};
|
||||
|
||||
try {
|
||||
const configService = getConfigService();
|
||||
|
||||
// Gate #1 — auto-grab feature toggle
|
||||
// Default ON when key is absent/null (matches organize-files.processor.ts).
|
||||
const autoGrab = await configService.get('ebook_auto_grab_enabled');
|
||||
if (autoGrab === 'false') {
|
||||
return zeroResult('Auto-grab disabled, skipping', 'skipped-auto-grab-off');
|
||||
}
|
||||
|
||||
// Gate #2 — at least one ebook source enabled
|
||||
// Includes legacy back-compat shim: ebook_sidecar_enabled === 'true' counts
|
||||
// as Anna's Archive ON if the new key is absent (mirrors manual fetch route).
|
||||
const [annasArchive, indexerSearch, legacy] = await Promise.all([
|
||||
configService.get('ebook_annas_archive_enabled'),
|
||||
configService.get('ebook_indexer_search_enabled'),
|
||||
configService.get('ebook_sidecar_enabled'),
|
||||
]);
|
||||
const annasOn = annasArchive === 'true' || (annasArchive == null && legacy === 'true');
|
||||
const indexerOn = indexerSearch === 'true';
|
||||
if (!annasOn && !indexerOn) {
|
||||
return zeroResult('No ebook sources enabled, skipping', 'skipped-no-source');
|
||||
}
|
||||
|
||||
// Anti-join: most-recent non-deleted ebook child per in-scope audiobook.
|
||||
// Broad form — branch fully in JS so per-skip counters and log lines are
|
||||
// observable. LIMIT is the per-run scan cap.
|
||||
const candidates = await prisma.$queryRaw<CandidateRow[]>`
|
||||
SELECT
|
||||
p.id AS parent_request_id,
|
||||
p.user_id AS user_id,
|
||||
p.audiobook_id AS audiobook_id,
|
||||
p.custom_search_terms AS custom_search_terms,
|
||||
a.title AS audiobook_title,
|
||||
a.author AS audiobook_author,
|
||||
a.audible_asin AS audible_asin,
|
||||
e.id AS ebook_request_id,
|
||||
e.status AS ebook_status,
|
||||
e.ebook_auto_retry_count AS ebook_auto_retry_count
|
||||
FROM requests p
|
||||
JOIN audiobooks a ON a.id = p.audiobook_id
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT id, status, ebook_auto_retry_count
|
||||
FROM requests
|
||||
WHERE parent_request_id = p.id
|
||||
AND type = 'ebook'
|
||||
AND deleted_at IS NULL
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 1
|
||||
) e ON TRUE
|
||||
WHERE p.status IN ('downloaded', 'available')
|
||||
AND (p.type IS NULL OR p.type <> 'ebook')
|
||||
AND p.deleted_at IS NULL
|
||||
ORDER BY p.created_at DESC
|
||||
LIMIT ${PER_RUN_LIMIT}
|
||||
`;
|
||||
|
||||
const scanned = candidates.length;
|
||||
logger.info(`Scanned ${scanned} in-scope audiobook request(s)`);
|
||||
|
||||
if (scanned === 0) {
|
||||
return {
|
||||
success: true,
|
||||
message: 'No in-scope audiobook requests',
|
||||
scanned: 0,
|
||||
gapsFound: 0,
|
||||
triggered: 0,
|
||||
created: 0,
|
||||
retried: 0,
|
||||
skippedInFlight: 0,
|
||||
skippedCancelled: 0,
|
||||
skippedCapHit: 0,
|
||||
};
|
||||
}
|
||||
|
||||
const jobQueue = getJobQueueService();
|
||||
let gapsFound = 0;
|
||||
let created = 0;
|
||||
let retried = 0;
|
||||
let skippedInFlight = 0;
|
||||
let skippedCancelled = 0;
|
||||
let skippedCapHit = 0;
|
||||
|
||||
for (const row of candidates) {
|
||||
let action:
|
||||
| 'created'
|
||||
| 'retried'
|
||||
| 'skipped-has-companion'
|
||||
| 'skipped-in-flight'
|
||||
| 'skipped-cancelled'
|
||||
| 'skipped-cap'
|
||||
| 'skipped-unknown'
|
||||
| null = null;
|
||||
let ebookRequestId: string | null = row.ebook_request_id;
|
||||
|
||||
try {
|
||||
await prisma.$transaction(async (tx) => {
|
||||
if (!row.ebook_request_id) {
|
||||
// No live ebook child — create one and seed counter at 1.
|
||||
const createdRow = await tx.request.create({
|
||||
data: {
|
||||
userId: row.user_id,
|
||||
audiobookId: row.audiobook_id,
|
||||
type: 'ebook',
|
||||
parentRequestId: row.parent_request_id,
|
||||
status: 'pending',
|
||||
progress: 0,
|
||||
customSearchTerms: row.custom_search_terms,
|
||||
ebookAutoRetryCount: 1,
|
||||
},
|
||||
});
|
||||
ebookRequestId = createdRow.id;
|
||||
action = 'created';
|
||||
return;
|
||||
}
|
||||
|
||||
const status = row.ebook_status;
|
||||
if (status === 'downloaded') {
|
||||
action = 'skipped-has-companion';
|
||||
return;
|
||||
}
|
||||
if (status && IN_FLIGHT_STATUSES.has(status)) {
|
||||
action = 'skipped-in-flight';
|
||||
return;
|
||||
}
|
||||
if (status === 'cancelled') {
|
||||
action = 'skipped-cancelled';
|
||||
return;
|
||||
}
|
||||
if (status === 'failed' || status === 'warn') {
|
||||
const current = row.ebook_auto_retry_count ?? 0;
|
||||
if (current >= AUTO_RETRY_CAP) {
|
||||
action = 'skipped-cap';
|
||||
return;
|
||||
}
|
||||
await tx.request.update({
|
||||
where: { id: row.ebook_request_id! },
|
||||
data: {
|
||||
status: 'pending',
|
||||
progress: 0,
|
||||
errorMessage: null,
|
||||
ebookAutoRetryCount: current + 1,
|
||||
},
|
||||
});
|
||||
action = 'retried';
|
||||
return;
|
||||
}
|
||||
// Defensive — unrecognized status (e.g. denied, awaiting_import on an
|
||||
// ebook child that crossed wires). Leave it alone; surface via log.
|
||||
action = 'skipped-unknown';
|
||||
});
|
||||
|
||||
if (action === 'created' || action === 'retried') {
|
||||
gapsFound++;
|
||||
try {
|
||||
await jobQueue.addSearchEbookJob(ebookRequestId!, {
|
||||
id: row.audiobook_id,
|
||||
title: row.audiobook_title,
|
||||
author: row.audiobook_author,
|
||||
asin: row.audible_asin || undefined,
|
||||
});
|
||||
if (action === 'created') created++;
|
||||
else retried++;
|
||||
} catch (enqueueErr) {
|
||||
// Roll counter back on enqueue failure so the cap reflects only
|
||||
// successful auto-retries. Per engineering brief: "increment only
|
||||
// when queue add succeeds." Failure to decrement is logged but
|
||||
// swallowed — primary error is the one that matters.
|
||||
await prisma.request.update({
|
||||
where: { id: ebookRequestId! },
|
||||
data: { ebookAutoRetryCount: { decrement: 1 } },
|
||||
}).catch((rollbackErr) => {
|
||||
logger.error(`Failed to roll back counter for ebook ${ebookRequestId}: ${rollbackErr instanceof Error ? rollbackErr.message : 'Unknown error'}`);
|
||||
});
|
||||
throw enqueueErr;
|
||||
}
|
||||
} else if (action === 'skipped-in-flight') skippedInFlight++;
|
||||
else if (action === 'skipped-cancelled') skippedCancelled++;
|
||||
else if (action === 'skipped-cap') skippedCapHit++;
|
||||
|
||||
logger.info('find_missing_ebooks iteration', {
|
||||
audiobookId: row.audiobook_id,
|
||||
parentRequestId: row.parent_request_id,
|
||||
ebookRequestId,
|
||||
action,
|
||||
});
|
||||
} catch (err) {
|
||||
logger.error(`Failed candidate ${row.parent_request_id}: ${err instanceof Error ? err.message : 'Unknown error'}`);
|
||||
}
|
||||
|
||||
// Spread DB operations over time to avoid connection pool exhaustion.
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
}
|
||||
|
||||
const triggered = created + retried;
|
||||
logger.info('find_missing_ebooks pass complete', {
|
||||
scanned,
|
||||
gapsFound,
|
||||
triggered,
|
||||
created,
|
||||
retried,
|
||||
skippedInFlight,
|
||||
skippedCancelled,
|
||||
skippedCapHit,
|
||||
});
|
||||
|
||||
return {
|
||||
success: true,
|
||||
message: 'find_missing_ebooks completed',
|
||||
scanned,
|
||||
gapsFound,
|
||||
triggered,
|
||||
created,
|
||||
retried,
|
||||
skippedInFlight,
|
||||
skippedCancelled,
|
||||
skippedCapHit,
|
||||
};
|
||||
} catch (error) {
|
||||
logger.error(`Error: ${error instanceof Error ? error.message : 'Unknown error'}`);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,7 @@ export type JobType =
|
||||
| 'audible_refresh'
|
||||
| 'retry_missing_torrents'
|
||||
| 'retry_failed_imports'
|
||||
| 'find_missing_ebooks'
|
||||
| 'cleanup_seeded_torrents'
|
||||
| 'monitor_rss_feeds'
|
||||
| 'sync_reading_shelves'
|
||||
@@ -105,6 +106,10 @@ export interface RetryFailedImportsPayload extends JobPayload {
|
||||
scheduledJobId?: string;
|
||||
}
|
||||
|
||||
export interface FindMissingEbooksPayload extends JobPayload {
|
||||
scheduledJobId?: string;
|
||||
}
|
||||
|
||||
export interface CleanupSeededTorrentsPayload extends JobPayload {
|
||||
scheduledJobId?: string;
|
||||
}
|
||||
@@ -386,6 +391,12 @@ export class JobQueueService {
|
||||
return await processRetryFailedImports(payloadWithJobId);
|
||||
});
|
||||
|
||||
this.queue.process('find_missing_ebooks', 1, async (job: BullJob<FindMissingEbooksPayload>) => {
|
||||
const { processFindMissingEbooks } = await import('../processors/find-missing-ebooks.processor');
|
||||
const payloadWithJobId = await this.ensureJobRecord(job, 'find_missing_ebooks');
|
||||
return await processFindMissingEbooks(payloadWithJobId);
|
||||
});
|
||||
|
||||
this.queue.process('cleanup_seeded_torrents', 1, async (job: BullJob<CleanupSeededTorrentsPayload>) => {
|
||||
const { processCleanupSeededTorrents } = await import('../processors/cleanup-seeded-torrents.processor');
|
||||
const payloadWithJobId = await this.ensureJobRecord(job, 'cleanup_seeded_torrents');
|
||||
@@ -756,6 +767,21 @@ export class JobQueueService {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Add find missing ebooks job
|
||||
*/
|
||||
async addFindMissingEbooksJob(scheduledJobId?: string): Promise<string> {
|
||||
return await this.addJob(
|
||||
'find_missing_ebooks',
|
||||
{
|
||||
scheduledJobId,
|
||||
} as FindMissingEbooksPayload,
|
||||
{
|
||||
priority: 7,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Add cleanup seeded torrents job
|
||||
*/
|
||||
|
||||
@@ -10,7 +10,7 @@ import { RMABLogger } from '../utils/logger';
|
||||
|
||||
const logger = RMABLogger.create('Scheduler');
|
||||
|
||||
export type ScheduledJobType = 'plex_library_scan' | 'plex_recently_added_check' | 'audible_refresh' | 'retry_missing_torrents' | 'retry_failed_imports' | 'cleanup_seeded_torrents' | 'monitor_rss_feeds' | 'sync_reading_shelves' | 'check_watched_lists';
|
||||
export type ScheduledJobType = 'plex_library_scan' | 'plex_recently_added_check' | 'audible_refresh' | 'retry_missing_torrents' | 'retry_failed_imports' | 'find_missing_ebooks' | 'cleanup_seeded_torrents' | 'monitor_rss_feeds' | 'sync_reading_shelves' | 'check_watched_lists';
|
||||
|
||||
export interface ScheduledJob {
|
||||
id: string;
|
||||
@@ -115,6 +115,13 @@ export class SchedulerService {
|
||||
enabled: true, // Enable by default
|
||||
payload: {},
|
||||
},
|
||||
{
|
||||
name: 'Find Missing Ebooks',
|
||||
type: 'find_missing_ebooks' as ScheduledJobType,
|
||||
schedule: '0 0 * * *', // Daily at midnight
|
||||
enabled: true, // Enable by default; gated by ebook_auto_grab_enabled + source-enablement at run time
|
||||
payload: {},
|
||||
},
|
||||
{
|
||||
name: 'Cleanup Seeded Torrents',
|
||||
type: 'cleanup_seeded_torrents' as ScheduledJobType,
|
||||
@@ -379,6 +386,9 @@ export class SchedulerService {
|
||||
case 'retry_failed_imports':
|
||||
bullJobId = await this.triggerRetryFailedImports(job);
|
||||
break;
|
||||
case 'find_missing_ebooks':
|
||||
bullJobId = await this.triggerFindMissingEbooks(job);
|
||||
break;
|
||||
case 'cleanup_seeded_torrents':
|
||||
bullJobId = await this.triggerCleanupSeededTorrents(job);
|
||||
break;
|
||||
@@ -645,6 +655,13 @@ export class SchedulerService {
|
||||
return await this.jobQueue.addRetryFailedImportsJob(job.id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger find missing ebooks safety-net pass
|
||||
*/
|
||||
private async triggerFindMissingEbooks(job: any): Promise<string> {
|
||||
return await this.jobQueue.addFindMissingEbooksJob(job.id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger RSS feed monitoring
|
||||
*/
|
||||
|
||||
@@ -18,6 +18,7 @@ export const createJobQueueMock = () => ({
|
||||
addAudibleRefreshJob: vi.fn(),
|
||||
addRetryMissingTorrentsJob: vi.fn(),
|
||||
addRetryFailedImportsJob: vi.fn(),
|
||||
addFindMissingEbooksJob: vi.fn(),
|
||||
addCleanupSeededTorrentsJob: vi.fn(),
|
||||
addNotificationJob: vi.fn().mockResolvedValue(undefined),
|
||||
});
|
||||
|
||||
@@ -0,0 +1,472 @@
|
||||
/**
|
||||
* Component: Find Missing Ebooks Processor Tests
|
||||
* Documentation: documentation/backend/services/scheduler.md
|
||||
*/
|
||||
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { createPrismaMock } from '../helpers/prisma';
|
||||
import { createJobQueueMock } from '../helpers/job-queue';
|
||||
|
||||
const prismaMock = createPrismaMock();
|
||||
const jobQueueMock = createJobQueueMock();
|
||||
const configMock = vi.hoisted(() => ({ get: vi.fn() }));
|
||||
|
||||
vi.mock('@/lib/db', () => ({
|
||||
prisma: prismaMock,
|
||||
}));
|
||||
|
||||
vi.mock('@/lib/services/job-queue.service', () => ({
|
||||
getJobQueueService: () => jobQueueMock,
|
||||
}));
|
||||
|
||||
vi.mock('@/lib/services/config.service', () => ({
|
||||
getConfigService: () => configMock,
|
||||
}));
|
||||
|
||||
type CandidateRow = {
|
||||
parent_request_id: string;
|
||||
user_id: string;
|
||||
audiobook_id: string;
|
||||
custom_search_terms: string | null;
|
||||
audiobook_title: string;
|
||||
audiobook_author: string;
|
||||
audible_asin: string | null;
|
||||
ebook_request_id: string | null;
|
||||
ebook_status: string | null;
|
||||
ebook_auto_retry_count: number | null;
|
||||
};
|
||||
|
||||
const baseRow = (overrides: Partial<CandidateRow> = {}): CandidateRow => ({
|
||||
parent_request_id: 'parent-1',
|
||||
user_id: 'user-1',
|
||||
audiobook_id: 'audio-1',
|
||||
custom_search_terms: null,
|
||||
audiobook_title: 'Test Book',
|
||||
audiobook_author: 'Test Author',
|
||||
audible_asin: 'ASIN0001',
|
||||
ebook_request_id: null,
|
||||
ebook_status: null,
|
||||
ebook_auto_retry_count: null,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
/**
|
||||
* Default: all gates pass (auto-grab default ON when null; Anna's enabled).
|
||||
* Tests that want a different gate state can override before calling.
|
||||
*/
|
||||
const installDefaultGates = () => {
|
||||
configMock.get.mockImplementation(async (key: string) => {
|
||||
switch (key) {
|
||||
case 'ebook_auto_grab_enabled':
|
||||
return null; // null/absent => ON
|
||||
case 'ebook_annas_archive_enabled':
|
||||
return 'true';
|
||||
case 'ebook_indexer_search_enabled':
|
||||
return 'false';
|
||||
case 'ebook_sidecar_enabled':
|
||||
return null;
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
installDefaultGates();
|
||||
// Default: $transaction runs the callback against the prismaMock surface.
|
||||
prismaMock.$transaction.mockImplementation(async (fn: any) => fn(prismaMock));
|
||||
// Default: each create call returns a stable ebook request id.
|
||||
prismaMock.request.create.mockImplementation(async (args: any) => ({
|
||||
id: 'new-ebook-1',
|
||||
...args.data,
|
||||
}));
|
||||
prismaMock.request.update.mockResolvedValue({});
|
||||
});
|
||||
|
||||
describe('processFindMissingEbooks — gating', () => {
|
||||
it('returns zeros when auto-grab is disabled (explicit false)', async () => {
|
||||
configMock.get.mockImplementation(async (key: string) =>
|
||||
key === 'ebook_auto_grab_enabled' ? 'false' : null
|
||||
);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-1' });
|
||||
|
||||
expect(result).toMatchObject({
|
||||
success: true,
|
||||
scanned: 0,
|
||||
gapsFound: 0,
|
||||
triggered: 0,
|
||||
created: 0,
|
||||
retried: 0,
|
||||
skippedInFlight: 0,
|
||||
skippedCancelled: 0,
|
||||
skippedCapHit: 0,
|
||||
});
|
||||
expect(prismaMock.$queryRaw).not.toHaveBeenCalled();
|
||||
expect(jobQueueMock.addSearchEbookJob).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('treats auto-grab unset (null) as ON and proceeds to source check', async () => {
|
||||
configMock.get.mockImplementation(async (key: string) => {
|
||||
switch (key) {
|
||||
case 'ebook_auto_grab_enabled':
|
||||
return null;
|
||||
case 'ebook_annas_archive_enabled':
|
||||
return 'true';
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
});
|
||||
prismaMock.$queryRaw.mockResolvedValue([]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-2' });
|
||||
|
||||
expect(result.scanned).toBe(0);
|
||||
expect(prismaMock.$queryRaw).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('returns zeros when both new source keys disabled AND no legacy key', async () => {
|
||||
configMock.get.mockImplementation(async (key: string) => {
|
||||
switch (key) {
|
||||
case 'ebook_auto_grab_enabled':
|
||||
return null;
|
||||
case 'ebook_annas_archive_enabled':
|
||||
return 'false';
|
||||
case 'ebook_indexer_search_enabled':
|
||||
return 'false';
|
||||
case 'ebook_sidecar_enabled':
|
||||
return null;
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
});
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-3' });
|
||||
|
||||
expect(result.scanned).toBe(0);
|
||||
expect(prismaMock.$queryRaw).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('legacy ebook_sidecar_enabled=true (with new keys absent) passes the source gate', async () => {
|
||||
configMock.get.mockImplementation(async (key: string) => {
|
||||
switch (key) {
|
||||
case 'ebook_auto_grab_enabled':
|
||||
return null;
|
||||
case 'ebook_annas_archive_enabled':
|
||||
return null;
|
||||
case 'ebook_indexer_search_enabled':
|
||||
return null;
|
||||
case 'ebook_sidecar_enabled':
|
||||
return 'true';
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
});
|
||||
prismaMock.$queryRaw.mockResolvedValue([]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-4' });
|
||||
|
||||
expect(result.scanned).toBe(0);
|
||||
expect(prismaMock.$queryRaw).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('processFindMissingEbooks — fresh-gap creation', () => {
|
||||
it('creates a new ebook request when no live ebook child exists (audiobook downloaded)', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
parent_request_id: 'parent-fresh',
|
||||
user_id: 'user-x',
|
||||
audiobook_id: 'audio-x',
|
||||
audiobook_title: 'Fresh Book',
|
||||
audiobook_author: 'Some Author',
|
||||
audible_asin: 'B09ABCDEFG',
|
||||
custom_search_terms: 'cst',
|
||||
}),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-5' });
|
||||
|
||||
expect(prismaMock.request.create).toHaveBeenCalledWith({
|
||||
data: {
|
||||
userId: 'user-x',
|
||||
audiobookId: 'audio-x',
|
||||
type: 'ebook',
|
||||
parentRequestId: 'parent-fresh',
|
||||
status: 'pending',
|
||||
progress: 0,
|
||||
customSearchTerms: 'cst',
|
||||
ebookAutoRetryCount: 1,
|
||||
},
|
||||
});
|
||||
expect(jobQueueMock.addSearchEbookJob).toHaveBeenCalledWith(
|
||||
'new-ebook-1',
|
||||
expect.objectContaining({
|
||||
id: 'audio-x',
|
||||
title: 'Fresh Book',
|
||||
author: 'Some Author',
|
||||
asin: 'B09ABCDEFG',
|
||||
})
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
scanned: 1,
|
||||
gapsFound: 1,
|
||||
triggered: 1,
|
||||
created: 1,
|
||||
retried: 0,
|
||||
skippedInFlight: 0,
|
||||
skippedCancelled: 0,
|
||||
skippedCapHit: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it('also creates for audiobook in `available` state (both statuses in scope)', async () => {
|
||||
// The query is responsible for filtering by status; here we just confirm
|
||||
// that the processor doesn't add a second status guard in JS that would
|
||||
// reject a row coming back from SQL.
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
parent_request_id: 'parent-available',
|
||||
audiobook_title: 'Available Book',
|
||||
}),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-6' });
|
||||
|
||||
expect(result.created).toBe(1);
|
||||
expect(prismaMock.request.create).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('omits asin when audiobook has no audibleAsin', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([baseRow({ audible_asin: null })]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
await processFindMissingEbooks({ jobId: 'job-6b' });
|
||||
|
||||
expect(jobQueueMock.addSearchEbookJob).toHaveBeenCalledWith(
|
||||
'new-ebook-1',
|
||||
expect.objectContaining({ asin: undefined })
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('processFindMissingEbooks — branch skips', () => {
|
||||
it('skips when most-recent ebook child is already downloaded (defensive)', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
ebook_request_id: 'ebook-1',
|
||||
ebook_status: 'downloaded',
|
||||
ebook_auto_retry_count: 0,
|
||||
}),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-7' });
|
||||
|
||||
expect(prismaMock.request.create).not.toHaveBeenCalled();
|
||||
expect(prismaMock.request.update).not.toHaveBeenCalled();
|
||||
expect(jobQueueMock.addSearchEbookJob).not.toHaveBeenCalled();
|
||||
// skipped-has-companion is intentionally not surfaced as its own counter —
|
||||
// admin can derive from scanned - gapsFound - skippedInFlight - skippedCancelled - skippedCapHit.
|
||||
expect(result).toMatchObject({
|
||||
scanned: 1,
|
||||
gapsFound: 0,
|
||||
triggered: 0,
|
||||
skippedInFlight: 0,
|
||||
skippedCancelled: 0,
|
||||
skippedCapHit: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
'pending',
|
||||
'awaiting_approval',
|
||||
'searching',
|
||||
'downloading',
|
||||
'processing',
|
||||
'awaiting_search',
|
||||
'awaiting_release',
|
||||
])('skips when most-recent ebook child status is in-flight: %s', async (status) => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({ ebook_request_id: 'ebook-1', ebook_status: status }),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: `job-inflight-${status}` });
|
||||
|
||||
expect(prismaMock.request.create).not.toHaveBeenCalled();
|
||||
expect(prismaMock.request.update).not.toHaveBeenCalled();
|
||||
expect(jobQueueMock.addSearchEbookJob).not.toHaveBeenCalled();
|
||||
expect(result.skippedInFlight).toBe(1);
|
||||
expect(result.gapsFound).toBe(0);
|
||||
});
|
||||
|
||||
it('skips when most-recent ebook child status is cancelled', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({ ebook_request_id: 'ebook-1', ebook_status: 'cancelled' }),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-cancelled' });
|
||||
|
||||
expect(jobQueueMock.addSearchEbookJob).not.toHaveBeenCalled();
|
||||
expect(result.skippedCancelled).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('processFindMissingEbooks — retry path', () => {
|
||||
it('retries a failed ebook child with counter < cap, increments counter', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
ebook_request_id: 'ebook-fail-1',
|
||||
ebook_status: 'failed',
|
||||
ebook_auto_retry_count: 3,
|
||||
}),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-retry' });
|
||||
|
||||
expect(prismaMock.request.update).toHaveBeenCalledWith({
|
||||
where: { id: 'ebook-fail-1' },
|
||||
data: {
|
||||
status: 'pending',
|
||||
progress: 0,
|
||||
errorMessage: null,
|
||||
ebookAutoRetryCount: 4,
|
||||
},
|
||||
});
|
||||
expect(jobQueueMock.addSearchEbookJob).toHaveBeenCalledWith(
|
||||
'ebook-fail-1',
|
||||
expect.objectContaining({ id: 'audio-1' })
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
retried: 1,
|
||||
created: 0,
|
||||
gapsFound: 1,
|
||||
triggered: 1,
|
||||
skippedCapHit: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it('skips a warn ebook child whose counter is at the cap (5)', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
ebook_request_id: 'ebook-cap',
|
||||
ebook_status: 'warn',
|
||||
ebook_auto_retry_count: 5,
|
||||
}),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-cap' });
|
||||
|
||||
expect(prismaMock.request.update).not.toHaveBeenCalled();
|
||||
expect(jobQueueMock.addSearchEbookJob).not.toHaveBeenCalled();
|
||||
expect(result.skippedCapHit).toBe(1);
|
||||
expect(result.retried).toBe(0);
|
||||
});
|
||||
|
||||
it('retries a failed ebook child with null counter, sets counter to 1', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
ebook_request_id: 'ebook-null-counter',
|
||||
ebook_status: 'failed',
|
||||
ebook_auto_retry_count: null,
|
||||
}),
|
||||
]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-null' });
|
||||
|
||||
expect(prismaMock.request.update).toHaveBeenCalledWith({
|
||||
where: { id: 'ebook-null-counter' },
|
||||
data: {
|
||||
status: 'pending',
|
||||
progress: 0,
|
||||
errorMessage: null,
|
||||
ebookAutoRetryCount: 1,
|
||||
},
|
||||
});
|
||||
expect(result.retried).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('processFindMissingEbooks — error isolation', () => {
|
||||
it('rolls back the counter when addSearchEbookJob throws, then continues with next candidate', async () => {
|
||||
// Two candidates: first one's enqueue will throw, second should still process.
|
||||
prismaMock.$queryRaw.mockResolvedValue([
|
||||
baseRow({
|
||||
parent_request_id: 'parent-throw',
|
||||
audiobook_id: 'audio-throw',
|
||||
ebook_request_id: 'ebook-throw',
|
||||
ebook_status: 'failed',
|
||||
ebook_auto_retry_count: 2,
|
||||
}),
|
||||
baseRow({
|
||||
parent_request_id: 'parent-ok',
|
||||
audiobook_id: 'audio-ok',
|
||||
ebook_request_id: 'ebook-ok',
|
||||
ebook_status: 'failed',
|
||||
ebook_auto_retry_count: 0,
|
||||
}),
|
||||
]);
|
||||
|
||||
jobQueueMock.addSearchEbookJob
|
||||
.mockRejectedValueOnce(new Error('queue blew up'))
|
||||
.mockResolvedValueOnce('bull-job-id');
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-throw' });
|
||||
|
||||
// Counter rolled back on the throwing candidate:
|
||||
expect(prismaMock.request.update).toHaveBeenCalledWith({
|
||||
where: { id: 'ebook-throw' },
|
||||
data: { ebookAutoRetryCount: { decrement: 1 } },
|
||||
});
|
||||
// Second candidate still processed:
|
||||
expect(prismaMock.request.update).toHaveBeenCalledWith({
|
||||
where: { id: 'ebook-ok' },
|
||||
data: {
|
||||
status: 'pending',
|
||||
progress: 0,
|
||||
errorMessage: null,
|
||||
ebookAutoRetryCount: 1,
|
||||
},
|
||||
});
|
||||
// gapsFound counts both attempted gaps; only the second succeeds in being triggered.
|
||||
expect(result.gapsFound).toBe(2);
|
||||
expect(result.retried).toBe(1);
|
||||
expect(result.created).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('processFindMissingEbooks — return shape', () => {
|
||||
it('exposes all observable counters in the result', async () => {
|
||||
prismaMock.$queryRaw.mockResolvedValue([]);
|
||||
|
||||
const { processFindMissingEbooks } = await import('@/lib/processors/find-missing-ebooks.processor');
|
||||
const result = await processFindMissingEbooks({ jobId: 'job-shape' });
|
||||
|
||||
expect(result).toEqual(
|
||||
expect.objectContaining({
|
||||
success: true,
|
||||
scanned: 0,
|
||||
gapsFound: 0,
|
||||
triggered: 0,
|
||||
created: 0,
|
||||
retried: 0,
|
||||
skippedInFlight: 0,
|
||||
skippedCancelled: 0,
|
||||
skippedCapHit: 0,
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -16,6 +16,7 @@ const jobQueueMock = vi.hoisted(() => ({
|
||||
addAudibleRefreshJob: vi.fn(),
|
||||
addRetryMissingTorrentsJob: vi.fn(),
|
||||
addRetryFailedImportsJob: vi.fn(),
|
||||
addFindMissingEbooksJob: vi.fn(),
|
||||
addCleanupSeededTorrentsJob: vi.fn(),
|
||||
addMonitorRssFeedsJob: vi.fn(),
|
||||
addSyncShelvesJob: vi.fn(),
|
||||
@@ -80,7 +81,7 @@ describe('SchedulerService', () => {
|
||||
const service = new SchedulerService();
|
||||
await service.start();
|
||||
|
||||
expect(prismaMock.scheduledJob.create).toHaveBeenCalledTimes(9);
|
||||
expect(prismaMock.scheduledJob.create).toHaveBeenCalledTimes(10);
|
||||
expect(jobQueueMock.addRepeatableJob).toHaveBeenCalledWith(
|
||||
'audible_refresh',
|
||||
{ scheduledJobId: 'job-1' },
|
||||
@@ -289,6 +290,7 @@ describe('SchedulerService', () => {
|
||||
['audible_refresh', 'addAudibleRefreshJob'],
|
||||
['retry_missing_torrents', 'addRetryMissingTorrentsJob'],
|
||||
['retry_failed_imports', 'addRetryFailedImportsJob'],
|
||||
['find_missing_ebooks', 'addFindMissingEbooksJob'],
|
||||
['cleanup_seeded_torrents', 'addCleanupSeededTorrentsJob'],
|
||||
['monitor_rss_feeds', 'addMonitorRssFeedsJob'],
|
||||
['sync_reading_shelves', 'addSyncShelvesJob'],
|
||||
|
||||
Reference in New Issue
Block a user