mirror of
https://github.com/kikootwo/ReadMeABook.git
synced 2026-06-02 20:30:10 +00:00
kikootwo
411b5f88a4
Introduce a per-indexer ratioLimit alongside seedingTimeMinutes to control torrent cleanup. Updates include: documentation (scheduler and settings pages), types and API (saved indexer config now includes ratioLimit), setup and management UI (new TorrentSeedingFields component, modal wiring, validation and handlers), and processor logic (cleanup-seeded-torrents now requires AND-semantics between time and ratio; 0 disables a criterion, both 0 = never cleaned, undefined client ratio with ratioLimit>0 = not met). Tests were added/updated to cover ratio-only, time+ratio, missing-ratio, and UI interactions. Default behavior: ratioLimit defaults to 0 (no ratio requirement).
174 lines
9.6 KiB
Markdown
174 lines
9.6 KiB
Markdown
# Recurring Jobs Scheduler
|
||
|
||
**Status:** ✅ Implemented
|
||
|
||
Manages recurring/scheduled jobs providing automated tasks (Plex scans, Audible refresh) with scheduled (cron) execution and manual triggering.
|
||
|
||
## Recent Updates
|
||
|
||
- Config validation before job execution
|
||
- Audible refresh persists to database
|
||
- Enhanced error handling with clear messages
|
||
- Schedule editing UI with toast notifications
|
||
- Human-friendly schedule descriptions and editor (preset/custom/advanced modes)
|
||
- Real-time cron expression preview
|
||
- Admin Jobs page shows per-job descriptions inline; startup auto-renames legacy "Plex *" job names to neutral defaults (type-gated, exact-literal match only)
|
||
|
||
## Scheduled Jobs
|
||
|
||
1. **plex_library_scan** - Default: every 6 hours, full library scan, disabled by default (enable after setup)
|
||
2. **plex_recently_added_check** - Default: every 5 minutes, lightweight polling of top 10 recently added items, enabled by default
|
||
3. **audible_refresh** - Default: daily midnight, fetches 200 popular + 200 new releases, stores with rankings, disabled by default
|
||
4. **retry_missing_torrents** - Default: daily midnight, processes union of `awaiting_search` ∪ `awaiting_release` (limit 50), handles both audiobook and ebook requests. Bidirectional transitions: `awaiting_search` → `awaiting_release` when release date is future + `indexer.skip_unreleased` ON; `awaiting_release` → `awaiting_search` + run search when release date has passed or setting OFF. Sole owner of these transitions. Enabled by default.
|
||
5. **retry_failed_imports** - Default: every 6 hours, re-attempts 'awaiting_import' status (limit 50), enabled by default
|
||
6. **find_missing_ebooks** - Default: daily midnight, scans `downloaded` ∪ `available` audiobook requests (limit 50) for missing ebook companions and triggers the existing ebook fetch flow (`addSearchEbookJob`). Gated by `ebook_auto_grab_enabled` AND at least one ebook source enabled (`ebook_annas_archive_enabled` or `ebook_indexer_search_enabled`; legacy `ebook_sidecar_enabled` accepted as Anna's fallback). Skips ebook children in-flight (`pending`, `awaiting_approval`, `searching`, `downloading`, `processing`, `awaiting_search`, `awaiting_release`) or `cancelled`. Retries `failed`/`warn` children up to **5 lifetime auto-retries** per audiobook, tracked in `Request.ebookAutoRetryCount` (nullable; processor-private — manual "Fetch Ebook" never reads/writes it). Per-candidate writes are wrapped in `prisma.$transaction` for race-safety with concurrent auto-grab; counter rolls back if `addSearchEbookJob` throws. Enabled by default. Returns `{ scanned, gapsFound, triggered, created, retried, skippedInFlight, skippedCancelled, skippedCapHit }`.
|
||
7. **cleanup_seeded_torrents** - Default: every 30 mins, deletes torrents after seeding requirements met. Respects per-indexer `seedingTimeMinutes` AND `ratioLimit` (BOTH required when set; `0` disables that criterion; both `0` = never cleaned up). Undefined ratio with `ratioLimit > 0` = not met (safe-deny). Enabled by default.
|
||
8. **monitor_rss_feeds** - Default: every 15 mins, checks RSS feeds from enabled indexers, matches against `awaiting_search` requests (audiobook and ebook, limit 100). Query is unchanged — release-date gate is applied AFTER a match is found: if matched book is unreleased + `indexer.skip_unreleased` ON, the match is skipped and request status is NOT mutated (retry job owns transitions). Enabled by default.
|
||
|
||
## Architecture: Bull + Cron
|
||
|
||
- Repeatable jobs with cron expressions (Bull's built-in scheduler)
|
||
- Manual trigger capability
|
||
- Job persistence and retry logic
|
||
- Admin UI management
|
||
- Automatic scheduling/unscheduling when jobs enabled/disabled
|
||
- Schedule updates handled by unscheduling old job and scheduling new one
|
||
|
||
## Human-Friendly Scheduling UI
|
||
|
||
**Three Modes:**
|
||
1. **Common Schedules** - Preset options (every 15min, hourly, daily, weekly, monthly)
|
||
2. **Custom Schedule** - Visual builder with dropdowns for minutes/hours/daily/weekly/monthly
|
||
3. **Advanced (Cron)** - Raw cron expression for power users
|
||
|
||
**Features:**
|
||
- Human-readable display: "Every 6 hours" instead of "0 */6 * * *"
|
||
- Real-time preview of cron expressions
|
||
- Visual schedule builder (no cron knowledge required)
|
||
- Cron validation before saving
|
||
- Shows both human text and cron expression in job list
|
||
|
||
**Utility Functions** (`src/lib/utils/cron.ts`):
|
||
- `cronToHuman(cron)` - Converts cron to readable text
|
||
- `customScheduleToCron(schedule)` - Builds cron from visual inputs (auto-converts 24+ hour intervals to daily)
|
||
- `cronToCustomSchedule(cron)` - Parses cron to visual inputs
|
||
- `isValidCron(cron)` - Validates cron expression
|
||
|
||
## Cron Expressions
|
||
|
||
```
|
||
* * * * *
|
||
│ │ │ │ └─ day of week (0-7)
|
||
│ │ │ └─── month (1-12)
|
||
│ │ └───── day of month (1-31)
|
||
│ └─────── hour (0-23)
|
||
└───────── minute (0-59)
|
||
```
|
||
|
||
**Examples:**
|
||
- `0 */6 * * *` - Every 6 hours
|
||
- `0 0 * * *` - Daily midnight
|
||
- `*/30 * * * *` - Every 30 mins
|
||
|
||
## API Endpoints
|
||
|
||
**GET /api/admin/jobs** - Get all scheduled jobs (admin auth)
|
||
|
||
**POST /api/admin/jobs** - Create job (admin auth)
|
||
```json
|
||
{
|
||
"name": "Daily Audible Refresh",
|
||
"type": "audible_refresh",
|
||
"schedule": "0 0 * * *",
|
||
"enabled": true
|
||
}
|
||
```
|
||
|
||
**PUT /api/admin/jobs/:id** - Update job (admin auth)
|
||
|
||
**DELETE /api/admin/jobs/:id** - Delete job (admin auth)
|
||
|
||
**POST /api/admin/jobs/:id/trigger** - Manually trigger job (admin auth)
|
||
|
||
**GET /api/admin/jobs/:id/history?limit=50** - Job execution history (admin auth)
|
||
|
||
## Data Model
|
||
|
||
```typescript
|
||
interface ScheduledJob {
|
||
id: string;
|
||
name: string;
|
||
type: JobType;
|
||
schedule: string; // cron
|
||
enabled: boolean;
|
||
lastRun: Date | null;
|
||
nextRun: Date | null;
|
||
payload: any;
|
||
}
|
||
```
|
||
|
||
## Implementation Details
|
||
|
||
**Scheduler Service (`scheduler.service.ts`):**
|
||
- `start()`: Initializes scheduler, creates default jobs, schedules all enabled jobs
|
||
- `scheduleJob()`: Adds job to Bull as repeatable job with cron expression
|
||
- `unscheduleJob()`: Removes repeatable job from Bull
|
||
- `updateScheduledJob()`: Unschedules old job, updates DB, schedules new job if enabled
|
||
- `deleteScheduledJob()`: Unschedules job before deleting from DB
|
||
|
||
**Job Queue Service (`job-queue.service.ts`):**
|
||
- `addRepeatableJob()`: Registers job type with Bull's repeat scheduler
|
||
- `removeRepeatableJob()`: Removes job from Bull's repeat scheduler
|
||
- Processors for each scheduled job type call `scheduler.triggerJobNow()`
|
||
- `setMaxListeners(20)`: Set on both Redis client and Bull queue to accommodate 12 job processors (6 regular + 6 scheduled)
|
||
|
||
**Flow:**
|
||
1. App starts → `scheduler.start()` → schedules all enabled jobs
|
||
2. Bull triggers job at cron time → processor calls `triggerJobNow()`
|
||
3. `triggerJobNow()` executes job-specific logic (Plex scan, Audible refresh, etc.)
|
||
4. Updates `lastRun` timestamp in database
|
||
|
||
## Audible Refresh Processor
|
||
|
||
**Implementation:**
|
||
1. Fetch 200 popular + 200 new releases (multi-page scraping)
|
||
2. Download and cache cover thumbnails locally (stored in `/app/cache/thumbnails`)
|
||
3. Wipe and re-populate `AudibleCacheCategory` entries with reserved IDs (`__popular__`, `__new_releases__`) and user-configured category IDs
|
||
4. Upsert book metadata in `AudibleCache`, ranked entries in `AudibleCacheCategory`
|
||
5. Record sync timestamp (`lastAudibleSync`)
|
||
6. Clean up unused thumbnails (removes covers for audiobooks no longer in cache)
|
||
7. Perform fuzzy matching (70% threshold) against Plex library
|
||
8. Set `plexGuid` when match found (with duplicate protection)
|
||
9. Update `availabilityStatus` to 'available' or 'unknown'
|
||
|
||
**Duplicate PlexGuid Handling:** Since `plexGuid` has UNIQUE constraint, only first match gets assigned to prevent violations.
|
||
|
||
**Thumbnail Caching:** Downloads cover images from Audible and stores them locally to reduce external requests. Cached thumbnails are served via `/api/cache/thumbnails/[filename]` endpoint. Unused thumbnails are automatically cleaned up after each sync.
|
||
|
||
## Fixed Issues ✅
|
||
|
||
- ✅ Jobs running without config validation
|
||
- ✅ Default alert() popups → toast notifications
|
||
- ✅ No UI for editing schedules → added edit modal
|
||
- ✅ Audible data not persisting → saves to database
|
||
- ✅ Download progress logging ~500x/s → 10s delay
|
||
- ✅ Requests failing permanently (no torrents) → retry system with 'awaiting_search'
|
||
- ✅ Requests failing permanently (no files) → retry system with max 5 retries + 'warn' status
|
||
- ✅ Failed requests blocking re-requests → allow re-requesting failed/warn/cancelled
|
||
- ✅ Files deleted immediately → kept until seeding requirements met
|
||
- ✅ No seeding time config → added `seeding_time_minutes`
|
||
- ✅ No ratio-based seeding policy → added per-indexer `ratioLimit` (AND-semantics with `seedingTimeMinutes`; `0` disables; undefined client ratio = safe-deny)
|
||
- ✅ Scheduled jobs not running on schedule → implemented Bull repeatable jobs with cron scheduling
|
||
- ✅ MaxListenersExceededWarning → increased maxListeners to 20 on both Redis client and Bull queue
|
||
- ✅ Cron expressions not user-friendly → added human-readable descriptions and visual schedule builder
|
||
- ✅ Scheduled jobs triggered by timer not appearing in system logs → Job records now created automatically for timer-triggered jobs
|
||
- ✅ Scheduled jobs triggered by timer not updating lastRun timestamp → Job queue now updates lastRun when processing timer-triggered jobs
|
||
- ✅ Daily cron patterns at non-midnight hours not recognized → Fixed `getIntervalFromCron` to parse any daily time (e.g., "0 4 * * *")
|
||
- ✅ "Every 24 hours" interval validation error → Auto-converts 24+ hour intervals to daily schedule (0 0 * * *)
|
||
|
||
## Tech Stack
|
||
|
||
- Bull repeatable jobs
|
||
- PostgreSQL (scheduled_jobs table)
|
||
- Bull/Redis infrastructure
|