mirror of
https://github.com/kikootwo/ReadMeABook.git
synced 2026-06-03 04:40:09 +00:00
Initial commit
This commit is contained in:
kikootwo
@@ -0,0 +1,49 @@
|
||||
# Phase 3: Automation Engine
|
||||
|
||||
**Status:** ⏳ In Development
|
||||
|
||||
Multi-stage pipeline transforming requests into downloaded, organized media in Plex.
|
||||
|
||||
## Pipeline
|
||||
|
||||
```
|
||||
Request → search_indexers → rank_results → download_torrent
|
||||
→ monitor_download → process_audiobook → update_plex
|
||||
```
|
||||
|
||||
## Job Types
|
||||
|
||||
1. **search_indexers** - Search Prowlarr for torrents
|
||||
2. **rank_results** - Apply ranking algorithm, select best
|
||||
3. **download_torrent** - Add to qBittorrent
|
||||
4. **monitor_download** - Poll progress (10s intervals)
|
||||
5. **process_audiobook** - Organize files to media directory
|
||||
6. **update_plex** - Trigger scan, fuzzy match
|
||||
|
||||
## Integration Points
|
||||
|
||||
**Indexers:** Prowlarr (primary), Jackett (fallback)
|
||||
**Download Clients:** qBittorrent (primary), Transmission (fallback)
|
||||
**Media Server:** Plex (scan + match)
|
||||
|
||||
## Job Queue (Bull)
|
||||
|
||||
- Redis-backed for persistence
|
||||
- Retry: 3 attempts, exponential backoff (2s, 4s, 8s)
|
||||
- Priorities: High (10), Medium (5), Low (1)
|
||||
- Concurrency: 3 concurrent per type
|
||||
- Jobs survive app restarts
|
||||
|
||||
## Config Keys
|
||||
|
||||
**Prowlarr:** `indexer.type=prowlarr`, `indexer.prowlarr_url`, `indexer.prowlarr_api_key`
|
||||
**qBittorrent:** `download_client.type=qbittorrent`, `download_client.qbittorrent_url/username/password`
|
||||
**Paths:** `paths.download_dir`, `paths.media_dir`
|
||||
|
||||
## Related Docs
|
||||
|
||||
- [Prowlarr](./prowlarr.md)
|
||||
- [qBittorrent](./qbittorrent.md)
|
||||
- [Ranking Algorithm](./ranking-algorithm.md)
|
||||
- [File Organization](./file-organization.md)
|
||||
- [Plex Integration](../integrations/plex.md)
|
||||
@@ -0,0 +1,124 @@
|
||||
# File Organization System
|
||||
|
||||
**Status:** ✅ Implemented
|
||||
|
||||
Copies completed downloads to standardized directory structure for Plex. Automatically tags audio files with correct metadata. Originals kept for seeding, cleaned up by scheduled job after requirements met.
|
||||
|
||||
## Target Structure
|
||||
|
||||
```
|
||||
/media/audiobooks/
|
||||
└── Author Name/
|
||||
└── Book Title (Year)/
|
||||
├── Book Title.m4b
|
||||
└── cover.jpg
|
||||
```
|
||||
|
||||
## Process
|
||||
|
||||
1. Download completes in `/downloads/[torrent-name]/` or `/downloads/[filename]` (single file)
|
||||
2. Identify audiobook files (.m4b, .m4a, .mp3) - supports both directories and single files
|
||||
3. Create `/media/audiobooks/[Author]/[Title]/`
|
||||
4. **Copy** files (not move - originals stay for seeding)
|
||||
5. **Tag metadata** (if enabled) - writes correct title, author, narrator to audio files
|
||||
6. Copy cover art if found, else download from Audible
|
||||
7. Originals remain until seeding requirements met
|
||||
|
||||
## Metadata Tagging
|
||||
|
||||
**Status:** ✅ Implemented
|
||||
|
||||
**Purpose:** Automatically writes correct metadata to audio files during file organization to improve Plex matching accuracy.
|
||||
|
||||
**Supported Formats:**
|
||||
- m4b, m4a, mp4 (AAC audiobooks)
|
||||
- mp3 (ID3v2 tags)
|
||||
|
||||
**Metadata Written:**
|
||||
- `title` - Book title
|
||||
- `album` - Book title (PRIMARY field for Plex matching)
|
||||
- `album_artist` - Author (PRIMARY field for Plex matching)
|
||||
- `artist` - Author (fallback)
|
||||
- `composer` - Narrator (standard audiobook field)
|
||||
- `date` - Year
|
||||
|
||||
**Configuration:**
|
||||
- Key: `metadata_tagging_enabled` (Configuration table)
|
||||
- Default: `true`
|
||||
- Configurable in: Setup wizard (Paths step), Admin settings (Paths tab)
|
||||
|
||||
**Implementation:**
|
||||
- Uses ffmpeg with `-codec copy` (no re-encoding, metadata only)
|
||||
- Fast (no audio transcoding)
|
||||
- Lossless (original audio preserved)
|
||||
- Runs after file copy, before cover art download
|
||||
- Non-blocking (errors don't fail file organization)
|
||||
- Logs success/failure per file
|
||||
|
||||
**Benefits:**
|
||||
- Fixes torrents with missing/incorrect metadata
|
||||
- Ensures Plex can match audiobooks correctly
|
||||
- Writes metadata from Audible/Audnexus (known accurate)
|
||||
- Prevents "[Various Albums]" and other metadata issues
|
||||
|
||||
**Tech Stack:**
|
||||
- ffmpeg (system dependency - included in Docker image)
|
||||
- `src/lib/utils/metadata-tagger.ts` - Tagging utility
|
||||
- Integrated into `src/lib/utils/file-organizer.ts`
|
||||
|
||||
**Requirements:**
|
||||
- ffmpeg must be installed in the container
|
||||
- **Multi-container setup** (`Dockerfile`): Added at line 56 via `apk add ffmpeg`
|
||||
- **Unified setup** (`dockerfile.unified`): Added at line 16 via `apt-get install ffmpeg`
|
||||
- **Verify installation:**
|
||||
- Multi-container: `docker exec readmeabook ffmpeg -version`
|
||||
- Unified: `docker exec readmeabook-unified ffmpeg -version`
|
||||
|
||||
## Seeding Support
|
||||
|
||||
**Config:** `seeding_time_minutes` (0 = unlimited, never cleanup)
|
||||
|
||||
**Cleanup Job:** `cleanup_seeded_torrents` (every 30 mins)
|
||||
1. Check 'available' and 'downloaded' status requests with download history
|
||||
2. Query qBittorrent for actual `seeding_time` field
|
||||
3. Delete torrent + files only after requirement met
|
||||
4. Respects config (0 = never cleanup)
|
||||
|
||||
## Interface
|
||||
|
||||
```typescript
|
||||
interface OrganizationResult {
|
||||
success: boolean;
|
||||
targetPath: string;
|
||||
filesMovedCount: number;
|
||||
errors: string[];
|
||||
audioFiles: string[];
|
||||
coverArtFile?: string;
|
||||
}
|
||||
|
||||
async function organize(
|
||||
downloadPath: string,
|
||||
audiobook: {title: string, author: string, year?: number, coverArtUrl?: string}
|
||||
): Promise<OrganizationResult>;
|
||||
```
|
||||
|
||||
## Path Sanitization
|
||||
|
||||
- Remove invalid chars: `<>:"/\|?*`
|
||||
- Trim dots/spaces
|
||||
- Collapse multiple spaces
|
||||
- Limit to 200 chars
|
||||
- Example: `Author: The <Best>! Book?` → `Author The Best! Book`
|
||||
|
||||
## Fixed Issues ✅
|
||||
|
||||
**1. EPERM errors** - Fixed with `fs.readFile/writeFile` instead of `copyFile`
|
||||
**2. Immediate deletion** - Changed to copy-only, scheduled cleanup after seeding
|
||||
**3. Files moved not copied** - Now copies to support seeding
|
||||
**4. Single file downloads** - Now supports files directly in downloads folder (not just directories)
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- Node.js `fs/promises`
|
||||
- `path` module
|
||||
- axios (cover art download)
|
||||
@@ -0,0 +1,91 @@
|
||||
# Prowlarr Integration
|
||||
|
||||
**Status:** ✅ Implemented | Manual search, interactive search, automatic search
|
||||
|
||||
Indexer aggregator for searching multiple torrent/usenet indexers simultaneously. Supports manual search, interactive torrent selection, and automatic RSS feed monitoring.
|
||||
|
||||
## API
|
||||
|
||||
**Base:** `http://prowlarr:9696/api/v1`
|
||||
**Auth:** `X-Api-Key` header
|
||||
|
||||
**GET /search?query={q}&categories=3030** - Search all indexers (3030 = audiobooks)
|
||||
**GET /indexer** - List configured indexers
|
||||
**GET /indexerstats** - Indexer statistics
|
||||
**GET /feed/{indexerId}/api?t=search&cat=3030&limit=100** - RSS feed for specific indexer
|
||||
|
||||
## Search
|
||||
|
||||
**Extended Search:** Enabled (`extended=1`) - searches title, tags, labels, and metadata fields
|
||||
|
||||
```typescript
|
||||
interface TorrentResult {
|
||||
indexer: string;
|
||||
title: string;
|
||||
size: number; // bytes
|
||||
seeders: number;
|
||||
leechers: number;
|
||||
publishDate: Date;
|
||||
downloadUrl: string; // magnet or .torrent
|
||||
infoHash?: string;
|
||||
guid: string;
|
||||
format?: 'M4B' | 'M4A' | 'MP3';
|
||||
bitrate?: string;
|
||||
hasChapters?: boolean;
|
||||
}
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
- `indexer.prowlarr_url`
|
||||
- `indexer.prowlarr_api_key`
|
||||
|
||||
## Error Handling
|
||||
|
||||
- 401: Invalid API key
|
||||
- 429: Rate limit (exponential backoff, max 3 retries)
|
||||
- 503: Service unavailable
|
||||
- Timeout: 30s per search
|
||||
|
||||
## Manual & Interactive Search
|
||||
|
||||
**Manual Search** (`POST /api/requests/{id}/manual-search`)
|
||||
- Triggers automatic search job for requests with status: pending, failed, awaiting_search
|
||||
- Uses ranking algorithm to select best torrent
|
||||
- Updates request status to 'pending'
|
||||
|
||||
**Interactive Search** (`POST /api/requests/{id}/interactive-search`)
|
||||
- Returns ranked torrent results for user selection
|
||||
- Shows table with: rank, title, size, quality score, seeders, indexer, publish date
|
||||
- Available for same statuses as manual search
|
||||
- User clicks "Download" button to select specific torrent
|
||||
|
||||
**Select Torrent** (`POST /api/requests/{id}/select-torrent`)
|
||||
- Downloads user-selected torrent from interactive search
|
||||
- Triggers download_torrent job
|
||||
- Updates request status to 'downloading'
|
||||
|
||||
**UI Integration:**
|
||||
- Manual Search button: Triggers automatic search
|
||||
- Interactive Search button: Opens modal with torrent results
|
||||
- Both buttons shown for requests with status: pending, failed, awaiting_search
|
||||
|
||||
## RSS Monitoring
|
||||
|
||||
**Automatic Feed Monitoring:** Enabled per-indexer via setup wizard or settings page
|
||||
**Schedule:** Every 15 minutes (default, configurable)
|
||||
**Process:**
|
||||
1. Fetch RSS feeds from all indexers with RSS enabled
|
||||
2. Fuzzy match results against requests in 'awaiting_search' status
|
||||
3. Trigger search jobs for matches
|
||||
4. Limit: 100 results per feed, 100 missing requests per check
|
||||
|
||||
**Matching Logic:**
|
||||
- Author name must appear in torrent title
|
||||
- At least 2 title words (>2 chars) must match
|
||||
- First match triggers search job (no duplicates)
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- axios
|
||||
- bottleneck (rate limiting)
|
||||
@@ -0,0 +1,91 @@
|
||||
# qBittorrent Integration
|
||||
|
||||
**Status:** ✅ Implemented
|
||||
|
||||
Free, open-source BitTorrent client with comprehensive Web API.
|
||||
|
||||
## Enterprise Torrent Addition
|
||||
|
||||
**Challenge:** `/api/v2/torrents/add` returns only "Ok." without torrent hash.
|
||||
|
||||
**Solution (Professional):**
|
||||
|
||||
**Magnet Links:**
|
||||
1. Extract `info_hash` from magnet URI (deterministic)
|
||||
2. Upload via `urls` parameter
|
||||
3. Return extracted hash immediately
|
||||
|
||||
**Torrent Files:**
|
||||
1. Download .torrent file to memory
|
||||
2. Parse with `parse-torrent` (bencode decoder)
|
||||
3. Extract `info_hash` (SHA-1 of info dict)
|
||||
4. Upload file content via `torrents` parameter (multipart/form-data)
|
||||
5. Return extracted hash immediately
|
||||
|
||||
**Benefits:** Deterministic, no race conditions, works with Docker networking, handles expired URLs
|
||||
|
||||
## API Endpoints
|
||||
|
||||
**Base:** `http://qbittorrent:8080/api/v2`
|
||||
**Auth:** Cookie-based (login required)
|
||||
|
||||
**POST /auth/login** - Get session cookie
|
||||
**POST /torrents/add** - Add torrent (supports `urls` and `torrents` params)
|
||||
**GET /torrents/info?hashes={hash}** - Get status/progress
|
||||
**POST /torrents/pause** - Pause torrent
|
||||
**POST /torrents/resume** - Resume
|
||||
**POST /torrents/delete** - Delete torrent
|
||||
**GET /torrents/files** - Get file list
|
||||
**POST /torrents/setCategory** - Set category
|
||||
|
||||
## Config
|
||||
|
||||
**Required (database only, no env fallbacks):**
|
||||
- `qbittorrent_url`
|
||||
- `qbittorrent_username`
|
||||
- `qbittorrent_password`
|
||||
- `paths_downloads`
|
||||
|
||||
Validation: All fields checked before service initialization.
|
||||
|
||||
## Data Models
|
||||
|
||||
```typescript
|
||||
interface TorrentInfo {
|
||||
hash: string;
|
||||
name: string;
|
||||
size: number;
|
||||
progress: number; // 0.0-1.0
|
||||
dlspeed: number; // bytes/s
|
||||
upspeed: number;
|
||||
eta: number; // seconds
|
||||
state: TorrentState;
|
||||
category: string;
|
||||
savePath: string;
|
||||
completionDate: number;
|
||||
}
|
||||
|
||||
type TorrentState = 'downloading' | 'uploading' | 'stalledDL' |
|
||||
'pausedDL' | 'queuedDL' | 'checkingDL' | 'error' | 'missingFiles';
|
||||
```
|
||||
|
||||
## Fixed Issues ✅
|
||||
|
||||
**1. Naive torrent identification** - Fixed with deterministic hash extraction
|
||||
**2. Docker networking issues** - Fixed by downloading .torrent ourselves
|
||||
**3. Duplicate detection** - Check if hash exists before adding
|
||||
**4. Config fallbacks to env** - Removed, database only
|
||||
**5. Unclear error messages** - List missing fields explicitly
|
||||
**6. Race condition on torrent availability** - Fixed with 3s initial delay + exponential backoff retry (500ms, 1s, 2s)
|
||||
**7. Error logging during duplicate check** - Removed console.error in getTorrent() during expected "not found" cases (duplicate checking)
|
||||
**8. Prowlarr magnet link redirects** - Some indexers return HTTP URLs that redirect to magnet: links. Fixed by intercepting 3xx redirects before axios follows them, extracting the Location header, and routing to magnet flow if target is a magnet: link
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- axios (HTTP + cookie mgmt)
|
||||
- parse-torrent (bencode + hash extraction)
|
||||
- form-data (multipart uploads)
|
||||
|
||||
## Related
|
||||
|
||||
- See [File Organization](./file-organization.md) for seeding support
|
||||
@@ -0,0 +1,53 @@
|
||||
# Intelligent Ranking Algorithm
|
||||
|
||||
**Status:** ❌ Not Implemented
|
||||
|
||||
Evaluates and scores torrents to automatically select best audiobook download.
|
||||
|
||||
## Scoring Criteria (100 points max)
|
||||
|
||||
**1. Format Quality (40 pts max)**
|
||||
- M4B with chapters: 40
|
||||
- M4B without chapters: 35
|
||||
- M4A: 25
|
||||
- MP3: 15
|
||||
- Other: 5
|
||||
|
||||
**2. Seeder Count (25 pts max)**
|
||||
- Formula: `Math.min(25, Math.log10(seeders + 1) * 10)`
|
||||
- 1 seeder: 0pts, 10 seeders: 10pts, 100 seeders: 20pts, 1000+: 25pts
|
||||
|
||||
**3. Size Reasonableness (20 pts max)**
|
||||
- Expected: 1-2 MB/min (64-128 kbps)
|
||||
- Deviation from expected → penalty
|
||||
|
||||
**4. Title Match Quality (15 pts max)**
|
||||
- Fuzzy match: title + author (Levenshtein distance)
|
||||
- Narrator bonus
|
||||
|
||||
## Interface
|
||||
|
||||
```typescript
|
||||
interface RankedTorrent extends TorrentResult {
|
||||
score: number;
|
||||
rank: number;
|
||||
breakdown: {
|
||||
formatScore: number;
|
||||
seederScore: number;
|
||||
sizeScore: number;
|
||||
matchScore: number;
|
||||
totalScore: number;
|
||||
notes: string[];
|
||||
};
|
||||
}
|
||||
|
||||
function rankTorrents(
|
||||
torrents: TorrentResult[],
|
||||
audiobook: AudiobookRequest
|
||||
): RankedTorrent[];
|
||||
```
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- string-similarity (fuzzy matching)
|
||||
- Regex for format detection
|
||||
Reference in New Issue
Block a user