Implement chapter merging feature and update ranking algorithm

Added automatic chapter merging to M4B with admin/config toggles, UI controls, and backend logic. Updated documentation to reflect implementation. Refactored ranking algorithm: increased Title/Author match points, removed size scoring, and improved Usenet/torrent handling. Enhanced Prowlarr integration for protocol detection and filtering. Improved file organizer to support chapter merging. Various bug fixes and logging improvements.
This commit is contained in:
kikootwo
2026-01-08 16:26:26 -05:00
parent 722a78ac33
commit 288421012d
21 changed files with 922 additions and 128 deletions
+2 -2
View File
@@ -48,7 +48,7 @@
- **qBittorrent integration (torrents)** → [phase3/qbittorrent.md](phase3/qbittorrent.md) - **qBittorrent integration (torrents)** → [phase3/qbittorrent.md](phase3/qbittorrent.md)
- **SABnzbd integration (Usenet/NZB)** → [phase3/sabnzbd.md](phase3/sabnzbd.md) - **SABnzbd integration (Usenet/NZB)** → [phase3/sabnzbd.md](phase3/sabnzbd.md)
- **File organization, seeding** → [phase3/file-organization.md](phase3/file-organization.md) - **File organization, seeding** → [phase3/file-organization.md](phase3/file-organization.md)
- **Chapter merging (PRD, not implemented)** → [features/chapter-merging.md](features/chapter-merging.md) - **Chapter merging (auto-merge to M4B)** → [features/chapter-merging.md](features/chapter-merging.md)
## Background Jobs ## Background Jobs
- **Bull queue, processors, retry logic** → [backend/services/jobs.md](backend/services/jobs.md) - **Bull queue, processors, retry logic** → [backend/services/jobs.md](backend/services/jobs.md)
@@ -93,7 +93,7 @@
**"How do I use the unified container?"** → [deployment/unified.md](deployment/unified.md) **"How do I use the unified container?"** → [deployment/unified.md](deployment/unified.md)
**"OAuth redirects to localhost / PUBLIC_URL not working"** → [backend/services/environment.md](backend/services/environment.md) **"OAuth redirects to localhost / PUBLIC_URL not working"** → [backend/services/environment.md](backend/services/environment.md)
**"What environment variables do I need?"** → [backend/services/environment.md](backend/services/environment.md) **"What environment variables do I need?"** → [backend/services/environment.md](backend/services/environment.md)
**"How does chapter merging work?"** → [features/chapter-merging.md](features/chapter-merging.md) (PRD only, not implemented) **"How does chapter merging work?"** → [features/chapter-merging.md](features/chapter-merging.md)
**"How does Audiobookshelf integration work?"** → [features/audiobookshelf-integration.md](features/audiobookshelf-integration.md) (PRD only, not implemented) **"How does Audiobookshelf integration work?"** → [features/audiobookshelf-integration.md](features/audiobookshelf-integration.md) (PRD only, not implemented)
**"How do I use OIDC/Authentik/Keycloak?"** → [features/audiobookshelf-integration.md](features/audiobookshelf-integration.md) (PRD only, not implemented) **"How do I use OIDC/Authentik/Keycloak?"** → [features/audiobookshelf-integration.md](features/audiobookshelf-integration.md) (PRD only, not implemented)
**"How does manual user registration work?"** → [features/audiobookshelf-integration.md](features/audiobookshelf-integration.md) (PRD only, not implemented) **"How does manual user registration work?"** → [features/audiobookshelf-integration.md](features/audiobookshelf-integration.md) (PRD only, not implemented)
+1 -1
View File
@@ -1,6 +1,6 @@
# Chapter Merging Feature # Chapter Merging Feature
**Status:** ❌ Not Started | Product Requirements Document **Status:** ✅ Implemented | Auto-merge multi-file chapters to M4B
## Overview ## Overview
+4 -10
View File
@@ -6,7 +6,7 @@ Evaluates and scores torrents to automatically select best audiobook download.
## Scoring Criteria (100 points max) ## Scoring Criteria (100 points max)
**1. Title/Author Match (50 pts max) - MOST IMPORTANT** **1. Title/Author Match (60 pts max) - MOST IMPORTANT**
**Multi-Stage Matching:** **Multi-Stage Matching:**
@@ -24,7 +24,7 @@ Evaluates and scores torrents to automatically select best audiobook download.
- "Dennis E. Taylor - Bobiverse - 01 - We Are Legion" → 3/3 = 100% → **PASSES** - "Dennis E. Taylor - Bobiverse - 01 - We Are Legion" → 3/3 = 100% → **PASSES**
- Prevents wrong series books from matching while handling common subtitle patterns - Prevents wrong series books from matching while handling common subtitle patterns
**Stage 2: Title Matching (0-35 pts)** **Stage 2: Title Matching (0-45 pts)**
- Only scored if Stage 1 passes - Only scored if Stage 1 passes
- **Tries full title first, then required title (without parentheses)** if no match - **Tries full title first, then required title (without parentheses)** if no match
- Example: "We Are Legion (We Are Bob)" tries both full title and "We Are Legion" - Example: "We Are Legion (We Are Bob)" tries both full title and "We Are Legion"
@@ -35,7 +35,7 @@ Evaluates and scores torrents to automatically select best audiobook download.
- Title preceded by metadata separator (` - `, `: `, `—`) — handles "Author - Series - 01 - Title" - Title preceded by metadata separator (` - `, `: `, `—`) — handles "Author - Series - 01 - Title"
- Author name appears in prefix — handles "Author Name - Title" - Author name appears in prefix — handles "Author Name - Title"
- **Acceptable suffix**: Followed by metadata markers: " by", " [", " -", " (", " {", " :", "," or end of string - **Acceptable suffix**: Followed by metadata markers: " by", " [", " -", " (", " {", " :", "," or end of string
- Complete match → 35 pts - Complete match → 45 pts
- Unstructured prefix (words without separators) → fuzzy similarity (partial credit) - Unstructured prefix (words without separators) → fuzzy similarity (partial credit)
- Prevents: "This Inevitable Ruin Dungeon Crawler Carl" matching "Dungeon Crawler Carl" - Prevents: "This Inevitable Ruin Dungeon Crawler Carl" matching "Dungeon Crawler Carl"
- Suffix continues with non-metadata → fuzzy similarity (partial credit) - Suffix continues with non-metadata → fuzzy similarity (partial credit)
@@ -61,12 +61,7 @@ Evaluates and scores torrents to automatically select best audiobook download.
**3. Seeder Count (15 pts max)** **3. Seeder Count (15 pts max)**
- Formula: `Math.min(15, Math.log10(seeders + 1) * 6)` - Formula: `Math.min(15, Math.log10(seeders + 1) * 6)`
- 1 seeder: 0pts, 10 seeders: 6pts, 100 seeders: 12pts, 1000+: 15pts - 1 seeder: 0pts, 10 seeders: 6pts, 100 seeders: 12pts, 1000+: 15pts
- Note: Usenet/NZB results without seeders get full 15 pts (centralized availability)
**4. Size Reasonableness (10 pts max)**
- Expected: 1-2 MB/min (64-128 kbps)
- Perfect match: 10 pts
- Deviation → penalty
- Unknown duration: 5 pts (neutral)
## Bonus Points System ## Bonus Points System
@@ -148,7 +143,6 @@ interface RankedTorrent extends TorrentResult {
breakdown: { breakdown: {
formatScore: number; formatScore: number;
seederScore: number; seederScore: number;
sizeScore: number;
matchScore: number; matchScore: number;
totalScore: number; // Same as score totalScore: number; // Same as score
notes: string[]; notes: string[];
@@ -0,0 +1,16 @@
-- Add chapter merging configuration
-- This allows admin to enable/disable automatic merging of multi-file chapter downloads into single M4B
-- Insert default configuration for chapter merging (disabled by default)
INSERT INTO configuration (id, key, value, encrypted, category, description, created_at, updated_at)
VALUES (
gen_random_uuid(),
'chapter_merging_enabled',
'false',
false,
'automation',
'Automatically merge multi-file chapter downloads into a single M4B audiobook with chapter markers. Improves playback experience and library organization.',
NOW(),
NOW()
)
ON CONFLICT (key) DO NOTHING;
+32
View File
@@ -81,6 +81,7 @@ interface Settings {
downloadDir: string; downloadDir: string;
mediaDir: string; mediaDir: string;
metadataTaggingEnabled: boolean; metadataTaggingEnabled: boolean;
chapterMergingEnabled: boolean;
}; };
ebook: { ebook: {
enabled: boolean; enabled: boolean;
@@ -1974,6 +1975,37 @@ export default function AdminSettings() {
</div> </div>
</div> </div>
{/* Chapter Merging Toggle */}
<div className="bg-gray-50 dark:bg-gray-800 rounded-lg p-4 border border-gray-200 dark:border-gray-700">
<div className="flex items-start gap-4">
<input
type="checkbox"
id="chapter-merging-settings"
checked={settings.paths.chapterMergingEnabled}
onChange={(e) => {
setSettings({
...settings,
paths: { ...settings.paths, chapterMergingEnabled: e.target.checked },
});
setValidated({ ...validated, paths: false });
}}
className="mt-1 h-5 w-5 rounded border-gray-300 text-blue-600 focus:ring-blue-500"
/>
<div className="flex-1">
<label
htmlFor="chapter-merging-settings"
className="block text-sm font-medium text-gray-900 dark:text-gray-100 cursor-pointer"
>
Auto-merge chapters to M4B
</label>
<p className="text-sm text-gray-600 dark:text-gray-400 mt-1">
Automatically merge multi-file chapter downloads into a single M4B audiobook with chapter
markers. Improves playback experience and library organization.
</p>
</div>
</div>
</div>
<div className="border-t border-gray-200 dark:border-gray-700 pt-6"> <div className="border-t border-gray-200 dark:border-gray-700 pt-6">
<Button <Button
onClick={testPaths} onClick={testPaths}
+13 -1
View File
@@ -11,7 +11,7 @@ export async function PUT(request: NextRequest) {
return requireAuth(request, async (req: AuthenticatedRequest) => { return requireAuth(request, async (req: AuthenticatedRequest) => {
return requireAdmin(req, async () => { return requireAdmin(req, async () => {
try { try {
const { downloadDir, mediaDir, metadataTaggingEnabled } = await request.json(); const { downloadDir, mediaDir, metadataTaggingEnabled, chapterMergingEnabled } = await request.json();
if (!downloadDir || !mediaDir) { if (!downloadDir || !mediaDir) {
return NextResponse.json( return NextResponse.json(
@@ -53,6 +53,18 @@ export async function PUT(request: NextRequest) {
}, },
}); });
// Update chapter merging setting
await prisma.configuration.upsert({
where: { key: 'chapter_merging_enabled' },
update: { value: String(chapterMergingEnabled ?? false) },
create: {
key: 'chapter_merging_enabled',
value: String(chapterMergingEnabled ?? false),
category: 'automation',
description: 'Automatically merge multi-file chapter downloads into single M4B with chapter markers',
},
});
console.log('[Admin] Paths settings updated'); console.log('[Admin] Paths settings updated');
// Invalidate qBittorrent service singleton to force reload of download_dir // Invalidate qBittorrent service singleton to force reload of download_dir
+1
View File
@@ -81,6 +81,7 @@ export async function GET(request: NextRequest) {
downloadDir: configMap.get('download_dir') || '/downloads', downloadDir: configMap.get('download_dir') || '/downloads',
mediaDir: configMap.get('media_dir') || '/media/audiobooks', mediaDir: configMap.get('media_dir') || '/media/audiobooks',
metadataTaggingEnabled: configMap.get('metadata_tagging_enabled') === 'true', metadataTaggingEnabled: configMap.get('metadata_tagging_enabled') === 'true',
chapterMergingEnabled: configMap.get('chapter_merging_enabled') === 'true',
}, },
ebook: { ebook: {
enabled: configMap.get('ebook_sidecar_enabled') === 'true', enabled: configMap.get('ebook_sidecar_enabled') === 'true',
@@ -28,8 +28,8 @@ const RequestWithTorrentSchema = z.object({
guid: z.string(), guid: z.string(),
title: z.string(), title: z.string(),
size: z.number(), size: z.number(),
seeders: z.number(), seeders: z.number().optional(), // Optional for NZB/Usenet results
leechers: z.number(), leechers: z.number().optional(), // Optional for NZB/Usenet results
indexer: z.string(), indexer: z.string(),
downloadUrl: z.string(), downloadUrl: z.string(),
publishDate: z.string().transform((str) => new Date(str)), publishDate: z.string().transform((str) => new Date(str)),
+11 -24
View File
@@ -88,39 +88,26 @@ export async function POST(request: NextRequest) {
// Rank torrents using the ranking algorithm with indexer priorities and flag configs // Rank torrents using the ranking algorithm with indexer priorities and flag configs
const rankedResults = rankTorrents(results, { title, author }, indexerPriorities, flagConfigs); const rankedResults = rankTorrents(results, { title, author }, indexerPriorities, flagConfigs);
// Dual threshold filtering: // No threshold filtering - show all results like interactive search
// 1. Base score must be >= 50 (quality minimum) // User can see scores and make their own decision
// 2. Final score must be >= 50 (not disqualified by negative bonuses) console.log(`[AudiobookSearch] Ranked ${rankedResults.length} results (no threshold filter - user decides)`);
const filteredResults = rankedResults.filter(result =>
result.score >= 50 && result.finalScore >= 50
);
const disqualifiedByNegativeBonus = rankedResults.filter(result =>
result.score >= 50 && result.finalScore < 50
).length;
console.log(`[AudiobookSearch] Ranked ${rankedResults.length} results, ${filteredResults.length} above threshold (50/100 base + final)`);
if (disqualifiedByNegativeBonus > 0) {
console.log(`[AudiobookSearch] ${disqualifiedByNegativeBonus} torrents disqualified by negative flag bonuses`);
}
// Log top 3 results with detailed score breakdown for debugging // Log top 3 results with detailed score breakdown for debugging
const top3 = filteredResults.slice(0, 3); const top3 = rankedResults.slice(0, 3);
if (top3.length > 0) { if (top3.length > 0) {
console.log(`[AudiobookSearch] ==================== RANKING DEBUG ====================`); console.log(`[AudiobookSearch] ==================== RANKING DEBUG ====================`);
console.log(`[AudiobookSearch] Requested Title: "${title}"`); console.log(`[AudiobookSearch] Requested Title: "${title}"`);
console.log(`[AudiobookSearch] Requested Author: "${author}"`); console.log(`[AudiobookSearch] Requested Author: "${author}"`);
console.log(`[AudiobookSearch] Top ${top3.length} results (out of ${filteredResults.length} above threshold):`); console.log(`[AudiobookSearch] Top ${top3.length} results (out of ${rankedResults.length} total):`);
console.log(`[AudiobookSearch] --------------------------------------------------------`); console.log(`[AudiobookSearch] --------------------------------------------------------`);
top3.forEach((result, index) => { top3.forEach((result, index) => {
console.log(`[AudiobookSearch] ${index + 1}. "${result.title}"`); console.log(`[AudiobookSearch] ${index + 1}. "${result.title}"`);
console.log(`[AudiobookSearch] Indexer: ${result.indexer}${result.indexerId ? ` (ID: ${result.indexerId})` : ''}`); console.log(`[AudiobookSearch] Indexer: ${result.indexer}${result.indexerId ? ` (ID: ${result.indexerId})` : ''}`);
console.log(`[AudiobookSearch] `); console.log(`[AudiobookSearch] `);
console.log(`[AudiobookSearch] Base Score: ${result.score.toFixed(1)}/100`); console.log(`[AudiobookSearch] Base Score: ${result.score.toFixed(1)}/100`);
console.log(`[AudiobookSearch] - Title/Author Match: ${result.breakdown.matchScore.toFixed(1)}/50`); console.log(`[AudiobookSearch] - Title/Author Match: ${result.breakdown.matchScore.toFixed(1)}/60`);
console.log(`[AudiobookSearch] - Format Quality: ${result.breakdown.formatScore.toFixed(1)}/25 (${result.format || 'unknown'})`); console.log(`[AudiobookSearch] - Format Quality: ${result.breakdown.formatScore.toFixed(1)}/25 (${result.format || 'unknown'})`);
console.log(`[AudiobookSearch] - Seeder Count: ${result.breakdown.seederScore.toFixed(1)}/15 (${result.seeders} seeders)`); console.log(`[AudiobookSearch] - Seeder Count: ${result.breakdown.seederScore.toFixed(1)}/15 (${result.seeders !== undefined ? result.seeders + ' seeders' : 'N/A for Usenet'})`);
console.log(`[AudiobookSearch] - Size Score: ${result.breakdown.sizeScore.toFixed(1)}/10 (${(result.size / (1024 ** 3)).toFixed(2)} GB)`);
console.log(`[AudiobookSearch] `); console.log(`[AudiobookSearch] `);
console.log(`[AudiobookSearch] Bonus Points: +${result.bonusPoints.toFixed(1)}`); console.log(`[AudiobookSearch] Bonus Points: +${result.bonusPoints.toFixed(1)}`);
if (result.bonusModifiers.length > 0) { if (result.bonusModifiers.length > 0) {
@@ -141,7 +128,7 @@ export async function POST(request: NextRequest) {
} }
// Add rank position to each result // Add rank position to each result
const resultsWithRank = filteredResults.map((result, index) => ({ const resultsWithRank = rankedResults.map((result, index) => ({
...result, ...result,
rank: index + 1, rank: index + 1,
})); }));
@@ -149,9 +136,9 @@ export async function POST(request: NextRequest) {
return NextResponse.json({ return NextResponse.json({
success: true, success: true,
results: resultsWithRank, results: resultsWithRank,
message: filteredResults.length > 0 message: rankedResults.length > 0
? `Found ${filteredResults.length} quality matches` ? `Found ${rankedResults.length} results`
: 'No quality matches found', : 'No results found',
}); });
} catch (error) { } catch (error) {
console.error('Failed to search for torrents:', error); console.error('Failed to search for torrents:', error);
@@ -141,10 +141,9 @@ export async function POST(
console.log(`[InteractiveSearch] Indexer: ${result.indexer}${result.indexerId ? ` (ID: ${result.indexerId})` : ''}`); console.log(`[InteractiveSearch] Indexer: ${result.indexer}${result.indexerId ? ` (ID: ${result.indexerId})` : ''}`);
console.log(`[InteractiveSearch] `); console.log(`[InteractiveSearch] `);
console.log(`[InteractiveSearch] Base Score: ${result.score.toFixed(1)}/100`); console.log(`[InteractiveSearch] Base Score: ${result.score.toFixed(1)}/100`);
console.log(`[InteractiveSearch] - Title/Author Match: ${result.breakdown.matchScore.toFixed(1)}/50`); console.log(`[InteractiveSearch] - Title/Author Match: ${result.breakdown.matchScore.toFixed(1)}/60`);
console.log(`[InteractiveSearch] - Format Quality: ${result.breakdown.formatScore.toFixed(1)}/25 (${result.format || 'unknown'})`); console.log(`[InteractiveSearch] - Format Quality: ${result.breakdown.formatScore.toFixed(1)}/25 (${result.format || 'unknown'})`);
console.log(`[InteractiveSearch] - Seeder Count: ${result.breakdown.seederScore.toFixed(1)}/15 (${result.seeders} seeders)`); console.log(`[InteractiveSearch] - Seeder Count: ${result.breakdown.seederScore.toFixed(1)}/15 (${result.seeders} seeders)`);
console.log(`[InteractiveSearch] - Size Score: ${result.breakdown.sizeScore.toFixed(1)}/10 (${(result.size / (1024 ** 3)).toFixed(2)} GB)`);
console.log(`[InteractiveSearch] `); console.log(`[InteractiveSearch] `);
console.log(`[InteractiveSearch] Bonus Points: +${result.bonusPoints.toFixed(1)}`); console.log(`[InteractiveSearch] Bonus Points: +${result.bonusPoints.toFixed(1)}`);
if (result.bonusModifiers.length > 0) { if (result.bonusModifiers.length > 0) {
+12
View File
@@ -412,6 +412,18 @@ export async function POST(request: NextRequest) {
}, },
}); });
// Chapter merging configuration
await prisma.configuration.upsert({
where: { key: 'chapter_merging_enabled' },
update: { value: String(paths.chapter_merging_enabled ?? false) },
create: {
key: 'chapter_merging_enabled',
value: String(paths.chapter_merging_enabled ?? false),
category: 'automation',
description: 'Automatically merge multi-file chapter downloads into single M4B with chapter markers'
},
});
// BookDate configuration (optional, global for all users) // BookDate configuration (optional, global for all users)
// Note: libraryScope and customPrompt are now per-user settings, not required here // Note: libraryScope and customPrompt are now per-user settings, not required here
if (bookdate && bookdate.provider && bookdate.apiKey && bookdate.model) { if (bookdate && bookdate.provider && bookdate.apiKey && bookdate.model) {
+4
View File
@@ -84,6 +84,7 @@ interface SetupState {
downloadDir: string; downloadDir: string;
mediaDir: string; mediaDir: string;
metadataTaggingEnabled: boolean; metadataTaggingEnabled: boolean;
chapterMergingEnabled: boolean;
bookdateProvider: string; bookdateProvider: string;
bookdateApiKey: string; bookdateApiKey: string;
bookdateModel: string; bookdateModel: string;
@@ -153,6 +154,7 @@ export default function SetupWizard() {
downloadDir: '/downloads', downloadDir: '/downloads',
mediaDir: '/media/audiobooks', mediaDir: '/media/audiobooks',
metadataTaggingEnabled: true, metadataTaggingEnabled: true,
chapterMergingEnabled: false,
bookdateProvider: 'openai', bookdateProvider: 'openai',
bookdateApiKey: '', bookdateApiKey: '',
bookdateModel: '', bookdateModel: '',
@@ -228,6 +230,7 @@ export default function SetupWizard() {
download_dir: state.downloadDir, download_dir: state.downloadDir,
media_dir: state.mediaDir, media_dir: state.mediaDir,
metadata_tagging_enabled: state.metadataTaggingEnabled, metadata_tagging_enabled: state.metadataTaggingEnabled,
chapter_merging_enabled: state.chapterMergingEnabled,
}, },
bookdate: state.bookdateConfigured ? { bookdate: state.bookdateConfigured ? {
provider: state.bookdateProvider, provider: state.bookdateProvider,
@@ -525,6 +528,7 @@ export default function SetupWizard() {
downloadDir={state.downloadDir} downloadDir={state.downloadDir}
mediaDir={state.mediaDir} mediaDir={state.mediaDir}
metadataTaggingEnabled={state.metadataTaggingEnabled} metadataTaggingEnabled={state.metadataTaggingEnabled}
chapterMergingEnabled={state.chapterMergingEnabled}
onUpdate={updateField} onUpdate={updateField}
onNext={() => goToStep(currentStepNumber + 1)} onNext={() => goToStep(currentStepNumber + 1)}
onBack={() => goToStep(currentStepNumber - 1)} onBack={() => goToStep(currentStepNumber - 1)}
+27
View File
@@ -13,6 +13,7 @@ interface PathsStepProps {
downloadDir: string; downloadDir: string;
mediaDir: string; mediaDir: string;
metadataTaggingEnabled: boolean; metadataTaggingEnabled: boolean;
chapterMergingEnabled: boolean;
onUpdate: (field: string, value: string | boolean) => void; onUpdate: (field: string, value: string | boolean) => void;
onNext: () => void; onNext: () => void;
onBack: () => void; onBack: () => void;
@@ -22,6 +23,7 @@ export function PathsStep({
downloadDir, downloadDir,
mediaDir, mediaDir,
metadataTaggingEnabled, metadataTaggingEnabled,
chapterMergingEnabled,
onUpdate, onUpdate,
onNext, onNext,
onBack, onBack,
@@ -235,6 +237,31 @@ export function PathsStep({
</div> </div>
</div> </div>
{/* Chapter Merging Toggle */}
<div className="bg-gray-50 dark:bg-gray-800 rounded-lg p-4 border border-gray-200 dark:border-gray-700">
<div className="flex items-start gap-4">
<input
type="checkbox"
id="chapter-merging"
checked={chapterMergingEnabled}
onChange={(e) => onUpdate('chapterMergingEnabled', e.target.checked)}
className="mt-1 h-5 w-5 rounded border-gray-300 text-blue-600 focus:ring-blue-500"
/>
<div className="flex-1">
<label
htmlFor="chapter-merging"
className="block text-sm font-medium text-gray-900 dark:text-gray-100 cursor-pointer"
>
Auto-merge chapters to M4B
</label>
<p className="text-sm text-gray-600 dark:text-gray-400 mt-1">
Automatically merge multi-file chapter downloads into a single M4B audiobook with chapter
markers. Improves playback experience and library organization.
</p>
</div>
</div>
</div>
<Button <Button
onClick={testPaths} onClick={testPaths}
loading={testing} loading={testing}
+81 -13
View File
@@ -46,8 +46,8 @@ interface ProwlarrSearchResult {
indexerId?: number; indexerId?: number;
title: string; title: string;
size: number; size: number;
seeders: number; seeders?: number; // Optional for NZB/Usenet results
leechers: number; leechers?: number; // Optional for NZB/Usenet results
publishDate: string; publishDate: string;
downloadUrl?: string; // Torrent file download URL (most indexers) downloadUrl?: string; // Torrent file download URL (most indexers)
magnetUrl?: string; // Magnet link (public trackers like TPB) magnetUrl?: string; // Magnet link (public trackers like TPB)
@@ -57,6 +57,7 @@ interface ProwlarrSearchResult {
downloadVolumeFactor?: number; downloadVolumeFactor?: number;
uploadVolumeFactor?: number; uploadVolumeFactor?: number;
indexerFlags?: string[] | number[]; // Can be string names or numeric IDs indexerFlags?: string[] | number[]; // Can be string names or numeric IDs
protocol?: string; // 'torrent' or 'usenet' - provided by Prowlarr API
[key: string]: any; // Allow any additional fields from Prowlarr API [key: string]: any; // Allow any additional fields from Prowlarr API
} }
@@ -77,9 +78,28 @@ export class ProwlarrService {
}, },
timeout: 30000, // 30 seconds timeout: 30000, // 30 seconds
paramsSerializer: { paramsSerializer: {
indexes: null, // Use repeat format: indexerIds=1&indexerIds=2 (not indexerIds[0]=1) serialize: (params) => {
// Custom serializer to handle arrays correctly for Prowlarr API
// indexerIds=[1,2,3] should become indexerIds=1&indexerIds=2&indexerIds=3
const parts: string[] = [];
for (const [key, value] of Object.entries(params)) {
if (Array.isArray(value)) {
value.forEach(v => parts.push(`${encodeURIComponent(key)}=${encodeURIComponent(v)}`));
} else if (value !== undefined && value !== null) {
parts.push(`${encodeURIComponent(key)}=${encodeURIComponent(value)}`);
}
}
return parts.join('&');
},
}, },
}); });
// Debug interceptor to log actual outgoing requests
this.client.interceptors.request.use((config) => {
console.log(`[Prowlarr] Actual request: ${config.method?.toUpperCase()} ${config.baseURL}${config.url}`);
console.log(`[Prowlarr] Request params:`, JSON.stringify(config.params));
return config;
});
} }
/** /**
@@ -91,25 +111,44 @@ export class ProwlarrService {
filters?: SearchFilters filters?: SearchFilters
): Promise<TorrentResult[]> { ): Promise<TorrentResult[]> {
try { try {
// Get configured download client type to determine if we should filter by category
const { getConfigService } = await import('../services/config.service');
const configService = getConfigService();
const clientType = (await configService.get('download_client_type')) || 'qbittorrent';
const params: Record<string, any> = { const params: Record<string, any> = {
query, query,
categories: filters?.category?.toString() || this.defaultCategory.toString(),
type: 'search', type: 'search',
limit: 100, // Maximum results to return from Prowlarr
extended: 1, // Enable searching in tags, labels, and metadata extended: 1, // Enable searching in tags, labels, and metadata
categories: filters?.category?.toString() || this.defaultCategory.toString(), // 3030 = Audiobooks (standard Newznab category)
}; };
// Filter by specific indexers if provided // Filter by specific indexers if provided
// Pass array directly - axios will serialize as indexerIds=1&indexerIds=2&indexerIds=3
if (filters?.indexerIds && filters.indexerIds.length > 0) { if (filters?.indexerIds && filters.indexerIds.length > 0) {
params.indexerIds = filters.indexerIds; params.indexerIds = filters.indexerIds;
} }
const response = await this.client.get('/search', { params }); const response = await this.client.get('/search', { params });
console.log(`[Prowlarr] Raw API response: ${response.data.length} results`);
// Debug: Log first raw result to see structure (debug mode only) // Debug: Log first raw result to see structure and protocol field
if (response.data.length > 0) {
const firstResult = response.data[0];
console.log(`[Prowlarr] First raw result - protocol: "${firstResult.protocol}", indexer: "${firstResult.indexer}", title: "${firstResult.title?.substring(0, 50)}..."`);
// Check protocol distribution in raw results
const rawProtocols = response.data.reduce((acc: Record<string, number>, r: any) => {
const proto = r.protocol || 'missing';
acc[proto] = (acc[proto] || 0) + 1;
return acc;
}, {});
console.log(`[Prowlarr] Raw protocol distribution:`, JSON.stringify(rawProtocols));
}
// Debug: Log first raw result full structure (debug mode only)
if (process.env.LOG_LEVEL === 'debug' && response.data.length > 0) { if (process.env.LOG_LEVEL === 'debug' && response.data.length > 0) {
console.log('[Prowlarr] Sample raw result from API:', JSON.stringify(response.data[0], null, 2)); console.log('[Prowlarr] Sample raw result from API:', JSON.stringify(response.data[0], null, 2));
console.log(`[Prowlarr] Received ${response.data.length} total results from API`);
} }
// Transform Prowlarr results to our format // Transform Prowlarr results to our format
@@ -130,7 +169,12 @@ export class ProwlarrService {
// Apply additional filters // Apply additional filters
if (filters?.minSeeders) { if (filters?.minSeeders) {
filtered = filtered.filter((r) => r.seeders >= (filters.minSeeders || 0)); // Only apply seeder filter to torrent results (NZB results don't have seeders)
filtered = filtered.filter((r) => {
// Skip filter for NZB results (undefined seeders)
if (r.seeders === undefined) return true;
return r.seeders >= (filters.minSeeders || 0);
});
} }
if (filters?.maxResults) { if (filters?.maxResults) {
@@ -318,6 +362,26 @@ export class ProwlarrService {
const config = await getConfigService(); const config = await getConfigService();
const clientType = (await config.get('download_client_type')) || 'qbittorrent'; const clientType = (await config.get('download_client_type')) || 'qbittorrent';
// Debug: Log protocol distribution
const protocolCounts = results.reduce((acc, r) => {
const proto = r.protocol || 'unknown';
acc[proto] = (acc[proto] || 0) + 1;
return acc;
}, {} as Record<string, number>);
console.log(`[Prowlarr] Protocol distribution in ${results.length} results:`, JSON.stringify(protocolCounts));
// Debug: Log first few results to see their protocols
if (results.length > 0 && results.length <= 5) {
results.forEach((r, i) => {
console.log(`[Prowlarr] Result ${i + 1}: protocol="${r.protocol || 'undefined'}", url="${r.downloadUrl.substring(0, 80)}..."`);
});
} else if (results.length > 5) {
console.log(`[Prowlarr] First 3 results:`);
results.slice(0, 3).forEach((r, i) => {
console.log(`[Prowlarr] ${i + 1}: protocol="${r.protocol || 'undefined'}", isNZB=${ProwlarrService.isNZBResult(r)}`);
});
}
if (clientType === 'sabnzbd') { if (clientType === 'sabnzbd') {
// Filter for NZB results only // Filter for NZB results only
const filtered = results.filter(result => ProwlarrService.isNZBResult(result)); const filtered = results.filter(result => ProwlarrService.isNZBResult(result));
@@ -340,6 +404,12 @@ export class ProwlarrService {
* Static method for protocol detection * Static method for protocol detection
*/ */
static isNZBResult(result: TorrentResult): boolean { static isNZBResult(result: TorrentResult): boolean {
// Check protocol field first (most reliable - provided by Prowlarr API)
if (result.protocol) {
return result.protocol.toLowerCase() === 'usenet';
}
// Fallback to URL pattern detection if protocol not provided
const url = result.downloadUrl.toLowerCase(); const url = result.downloadUrl.toLowerCase();
// Check file extension // Check file extension
@@ -347,14 +417,11 @@ export class ProwlarrService {
return true; return true;
} }
// Check URL path // Check URL path patterns common in Newznab APIs
if (url.includes('/nzb/') || url.includes('&t=get')) { if (url.includes('/nzb/') || url.includes('&t=get') || url.includes('/getnzb')) {
return true; return true;
} }
// Check categories (3030 is audiobooks, but some indexers use Usenet-specific codes)
// Note: This is less reliable, so we prioritize URL patterns
return false; return false;
} }
@@ -394,6 +461,7 @@ export class ProwlarrService {
bitrate: metadata.bitrate, bitrate: metadata.bitrate,
hasChapters: metadata.hasChapters, hasChapters: metadata.hasChapters,
flags: flags.length > 0 ? flags : undefined, flags: flags.length > 0 ? flags : undefined,
protocol: result.protocol, // 'torrent' or 'usenet'
}; };
} catch (error) { } catch (error) {
console.error('Failed to transform result:', result, error); console.error('Failed to transform result:', result, error);
+2
View File
@@ -267,6 +267,8 @@ export class SABnzbdService {
* Returns the NZB ID * Returns the NZB ID
*/ */
async addNZB(url: string, options?: AddNZBOptions): Promise<string> { async addNZB(url: string, options?: AddNZBOptions): Promise<string> {
console.log(`[SABnzbd] Adding NZB from URL: ${url.substring(0, 150)}...`);
const response = await this.client.get('/api', { const response = await this.client.get('/api', {
params: { params: {
mode: 'addurl', mode: 'addurl',
@@ -132,7 +132,7 @@ export async function processDownloadTorrent(payload: DownloadTorrentPayload): P
torrentSizeBytes: torrent.size, torrentSizeBytes: torrent.size,
torrentUrl: torrent.guid, torrentUrl: torrent.guid,
magnetLink: torrent.downloadUrl, magnetLink: torrent.downloadUrl,
seeders: torrent.seeders, seeders: torrent.seeders || 0,
leechers: torrent.leechers || 0, leechers: torrent.leechers || 0,
downloadStatus: 'downloading', downloadStatus: 'downloading',
selected: true, selected: true,
@@ -163,7 +163,7 @@ export async function processDownloadTorrent(payload: DownloadTorrentPayload): P
torrent: { torrent: {
title: torrent.title, title: torrent.title,
size: torrent.size, size: torrent.size,
seeders: torrent.seeders, seeders: torrent.seeders || 0,
format: torrent.format, format: torrent.format,
}, },
}; };
@@ -104,7 +104,6 @@ export async function processSearchIndexers(payload: SearchIndexersPayload): Pro
const rankedResults = ranker.rankTorrents(searchResults, { const rankedResults = ranker.rankTorrents(searchResults, {
title: audiobook.title, title: audiobook.title,
author: audiobook.author, author: audiobook.author,
durationMinutes: undefined, // We don't have duration from Audible
}, indexerPriorities, flagConfigs); }, indexerPriorities, flagConfigs);
// Dual threshold filtering: // Dual threshold filtering:
@@ -160,10 +159,9 @@ export async function processSearchIndexers(payload: SearchIndexersPayload): Pro
await logger?.info(` Indexer: ${result.indexer}${result.indexerId ? ` (ID: ${result.indexerId})` : ''}`); await logger?.info(` Indexer: ${result.indexer}${result.indexerId ? ` (ID: ${result.indexerId})` : ''}`);
await logger?.info(``); await logger?.info(``);
await logger?.info(` Base Score: ${result.score.toFixed(1)}/100`); await logger?.info(` Base Score: ${result.score.toFixed(1)}/100`);
await logger?.info(` - Title/Author Match: ${result.breakdown.matchScore.toFixed(1)}/50`); await logger?.info(` - Title/Author Match: ${result.breakdown.matchScore.toFixed(1)}/60`);
await logger?.info(` - Format Quality: ${result.breakdown.formatScore.toFixed(1)}/25 (${result.format || 'unknown'})`); await logger?.info(` - Format Quality: ${result.breakdown.formatScore.toFixed(1)}/25 (${result.format || 'unknown'})`);
await logger?.info(` - Seeder Count: ${result.breakdown.seederScore.toFixed(1)}/15 (${result.seeders} seeders)`); await logger?.info(` - Seeder Count: ${result.breakdown.seederScore.toFixed(1)}/15 (${result.seeders !== undefined ? result.seeders + ' seeders' : 'N/A for Usenet'})`);
await logger?.info(` - Size Score: ${result.breakdown.sizeScore.toFixed(1)}/10`);
await logger?.info(``); await logger?.info(``);
await logger?.info(` Bonus Points: +${result.bonusPoints.toFixed(1)}`); await logger?.info(` Bonus Points: +${result.bonusPoints.toFixed(1)}`);
if (result.bonusModifiers.length > 0) { if (result.bonusModifiers.length > 0) {
@@ -199,7 +197,7 @@ export async function processSearchIndexers(payload: SearchIndexersPayload): Pro
selectedTorrent: { selectedTorrent: {
title: bestResult.title, title: bestResult.title,
score: bestResult.score, score: bestResult.score,
seeders: bestResult.seeders, seeders: bestResult.seeders || 0,
format: bestResult.format, format: bestResult.format,
}, },
}; };
+19 -4
View File
@@ -344,11 +344,18 @@ async function searchByAsin(
const html = await fetchHtml(searchUrl, flaresolverrUrl, logger); const html = await fetchHtml(searchUrl, flaresolverrUrl, logger);
const $ = cheerio.load(html); const $ = cheerio.load(html);
// Exclude MD5 links from "Recent downloads" banner (they're in .js-recent-downloads-container) // Exclude MD5 links from "Recent downloads" banner and "Partial matches" section
// Only look for actual search result links // Only look for actual search result links
const searchResultLinks = $('a[href*="/md5/"]').filter((i, elem) => { const searchResultLinks = $('a[href*="/md5/"]').filter((i, elem) => {
// Exclude links inside the recent downloads banner // Exclude links inside the recent downloads banner
return $(elem).closest('.js-recent-downloads-container').length === 0; if ($(elem).closest('.js-recent-downloads-container').length > 0) {
return false;
}
// Exclude links inside the partial matches section
if ($(elem).closest('.js-partial-matches-show').length > 0) {
return false;
}
return true;
}); });
if (DEBUG_ENABLED) { if (DEBUG_ENABLED) {
@@ -451,9 +458,17 @@ async function searchByTitle(
const html = await fetchHtml(searchUrl, flaresolverrUrl, logger); const html = await fetchHtml(searchUrl, flaresolverrUrl, logger);
const $ = cheerio.load(html); const $ = cheerio.load(html);
// Exclude MD5 links from "Recent downloads" banner (they're in .js-recent-downloads-container) // Exclude MD5 links from "Recent downloads" banner and "Partial matches" section
const searchResultLinks = $('a[href*="/md5/"]').filter((i, elem) => { const searchResultLinks = $('a[href*="/md5/"]').filter((i, elem) => {
return $(elem).closest('.js-recent-downloads-container').length === 0; // Exclude links inside the recent downloads banner
if ($(elem).closest('.js-recent-downloads-container').length > 0) {
return false;
}
// Exclude links inside the partial matches section
if ($(elem).closest('.js-partial-matches-show').length > 0) {
return false;
}
return true;
}); });
if (DEBUG_ENABLED) { if (DEBUG_ENABLED) {
+547
View File
@@ -0,0 +1,547 @@
/**
* Component: Chapter Merger Utility
* Documentation: documentation/features/chapter-merging.md
*
* Merges multi-file audiobook chapter downloads into a single M4B file
* with proper chapter markers.
*/
import { exec } from 'child_process';
import { promisify } from 'util';
import path from 'path';
import fs from 'fs/promises';
import { JobLogger } from './job-logger';
const execPromise = promisify(exec);
// Supported audio formats for chapter merging
const SUPPORTED_FORMATS = ['.mp3', '.m4a', '.m4b', '.mp4', '.aac'];
// Patterns that indicate chapter-based files
const CHAPTER_PATTERNS = [
/^(\d{1,3})[\s._-]/, // "01 - Title.mp3", "1.mp3", "001_chapter.mp3"
/chapter\s*(\d+)/i, // "Chapter 1.mp3", "chapter01.mp3"
/ch\s*(\d+)/i, // "Ch1.mp3", "ch 01.mp3"
/part\s*(\d+)/i, // "Part 1.mp3"
/disc\s*(\d+)/i, // "Disc 1.mp3"
/track\s*(\d+)/i, // "Track 1.mp3"
];
// Generic title patterns to ignore when extracting chapter names
const GENERIC_TITLE_PATTERNS = [
/^track\s*\d+$/i,
/^chapter\s*\d+$/i,
/^\d+$/,
/^part\s*\d+$/i,
];
export interface ChapterFile {
path: string;
filename: string;
duration: number; // milliseconds
bitrate?: number; // kbps
trackNumber?: number; // from metadata
titleMetadata?: string; // from metadata
chapterTitle: string; // final computed title
}
export interface AudioProbeResult {
duration: number; // milliseconds
bitrate?: number; // kbps
trackNumber?: number;
title?: string;
format: string;
}
export interface MergeOptions {
title: string;
author: string;
narrator?: string;
year?: number;
asin?: string;
outputPath: string;
}
export interface MergeResult {
success: boolean;
outputPath?: string;
chapterCount?: number;
totalDuration?: number; // milliseconds
error?: string;
}
/**
* Detect if the given files appear to be chapter files that should be merged
*/
export async function detectChapterFiles(files: string[]): Promise<boolean> {
// Need at least 2 files to merge
if (files.length < 2) {
return false;
}
// All files must have same audio format
const extensions = new Set(files.map(f => path.extname(f).toLowerCase()));
if (extensions.size > 1) {
return false;
}
// Must be a supported format
const ext = [...extensions][0];
if (!SUPPORTED_FORMATS.includes(ext)) {
return false;
}
// Check if files match chapter patterns
const filenames = files.map(f => path.basename(f));
const matchingFiles = filenames.filter(filename =>
CHAPTER_PATTERNS.some(pattern => pattern.test(filename))
);
// At least 80% of files should match chapter patterns
const matchRatio = matchingFiles.length / filenames.length;
return matchRatio >= 0.8;
}
/**
* Probe an audio file to extract duration and metadata
*/
export async function probeAudioFile(filePath: string): Promise<AudioProbeResult> {
const command = `ffprobe -v quiet -print_format json -show_format -show_streams "${filePath}"`;
try {
const { stdout } = await execPromise(command, { timeout: 30000 });
const data = JSON.parse(stdout);
const format = data.format || {};
const tags = format.tags || {};
// Duration in milliseconds
const duration = Math.round((parseFloat(format.duration) || 0) * 1000);
// Bitrate in kbps
const bitrate = format.bit_rate ? Math.round(parseInt(format.bit_rate) / 1000) : undefined;
// Track number (various possible tag names)
let trackNumber: number | undefined;
const trackStr = tags.track || tags.TRACK || tags['track-number'];
if (trackStr) {
// Handle "1/10" format
const match = String(trackStr).match(/^(\d+)/);
if (match) {
trackNumber = parseInt(match[1]);
}
}
// Title
const title = tags.title || tags.TITLE || undefined;
// File extension as format indicator
const fileFormat = path.extname(filePath).toLowerCase().slice(1);
return {
duration,
bitrate,
trackNumber,
title,
format: fileFormat,
};
} catch (error) {
throw new Error(`Failed to probe audio file: ${error instanceof Error ? error.message : 'Unknown error'}`);
}
}
/**
* Natural sort comparison for filenames
* Handles numeric sequences correctly: ch1, ch2, ch10 (not ch1, ch10, ch2)
*/
function naturalSortCompare(a: string, b: string): number {
const aParts = a.split(/(\d+)/);
const bParts = b.split(/(\d+)/);
for (let i = 0; i < Math.max(aParts.length, bParts.length); i++) {
const aPart = aParts[i] || '';
const bPart = bParts[i] || '';
// Check if both parts are numeric
const aNum = parseInt(aPart);
const bNum = parseInt(bPart);
if (!isNaN(aNum) && !isNaN(bNum)) {
if (aNum !== bNum) return aNum - bNum;
} else {
const cmp = aPart.localeCompare(bPart, undefined, { sensitivity: 'base' });
if (cmp !== 0) return cmp;
}
}
return 0;
}
/**
* Check if a title is generic (should be ignored)
*/
function isGenericTitle(title: string): boolean {
return GENERIC_TITLE_PATTERNS.some(pattern => pattern.test(title.trim()));
}
/**
* Extract chapter name from filename
*/
function extractChapterNameFromFilename(filename: string): string | null {
const basename = path.basename(filename, path.extname(filename));
// Try to extract meaningful name after chapter indicator
// "01 - The Beginning" -> "The Beginning"
// "Chapter 1 - Introduction" -> "Introduction"
const patterns = [
/^\d+[\s._-]+(.+)$/, // "01 - Title" or "01_Title"
/^chapter\s*\d+[\s._-]+(.+)$/i, // "Chapter 1 - Title"
/^ch\s*\d+[\s._-]+(.+)$/i, // "Ch1 - Title"
/^part\s*\d+[\s._-]+(.+)$/i, // "Part 1 - Title"
];
for (const pattern of patterns) {
const match = basename.match(pattern);
if (match && match[1]) {
const extracted = match[1].trim();
if (extracted.length > 0 && !isGenericTitle(extracted)) {
return extracted;
}
}
}
return null;
}
/**
* Get chapter title with priority: metadata > filename > fallback
*/
function getChapterTitle(file: ChapterFile, index: number): string {
// Priority 1: Title metadata (if meaningful)
if (file.titleMetadata && !isGenericTitle(file.titleMetadata)) {
return file.titleMetadata;
}
// Priority 2: Extract from filename
const extracted = extractChapterNameFromFilename(file.filename);
if (extracted) {
return extracted;
}
// Priority 3: Fallback to "Chapter X"
return `Chapter ${index + 1}`;
}
/**
* Analyze and order chapter files
* Returns files in correct order with metadata populated
*/
export async function analyzeChapterFiles(
filePaths: string[],
logger?: JobLogger
): Promise<ChapterFile[]> {
// Probe all files in parallel
const probePromises = filePaths.map(async (filePath) => {
const probe = await probeAudioFile(filePath);
return {
path: filePath,
filename: path.basename(filePath),
duration: probe.duration,
bitrate: probe.bitrate,
trackNumber: probe.trackNumber,
titleMetadata: probe.title,
chapterTitle: '', // Will be computed after ordering
};
});
const files = await Promise.all(probePromises);
// Create filename-based order (natural sort)
const filenameOrder = [...files].sort((a, b) =>
naturalSortCompare(a.filename, b.filename)
);
// Check if metadata order is available and valid
const hasAllTrackNumbers = files.every(f => f.trackNumber !== undefined && f.trackNumber > 0);
let useMetadataOrder = false;
let metadataOrder: ChapterFile[] = [];
if (hasAllTrackNumbers) {
metadataOrder = [...files].sort((a, b) => (a.trackNumber || 0) - (b.trackNumber || 0));
// Check if track numbers are sequential
const isSequential = metadataOrder.every((f, i) => {
const expectedTrack = i + 1;
return f.trackNumber === expectedTrack;
});
if (isSequential) {
// Compare orders
const ordersMatch = filenameOrder.every((f, i) => f.path === metadataOrder[i].path);
if (ordersMatch) {
await logger?.info('Chapter ordering: filename and metadata orders match - high confidence');
} else {
await logger?.warn('Chapter ordering: filename order differs from metadata - using metadata order (more reliable)');
useMetadataOrder = true;
}
} else {
await logger?.warn('Chapter ordering: metadata track numbers not sequential - using filename order');
}
} else {
await logger?.info('Chapter ordering: incomplete metadata track numbers - using filename order');
}
// Use the determined order
const orderedFiles = useMetadataOrder ? metadataOrder : filenameOrder;
// Compute chapter titles
for (let i = 0; i < orderedFiles.length; i++) {
orderedFiles[i].chapterTitle = getChapterTitle(orderedFiles[i], i);
}
return orderedFiles;
}
/**
* Generate FFMETADATA1 format chapter metadata
*/
function generateChapterMetadata(chapters: ChapterFile[]): string {
let metadata = ';FFMETADATA1\n';
let currentTime = 0; // milliseconds
for (const chapter of chapters) {
const startTime = currentTime;
const endTime = currentTime + chapter.duration;
// Escape special characters in title
const escapedTitle = chapter.chapterTitle
.replace(/\\/g, '\\\\')
.replace(/=/g, '\\=')
.replace(/;/g, '\\;')
.replace(/#/g, '\\#')
.replace(/\n/g, '');
metadata += '\n[CHAPTER]\n';
metadata += 'TIMEBASE=1/1000\n';
metadata += `START=${startTime}\n`;
metadata += `END=${endTime}\n`;
metadata += `title=${escapedTitle}\n`;
currentTime = endTime;
}
return metadata;
}
/**
* Determine optimal bitrate for MP3 conversion
* Uses source bitrate if < 128kbps, otherwise 128k
*/
function determineOutputBitrate(chapters: ChapterFile[]): string {
// Find minimum bitrate across all files
const bitrates = chapters
.filter(c => c.bitrate !== undefined)
.map(c => c.bitrate as number);
if (bitrates.length === 0) {
return '128k';
}
const minBitrate = Math.min(...bitrates);
// Use source bitrate if lower than 128k, otherwise cap at 128k
if (minBitrate < 128) {
return `${minBitrate}k`;
}
return '128k';
}
/**
* Merge chapter files into a single M4B with chapter markers
*/
export async function mergeChapters(
chapters: ChapterFile[],
options: MergeOptions,
logger?: JobLogger
): Promise<MergeResult> {
if (chapters.length === 0) {
return { success: false, error: 'No chapters to merge' };
}
const tempDir = path.dirname(options.outputPath);
const concatFile = path.join(tempDir, `concat_${Date.now()}.txt`);
const metadataFile = path.join(tempDir, `chapters_${Date.now()}.txt`);
try {
// Ensure temp directory exists
await fs.mkdir(tempDir, { recursive: true });
// Create concat file
const concatContent = chapters
.map(c => `file '${c.path.replace(/'/g, "'\\''")}'`)
.join('\n');
await fs.writeFile(concatFile, concatContent);
// Create chapter metadata file
const chapterMetadata = generateChapterMetadata(chapters);
await fs.writeFile(metadataFile, chapterMetadata);
// Determine if we need to re-encode (MP3 input requires conversion to AAC)
const inputFormat = path.extname(chapters[0].path).toLowerCase();
const needsReencode = inputFormat === '.mp3';
// Build ffmpeg command
const args: string[] = [
'ffmpeg',
'-y',
'-f', 'concat',
'-safe', '0',
'-i', `"${concatFile}"`,
'-i', `"${metadataFile}"`,
'-map_metadata', '1',
];
if (needsReencode) {
// MP3 -> M4B requires re-encoding to AAC
const bitrate = determineOutputBitrate(chapters);
args.push('-codec:a', 'aac', '-b:a', bitrate);
await logger?.info(`Re-encoding MP3 to AAC at ${bitrate}`);
} else {
// M4A/M4B -> M4B can use codec copy (fast, lossless)
args.push('-codec', 'copy');
await logger?.info('Using codec copy (no re-encoding)');
}
// Add book metadata
const escapeMetadata = (val: string): string =>
val.replace(/"/g, '\\"').replace(/'/g, "\\'");
args.push('-metadata', `title="${escapeMetadata(options.title)}"`);
args.push('-metadata', `album="${escapeMetadata(options.title)}"`);
args.push('-metadata', `album_artist="${escapeMetadata(options.author)}"`);
args.push('-metadata', `artist="${escapeMetadata(options.author)}"`);
if (options.narrator) {
args.push('-metadata', `composer="${escapeMetadata(options.narrator)}"`);
}
if (options.year) {
args.push('-metadata', `date="${options.year}"`);
}
if (options.asin) {
// Custom iTunes tag for ASIN
args.push('-metadata', `----:com.apple.iTunes:ASIN="${escapeMetadata(options.asin)}"`);
}
// Output format
args.push('-f', 'mp4');
args.push(`"${options.outputPath}"`);
const command = args.join(' ');
// Calculate timeout: base 5 minutes + 30 seconds per chapter
const timeout = (5 * 60 * 1000) + (chapters.length * 30 * 1000);
await logger?.info(`Merging ${chapters.length} chapters...`);
try {
await execPromise(command, { timeout });
} catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
throw new Error(`FFmpeg merge failed: ${errorMsg}`);
}
// Verify output file exists
try {
await fs.access(options.outputPath);
} catch {
throw new Error('Merged file not created');
}
// Calculate total duration
const totalDuration = chapters.reduce((sum, c) => sum + c.duration, 0);
await logger?.info(`Merge complete: ${chapters.length} chapters, ${formatDuration(totalDuration)}`);
return {
success: true,
outputPath: options.outputPath,
chapterCount: chapters.length,
totalDuration,
};
} catch (error) {
const errorMsg = error instanceof Error ? error.message : 'Unknown error';
return { success: false, error: errorMsg };
} finally {
// Clean up temp files
try {
await fs.unlink(concatFile);
} catch {
// Ignore cleanup errors
}
try {
await fs.unlink(metadataFile);
} catch {
// Ignore cleanup errors
}
}
}
/**
* Format duration in milliseconds to human readable string
*/
export function formatDuration(ms: number): string {
const seconds = Math.floor(ms / 1000);
const hours = Math.floor(seconds / 3600);
const minutes = Math.floor((seconds % 3600) / 60);
const secs = seconds % 60;
if (hours > 0) {
return `${hours}h ${minutes}m ${secs}s`;
} else if (minutes > 0) {
return `${minutes}m ${secs}s`;
}
return `${secs}s`;
}
/**
* Check available disk space in directory
* Returns available bytes, or null if unable to determine
*/
export async function checkDiskSpace(directory: string): Promise<number | null> {
try {
// Use df on Unix-like systems
const { stdout } = await execPromise(`df -k "${directory}" | tail -1 | awk '{print $4}'`);
const availableKb = parseInt(stdout.trim());
if (!isNaN(availableKb)) {
return availableKb * 1024; // Convert to bytes
}
} catch {
// df not available (Windows) or other error
}
return null;
}
/**
* Estimate output file size (sum of inputs + 10% overhead)
*/
export async function estimateOutputSize(filePaths: string[]): Promise<number> {
let totalSize = 0;
for (const filePath of filePaths) {
try {
const stats = await fs.stat(filePath);
totalSize += stats.size;
} catch {
// Ignore errors, estimate conservatively
}
}
// Add 10% overhead for metadata and format differences
return Math.ceil(totalSize * 1.1);
}
+111 -2
View File
@@ -8,6 +8,14 @@ import path from 'path';
import axios from 'axios'; import axios from 'axios';
import { createJobLogger, JobLogger } from './job-logger'; import { createJobLogger, JobLogger } from './job-logger';
import { tagMultipleFiles, checkFfmpegAvailable } from './metadata-tagger'; import { tagMultipleFiles, checkFfmpegAvailable } from './metadata-tagger';
import {
detectChapterFiles,
analyzeChapterFiles,
mergeChapters,
formatDuration,
estimateOutputSize,
checkDiskSpace,
} from './chapter-merger';
import { prisma } from '../db'; import { prisma } from '../db';
import { downloadEbook } from '../services/ebook-scraper'; import { downloadEbook } from '../services/ebook-scraper';
@@ -83,6 +91,86 @@ export class FileOrganizer {
// Determine base path for source files // Determine base path for source files
const baseSourcePath = isFile ? path.dirname(downloadPath) : downloadPath; const baseSourcePath = isFile ? path.dirname(downloadPath) : downloadPath;
// Track if we created a merged file that needs cleanup
let tempMergedFile: string | null = null;
// Check for chapter merging if multiple files
if (audioFiles.length > 1) {
try {
const chapterMergingConfig = await prisma.configuration.findUnique({
where: { key: 'chapter_merging_enabled' },
});
const chapterMergingEnabled = chapterMergingConfig?.value === 'true';
if (chapterMergingEnabled) {
// Build full paths to source files
const sourceFilePaths = audioFiles.map((audioFile) =>
isFile ? downloadPath : path.join(downloadPath, audioFile)
);
const isChapterDownload = await detectChapterFiles(sourceFilePaths);
if (isChapterDownload) {
await logger?.info(`Detected ${audioFiles.length} chapter files, attempting merge...`);
// Check disk space
const estimatedSize = await estimateOutputSize(sourceFilePaths);
const availableSpace = await checkDiskSpace(this.tempDir);
if (availableSpace !== null && availableSpace < estimatedSize) {
await logger?.warn(`Insufficient disk space for merge (need ${Math.round(estimatedSize / 1024 / 1024)}MB, have ${Math.round(availableSpace / 1024 / 1024)}MB). Skipping merge.`);
} else {
// Analyze and order chapter files
const chapters = await analyzeChapterFiles(sourceFilePaths, logger ?? undefined);
// Create output path in temp directory
const outputFilename = `${this.sanitizePath(audiobook.title)}.m4b`;
const outputPath = path.join(this.tempDir, outputFilename);
// Perform merge
const mergeResult = await mergeChapters(
chapters,
{
title: audiobook.title,
author: audiobook.author,
narrator: audiobook.narrator,
year: audiobook.year,
asin: audiobook.asin,
outputPath,
},
logger ?? undefined
);
if (mergeResult.success && mergeResult.outputPath) {
await logger?.info(
`Merge successful: ${mergeResult.chapterCount} chapters, ${formatDuration(mergeResult.totalDuration || 0)}`
);
// Replace audioFiles array with single merged file
audioFiles.length = 0;
audioFiles.push(mergeResult.outputPath);
// Mark for cleanup after copy
tempMergedFile = mergeResult.outputPath;
// Update isFile flag since we now have a single file path
// (not in the download directory structure)
} else {
await logger?.warn(`Chapter merge failed: ${mergeResult.error}. Falling back to individual files.`);
result.errors.push(`Chapter merge failed: ${mergeResult.error}`);
// Continue with original audioFiles array
}
}
}
}
} catch (error) {
await logger?.error(`Chapter merging error: ${error instanceof Error ? error.message : 'Unknown error'}`);
result.errors.push(`Chapter merging error: ${error instanceof Error ? error.message : 'Unknown error'}`);
// Continue with original audioFiles array
}
}
// Tag metadata BEFORE moving files (prevents Plex race condition) // Tag metadata BEFORE moving files (prevents Plex race condition)
// Map from original file path to tagged file path (for successful tags) // Map from original file path to tagged file path (for successful tags)
const taggedFileMap = new Map<string, string>(); const taggedFileMap = new Map<string, string>();
@@ -103,8 +191,13 @@ export class FileOrganizer {
await logger?.info(`Tagging ${audioFiles.length} audio files with metadata (before move)...`); await logger?.info(`Tagging ${audioFiles.length} audio files with metadata (before move)...`);
// Build full paths to source files for tagging // Build full paths to source files for tagging
// Handle merged files (absolute paths) vs original files (relative paths)
const sourceFilePaths = audioFiles.map((audioFile) => const sourceFilePaths = audioFiles.map((audioFile) =>
isFile ? downloadPath : path.join(downloadPath, audioFile) path.isAbsolute(audioFile)
? audioFile // Merged file - use path directly
: isFile
? downloadPath
: path.join(downloadPath, audioFile)
); );
const taggingResults = await tagMultipleFiles(sourceFilePaths, { const taggingResults = await tagMultipleFiles(sourceFilePaths, {
@@ -167,7 +260,13 @@ export class FileOrganizer {
// Copy audio files (do NOT delete originals - needed for seeding) // Copy audio files (do NOT delete originals - needed for seeding)
for (const audioFile of audioFiles) { for (const audioFile of audioFiles) {
const originalSourcePath = isFile ? downloadPath : path.join(downloadPath, audioFile); // Handle merged files (absolute paths) vs original files (relative paths)
const isAbsolutePath = path.isAbsolute(audioFile);
const originalSourcePath = isAbsolutePath
? audioFile // Merged file - use path directly
: isFile
? downloadPath
: path.join(downloadPath, audioFile);
const filename = path.basename(audioFile); const filename = path.basename(audioFile);
const targetFilePath = path.join(targetPath, filename); const targetFilePath = path.join(targetPath, filename);
@@ -234,6 +333,16 @@ export class FileOrganizer {
} }
} }
// Clean up temp merged file after successful copy
if (tempMergedFile) {
try {
await fs.unlink(tempMergedFile);
await logger?.info(`Cleaned up temp merged file: ${path.basename(tempMergedFile)}`);
} catch (cleanupError) {
await logger?.warn(`Failed to clean up temp merged file: ${path.basename(tempMergedFile)}`);
}
}
// Handle cover art // Handle cover art
if (coverFile) { if (coverFile) {
const sourcePath = path.join(baseSourcePath, coverFile); const sourcePath = path.join(baseSourcePath, coverFile);
+31 -60
View File
@@ -10,8 +10,8 @@ export interface TorrentResult {
indexerId?: number; indexerId?: number;
title: string; title: string;
size: number; size: number;
seeders: number; seeders?: number; // Optional for NZB/Usenet results (no seeders concept)
leechers: number; leechers?: number; // Optional for NZB/Usenet results (no leechers concept)
publishDate: Date; publishDate: Date;
downloadUrl: string; downloadUrl: string;
infoUrl?: string; // Link to indexer's info page (for user reference) infoUrl?: string; // Link to indexer's info page (for user reference)
@@ -21,6 +21,7 @@ export interface TorrentResult {
bitrate?: string; bitrate?: string;
hasChapters?: boolean; hasChapters?: boolean;
flags?: string[]; // Indexer flags like "Freeleech", "Internal", etc. flags?: string[]; // Indexer flags like "Freeleech", "Internal", etc.
protocol?: string; // 'torrent' or 'usenet' - from Prowlarr API
} }
export interface AudiobookRequest { export interface AudiobookRequest {
@@ -45,7 +46,6 @@ export interface BonusModifier {
export interface ScoreBreakdown { export interface ScoreBreakdown {
formatScore: number; formatScore: number;
seederScore: number; seederScore: number;
sizeScore: number;
matchScore: number; matchScore: number;
totalScore: number; totalScore: number;
notes: string[]; notes: string[];
@@ -78,10 +78,9 @@ export class RankingAlgorithm {
// Calculate base scores (0-100) // Calculate base scores (0-100)
const formatScore = this.scoreFormat(torrent); const formatScore = this.scoreFormat(torrent);
const seederScore = this.scoreSeeders(torrent.seeders); const seederScore = this.scoreSeeders(torrent.seeders);
const sizeScore = this.scoreSize(torrent.size, audiobook.durationMinutes);
const matchScore = this.scoreMatch(torrent, audiobook); const matchScore = this.scoreMatch(torrent, audiobook);
const baseScore = formatScore + seederScore + sizeScore + matchScore; const baseScore = formatScore + seederScore + matchScore;
// Calculate bonus modifiers // Calculate bonus modifiers
const bonusModifiers: BonusModifier[] = []; const bonusModifiers: BonusModifier[] = [];
@@ -138,13 +137,11 @@ export class RankingAlgorithm {
breakdown: { breakdown: {
formatScore, formatScore,
seederScore, seederScore,
sizeScore,
matchScore, matchScore,
totalScore: baseScore, totalScore: baseScore,
notes: this.generateNotes(torrent, { notes: this.generateNotes(torrent, {
formatScore, formatScore,
seederScore, seederScore,
sizeScore,
matchScore, matchScore,
totalScore: baseScore, totalScore: baseScore,
notes: [], notes: [],
@@ -180,20 +177,17 @@ export class RankingAlgorithm {
): ScoreBreakdown { ): ScoreBreakdown {
const formatScore = this.scoreFormat(torrent); const formatScore = this.scoreFormat(torrent);
const seederScore = this.scoreSeeders(torrent.seeders); const seederScore = this.scoreSeeders(torrent.seeders);
const sizeScore = this.scoreSize(torrent.size, audiobook.durationMinutes);
const matchScore = this.scoreMatch(torrent, audiobook); const matchScore = this.scoreMatch(torrent, audiobook);
const totalScore = formatScore + seederScore + sizeScore + matchScore; const totalScore = formatScore + seederScore + matchScore;
return { return {
formatScore, formatScore,
seederScore, seederScore,
sizeScore,
matchScore, matchScore,
totalScore, totalScore,
notes: this.generateNotes(torrent, { notes: this.generateNotes(torrent, {
formatScore, formatScore,
seederScore, seederScore,
sizeScore,
matchScore, matchScore,
totalScore, totalScore,
notes: [], notes: [],
@@ -231,43 +225,23 @@ export class RankingAlgorithm {
* 10 seeders: 6 points * 10 seeders: 6 points
* 100 seeders: 12 points * 100 seeders: 12 points
* 1000+ seeders: 15 points * 1000+ seeders: 15 points
*
* Note: NZB/Usenet results don't have seeders concept - centralized servers provide guaranteed availability
*/ */
private scoreSeeders(seeders: number): number { private scoreSeeders(seeders: number | undefined): number {
// Handle undefined/null (NZB results) - give full score since Usenet has centralized availability
if (seeders === undefined || seeders === null || isNaN(seeders)) {
return 15; // Full score - Usenet doesn't need seeders, content is on centralized servers
}
if (seeders === 0) return 0; if (seeders === 0) return 0;
return Math.min(15, Math.log10(seeders + 1) * 6); return Math.min(15, Math.log10(seeders + 1) * 6);
} }
/**
* Score size reasonableness (10 points max)
* Expected: 1-2 MB per minute (64-128 kbps)
* Perfect match: 10 points
* Too small/large: Reduced points
*/
private scoreSize(size: number, durationMinutes?: number): number {
if (!durationMinutes) {
return 5; // Neutral score if duration unknown
}
// Expected size: 1-2 MB per minute
const minExpected = durationMinutes * 1024 * 1024; // 1 MB/min
const maxExpected = durationMinutes * 2 * 1024 * 1024; // 2 MB/min
if (size >= minExpected && size <= maxExpected) {
return 10; // Perfect size
}
// Calculate deviation penalty
const deviation =
size < minExpected
? (minExpected - size) / minExpected
: (size - maxExpected) / maxExpected;
return Math.max(0, 10 - deviation * 10);
}
/** /**
* Score title/author match quality (50 points max) * Score title/author match quality (60 points max)
* Title similarity: 0-35 points (heavily weighted!) * Title similarity: 0-45 points (heavily weighted!)
* Author presence: 0-15 points * Author presence: 0-15 points
*/ */
private scoreMatch( private scoreMatch(
@@ -392,7 +366,7 @@ export class RankingAlgorithm {
if (isCompleteTitle) { if (isCompleteTitle) {
// Complete title match → full points // Complete title match → full points
titleScore = 35; titleScore = 45;
bestMatch = true; bestMatch = true;
break; // Found a good match, stop trying break; // Found a good match, stop trying
} }
@@ -403,7 +377,7 @@ export class RankingAlgorithm {
// No complete match found, use fuzzy similarity as fallback // No complete match found, use fuzzy similarity as fallback
// Try against full title first, then required title // Try against full title first, then required title
const fuzzyScores = titlesToTry.map(title => compareTwoStrings(title, torrentTitle)); const fuzzyScores = titlesToTry.map(title => compareTwoStrings(title, torrentTitle));
titleScore = Math.max(...fuzzyScores) * 35; titleScore = Math.max(...fuzzyScores) * 45;
} }
// ========== STAGE 3: AUTHOR MATCHING (0-15 points) ========== // ========== STAGE 3: AUTHOR MATCHING (0-15 points) ==========
@@ -427,7 +401,7 @@ export class RankingAlgorithm {
authorScore = compareTwoStrings(requestAuthor, torrentTitle) * 15; authorScore = compareTwoStrings(requestAuthor, torrentTitle) * 15;
} }
return Math.min(50, titleScore + authorScore); return Math.min(60, titleScore + authorScore);
} }
/** /**
@@ -474,26 +448,23 @@ export class RankingAlgorithm {
notes.push('Unknown or uncommon format'); notes.push('Unknown or uncommon format');
} }
// Seeder notes // Seeder notes (skip for NZB/Usenet results which don't have seeders)
if (torrent.seeders === 0) { if (torrent.seeders !== undefined && torrent.seeders !== null && !isNaN(torrent.seeders)) {
notes.push('⚠️ No seeders available'); if (torrent.seeders === 0) {
} else if (torrent.seeders < 5) { notes.push('⚠️ No seeders available');
notes.push(`Low seeders (${torrent.seeders})`); } else if (torrent.seeders < 5) {
} else if (torrent.seeders >= 50) { notes.push(`Low seeders (${torrent.seeders})`);
notes.push(`Excellent availability (${torrent.seeders} seeders)`); } else if (torrent.seeders >= 50) {
notes.push(`Excellent availability (${torrent.seeders} seeders)`);
}
} }
// Size notes // Match notes (now worth 60 points!)
if (breakdown.sizeScore < 5) { if (breakdown.matchScore < 24) {
notes.push('⚠️ Unusual file size');
}
// Match notes (now worth 50 points!)
if (breakdown.matchScore < 20) {
notes.push('⚠️ Poor title/author match'); notes.push('⚠️ Poor title/author match');
} else if (breakdown.matchScore < 35) { } else if (breakdown.matchScore < 42) {
notes.push('⚠️ Weak title/author match'); notes.push('⚠️ Weak title/author match');
} else if (breakdown.matchScore >= 45) { } else if (breakdown.matchScore >= 54) {
notes.push('✓ Excellent title/author match'); notes.push('✓ Excellent title/author match');
} }