mirror of
https://github.com/kikootwo/ReadMeABook.git
synced 2026-06-02 20:30:10 +00:00
Audible: HTML refresh, multi-narrator & works dedup
Switch nightly discovery refresh to scrape Audible's curated HTML storefronts (popular, new releases, category pages) while keeping real-time user paths on the JSON catalog API. Add robust HTML resilience knobs (increased retries, capped jittered backoff, AdaptivePacer changes and per-batch cooldowns) to avoid failing nightly jobs during 503 storms. Implement multi-narrator capture via a new extractAllNarrators helper and update parsers to preserve all narrator anchors. Introduce two-pass dedup: in-memory deduplicateAndCollectGroups + collapseByExistingWorks that consults the works table, export metadataScore for consistent representative selection, and persist dedup groups (fire-and-forget). Wire collapseByExistingWorks into search/author/series routes and make defensive dedup in the refresh processor. Add HTML parsing helpers, runtime/lang-aware parsing, jitteredBackoff cap, and tests for the new behaviors.
This commit is contained in:
@@ -67,6 +67,24 @@ describe('jitteredBackoff', () => {
|
||||
expect(value).toBeGreaterThanOrEqual(250);
|
||||
expect(value).toBeLessThanOrEqual(750);
|
||||
});
|
||||
|
||||
it('caps the result at maxBackoffMs when the raw backoff would exceed it', () => {
|
||||
// attempt=10 with base=1000 produces 2^10 * 1000 * [0.5..1.5] = 512_000..1_536_000,
|
||||
// all of which exceed a 60_000ms cap.
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const value = jitteredBackoff(10, 1000, 60_000);
|
||||
expect(value).toBeLessThanOrEqual(60_000);
|
||||
}
|
||||
});
|
||||
|
||||
it('returns the un-capped jittered value when below the cap', () => {
|
||||
// attempt=0 with base=1000 produces 500..1500, all below a 60_000ms cap.
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const value = jitteredBackoff(0, 1000, 60_000);
|
||||
expect(value).toBeGreaterThanOrEqual(500);
|
||||
expect(value).toBeLessThanOrEqual(1500);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('randomDelay', () => {
|
||||
|
||||
Reference in New Issue
Block a user