diff --git a/scripts/README.md b/scripts/README.md index b6b6faf78..44973b657 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -33,7 +33,7 @@ A Git push does not deploy this pipeline. Changes to `scripts/sync/` or `scripts | [`scripts/sync/lib/catalog-rows.js`](sync/lib/catalog-rows.js) | Scrubbed values for stage 2's API import and stage 3's latest-episode update: an odd optional field from Podcast Index becomes empty instead of failing the write. The latest-episode update stores the episode's served ID in `latest_ep_id` (what Home, trending and bootstrap send), plus `latest_ep_key` and `latest_ep_eid`, from `tipIdentity` in `lib/episode-identity.js` (by catalog key, else Podcast Index's ID as an alias in the same show, else that ID). Change detection and the tip queue keep Podcast Index's own ID | | [`scripts/sync/lib/stale-tips.js`](sync/lib/stale-tips.js) | Stage 3's tips for stale shows (`R5_MODE`: `off` by default, `shadow`, `on`). After the Podcast Index pass, checked chart shows that the published feed-address registry (`REGISTRY_DB`) flags `episode_stale` ask the local feed reader (`FEEDS_URL`, `POST /feeds/latest`, 40 shows per call) for their newest episode. When it is playable, a different episode and newer than the stored tip, it becomes the tip under its served ID, catalog key and recipe ID. Only the tip columns change: `qdrant_vectorized` and the tip queue stay with the Podcast Index pass. `shadow` reads only the feed's dates and never writes. A registry or feed-reader failure skips it, and the Podcast Index tips stay | | [`scripts/sync/lib/show-window.js`](sync/lib/show-window.js) | Saved episode windows (`WINDOWS_MODE`: `off` by default, `shadow`, `on`). Stage 3 records each full list it fetches as episode-book rows, with descriptions, and a `show_window` row: the book row IDs in list order. It fetches at the cap stage 4 enforces, chart or earned. Shows whose R5 tip came from the feed also save the feed reader's merged list (`GET /feeds/episodes`), so their feed-only episodes get vectors. With `on`, stage 4 reads a saved window instead of fetching the list from Podcast Index, when it is usable, fresh (3 days) and long enough. `shadow` records windows and has stage 4 compare them with its own fetch. A window with an episode that has no catalog key, or that another show serves, is unusable, and stage 4 takes the fetch path. Tip-queue rows carry `gen`, and stage 4 clears only rows enqueued before it started | -| [`scripts/sync/lib/cadence.js`](sync/lib/cadence.js) | Per-show check schedule for stage 3, in shadow (`CADENCE_MODE`: `shadow` by default, `off`). Checks still follow the news/core/relaxed tiers; the run report adds what the schedule would have checked. Each show's interval follows its pace, the average gap between its newest 11 episodes: news shows hourly, others a quarter of the gap within 3 to 48 hours, and dormant shows weekly when both the newest episode and a fresh Apple answer are over 180 days old. A show with no known pace keeps its tier. A show Apple lists a newer episode for (`APPLE_LATEST_DB`, the applefeeds snapshot) is due until it has been checked 6 hours past that date. Pace and the newest date come from each fetched list and from the episode book about daily (400 shows per read); show records gain `g`, `p` and `i` (iTunes ID). The report line counts shows the schedule would skip or check earlier, new episodes it would have found later and by how much (only tips dated after the newest episode known, so Podcast Index re-indexing an old episode doesn't count), and checks a day against today's tiers. A book or snapshot failure only skips that part | +| [`scripts/sync/lib/cadence.js`](sync/lib/cadence.js) | Per-show check schedule for stage 3, in shadow (`CADENCE_MODE`: `shadow` by default, `off`). Checks still follow the news/core/relaxed tiers; the run report adds what the schedule would have checked. Each show's interval follows its pace, the average gap between its newest 11 episodes: news shows hourly, others a quarter of the gap within 3 to 48 hours, and dormant shows weekly when both the newest episode and a fresh Apple answer are over 180 days old. A show with no known pace keeps its tier. A show Apple lists a newer episode for (`APPLE_LATEST_DB`, the applefeeds snapshot) is due until it has been checked 6 hours past that date. Pace and the newest date come from each fetched list and from the episode book about daily (400 shows per read); show records gain `g`, `p` and `i` (iTunes ID). The report line counts shows the schedule would skip or check earlier, new episodes it would have found later and by how much (only tips dated after the newest episode known, so Podcast Index re-indexing an old episode doesn't count), and checks a day against today's tiers. Drop times (24 Oct): for each new episode it reads the show's release pattern from the dates known before it (`arrivalPattern`: daily, weekly or a regular gap when most of the last 10 episodes fit; irregular otherwise) and reports how many landed on schedule, where timing checks would look, against off-schedule ones (bonus episodes, irregular shows) that the usual checks catch, plus how many hours after release this run found each kind. Timing checks would only ever add checks: every show keeps its usual ones. A book or snapshot failure only skips that part | | [`scripts/sync/lib/attribution.js`](sync/lib/attribution.js) | Vector attribution log: one JSON line per episode point stored or deleted (prune, cap trim), show vector stored, and show imported or deleted. Written to `SYNC_ATTRIBUTION_DIR` (default `scripts/data/attribution/`, gitignored) as daily files, gzipped after the day ends and kept 60 days. Best effort: a write failure never fails a stage. Stage 4 adds `stored`, `pruned` and `reasons …` to its run line | | [`scripts/sync/attribution-report.js`](sync/attribution-report.js) | Read-only report over that log: embeddings by reason (new episode, first coverage, returning show, re-embed after cap trim or prune, Podcast Index ID change, backfill), deletions, and how long chart entrants stay. `--days N`, `--json` | | [`scripts/sync/lib/episode-book.js`](sync/lib/episode-book.js) + [`episode-book-cli.js`](sync/episode-book-cli.js) | The episode book in sqld: one permanent row per (show, catalog key) with its recipe ID and served ID, plus Podcast Index aliases, `served_show` and `show_window`. Every book write also marks the show in `served_show` when one of its episodes is listed by Podcast Index under an ID other than its served ID (renumbered, duplicated or recipe-served), so the Worker's v1 overlay translates that ID back. That check reaches a show's aliases through `episode_pi_alias_rid` (an index on `rid`); without it each check scanned every alias inside the write. `ensure` creates the tables, indexes and columns; `stats` counts. Writes go through `turso.atomic()` (one Hrana batch: every step runs only after the previous one succeeded, then COMMIT, else ROLLBACK). Rules: [`docs/EPISODE_IDS.md`](../docs/EPISODE_IDS.md) | diff --git a/scripts/sync/03-sync-episodes.js b/scripts/sync/03-sync-episodes.js index 8d47caab2..f28196b1a 100644 --- a/scripts/sync/03-sync-episodes.js +++ b/scripts/sync/03-sync-episodes.js @@ -186,6 +186,8 @@ async function main() { } } const newTipIds = []; + // New episodes and whether each landed on its show's usual schedule (cadence shadow). + const timing = []; const plan = staleness.planDue(st.candidateIds, st.shows); const { allDue, neverChecked, staleNews, staleCore, staleRelaxed, @@ -315,8 +317,18 @@ async function main() { checked.m = medium; if (shadow) { const knewNewest = checked.p; - cadence.notePace(checked, (full || tip).map((ep) => ep.datePublished)); - if (cadence.isNewEpisode(knewNewest, latest.datePublished)) newTipIds.push(podId); + const dates = (full || tip).map((ep) => Number(ep.datePublished) || 0); + cadence.notePace(checked, dates); + if (cadence.isNewEpisode(knewNewest, latest.datePublished)) { + newTipIds.push(podId); + // The pattern as it stood before this episode: only the dates we already knew. + const pattern = cadence.arrivalPattern(dates.filter((d) => d > 0 && d <= (knewNewest || 0))); + timing.push({ + kind: pattern.kind, + on: cadence.onSchedule(pattern, knewNewest, latest.datePublished), + lagH: (Date.now() / 1000 - Number(latest.datePublished)) / 3600, + }); + } } } catch (e) { if (pi.isSuspended(e)) { @@ -365,7 +377,7 @@ async function main() { state.save(st); const paceShows = st.candidateIds.filter((id) => st.shows[id]?.g != null).length; cadenceLine = cadence.summary(cfg.CADENCE_MODE, shadow, cadence.compare(shadow, allDue, newTipIds), { - appleError, paceShows, candidates: st.candidateIds.length, + appleError, paceShows, candidates: st.candidateIds.length, timing: cadence.timingTally(timing), }); log.info(`[CADENCE] ${cadenceLine}`); } diff --git a/scripts/sync/lib/cadence.js b/scripts/sync/lib/cadence.js index be3533d17..db0333604 100644 --- a/scripts/sync/lib/cadence.js +++ b/scripts/sync/lib/cadence.js @@ -230,10 +230,119 @@ function openApple(path) { } } +// --- Drop times (shadow, 24 Oct): does a new episode land on its show's usual schedule? --- +// Timing checks would only add checks around a show's expected drop; every show keeps its usual +// checks too, so a bonus or off-schedule episode is still found as it is today (user, 24 Oct). + +const DAY_S = 86_400; +const WEEK_S = 7 * DAY_S; +/** Episodes a pattern is read from, and the share that must fit it. */ +const PATTERN_DATES = 10; +const PATTERN_SHARE = 0.6; + +/** Seconds between a and b on a circle of `period` seconds. */ +function circularGap(a, b, period) { + const d = Math.abs((((a - b) % period) + period) % period); + return Math.min(d, period - d); +} + +/** + * The point on a `period` cycle (hour steps) that most dates fall near, and how many do; among + * equally good points, the one closest to them. + */ +function bestSlot(dates, period, tolerance) { + let best = { at: 0, hits: 0, spread: Infinity }; + for (let at = 0; at < period; at += 3600) { + const near = dates.map((d) => circularGap(d, at, period)).filter((g) => g <= tolerance); + const spread = near.reduce((a, g) => a + g, 0); + if (near.length > best.hits || (near.length === best.hits && near.length > 0 && spread < best.spread)) { + best = { at, hits: near.length, spread }; + } + } + return { at: best.at, hits: best.hits }; +} + +function median(values) { + const s = [...values].sort((a, b) => a - b); + if (!s.length) return 0; + const mid = s.length >> 1; + return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2; +} + +/** + * A show's release pattern from its recent episode dates (epoch seconds, before the new one): + * daily — most episodes within an hour of the same time of day, about a day apart; + * weekly — most within two hours of the same weekday and time, about a week apart; + * gap — most gaps within 10% of the typical gap (fortnightly, monthly); + * irregular otherwise, or with under 4 dated episodes. Bonus episodes don't break a pattern + * while most episodes still fit it. + * @returns {{ kind: 'daily'|'weekly'|'gap'|'irregular', slot?: number, gap?: number }} + */ +function arrivalPattern(dates) { + const ds = (dates || []).map(Number).filter((d) => d > 0).sort((a, b) => b - a).slice(0, PATTERN_DATES); + if (ds.length < 4) return { kind: 'irregular' }; + const gaps = []; + for (let i = 1; i < ds.length; i++) gaps.push(ds[i - 1] - ds[i]); + const g = median(gaps); + const need = Math.ceil(ds.length * PATTERN_SHARE); + if (g >= 0.6 * DAY_S && g <= 1.6 * DAY_S) { + const s = bestSlot(ds, DAY_S, 3600); + if (s.hits >= need) return { kind: 'daily', slot: s.at }; + } + if (g >= 4 * DAY_S && g <= 9 * DAY_S) { + const s = bestSlot(ds, WEEK_S, 2 * 3600); + if (s.hits >= need) return { kind: 'weekly', slot: s.at }; + } + if (g >= DAY_S && gaps.filter((x) => Math.abs(x - g) <= 0.1 * g).length >= Math.ceil(gaps.length * PATTERN_SHARE)) { + return { kind: 'gap', gap: g }; + } + return { kind: 'irregular' }; +} + +/** True when an episode dated `dateS` lands on the pattern: where timing checks would look. */ +function onSchedule(pattern, prevNewestS, dateS) { + const d = Number(dateS) || 0; + if (!d || !pattern || pattern.kind === 'irregular') return false; + if (pattern.kind === 'daily') return circularGap(d, pattern.slot, DAY_S) <= 1.5 * 3600; + if (pattern.kind === 'weekly') return circularGap(d, pattern.slot, WEEK_S) <= 3 * 3600; + const prev = Number(prevNewestS) || 0; + if (!prev || d <= prev) return false; + const k = Math.max(1, Math.round((d - prev) / pattern.gap)); + return Math.abs(d - (prev + k * pattern.gap)) <= Math.max(6 * 3600, 0.1 * pattern.gap); +} + +/** Tally of new episodes by pattern, and how long after release this run found them. */ +function timingTally(entries) { + const out = { total: 0, onSchedule: 0, byKind: { daily: 0, weekly: 0, gap: 0 }, offSchedule: 0, lagOnH: [], lagOffH: [] }; + for (const e of entries || []) { + out.total++; + if (e.on) { + out.onSchedule++; + out.byKind[e.kind] = (out.byKind[e.kind] || 0) + 1; + if (e.lagH >= 0) out.lagOnH.push(e.lagH); + } else { + out.offSchedule++; + if (e.lagH >= 0) out.lagOffH.push(e.lagH); + } + } + return out; +} + const fmtH = (h) => (h >= 10 ? String(Math.round(h)) : h.toFixed(1)); +const lag = (hours) => (hours.length ? `${fmtH(median(hours))} h` : 'n/a'); + +/** The timing part of the report line. */ +function timingSummary(t) { + if (!t || !t.total) return ''; + const k = t.byKind; + return ` · timing: ${t.onSchedule} of ${t.total} new episodes on schedule (daily ${k.daily}, weekly ${k.weekly}, gap ${k.gap}), ` + + `${t.offSchedule} off schedule; found today ${lag(t.lagOnH)} after release on schedule, ` + + `${lag(t.lagOffH)} off schedule (median)`; +} + /** One line for the run report. */ -function summary(mode, plan, cmp, { appleError = null, paceShows = 0, candidates = 0 } = {}) { +function summary(mode, plan, cmp, { appleError = null, paceShows = 0, candidates = 0, timing = null } = {}) { if (mode !== 'shadow' || !plan) return 'cadence off'; const c = plan.counts; const pct = candidates ? Math.round((paceShows / candidates) * 100) : 0; @@ -241,7 +350,7 @@ function summary(mode, plan, cmp, { appleError = null, paceShows = 0, candidates `${c.appleForced} Apple-forced, ${c.never} never checked) · ${cmp.late} new episodes not yet due` + (cmp.late ? ` (median ${fmtH(cmp.lateMedianH)} h, max ${fmtH(cmp.lateMaxH)} h late)` : '') + ` · checks/day ${plan.perDay} vs ${plan.perDayNow} · news ${c.news}, dormant ${c.dormant}, pace ${c.pace}, tier ${c.tier}` + - ` · pace known ${pct}%` + (appleError ? ` · Apple unread (${appleError})` : ''); + ` · pace known ${pct}%` + (appleError ? ` · Apple unread (${appleError})` : '') + timingSummary(timing); } module.exports = { @@ -250,6 +359,10 @@ module.exports = { notePace, noteNewest, isNewEpisode, + arrivalPattern, + onSchedule, + timingTally, + timingSummary, intervalFor, appleForces, shadowPlan, diff --git a/scripts/sync/lib/cadence.test.js b/scripts/sync/lib/cadence.test.js index 36fcc3890..8d77e9666 100644 --- a/scripts/sync/lib/cadence.test.js +++ b/scripts/sync/lib/cadence.test.js @@ -160,3 +160,50 @@ test('a changed tip counts as a new episode only when it is newer than the newes assert.equal(isNewEpisode(0, NOW_S), true, 'no known date'); assert.equal(isNewEpisode(undefined, '0'), false, 'undated tip'); }); + +const { arrivalPattern, onSchedule, timingTally, timingSummary } = require('./cadence'); +const H = 3600; +// Tuesday 2026-10-20 05:00 UTC. +const TUE5 = Date.UTC(2026, 9, 20, 5) / 1000; + +test('drop times: weekly, daily and regular-gap shows are recognised; bonus episodes don\'t break a pattern', () => { + const weekly = Array.from({ length: 10 }, (_, i) => TUE5 - i * 7 * DAY_S + (i % 3) * 600); + assert.equal(arrivalPattern(weekly).kind, 'weekly'); + const withBonus = [...weekly.slice(0, 8), TUE5 - 3 * DAY_S + 13 * H, TUE5 - 10 * DAY_S + 20 * H]; + assert.equal(arrivalPattern(withBonus).kind, 'weekly', '2 bonus episodes in 10'); + const daily = Array.from({ length: 10 }, (_, i) => TUE5 - i * DAY_S + (i % 2) * 1200); + assert.equal(arrivalPattern(daily).kind, 'daily'); + const fortnightly = Array.from({ length: 8 }, (_, i) => TUE5 - i * 14 * DAY_S + (i % 2) * 5 * H); + assert.deepEqual(arrivalPattern(fortnightly).kind, 'gap'); + const random = [TUE5, TUE5 - 2.3 * DAY_S, TUE5 - 9.1 * DAY_S, TUE5 - 11 * DAY_S, TUE5 - 25 * DAY_S, TUE5 - 26.5 * DAY_S]; + assert.equal(arrivalPattern(random).kind, 'irregular'); + assert.equal(arrivalPattern([TUE5, TUE5 - 7 * DAY_S, TUE5 - 14 * DAY_S]).kind, 'irregular', 'too few dates'); +}); + +test('drop times: a new episode is on schedule only near the expected slot', () => { + const weekly = arrivalPattern(Array.from({ length: 10 }, (_, i) => TUE5 - i * 7 * DAY_S)); + assert.equal(onSchedule(weekly, TUE5, TUE5 + 7 * DAY_S + 2 * H), true); + assert.equal(onSchedule(weekly, TUE5, TUE5 + 14 * DAY_S - H), true, 'a skipped week still lands on the slot'); + assert.equal(onSchedule(weekly, TUE5, TUE5 + 3 * DAY_S), false, 'a Friday bonus is off schedule'); + const daily = arrivalPattern(Array.from({ length: 10 }, (_, i) => TUE5 - i * DAY_S)); + assert.equal(onSchedule(daily, TUE5, TUE5 + DAY_S + H), true); + assert.equal(onSchedule(daily, TUE5, TUE5 + DAY_S + 6 * H), false); + const gap = { kind: 'gap', gap: 14 * DAY_S }; + assert.equal(onSchedule(gap, TUE5, TUE5 + 14 * DAY_S + 5 * H), true); + assert.equal(onSchedule(gap, TUE5, TUE5 + 9 * DAY_S), false); + assert.equal(onSchedule(gap, 0, TUE5), false, 'no known newest date'); + assert.equal(onSchedule({ kind: 'irregular' }, TUE5, TUE5 + DAY_S), false); +}); + +test('drop times: tally and report wording', () => { + const t = timingTally([ + { kind: 'weekly', on: true, lagH: 20 }, + { kind: 'daily', on: true, lagH: 4 }, + { kind: 'irregular', on: false, lagH: 30 }, + ]); + assert.deepEqual([t.total, t.onSchedule, t.offSchedule, t.byKind], [3, 2, 1, { daily: 1, weekly: 1, gap: 0 }]); + assert.equal(timingSummary(t), ' · timing: 2 of 3 new episodes on schedule (daily 1, weekly 1, gap 0), 1 off schedule; ' + + 'found today 12 h after release on schedule, 30 h off schedule (median)'); + assert.equal(timingSummary(timingTally([])), ''); + assert.match(timingSummary(timingTally([{ kind: 'irregular', on: false, lagH: 2 }])), /n\/a after release on schedule, 2\.0 h off schedule/); +});