Anekdotnik analytics: can this become a recommendation and joke-quality engine?

Review of the shipped implementation · 16 August 2026 · ellsworth

Three findings that change the picture

1. traffic_source is measuring the wrong thing — on every event

src/app/api/events/route.ts:66 derives it from the Referer header of the /api/events POST:

classifyTrafficSource(req.headers.get("referer") ?? "", selfHost)

That referer is the Anekdotnik page the reader is currently on — our own host. And classifyTrafficSource (core.ts:162) returns "internal" whenever host === selfHost. So a reader arriving from Yandex, from Telegram, or by typing the URL produces identical rows: internal.

Consequences, all silent:

• the search_one_and_done cohort matches traffic_source === "search" and is therefore near-empty by construction, not by behaviour;
• every cohort card's traffic split is one bar;
• the acquisition channel — the most obvious proxy for "was this the wrong audience?" — does not exist in the dataset at all.

The true value is document.referrer at first page load, client-side. It is never captured. Fixing this is perhaps thirty lines and it unlocks the audience question.

2. feed_position is collected, indexed, and used by nothing

JokeCard.tsx:43 passes feedPosition: index, the validator accepts it, events.ts writes it. Then: zero references to feed_position in src/lib/repositories/analytics.ts. Not in the overview, not in the cohorts, not in content health.

This is the confounder the ranker's own documentation names as disqualifying: "97 quality-8 jokes absorb 63% of all impressions because they sit on /best." The control variable is already in the table. Nothing divides by it.

3. Completion is an inverted quality signal as currently computed

Measured 2026-08-14 and recorded in CLAUDE.md:

Higher-quality jokes complete nine times less often. Two confounders: length (a one-liner completes trivially) and placement. getContentHealth normalises against a length-bucket benchmark — genuinely good, and half the fix. It does not normalise against position. So the metric on the Overview tab, and the benchmark driving every health diagnosis, still carries the placement bias.

Metric by metric

Can we identify good and bad jokes today?

For roughly 1.8% of the corpus, weakly. For the rest, no.

• 789 of 43,809 jokes have any behavioural data at all.
• classifyContentHealth requires 40 unique readers before it will say anything, and calls a sample "high confidence" only at 200. At current traffic very few jokes clear the first bar and almost none the second.
• The strongest available quality proxy is not behavioural at all — it is quality_score, assigned by an LLM in scripts/tag-jokes.ts. It already covers the corpus, and the ranker leans on it at weight 0.6.

The honest framing: you have an AI quality score with full coverage and no validation, and a behavioural signal with validation and no coverage. The single highest-value analysis available right now is to correlate the two on the 789 — position- and length-normalised. That tells you whether the LLM score is trustworthy enough to rank 43,809 jokes, which is a question worth far more than any new metric.

Can we tell a bad joke from a joke shown to the wrong audience?

No — and the current answer is a leftover bucket, not a measurement.

possible_audience_mismatch is rule 5 in health.ts: it fires when completion is below benchmark and none of the other four rules matched. It is the residual. Any joke that under-performs for an unmodelled reason lands there.

To answer the question properly you need the same joke observed across different audience slices. Three ways, none currently available:

1. By acquisition channel — blocked by the traffic_source defect.
2. By topic affinity — the events table has no tags, and no query joins them. Health is computed per joke, never per joke × topic.
3. By slot — feed_position is unused, so "underperformed at position 18" and "underperformed on merit" are indistinguishable.

All three are unlocked by data you already store or can capture trivially. None needs a model.

Readiness for user × joke × context × action × outcome

Every event carries session_id and occurred_at, so outcome is derivable without new instrumentation: order a session's events, and for each joke compute jokes-seen-after, seconds-to-session-end, and whether the session ended there. That is a materialised view, not a tracking change. It is the cheapest large win on this list.

What to track for individual humour preference

What you already have that most sites don't: reading_ratio and the nine dislike reasons. What is missing is not more events but the joining of events to content attributes:

• Topic affinity per device — completion and reading-ratio grouped by tag, per voter_key. Needs one join, no new tracking. Note the ranker already proved the concept: an in-page session affinity model matched 9–10 of 10 to a stated preference, and was removed only because the append-style surface it needed was dropped.
• Length and format preference — the buckets exist in LENGTH_BUCKETS; nothing is computed per reader.
• Negative preference — topic_dislike and too_vulgar are explicit statements of taste, currently aggregated into a site-wide rate and never attached to the reader who said it.
• Position-corrected preference — without it a reader's "taste" is partly a record of what the feed put in front of them first.
• Dwell on the punchline — not the whole card. Would separate "read it" from "got it". Genuinely new instrumentation; lowest priority of these.

1. What is good now

1. The event vocabulary. 21 allowlisted names, rejected at ingestion, so the table cannot fill with junk. Documented one-line rationale per event.
2. Active reading time. Excludes hidden tabs, requires ≥50% visibility for ≥1s, accrues only while engaged. This is the hard part and it is done properly.
3. Length normalisation. expectedReadingMs plus a per-bucket completion benchmark. Explicitly documented as an estimate, never a truth claim.
4. The nine dislike reasons. Rare, cheap, and directly actionable — see §5.
5. Rates always carry their denominator (rate()), and cohorts carry MIN_SAMPLE guards. Very few analytics stacks do either.
6. The content-health engine's epistemics. Hypotheses with confidence and contradicting signals, never auto-acting. Exactly right for a quality engine.
7. A working A/B framework — assignArm(), device-sticky, salted per experiment, with experiment_id on every event. Built for the audio card; generalises to AI-edited joke variants for free.
8. Privacy by design. No IP, no user agent, no joke text, anonymous key. Nothing here needs unwinding later.

2. What is misleading or useless

1. traffic_source — wrong on every row. Worse than missing, because it looks populated.
2. Completion rate as a headline. Inverted against LLM quality (36.4% → 3.8%) until normalised by position.
3. firstJokeAbandonment — inherits that bias; largely a length metric.
4. search_one_and_done cohort — dead by construction.
5. Journey steps 2 and 3 — became_visible and read_started are the same predicate; the funnel shows a stage that cannot drop.
6. returnRate at this scale — true, and uninformative with 49 multi-session devices.
7. joke_impression as a denominator. A DOM mount is not a view; joke_visible exists and is the right one.

3. What is missing

1. Outcome. No per-joke linkage to what happened next. Derivable from data already stored.
2. True acquisition source — document.referrer at session start.
3. Topics on the analysis path. No tag join anywhere in the analytics repository, so no topic-level quality and no topic affinity.
4. Position normalisation — the column exists, the divide does not.
5. Text embeddings. Nothing represents joke content numerically. This is what makes duplicate detection, similarity and cold-start recommendation possible, and for 43,809 short texts it is cheap and offline.
6. Validation of quality_score against observed behaviour. The ranker's largest weight has never been checked against a reader.
7. Scroll-past surfaced — impression-without-visible, derivable, unexposed.
8. A ranker holdout. No arm measures the ranker against a control, so its value is unknown. The machinery to do it now exists.

4. What to implement next

Ordered by value ÷ effort. The first three need no new tracking and no model.

1. Fix traffic_source. Capture document.referrer client-side at session_start, classify it there, carry it on the session. ~30 lines. Unblocks the audience question and one dead cohort.
2. Divide by feed_position. Add position-normalised completion to getContentHealth — expected completion for a slot, actual over expected. Data is already in the table. This is what makes every downstream quality number honest.
3. Derive the outcome table. A view over session_id + occurred_at: per joke, jokes-seen-after, seconds-to-session-end, session-ended-here. No instrumentation. This is the label column for any future model.
4. Join tags into content health → topic × completion, and per-device topic affinity. One join, and it makes "wrong audience" measurable.
5. Validate quality_score against position-normalised behaviour on the 789. Cheap, and it either justifies or dethrones the ranker's dominant weight.
6. Embed the corpus (offline, one pass). Near-duplicate clusters immediately; similarity-based recommendation that works on a first-time visitor, which matters when 98.5% of them are.

5. How this becomes a recommendation + AI joke-quality engine

The path is unusually short because two of the hard parts are already built: a defensible reading signal and a working experiment framework.

1. Make the numbers honest (items 1–3 above). Everything downstream is worthless without position normalisation and a real outcome variable.
2. Content-based recommendation first, not collaborative. With 49 multi-session devices, collaborative filtering has nothing to learn from. Embeddings plus quality_score plus topic affinity within the session cold-starts correctly — and the ranker's own history shows session affinity worked (9–10/10) when it had a surface to run on.
3. Turn content health into a work queue. The seven diagnoses already map onto actions: possible_duplicate → dedupe candidate; possible_long_intro → AI-shorten candidate; possible_missing_context → add a footnote; possible_mobile_formatting → reformat. Add "AI-edit candidate" as a filtered view over jokes whose dislike reasons are dominated by too_long or poorly_written. Those two reasons are a labelled dataset for exactly this and are currently only counted.
4. A/B the AI edits with the framework that exists. assignArm() salted with a per-joke experiment name gives a sticky split per reader; the variant travels in experiment_id; every existing reading event is already stamped with it. The comparison metric should be position-normalised completion and reading ratio, not raw completion — otherwise the shorter variant always "wins" by being shorter.
5. Close the loop last. Only once a ranker holdout shows a measured lift should ordering be driven automatically. Ship it as an arm, never as a silent replacement.

6. Other uses for the same dataset

• Publishing gate for the ~14k pending jokes. Named as still-open in CLAUDE.md. A validated quality_score turns that backlog into months of scheduled content with no human review.
• Corpus pruning. Duplicate clusters plus persistent low performance identify what to unpublish — improving the average without writing anything.
• Tag-quality audit. The 60-topic taxonomy draws 0.64% of entries at 62.5% bounce. Per-topic reading behaviour would show which topics are genuinely weak versus badly titled — a question the SEO data alone cannot separate.
• РСЯ revenue modelling. Per-session depth is pageviews, and pageviews are the whole ad-revenue model. This dataset predicts revenue per acquisition channel directly — once the channel is real.
• Distribution targeting. Which jokes get shared (not liked) is the best available predictor of what will travel on Telegram or Дзен. That signal is currently pooled into a combined "shares" rate.
• Ad-placement design. Position-normalised attention tells you where a unit can sit without costing reading depth — measurable before РСЯ is live, not after.
• Editorial feedback for the thematic pages. Reading ratio per chapter would show which authored sections hold people, feeding the decision on the next page after Штирлиц and Вовочка.

Method

Read from origin/main at 44b0c1d: src/lib/analytics/{core,client,health,useJokeTracking,jokeReadingEvents,rate-limit}.ts, src/lib/repositories/{analytics,events,recommendations}.ts, src/lib/validators/events.ts, src/app/api/events/route.ts, src/app/admin/analytics/visitors/page.tsx, src/components/JokeCard.tsx, migrations/008_analytics_events.sql, and the roadmap sections of CLAUDE.md. Event-emission coverage was checked by grepping all 21 names across src/; all 21 are wired. The feed_position and recommendation_surface findings are absence-of-reference results, not absence of mention.

Not verified: every quantitative figure here (789/43,809 jokes, 2,769 votes, the quality × completion table, 49 multi-session devices) is quoted from measurements recorded in-repo on 2026-08-14, not re-run — I have no database credentials. They are two days old and the direction is unlikely to have changed, but the exact numbers will have drifted. The traffic_source defect is a code-reading result and should be confirmed with one query (SELECT traffic_source, COUNT(*) FROM anekdotnik_events GROUP BY 1) before anyone spends a day on it.