How do news aggregators work?

This page explains how a news aggregator like WeSearch, Google News, Apple News, Feedly, Reddit's news subreddits, or Hacker News actually works under the hood. Some plumbing is shared across all of them; the editorial layer is where they diverge dramatically.

Step 1: Source list

Every aggregator starts with a list of sources. Some maintain a hand-curated catalog (WeSearch, Apple News in part, Ground News). Some let users bring their own list (Feedly, Inoreader). Some have algorithmic source discovery (Google News, SmartNews). The source list is the most consequential editorial decision — what's in the catalog determines what readers can possibly see.

Source lists range from dozens (a personal RSS reader) to hundreds (WeSearch is at 700+) to tens of thousands (Google News). More isn't always better; a wider list usually means weaker curation.

Step 2: Fetching

Aggregators pull from sources at intervals. Most editorial publishers expose RSS or Atom feeds — XML files that list recent items with title, link, publish date, optional description, optional image. The aggregator polls each feed every few minutes (5-minute intervals are typical for an active hub; some big aggregators push to every-minute).

Polite aggregators respect HTTP caching: If-Modified-Since and ETag headers, so a feed that hasn't changed costs a 304 instead of a full body. Polite aggregators also rotate through sources rather than hammering all at once. Bad behavior gets you blocked.

For sources without RSS, some aggregators crawl HTML directly — parse the homepage, extract article links, follow them, extract metadata. This is more fragile and gets blocked more often.

Step 3: Normalization

Each source's RSS schema is slightly different — some include a media:thumbnail, some include byline, some include category tags, some don't. The aggregator normalizes everything into one canonical schema (title, link, publish time, source, image, description, byline, language). Without normalization, downstream rendering is impossible.

Step 4: Deduplication

Many news stories show up in multiple feeds — a Reuters wire that gets republished by AP, the Guardian, and a national paper. The aggregator deduplicates so you don't see the same headline three times. Dedup happens by canonical URL (the article's "real" URL, found in canonical link tags or by following redirects), or by content-hash, or by title-similarity.

WeSearch dedupes by canonical URL. This catches most cases. It misses cases where the same wire copy is republished under different URLs by different desks; better dedup would require content-hashing.

Step 5: Ranking / sorting

Here's where aggregators diverge dramatically:

• Chronological. Sort by publish time. Simplest, most predictable. WeSearch, raw RSS readers.
• Algorithmic. Score each item via a model trained on engagement, recency, source-trust, etc. Reorder. Google News, Apple News, SmartNews, Reddit's "Best."
• Voted. Let readers upvote items; sort by score-over-time. Hacker News, Reddit's classic Hot.
• Personalized. Score against the individual reader's history. Strongest version of algorithmic. Google News, Flipboard.

Each approach makes a different trade. Algorithmic is convenient but bubble-prone. Chronological is honest but firehose-shaped. Voted is community-driven but vulnerable to brigading. WeSearch chose chronological; this page just describes the menu.

Step 6: Display

The aggregator renders items as cards in a feed surface. Modern aggregators include images, summaries, sometimes AI-generated TL;DRs. The feed scrolls; old items eventually drop off the visible window.

Step 7 (optional): Discussion

A subset of aggregators add a discussion layer — comments, reactions, shares. Reddit and Hacker News are discussion-first. Most "news app" style aggregators (Apple News, Google News, Feedly) have no discussion. WeSearch's discussion layer is anonymous and threaded.

Step 8 (optional): Notifications

Aggregators that support push notifications need: a list of subscribers, a list of subscriber preferences (which topics, which sources, which keywords), and a delivery mechanism (Web Push via VAPID, native APNs/FCM, or email). When a new story matches a watch, the aggregator fires a push.

Why aggregators exist

The honest answer: because following many sources directly costs too much in time and subscription fees. An aggregator pays for its discovery work by being a single attention surface for many publishers. The editorial value is in the curation — which sources are in the catalog, how the ranking sorts, what gets filtered.

How specific aggregators handle each step

Why aggregator design choices matter

Each step in the pipeline above is a design decision with downstream consequences. A crawler-driven source list scales but is less editorial; a hand-curated source list is editorially stronger but doesn't scale. Algorithmic ranking is convenient but bubble-prone; chronological is honest but firehose-shaped. A discussion layer adds reader value but raises moderation cost; no discussion saves cost but loses the community signal. Advertising-auction infrastructure can fund free aggregation but expands the reader-data surface; avoiding that model requires alternative funding such as subscriptions, donations, or clearly separated first-party services. The aggregator you pick reflects which trade-offs you're willing to accept.

Bottom line

• News aggregation is technically simple (RSS/Atom pull, deduplicate, render); the editorial choices are where products diverge.
• The major commercial aggregators converge on personalization-driven engagement-optimization because the business model rewards it.
• Non-commercial alternatives (RSS readers, Hacker News, WeSearch) keep chronological/voted ordering because they don't have the same ad-revenue pressure.
• The right aggregator for any reader depends on which trade-offs they're willing to accept.

Frequently asked