Research and datasets

Where researchers get Reddit text now

For a decade the answer was Pushshift, and since 2023 Pushshift has been limited to moderators of the communities they moderate. The torrent archives that fill the gap are large, arrive in batches, and lag the site by weeks or months. Reddit's own Data API closes to the public in March 2027, and its search never indexed comments. A study that needs this month's conversation, or a dataset that needs to stay current, has been working around all three.

Redlisten takes the current window. Every public post and comment, searchable in full text for 30 days on paid plans, streamed in order with a cursor, and exported daily as JSONL for any keyword stream on the Business plan. For history before that, back to 2013, email us; it is an Enterprise conversation rather than a self serve button.

Three ways to build a dataset

Search when the corpus is defined by a query. /search?q= takes AND, OR, NOT, quoted phrases, subreddit: and author:, and returns posts, comments or both. Search pages up to 100 results a call, or 50 on a free key, so a corpus of a few thousand documents is a short loop over after.

Streams when the corpus is defined by a topic and keeps growing. A keyword stream collects every matching post and comment from the moment it is created, backfilled 30 days on paid plans. Read it with a cursor or, on Business, download the previous day's matches as a compressed JSONL file produced once a day and kept for 30 days.

The native feeds when the corpus is everything. /v1/posts and /v1/comments deliver the whole site in order, up to 1,000 items a page, with next_cursor and has_more so a collector resumes where it stopped. Items on those two feeds are metered: 1M a month on Pro, 5M on Team and 25M on Business, as listed on 6 Nov 2026.

# List and download a stream's daily exports (Business plan)
curl "https://api.redlisten.com/v1/streams/st_8f2k/exports" \
  -H "Authorization: Bearer rl_live_YOUR_KEY"
# {"data":[{"day":"2026-11-05","url":"/v1/streams/st_8f2k/exports/2026-11-05.jsonl.gz","bytes":4182233,"items":12894}, ...]}

curl -o 2026-11-05.jsonl.gz \
  "https://api.redlisten.com/v1/streams/st_8f2k/exports/2026-11-05.jsonl.gz" \
  -H "Authorization: Bearer rl_live_YOUR_KEY"

# Or walk the live comment feed with a cursor in Python
import os, gzip, json, requests
H = {"Authorization": f"Bearer {os.environ['REDLISTEN_KEY']}"}
cursor = "latest"
with gzip.open("comments.jsonl.gz", "at") as out:
    while True:
        page = requests.get("https://api.redlisten.com/v1/comments", headers=H,
                            params={"cursor": cursor, "limit": 1000}).json()
        for c in page["data"]:
            out.write(json.dumps(c) + "\n")
        cursor = page["next_cursor"]
        if not page["has_more"]:
            break

What a record carries

The native post object carries the id, title, Markdown body, author name and stable account id, community name and id with subscriber count, flair, detected language, NSFW and spoiler flags, score, upvote ratio, comment count, media URLs, permalink, and a status of live, removed or deleted with the time the change was observed. Comments carry the post id, parent id and depth, so a thread can be rebuilt offline. The docs list every field.

Removed and deleted items stay in the feed with their captured text and the status flag, which matters for moderation research: the record of what was removed is often the dataset. Scores and comment counts keep refreshing as a thread grows, so a post can appear more than once with a new cursor, and the latest version is the current state.

Tools people pair it with

Python covers most of it: requests for the calls, pandas or polars over the JSONL, and the usual text stack for the analysis. A dataset built from exports loads into a Hugging Face dataset with load_dataset("json", ...) in one line. Postman is useful before the code exists: import the OpenAPI spec from api.redlisten.com/openapi.json and try each query against a saved key.

Cost for a study

A free key, issued at signup, has 250 requests a month and searches the last 7 days, enough to size a corpus. Pro at $29 a month searches 30 days with 50,000 requests and 1M results. Team at $99 raises that to 250,000 requests, 5M items and 1,000 items a page on the native feeds. Business at $299 adds daily exports, a live streaming connection and 10M results. Yearly billing takes two months off. The data you retrieve stays yours after a subscription ends, under the terms, so a one month pull can back a paper written in six.