AI and RAG pipelines

Why Reddit text is worth retrieving

Ask a model which project management tool a ten person agency should pick and it will answer from training data that is months old and written by vendors. Ask the same question with five Reddit threads in the context window and it answers from people who tried three of them and switched. Reddit threads carry the comparison, the complaint and the workaround, in the words buyers and users use. That makes them a strong retrieval source for support agents, research assistants and product copilots.

Reddit's public Data API closes in March 2027, and its search never covered comments. Redlisten gives a pipeline full text search over every post and comment from the last 30 days and the complete thread behind any hit.

Retrieve: search, then expand

A retrieval step is two calls. /search?q= finds posts and comments that match the question, with AND, OR, NOT, quoted phrases, subreddit: and author: in the query and t to bound the window. /comments/{id} then returns the post and its comment tree, with parent_id and depth on every comment, so the chunk the model reads is the whole discussion rather than one detached sentence.

import os, requests

BASE = "https://api.redlisten.com"
H = {"Authorization": f"Bearer {os.environ['REDLISTEN_KEY']}"}

def retrieve(question, k=5):
    # 1. search posts and comments from the last 30 days
    hits = requests.get(f"{BASE}/search", headers=H,
                        params={"q": question, "t": "month", "limit": k}).json()
    docs = []
    for child in hits["data"]["children"]:
        d = child["data"]
        post_id = d.get("link_id", d["name"]).split("_")[1]
        # 2. pull the full thread so the model sees the replies, not just the hit
        thread = requests.get(f"{BASE}/comments/{post_id}", headers=H,
                              params={"limit": 200}).json()
        post = thread[0]["data"]["children"][0]["data"]
        comments = [c["data"].get("body", "") for c in thread[1]["data"]["children"]]
        docs.append({"title": post["title"], "url": "https://www.reddit.com" + post["permalink"],
                     "text": (post.get("selftext") or "") + "\n\n" + "\n".join(comments)})
    return docs

for d in retrieve('"best budget 3d printer" NOT subreddit:3Dprintingdeals'):
    print(d["title"], d["url"], len(d["text"]))

Each result carries score, created_utc and subreddit, which make good re-ranking features: a highly voted comment from a specialist community usually beats a fresh one from a general one. Search pages up to 100 results a call, or 50 on a free key, which is a wide enough candidate set for a re-ranker in one request.

Keep an index fresh

For a corpus rather than a per question lookup, the native feeds do the work. /v1/posts and /v1/comments deliver every public post and comment in order with a cursor, so an embedding pipeline reads the feed, chunks, embeds and stores, then saves the cursor and resumes. A keyword stream narrows that to a topic: match on the terms your product cares about and embed the matches alone. Business keys add daily JSONL exports of each stream, which fit a nightly batch job better than a cursor loop.

Agents: the MCP server

Redlisten ships an MCP server, so an assistant can search Reddit, open a thread and follow an author without any glue code. Connected with no key at all, it serves the keyless previews: mention counts for a keyword across Reddit and what a subreddit is talking about this week. In Claude it registers with claude mcp add redlisten and your key. Cursor and other MCP aware editors take the same server in their MCP settings, which lets a coding agent read how people actually hit a library's edge cases. ChatGPT and custom GPTs can call the same endpoints through an action built from the OpenAPI spec at api.redlisten.com/openapi.json.

For a framework agent, a LangChain tool wrapping /search and /comments/{id} is a dozen lines, and the Listing JSON parses without a schema. The same two calls back a retriever class for a RAG chain.

Cost and limits for pipelines

Search and thread calls count one request each, whatever the page size. A Pro key at $29 a month carries 50,000 requests and 1M results, a Team key at $99 carries 250,000 requests and 5M items, and a Business key at $299 carries 1M requests and 10M results plus exports and a live streaming connection. The per minute limit runs from 30 on Free to 600 on Business. Prices are as listed on 6 Nov 2026 and drop by two months when billed yearly.

Content is public Reddit text and the data you retrieve is yours to store and use inside your product under the terms. The docs have the full field list for posts and comments, and the search page covers the query syntax.