{ "subreddit": "r/ETL", "page": "https://redlisten.com/r/etl", "updated": "2026-11-11T04:08:16.102Z", "members": 13945, "activity": { "posts_last_24_hours": 2, "posts_per_hour": 0.1, "comments_per_hour": 0.3, "newest_post_utc": 1794341928.614, "newest_comment_utc": 1794246371.674 }, "top_threads_this_week": [ { "title": "Built an End to End Azure Lakehouse (ADF, Databricks, Delta lake, dbt, Airflow) - Key Learnings & Architecture Patterns", "score": 14, "comments": 3, "created_utc": 1793859512.339, "snippet": "Hi Everyone! I recently completed a hands on data engineering project called WeatherFlow - Azure based Weather Data Lakehouse. I built to gain practical experience with tools and workflows. The...", "url": "https://www.reddit.com/r/ETL/comments/1xobgu5/built_an_end_to_end_azure_lakehouse_adf/" }, { "title": "Built an AI extension that generates complete Python data pipelines from data samples", "score": 9, "comments": 1, "created_utc": 1793967316.78, "snippet": "Hey, I'm Kevin, I run product at PrettyWhale.ai, a small team based in France. We've spent the last few months building a VS Code extension that writes the tedious part of ingestion code from a...", "url": "https://www.reddit.com/r/ETL/comments/1xpbc3e/built_an_ai_extension_that_generates_complete/" }, { "title": "Managed versus self-hosted ingestion", "score": 9, "comments": 11, "created_utc": 1793907035.771, "snippet": "Posting this here. For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the...", "url": "https://www.reddit.com/r/ETL/comments/1xorw70/managed_versus_selfhosted_ingestion/" }, { "title": "Snowflake into PostgreSQL using Hydra ETL", "score": 7, "comments": 0, "created_utc": 1794166319.199, "snippet": "Moving data out of Snowflake into PostgreSQL looks easy until the types come out wrong! That's the kind of silent change I wanted Hydra ETL's new Snowflake source to rule out. Tested on a real...", "url": "https://www.reddit.com/r/ETL/comments/1xr6vtr/snowflake_into_postgresql_using_hydra_etl/" }, { "title": "How does your team deal with source schema changes breaking downstream pipelines?", "score": 6, "comments": 12, "created_utc": 1793945050.435, "snippet": "I've been thinking about how much time data teams spend dealing with unexpected schema changes. A source column gets renamed, a new field appears, a data type changes, or an API response gets...", "url": "https://www.reddit.com/r/ETL/comments/1xp5gsi/how_does_your_team_deal_with_source_schema/" }, { "title": "Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.", "score": 5, "comments": 10, "created_utc": 1793880289.546, "snippet": "The title. I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice...", "url": "https://www.reddit.com/r/ETL/comments/1xogyli/notsobeginner_wanting_to_independently_build_etl/" }, { "title": "How do you stop an agent from making unnecessary tool calls?", "score": 3, "comments": 4, "created_utc": 1793970223.618, "snippet": "I’m building a company research agent that can query external data through MCP, and I’m using Coresignal’s MCP for the company and employee lookups. Letting the model decide what to fetch works...", "url": "https://www.reddit.com/r/ETL/comments/1xpca8m/how_do_you_stop_an_agent_from_making_unnecessary/" }, { "title": "Added institutions, funders and code in Research @ foo🦍", "score": 3, "comments": 0, "created_utc": 1793862921.529, "snippet": "foorilla.com v1.14.15 | foorilla API v1.10.1 Research papers now come with their scholarly record from OpenAlex: citation counts, the journals and conferences they also appeared in, retraction...", "url": "https://www.reddit.com/r/ETL/comments/1xocdqb/added_institutions_funders_and_code_in_research/" }, { "title": "Added weekly and monthly company job history", "score": 2, "comments": 0, "created_utc": 1794216957.92, "snippet": "jobdataapi.com v4.39 / API version 1.37 We’ve added a new /api/companyjobhistory/ endpoint with weekly and monthly job count history for every company. Each row covers one completed period and...", "url": "https://www.reddit.com/r/ETL/comments/1xrnf6m/added_weekly_and_monthly_company_job_history/" }, { "title": "Added API endpoints for entities @ foo🦍", "score": 2, "comments": 0, "created_utc": 1794201689.6, "snippet": "foorilla.com v1.14.18 | foorilla API v1.11 November 9, 2026 The organizations and people behind jobs, stories, papers, code and events are now in the API. Each entity comes with what identifies...", "url": "https://www.reddit.com/r/ETL/comments/1xrjaji/added_api_endpoints_for_entities_foo/" } ], "newest_posts": [ { "title": "Has anyone used Drogon framework to build a Data Pipeline????", "score": 1, "comments": 0, "created_utc": 1794341928.614, "url": "https://www.reddit.com/r/ETL/comments/1xsxhpc/has_anyone_used_drogon_framework_to_build_a_data/" }, { "title": "I built an open-source tool for migrating data between PostgreSQL and Neo4j (RtoGtoR)", "score": 0, "comments": 0, "created_utc": 1794334839.136, "url": "https://www.reddit.com/r/ETL/comments/1xsudym/i_built_an_opensource_tool_for_migrating_data/" }, { "title": "Added weekly and monthly company job history", "score": 2, "comments": 0, "created_utc": 1794216957.92, "url": "https://www.reddit.com/r/ETL/comments/1xrnf6m/added_weekly_and_monthly_company_job_history/" }, { "title": "Added API endpoints for entities @ foo🦍", "score": 2, "comments": 0, "created_utc": 1794201689.6, "url": "https://www.reddit.com/r/ETL/comments/1xrjaji/added_api_endpoints_for_entities_foo/" }, { "title": "Suggestions for streamlining data processing", "score": 1, "comments": 3, "created_utc": 1794182085.118, "url": "https://www.reddit.com/r/ETL/comments/1xrcvjs/suggestions_for_streamlining_data_processing/" }, { "title": "Snowflake into PostgreSQL using Hydra ETL", "score": 7, "comments": 0, "created_utc": 1794166319.199, "url": "https://www.reddit.com/r/ETL/comments/1xr6vtr/snowflake_into_postgresql_using_hydra_etl/" }, { "title": "How do you stop an agent from making unnecessary tool calls?", "score": 3, "comments": 4, "created_utc": 1793970223.618, "url": "https://www.reddit.com/r/ETL/comments/1xpca8m/how_do_you_stop_an_agent_from_making_unnecessary/" }, { "title": "Built an AI extension that generates complete Python data pipelines from data samples", "score": 9, "comments": 1, "created_utc": 1793967316.78, "url": "https://www.reddit.com/r/ETL/comments/1xpbc3e/built_an_ai_extension_that_generates_complete/" }, { "title": "Automating Data Preprocessing: A Guide to Python AI Data Cleaning", "score": 2, "comments": 0, "created_utc": 1793966762.941, "url": "https://www.reddit.com/r/ETL/comments/1xpb655/automating_data_preprocessing_a_guide_to_python/" }, { "title": "How does your team deal with source schema changes breaking downstream pipelines?", "score": 6, "comments": 12, "created_utc": 1793945050.435, "url": "https://www.reddit.com/r/ETL/comments/1xp5gsi/how_does_your_team_deal_with_source_schema/" } ], "words_this_week": [ { "word": "data", "titles": 26 }, { "word": "pipeline", "titles": 12 }, { "word": "pipelines", "titles": 9 }, { "word": "built", "titles": 7 }, { "word": "source", "titles": 7 }, { "word": "real", "titles": 7 }, { "word": "lakehouse", "titles": 6 }, { "word": "dbt", "titles": 6 }, { "word": "postgresql", "titles": 6 }, { "word": "build", "titles": 6 }, { "word": "tool", "titles": 6 }, { "word": "added", "titles": 6 } ], "preview": "This is a preview. Every post and comment in r/ETL, live, as JSON or RSS on the same paths a Reddit client uses, plus keyword streams to Slack, Discord, webhook or email, needs a free key: https://app.redlisten.com/?signup=1&utm_source=redlisten&utm_medium=subreddit", "docs": "https://redlisten.com/docs", "attribution": "Source: Redlisten, https://redlisten.com/r/etl" }