- Built an End to End Azure Lakehouse (ADF, Databricks, Delta lake, dbt, Airflow) - Key Learnings & Architecture Patterns
Hi Everyone! I recently completed a hands on data engineering project called WeatherFlow - Azure based Weather Data Lakehouse. I built to gain practical experience with tools and workflows. The...
14 points3 comments6 d ago
- Built an AI extension that generates complete Python data pipelines from data samples
Hey, I'm Kevin, I run product at PrettyWhale.ai, a small team based in France. We've spent the last few months building a VS Code extension that writes the tedious part of ingestion code from a...
9 points1 comment5 d ago
- Managed versus self-hosted ingestion
Posting this here. For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the...
9 points11 comments5 d ago
- Snowflake into PostgreSQL using Hydra ETL
Moving data out of Snowflake into PostgreSQL looks easy until the types come out wrong! That's the kind of silent change I wanted Hydra ETL's new Snowflake source to rule out. Tested on a real...
7 points0 comments2 d ago
- How does your team deal with source schema changes breaking downstream pipelines?
I've been thinking about how much time data teams spend dealing with unexpected schema changes. A source column gets renamed, a new field appears, a data type changes, or an API response gets...
6 points12 comments5 d ago
- Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.
The title. I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice...
5 points10 comments6 d ago
- How do you stop an agent from making unnecessary tool calls?
I’m building a company research agent that can query external data through MCP, and I’m using Coresignal’s MCP for the company and employee lookups. Letting the model decide what to fetch works...
3 points4 comments5 d ago
- Added institutions, funders and code in Research @ foo🦍
foorilla.com v1.14.15 | foorilla API v1.10.1 Research papers now come with their scholarly record from OpenAlex: citation counts, the journals and conferences they also appeared in, retraction...
3 points0 comments6 d ago
- Added weekly and monthly company job history
jobdataapi.com v4.39 / API version 1.37 We’ve added a new /api/companyjobhistory/ endpoint with weekly and monthly job count history for every company. Each row covers one completed period and...
2 points0 comments2 d ago
- Added API endpoints for entities @ foo🦍
foorilla.com v1.14.18 | foorilla API v1.11 November 9, 2026 The organizations and people behind jobs, stories, papers, code and events are now in the API. Each entity comes with what identifies...
2 points0 comments2 d ago