What r/dataengineering is talking about this week

Not a troll post but an honest question, and maybe more about Microsoft vs Snowflake/Databricks. Why is Fabric so far behind its competitors. Not so much from the perspective of specific UI/UX or...

Hey all, so, I have been doing this for about 6 years now on a legacy on-prem stack. Every time I try to transition to something more modern - dlt/dbt I get shot down. Stick to .NET, that's what...

I have been offered a position as a Junior Data Engineer at QB. The problem is that I don't really know too much about their reputation and if I would gain experiences and skills that would...

My fellow data wranglers, Has anyone built a real production solution using DuckLake? I mean something with an ELT pipeline and actual users, that would have sid ducklake as a component?

The title. I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice...

Current setup Azure SQL Database (Standard S2, DTU model). No linked servers or cross-database queries. Power BI Pro for dashboards (40 users, 3 authors). We have no tool for scheduled or...

hey everyone! I have about a few TB worth of code repository data (from git-client compatible source control) and I need to be able to run recurrent scans on the code itself (not on the repo...

Hi, I'm close to releasing our Dataform branching and release process, but I'm unsure about the hotfix approach. Current setup: Dataform's default branch is main, so every workspace is cut from...

My org has only OLTP, we are pulling data from it directly to powerbi and building facts and dims in semantic model. Now my org is thinking about creating datamarts for each department since there...

So at work I’m using Azure Pipelines as an orchestrator to materialize several tables in parallel. Each one is running a Create Table As Select sql statement and uses a DBT merge strategy so that...