Robert Chang's Blog

Airbnb's Semantic Layer: MinervaSQL

Sun, 16 Aug 2026 01:00:00 +0000

Introduction

This is the third and final post in the Airbnb Semantic Layer series (see Parts I and II), covering the engineering decisions that shaped Minerva, our semantic layer, into what it is today. Minerva has 4,000+ consumers at Airbnb, and their use cases vary widely. The query layer is what makes data access possible at that scale. If you want a primer before reading on, Barak gave a lightning talk that introduces MinervaSQL, our query layer.

MinervaSQL

In the previous post, we highlighted that Minerva Compute creates more than 10,000 materialized tables. At that scale, no user can reasonably remember which tables to query. One common solution is to denormalize data that gets queried together, so users only have to hit a handful of wide tables. The logical extreme of this is the One Big Table (OBT) approach, where everything is denormalized into a single wide table. At Minerva’s scale, OBT is prohibitively expensive. Imagine how many dependencies such a table would carry, not to mention the time it would take to backfill or the storage it would consume.

Minerva’s query layer applies the OBT idea to a virtual table (magic.all), so nothing has to be materialized into one giant physical table. Users write queries in a custom dialect called MinervaSQL against that virtual table, and the query layer rewrites them against the physical tables, working out the joins automatically. Metrics are queried through a special AGG syntax, which gets rewritten into the right aggregate expression based on the metric definition.

This abstraction cut down how much SQL users had to write to answer their questions. Over time, we added more syntax to the dialect to support sophisticated query patterns. FILTER lets users define filtered metrics on the fly. With SHIFT, time-over-time comparisons became easy to express. We also added configuration that lets advanced users control which source to query when several are eligible to serve the same data.

Rewriting

A big challenge of the One Big Virtual Table (OBVT) approach is turning a query against the virtual table into queries against the physical tables underneath. We made this possible with Rewriter, a pipeline of rewriting rules over SQL ASTs. Each rule receives a syntax tree and returns a transformed one, and at every step the tree remains a valid MinervaSQL expression.

The pipeline has many rules, but the rewriting comes down to a few key steps. The first is temporal and metric resolution. Earlier we mentioned the AGG expression, which doesn’t contain the real metric aggregate expression. The metric resolution rule expands it into that expression and adds a source hint tracking where the metric came from. Those hints are the breadcrumbs the rest of the pipeline follows to expand and rewrite the query.

Working from the source hints, the rewriter then tries to create a single SELECT per scope, which is what eventually lets us swap entities for physical tables. For a metric from a single source, this is relatively straightforward. Derived metrics and drill-across queries take more work: the source-tagged aggregates get split into one subquery per source, then joined back together with a FULL OUTER JOIN on the shared dimensions. The last step replaces all entity references with physical tables, leaving no magic tables and no @ columns anywhere in the tree.

This only touches the surface of the pipeline. We’ve added many rules over the years to accommodate how people actually query. Since rules are pure functions and independently testable, we’ve built detailed test cases and fixtures for different scenarios, which is why we can change the rewriting logic without worrying much about breaking it.

Source Selection

The section above reads as if there’s always exactly one physical table that can answer a metric-by-dimension query. In practice there usually isn’t, because Minerva builds precomputed datasets like dimension sets and rollup sources to serve queries from. Paying extra cost on write buys enormous savings on read, and picking the cheapest correct dataset for a given query is a big part of what makes Minerva both performant and trustworthy.

The algorithm walks a fallback chain. Rollup sources get tried first, since pre-aggregated data is the cheapest thing to read. A rollup qualifies when its dimensions are a superset of the query’s group-by dimensions, so the query can re-aggregate up to a coarser grain, and when every aggregate in the query decomposes into additive components the rollup already stores.

Plenty of queries fail those checks, whether because they aren’t aggregate queries at all, or because a metric or a grain isn’t covered. Those go to dimension sets, where we look for a set containing all the dimensions the query touches. Serving from a dimension set still helps enormously, since joins are usually the expensive part of a wide multi-dimension query. If nothing precomputed qualifies, we fall back to joining and aggregating on the fly.

All of this rests on every option returning the same result, since otherwise the numbers users see would depend on which source happened to win. That turns out to be hard, because the pipelines behind these datasets land on different schedules, which leaves the data across them eventually consistent.

Inconsistency was a big problem when MinervaSQL first launched, and we introduced WATERMARK to address it. Watermarking reads everything as of the oldest available partition, so all sources agree on what “now” means. Dashboards that need both speed and recency get HYBRID DENORMALIZATION, which UNIONs precomputed data with recent data at a split date. This area is still under active development, since freshness and consistency trade against each other.

Auto-denormalization

As described above, precomputed datasets are much more efficient to read from, but curating them can be tedious. In the early days of Minerva, users curated dimension sets by hand. That took time, and different teams would end up creating very similar sets. Minerva 1.0 had no mechanism for pre-aggregated datasets at all, so everything got ingested into Druid instead. That worked for a while, though operating it was always a pain because it required specialized Druid knowledge.

In Minerva 2.0, we flipped the model so that most datasets are created in the background without users knowing about it. An auto-denormalization framework does this by reading MinervaSQL query history, modeling the space of possible denormalizations, and proposing denormalized datasets whose query savings most outweigh their materialization costs.

At Airbnb, query history lives in Elasticsearch, with every query MinervaSQL serves logged as a document. Rewriting a query means the system already knows which sources get touched, which metrics were aggregated, and which dimensions were grouped, so for any given query we can recover what was queried, what was aggregated, and at what grain. From there we de-duplicate queries and collect query counts, which give us a way to estimate query cost. We also track dimension cardinality, row counts, and the number of joins, all of which feed into the materialization cost calculation later.

That leaves the selection algorithm itself. Precomputed datasets aren’t free, so the job is to find the sets whose query savings most outweigh what they cost to materialize. Harinarayan, Rajaraman and Ullman worked this out in Implementing Data Cubes Efficiently, which describes the greedy selection procedure at the heart of our implementation. Glossing over a lot, the selector tells us which precomputed datasets would pay off most, and we create those behind the scenes. It runs on a regular schedule, since query patterns keep shifting.

Summary

When MinervaSQL was first introduced, it didn’t immediately become the workhorse for our users. Many people were still used to writing their own queries against physical datasets. We spent a lot of time evangelizing what MinervaSQL could do, and over time users started to see how complex queries could be expressed in a much simpler dialect, which is when it began to gain traction.

With AI, MinervaSQL became the engine behind data agents. Natural language can be translated into MinervaSQL, which then translates deterministically into the dialect the compute engine understands. That gives us self-serve analytics without giving up correctness. A lot of good engineering went into this, and I think it’s what makes Minerva’s semantic layer state of the art in the industry. Huge kudos to coworkers like Barak, who built this from the ground up.

That concludes the three-part series. If you’ve read this far, I hope you came away with a better sense of how we built and scaled Minerva at Airbnb. It’s been a fun ride, and I feel lucky to have been part of it.

Airbnb's Semantic Layer: Compute

Thu, 13 Aug 2026 01:00:00 +0000

Introduction

This is a follow-up to Airbnb’s Semantic Layer: Developer Experience, where I covered the engineering decisions that shaped Minerva, our semantic layer, into what it is today. Minerva is the largest data framework at Airbnb by Airflow DAG count, and it regularly accounts for more than half of the company’s backfill capacity. This post is about the compute side: how we keep 10,000+ Minerva sources up-to-date across 20,000+ Airflow DAGs.

Compute

Once semantics are defined in the semantic layer, we want to materialize those definitions so the query layer can read from them. Thousands of sources and definitions change constantly, and with 100+ PRs updating these definitions every week, we need a framework that can manage those changes at scale.

To start, we model the entire semantic layer as a graph. Each node is a dataset registered in Minerva, and two nodes connect if one was created downstream of the other. That structure lets us identify which subset of the graph changes whenever a definition gets updated. The compute framework then figures out what changed and backfills the affected datasets automatically.

For every source, Minerva continuously reconciles the desired state of a dataset against its actual state. Any gap between the two becomes a unit of backfill work, and the framework dispatches it. As datasets get updated, we track their state. Compute uses these states to trigger downstream processing, and the query layer relies on them to make sure it only queries up-to-date datasets. When a dataset goes stale or breaks, our janitor process handles lifecycle cleanup.

The sections below cover each of these in more detail. We will use dataset and source interchangeably for the remainder of the post.

Detecting Change

Each dataset in the Minerva graph has a data version, an MD5 hash of every field in the YAML file that encodes the dataset’s semantics. Each versioned dataset materializes into an Iceberg table in the warehouse. When a producer edits a config, say by adding a dimension filter or changing a column’s projection, the data version updates, telling Minerva to recompute the physical data to match the new definition. That covers a single dataset. Things get more interesting when datasets depend on each other.

Some datasets in Minerva are derived from others. If a user defines an event source from a fact table and another user defines a dimension source from a dimension table, Minerva can pre-join them into a dataset called a dimension set, or pre-aggregate the events up to a particular grain to create a rollup source. These pre-joined and pre-aggregated datasets depend on the event source, so when that event source changes, they have to be updated accordingly.

We manage these dependencies through chained data versions. A downstream source’s data version includes the upstream source’s data version as one of its inputs. In the example above, the event source’s data version is folded into the data versions of both the dimension set and the rollup source. When the event source changes, those two downstream versions change with it. A single YAML edit can turn into a wave of updates across the graph.

This design cascades change well, but its change detection can be overly aggressive. Say we add a new column to the event source. Downstream sources that never reference that column don’t need to be recomputed. They get backfilled anyway, because all we know is that the upstream version changed. Workarounds exist, like pinning a source to a fixed data version, though they get complex fast for large-scale changes. This is an area I wish we’d invested in more, since better change detection could meaningfully cut how much Minerva has to backfill.

Some tools have taken this further. SQLMesh has invested heavily in sophisticated change detection, introducing concepts like breaking and non-breaking changes based on comparing canonicalized SQL ASTs. I believe this is the direction transformation tools should go.

Reconciliation

With change detection covered, we can zoom in on a single dataset and see how the compute framework reconciles changes. Since most datasets at Airbnb are date-partitioned, the reconciliation algorithm depends heavily on partition-level operations.

Reconciliation Algorithm

Every day, each dataset works out which input partitions exist and which output partitions have already been written. From that difference it generates a plan, batching the missing date partitions into windows and backfilling them in parallel.

In practice, this reconciliation runs per dataset, per day, in an Airflow DAG. A centralized control plane that loops through all datasets could work just as well, since the reconciliation algorithm itself doesn’t depend on how the work gets scheduled. The same algorithm handles several scenarios, and each one dispatches work differently.

Recurring Runs

The most common scenario is a source with no changes to its definition. New input data lands each day, and Minerva’s job is to materialize it. The algorithm notices that all input partitions exist while the output partitions stop at the previous day, so it generates a single batch covering one date partition. Once that partition is written, the dataset is up-to-date.

Partitioning is what makes incremental compute possible. Some datasets, though, have late-arriving events like cancellations or alterations, so their history keeps changing well after the fact and can’t be processed incrementally. We reprocess those from the beginning every day, which makes every run a full historical backfill. That’s expensive, so we later extended the algorithm to let users specify an output window, where only the most recent X days get reprocessed instead of the whole history. This works in practice because most users don’t need data that far back.

Offline Backfills

In the first version of Minerva, every change was backfilled directly in production. We soon found that as users changed business definitions more often, Minerva couldn’t keep up, since data stayed unavailable while a backfill was running. That caused a string of availability problems and, eventually, a major incident I described in Reflections on Airbnb. We learned that users needed an isolated environment to backfill in.

Luckily, each versioned dataset already materializes into its own table, so we can kick off backfills and write new data into a table completely isolated from production. The same reconciliation algorithm applies here. It sees that all input partitions exist while the output table is empty, so it backfills the table from the beginning of time to the current date. This can take a while, so the algorithm splits the work into disjoint batches and runs them concurrently, which cuts backfill time considerably.

Offline backfills mean users can rebuild their datasets without time pressure, and when a dataset is backfilled, promotion to production is instant. Tools like SQLMesh have reached the same conclusion and shipped features such as Virtual Data Environments.

Online Backfills

The last scenario sits between recurring runs and a full historical backfill. Occasionally, users discovered that the input tables feeding Minerva were corrupted. Fixing means restating the affected Minerva data from the corrected input table, without touching the other partitions. This is exactly what online backfill does. Users surgically re-materialize a subset of Minerva partitions in place.

We adapted the self-healing algorithm to handle this. When an online backfill is triggered, we force a re-run of a subset of the output partitions by telling the algorithm they do not exist. From there it takes its usual course, creating batch windows and dispatching backfills accordingly. The data lands in the same production table under the same data version, since no new semantics were introduced.

Putting Everything Together

Right before I left Airbnb, the team had been working to unify these disparate workflows into a single workflow with distinct phases. A new dataset typically moves through a dry run, then an offline backfill, and eventually gets promoted to production, where it runs on a recurring basis.

Tracking Source State

With thousands of sources that need to be processed in the correct order, we need to track the state of each one closely. That matters even more because backfilling is a long, async process that depends on how much compute capacity or orchestration slots are available at any given time.

We addressed this with source state, a suite of API endpoints for writing and reading the current state of a source. It tracks the source’s latest fingerprint, data version, and available partitions. Once a source is fully backfilled, we post an update to the API, and compute and the query layer both read from it.

For compute, each dataset is scheduled in its own Airflow DAG, so we built a custom source state sensor to coordinate dependencies between them. It pokes the state of the upstream datasets a DAG depends on, checking whether they’ve landed, and only passes once they have. That keeps a downstream dataset from starting before its inputs are ready.

The query layer uses source state through a fingerprint, a hash of the physical data that was just written. Unlike a data version, a fingerprint tells you whether the physical data actually changed. For Iceberg tables, it’s the snapshot_id of the new Iceberg snapshot. Other engines use a SHA256 hash of the table name and the latest partition date instead. With the fingerprint, the query layer can use it as a cache to serve common queries.

Lifecycle Management

When new datasets come online, older ones go stale, and Minerva runs a janitor process to clean them up. Deletion happens in two phases. Soft deletion comes first, then hard deletion. Candidates for soft deletion are flagged based on usage, aggregated across query logs. When a source is soft deleted, we hide its data assets from search in the catalog and pause the associated pipelines, without deleting any data.

After a grace period, hard deletion happens. Hard deletion cleans up both the configurations and the underlying data. For configuration, we have a set of deleters that walk the Minerva graph and clean everything defined downstream of a source. Once that change merges, the Airflow DAGs get dismantled, and a separate janitor drops the underlying tables and their storage.

We have a robust lifecycle management process today. I wish we’d built it this way earlier in Minerva’s history. It’s hard to ask users to do this kind of cleanup unless most of it is automated. During my time operating Minerva, we found many opportunities to reduce waste and cost, thanks to these lifecycle management tools.

Summary

This post covered Minerva’s core challenge. Semantic definitions get created, updated, and deleted constantly, and building a compute framework that keeps these datasets up-to-date and consistent sits at the heart of the system.

I walked through the platform’s key capabilities: change detection, reconciliation, state tracking, and lifecycle management, and how they fit together to keep the system running. In the next and final post, we’ll get into the query layer and see how it reads the datasets Compute produces and serves them to consumers at scale.

Airbnb's Semantic Layer: Developer Experience

Mon, 10 Aug 2026 01:00:00 +0000

Introduction

By the time I left Airbnb, Minerva, Airbnb’s semantic layer, had reached every part of the company. We had about 200 data producers submitting 100+ PRs per week, close to 10,000 configs generating 20,000+ DAGs and 10,000+ Iceberg tables. Minerva regularly accounted for more than half of Airbnb’s backfill capacity and was the largest data framework at the company by Airflow DAG count. On the consumption side, it powered 60,000+ Superset charts and Tableau dashboards for more than 4,000 consumers.

Almost none of that was in view when we started. Over the years we made engineering decisions, some good and some bad, that shaped where the platform is today. I want to use this post to share some of those decisions and what we learned from them.

Developer Experience

With so many data producers using our tools daily, workflow design mattered a lot. The number one concern from our users was always iteration speed. People wanted to create, test, and iterate on their changes with a fast feedback loop. That became our north star, and it shaped how we treated configuration, validation, review, and dry runs.

Configuration as Code

Minerva was a configuration-driven framework. Users declared semantic definitions in YAML, and Minerva treated those definitions as code, stored in git, version controlled, and reviewed before they landed. Treating data as code is popular now, thanks partly to tools like dbt, and Minerva did this at Airbnb scale.

Storing semantics in git creates a real challenge around discoverability. Without conventions, there’s no easy way to navigate the files, and when definitions are shared across hundreds of teams, people who can’t find existing ones end up rewriting them. In the first version of Minerva, we integrated with Dataportal, Airbnb’s data catalog, and surfaced configs inline when people searched for metrics and dimensions. The second version went further, adding a dedicated UI, Minerva Studio, that surfaced the same information plus richer metadata.

Readability inside the config was its own problem. The first version had no tool like sqlglot to parse SQL, so we relied on custom fields and a DSL to define metrics, adding more of both whenever people needed to reuse a definition. Over time this got unwieldy. Understanding a definition meant learning the DSL, then piecing it together across several places in the config. The second version moved metric definitions to plain SQL and used sqlglot to parse and validate them, which simplified things considerably, since you could read a definition directly, inline.

Validation

Not every configuration was correct, so the framework needed validation to catch problems before merging. We built a validation suite that ran at two points in the developer lifecycle, locally as users iterated on their configs, and in CI when a PR was created.

In the first version of Minerva, validations were strung together through a series of fragile shell scripts. The same scripts ran for both users and CI, so we often had to add branching logic based on the environment. They also lacked single responsibilities. They called other scripts that did specific validations, and there was no easy way to know who owned what. Being shell scripts, they were close to impossible to unit test.

When we developed the second version of Minerva, my colleagues Philip and Krist took on the daunting task of revamping our validation suite. They used Cerberus, a data validation framework for Python, to standardize the basic YAML validation tasks. In Cerberus you define a validation schema, a mapping of schema keys to schema values, where the values are predefined rules for what each key can take. For example:

schema = {'name': {'type': 'string', 'maxlength': 10}}

This says the name field has to be a string and cannot exceed a length of 10. Cerberus handled a large share of this type of standardized validations well, and we only added custom rules when necessary, which kept validations relatively cheap.

Another improvement Krist introduced was what he called the Spell framework, a CLI tool built on top of Typer. Different validation tasks could be implemented as Spells, and each Spell could be unit tested in our codebase. Each Spell was a CLI command with its own arguments, which users could run locally:

minerva validate

The same Spells users ran locally also ran in CI. One particularly nice set was the auto-fix Spells, which detected issues and fixed them on the user’s behalf. Some of these operations were slow, so we started tracking how long CI jobs took to make sure we weren’t introducing regressions. Every second of CI time showed up in someone’s iteration loop.

We also put considerable effort into documenting what each field means, and we added clear error messages along with breadcrumbs that showed users how to fix a misconfiguration. This cut down how often people had to come to the oncall channel.

Ownership, Review, and PR Approvals

As more business definitions got codified in Minerva, we realized the platform was not just a store for the source of truth. It was also the machine that facilitated change management and data governance at scale. Users had to see what changed and who changed it. They also needed a way to reach the owners when something needed discussion. We treated ownership as a first-class concept from day one, and that had real implications for review, approval, and lifecycle management.

Minerva had a concept called a team. Each team carried information like a list of maintainers, PagerDuty emails, and Slack channels. Teams could be attached as owners to specific datasets, and owning a dataset came with real responsibilities.

Owners were tagged whenever someone else edited one of their datasets, and they could approve or request changes before the edit merged. The PR review process became a place for different teams to debate and reconcile how business semantics should be defined. It did introduce friction into the developer flow, and collecting all the stamps from owners could take a while. We introduced different levels of reviewers, so lighter changes went through a lighter process, and we partnered with analytics engineering to build a Minerva reviewer program, where reviewers could veto or approve changes.

Owners were also the first line of defense for pipeline failures and delays. Any alert triggered by a failure went to the owning team, and they investigated why a pipeline was delayed or failed, escalating to the Minerva team only when they believed there was a wider infrastructure issue. This worked well for teams with dedicated oncall engineers. Teams without an oncall rotation often got fatigued by the alert emails, ignored them, and came straight to our support channel, which became a heavy source of operational load for us.

Owners were also the ones who could answer incoming questions about what a business definition actually meant. They understood the context better than the Minerva platform team did, so we surfaced them on every Minerva asset page in Dataportal. We did the same in SQL Lab, our SQL editor, so someone who hit a problem querying a metric or dimension could reach the owner directly. In the age of AI, it’s not hard to imagine that these owners would become the curators and stewards of specific domain knowledge in their respective areas. This I imagine will be a key thing that drives context engineering at scale.

Dry Run

In the early days of Minerva, we had no mechanism that let users dry run their changes. All changes were tested in production, and that caused real problems. The iteration cycle was slow. Users had to go through the PR review process and get approvals just to test their changes. It was expensive and risky for data integrity too. We wasted compute backfilling data that could be wrong, and we might publish it without consumers knowing the data was bad.

This shortfall was enough that we introduced a new workflow, dry run. The idea was that users should test their changes before putting them into production. When a user put up a PR, we serialized the contents of the YAML configs and diffed them against what was in production. From that diff we calculated which datasets had changed, similar to how git diff works, and surfaced useful information like the diff tree and the estimated backfill cost of the change. Philip demonstrates this in more detail in his talk here, for anyone curious.

Dry runs did not trigger a full backfill. They ran a targeted date range and wrote the data into a temp namespace isolated from production. When a dry run completed, users ran a command to generate a receipt as proof of work. Without it, validation flagged that the change had not been tested. Dry run was popular once it landed. People no longer had to go through an elaborate review workflow just to test something. They could play with the data in the temp namespace and make further modifications, which sped up the iteration cycle considerably.

Right before I left Airbnb, the team had been investing in the next iteration of the dry run experience. The key change is that users will be able to dry run without even putting up a PR, and the temp data will show up not only in the warehouse, but also in the query layer. This means that users can now query these test data in our BI tools directly! Much of this workflow is CLI driven, which means AI can run it too. The goal is for AI to eventually complete the loop with little human intervention, which will speed iteration even more. I think it could be the next step-function change to developer experience.

Summary

When you’re building a data framework, developer experience matters. This matters less with a handful of users. At a certain scale, small improvements to that experience end up having an outsized impact.

Building Minerva taught us that users care deeply about iteration speed, and that hasn’t changed in the age of AI. If anything, the bar is higher now. Keeping semantics as code in git gave us versioning and review for free, though it put the burden on us to make definitions discoverable and readable. We spent a lot of time on validation, catching errors as early as we could, and auto-fixing issues in the background when that made sense. The dry run workflow got heavy investment too, since it let people test changes end to end. Finally, we invested in ownership. We codified it across review and oncall, then put owners front and center in the data catalog, and that’s what let us scale ownership in a distributed way.

In the next post I’ll go into the second big component of Minerva — compute, and how we built a system that keeps data in the warehouse in sync with the latest business definitions.

Reflections on Airbnb

Wed, 05 Aug 2026 09:00:00 +0000

Introduction

I left Airbnb a few weeks ago, a decade after joining in early 2016.

Taking inspiration from Reflections on Palantir and Reflections on OpenAI, I want to write down some reflections on my time there while the memory is still fresh. This post is part about what I think makes Airbnb unique, and part a recollection of lessons learned over the years. All in all, I left the company with a lot of gratitude, and that’s the spirit I’m writing this in.

Company

Business

When I joined in early 2016, the company was in full growth mode, and I spent my first few years growing the host side of the marketplace. That’s where I learned that running a two-sided marketplace is mostly about designing incentives and rules that don’t overly favor one side or the other. Take Instant Book, where guests can book without host approval. It’s convenient for guests, and it also means hosts have less control over who stays in their home. Or House Rules, useful for setting expectations. Pile up too many of them and you’ve created a hassle for guests. Growing a marketplace means the market design matters just as much as the growth curve.

After several years of high growth, Airbnb announced its intention to go public in late 2019. Then COVID hit: by April 2020, gross nights booked were down 72% year over year, and net bookings turned negative, an unprecedented situation, as cancellations skyrocketed. After raising capital and cutting costs everywhere else, Airbnb made the hard decision to lay off about 25% of its workforce, roughly 1,900 people, which remains the largest layoff in Airbnb’s history.

As the world slowly adapted to life with COVID, business recovery came faster than it did for most of our peers, as travel demand shifted toward domestic trips and rural destinations. Having supply across 100,000 cities and towns and 220 countries and regions meant that wherever demand reappeared, we already had listings there. In a miraculous turn of events, Airbnb went public that December, ringing the bell with hosts around the world. After the year everyone had just been through, it was an emotional moment.

Airbnb has spent the years since perfecting the core product, and it’s now expanding beyond it. Services, Experiences, and a push into boutique hotels are among the newest bets. It’s also exploring what AI can do for the product and the business. Most of these are early and unproven, but a new chapter has begun.

Culture

For a company whose business is predicated on 99% of humans being good (after all, you’re staying at a stranger’s home), Airbnb looks for many of those same qualities in hiring. Beyond the standard interview process, Airbnb runs culture interviews to assess a candidate’s fit with its core values. In mine, I talked about a trip I’d helped plan to Peru. I was told later that the stories I shared embodied the core value of “Be a Host.”

It’s no surprise, then, that my favorite core value is “Be a Host,” and it shows up in small but telling ways. On my first day at Airbnb’s HQ, people were opening and holding doors for me and for each other. That kind of behavior extended beyond small daily etiquette into how people actually worked together. My coworkers were brilliant, and brilliant assholes were rare. People were competitive, rarely combative. It was, on the whole, a place of low drama, where people treated each other with a lot of respect.

On the flip side, this desire to be a great host can sometimes get overextended into playing nice, or even conflict avoidance. This shows up often in consensus building, which can be a slow and sometimes painful process. It often comes as a shock to those used to a culture of direct, blunt feedback.

I do think the cofounders don’t want to fuck up the culture, and they’ve tried to hold onto the early spirit every time the company outgrows itself. That said, it’s hard not to miss some of the early days, when the company was smaller and less formal. Every Friday we had nerds@, where engineers shared what they’d learned or built that week. At world@, product leads frequently showed early snippets of what we were about to launch. Some of that faded as we grew, and I wish we’d kept more of it.

Operating Model

Given that Brian Chesky, co-founder and CEO, is a designer by training, it’s no surprise that he applies that same design mindset to building the company itself, and he’s never shy about learning from iconic companies.

On the fourth floor of Airbnb HQ, frames hang on the walls describing the host and guest journey, inspired by how Walt Disney designed the storyline of Snow White. In the years leading up to COVID, Brian drew heavily on Amazon, with the ambition to grow Airbnb into a Trip platform that went beyond just accommodation. More recently, he’s turned to Apple, reshaping how we do product marketing, launches, and roadmap planning. Each era brought its own ambitions and its own way of working, often under new leadership too.

Every change, though, came with its own growing pains. Pre-COVID, that meant unfocused sprawl: Airbnb was simultaneously pursuing experiences, magazine, business travel, hotels, and even flights, spread across four largely independent business units. Post-COVID, the pendulum swung toward bi-annual, big-bang releases: high-stakes, all-or-nothing launches that made it harder to isolate what was working. And Brian’s founder mode talk spun up its own debates, internally and externally. Despite this thrash, I appreciate that Brian has the ambition to build an iconic company and isn’t afraid to adapt and experiment, treating the company itself as an iterative design problem.

One more operating model choice worth mentioning: Airbnb is one of the few companies that genuinely committed to remote-first since COVID. I’ve benefited from the flexibility, but I miss standing at a whiteboard with a coworker and working a problem out. Keeping that kind of collaboration alive is a hard problem. We’ve hired strong people in places we couldn’t have reached otherwise, though it’s harder to screen for candidates who want the job rather than the flexibility. The remote-versus-office question is challenging, and Airbnb is still working hard to find the right balance.

Data at Airbnb

Teams

In the early days, the entire data team, known internally as the A-team, was small enough to fit in a single room. It skewed toward early-career PhDs from a mix of backgrounds: economics, statistics, operations research, and the social sciences. All of them were sharp thinkers, deeply data-driven, and product-focused.

Organizationally, individual contributors owned specific domains and became experts in them, but they all reported up to a Head of Data. In more recent years, they report to various leads in engineering in a decentralized fashion. By the time I left, Airbnb had hired its first VP of Data Science, so perhaps the pendulum will swing back toward a more centralized data organization. These swings are not uncommon: LinkedIn and Facebook went through similar evolutions.

Airbnb’s relationship with Data Engineering has been a complicated one. The investment started early: many of the founding data engineers came from Facebook, and our early warehouse showed it. The medallion architecture and core data were heavily inspired by what they’d built there. Around 2018, the org dismantled the Data Engineering team, and in my opinion it was one of the costliest mistakes data leadership at Airbnb made.

Luckily, we retained some of the strongest engineers, many of whom moved to Data Platform. Under new leadership, Airbnb reinvested in data engineering hiring in 2019, and the community is strong again today. Airbnb was also one of the first companies at this scale to create an Analytics Engineering organization, thanks in part to its investment in tooling like Minerva, our semantic layer.

Overall, the roles are increasingly specialized: data scientists and analysts focus on product work, analytics and data engineers build company-wide datasets, and software engineers build the underlying platform.

Platform

Airbnb historically leaned “build” over “buy,” and several successful open-source projects were born here, most notably Airflow and Superset.

For offline data, everything lives in a lakehouse: data on S3, stored as Parquet files in Iceberg tables. Data is typically date-partitioned, and we compute incrementally wherever possible, though some late-arriving data (think cancellations or alterations) forces a full history rewrite. Spark is the main engine for batch, Flink for streaming, and Trino for interactive queries. For orchestration, Airbnb runs one of the largest Airflow deployments in the world, often at a scale the open source community isn’t equipped for.

Airbnb leans heavily on config-driven frameworks, so much so that some joke that our data contracts are entirely built on fragile YAML files. The frameworks built this way are still widely adopted: ML feature platform Chronon, Minerva as our semantic layer, and an experimentation platform called ERF, to name a few. Python is the primary language for building these data frameworks.

We invest just as heavily on the consumption side. Dataportal is a catalog and UI that helps people find the right data, and a unified metadata service sits underneath it, storing the metadata every other data tool depends on, such as ownership, landing times, and asset tagging. More recently we built an internal data agent, and it has shown real promise, largely because the semantic layer and metadata service were already there for it to stand on.

I’m biased here. I think Airbnb’s data ecosystem is sophisticated and underrated compared to peer companies. Investment in data was one of the reasons I joined, and it held up.

Semantic Layer

For seven of my ten years at Airbnb, I worked on the company’s semantic layer, Minerva. People at other companies often ask how we scaled it across the entire organization. There wasn’t one standout strategy. It came down to aligning with company initiatives, finding champions, and a relentless ownership mindset.

As early as mid-2018, we were already thinking about consolidating definitions across business metrics and experimentation metrics under a single source of truth. The real catalyst came around 2019, as Airbnb prepared to go public and data quality, or the lack of it, became an existential problem. Years earlier, we’d made the mistake of dismantling our Data Engineering team, and we’d been paying for it ever since. Different teams built their own versions of “bookings,” “active listings,” and “revenue.” When Brian asked for last week’s bookings number, he’d get a different answer depending on who he asked. For a company about to report to public markets, that was untenable. Our CTO was so concerned he declared “data bankruptcy.”

Out of that urgency came three efforts. First, a company-wide data quality initiative to rebuild the most business-critical data models from the ground up. Rebuilding models once wasn’t enough to keep them trustworthy, so we created a certification process called MIDAS to hold data to a consistent bar of quality. Finally, we invested in infrastructure: a way for data producers to define a single source of truth for business metrics and dimensions that could actually be certified. Minerva became the natural home for that.

IPO-readiness created the urgency. MIDAS gave us the program and process to act on it, and Minerva ended up being the paved path that came out of it. We worked closely with key leaders early on to position it that way, then expanded team by team as wins built momentum. It took about two years before Minerva became the standard tool for Analytics. In some ways, the analytics engineering role at Airbnb exists because we had this central piece of technology sitting in the middle.

Work Lessons

On Building Software

Working at the intersection of software and data engineering, I got to learn from several exceptional software and data engineers who taught me how to think about the craft.

My first big takeaway is that “software engineering is programming integrated over time,” a definition popularized by the book Software Engineering at Google. Design patterns and abstractions are tools for managing complexity, more than ends in themselves. When a group of people with different levels of understanding and mental models work on the same codebase, having these at our disposal is a proven way to evolve the software while keeping everyone’s understanding aligned.

Speaking of design patterns, I learned that they are a useful vocabulary for identifying common problems and possible solutions. Several came up again and again: Adapter pattern, which let us wrap different backend databases behind a consistent interface; Bridge pattern, which we used to decouple the Airflow operator from the unit of work it carries (what we call Step); and Strategy pattern, which became the backbone of our audit framework for checking data quality.

On abstraction: I started off writing procedural code to implement our write-audit-publish (WAP) pattern, then watched more experienced engineers replace it with abstractions that simplified the code I’d written and made writing new code easier. In another example, we introduced a Spell abstraction that lets developers add codemod capabilities, invoked by both developers and CI for config validation. It replaced a series of very fragile shell scripts strung together over the years. Nowadays, if I find myself writing similar code with unnecessary implementation detail for the task at hand more than once, I pause and ask whether there’s a useful abstraction we can introduce to hide that complexity.

I’ve also picked up several other technical lessons along the way: domain modeling, why choosing boring technology is usually the wiser choice, composition over inheritance, dependency injection and inversion, and snapshot testing. Each one probably deserves its own blog post. The common thread is that all of these took time to internalize. I only appreciated them after working in the same codebase long enough to see why they mattered.

On Maintaining Software

There was a period where we were heads down building and re-architecting. There are also periods that call for reliability and stability over agility and speed of change. Navigating different phases of a software lifecycle requires shifting how you think and where you focus, and for me, it took a few painful incidents before I really internalized that.

For a while we treated every dataset roughly with the same priority. Once the platform was adopted company-wide, it became clear that financial reporting and executive dashboards were far more critical than someone’s ad-hoc report. We introduced a data tiering system that lets us prioritize compute resources according to each dataset’s tier and SLA requirements. We also invested in observability. For a long time we didn’t know what “healthy” looked like for most of our systems, so we built layered observability: real-time, intraday, daily, and defined “good” for each tier. To keep low-signal alerts from burying the real issues, we routed alerts to the right owners and tuned thresholds until the ones that fired were worth acting on.

To harden our system, we learned from every incident for corrective action. One recurring lesson: big-bang releases are risky because the blast radius is enormous. We leaned into gradual rollouts, feature flags, and staged deployments, and over time deployments got calmer. We also surveyed the team on which parts of the job were intolerably repetitive and automated as much of it as we could. Restarting failed jobs, validating releases, and clearing alerts by hand were the biggest offenders, and we automated most of them over time.

At times, we needed to make much bigger changes. The accumulated weight of new use cases and tech debt made the architecture itself unmaintainable, and the only real fix was to rebuild the foundation. We did that twice with Minerva over seven years, and I expect it will happen again as requirements and use cases continue to evolve.

On Ownership Mindset

Having worked on Airbnb’s Data Platform for many years, I’ve come to believe that one of the keys separating a successful project from a mediocre one is the level of care contributors bring, the ownership mindset that makes you take pride in your work.

For us, that mindset showed up in everyday choices. While it was often tempting to build the intellectually interesting thing, we pushed back on our own over-designed proposals, and more than once killed them, in favor of something simpler that unblocked users. We cleaned up technical debt when the opportunity came, not because we treated debt as something to avoid at all costs, but because we knew it would help other developers down the line, even if users never noticed. And when users ran into issues, we took pride in unblocking them fast. Our on-call rotation was where we learned what was broken, not just a chore to rotate through.

Ownership mindset mattered even more during difficult times. There were periods when reliability problems hurt our team’s reputation, voluntary attrition left us severely under-resourced, and we couldn’t prioritize the work users were asking for. In moments like those, we called out the problems honestly and took steps to address them before they turned into fires. When the difficulty was structural, organizational misalignment rather than a technical gap, we weren’t shy about rallying leadership to make bigger changes.

I was lucky to work on a team where everyone showed a high level of agency and ownership. That’s part of why Minerva stuck with Airbnb’s data community for as long as it did.

Personal Lessons

Switching Roles

Some of the defining moments of my time at Airbnb came when I stepped outside my existing role and started over from scratch.

From 2016-2019, I worked as a data scientist helping grow our host community across several marketplace tiers (remember Airbnb Plus?). When the company dismantled its data engineering org, I was one of the few data scientists who dove into the wreckage and rebuilt from there, and found I enjoyed the work. Building pipelines and producing high-quality datasets taught me how much leverage comes from the right tooling, so from 2019 to 2021, I pivoted into product management to help scale Airbnb’s semantic layer from 0.5 to 100. From 2021 on, I worked as a SWE on the Minerva team to rebuild our stack from the ground up.

A shift in your interests doesn’t automatically earn the organization’s trust that you can execute in a new role. Every stretch into something new is a bet the org makes on you, and it’s on you to make it pay off. My playbook stayed the same each time: do great work, earn a reputation that precedes you, find an adjacent area that interests you, identify a sponsor, then pivot if that move propels growth.

Switching functions re-accelerates your growth. It isn’t free, either. It can slow your climb up any single ladder, and the expertise you’ve built fades into the background, at least for a while. You have to be willing to sit with that discomfort and be honest about your own learning curve (more on that in the next section).

Looking back, this is the main reason I stayed at Airbnb as long as I did. It felt like three different jobs packed into one decade, each giving me a distinct experience and its own growth opportunities. In the age of AI, I think the people who can operate beyond any single role are the ones best positioned to thrive, and I’m curious where that takes me next.

Growing Pains

With each transition come growing pains. Two stories from my time at Airbnb still stick with me. In one, I leaned in immediately. In the other, I flinched for months before turning it around.

In my first week (yes, first!) as the new product manager for the Minerva team, we had the largest incident in the team’s history, internally known as CIM-198. It was big enough that we had to put a moratorium on Minerva, blocking users from contributing new semantics to our platform. My immediate job was to communicate the scope of the incident and how we’d fix it. My broader job was to work with engineers to find the gaps in our system and build both short- and long-term fixes so it couldn’t happen again.

The whole thing felt like an extended interview, testing product management skills I barely had yet. Our users were patient, leadership gave us room to work, and I partnered closely with engineering to build a solid plan. We shipped several new features, including offline backfill, which still plays a key role in the architecture today. The incident was memorable enough that we eventually printed T-shirts reading “I survived CIM-198,” a badge of honor I hope we never have to award again. I had fond memories of that first week and thought I handled it well.

The second story is from when I first transitioned to software engineering. I worked with a mentor who was extremely capable and had a high bar. His style and presence could be dominating, which made me feel insecure. Unlike the CIM-198 crisis, there was no deadline compelling me to act here, so instead of leaning in, I did the opposite. I worried constantly about people’s perception of me, afraid of wasting his time, asking naive questions, and looking dumb. I’d charge ahead on implementations without checking in, only to surface a PR too late for meaningful feedback. It got bad enough that one day he asked me, point blank, “Are you avoiding me?” That was a rude awakening.

Something he told me afterward has stuck with me since: even if my judgment was 99% bad starting out, as long as I was willing to reflect on why, my taste would improve over time. That was the actual path to growth. It took some psychological work, but I realized a strong re-start meant embracing the growing pains and letting go of my ego. I eventually leaned into the discomfort and changed how I conduct myself. I started pair programming more, asking questions earlier, and letting myself look unsure. Six months later, that same mentor told me another respected engineer on the team considered me one of the most reliable people he worked with.

The difference between the two stories was how quickly I let go of my ego and embraced the growing pains. I’d like to think I’ve gotten faster at that over time.

Seeking Impact

In most companies, growth is measured by your level on the career ladder. It’s a useful external scorecard, but orient your whole career around it and you might end up chasing levels and labels instead of the work that energizes you.

I learned this the hard way as a PM scaling Minerva. By most external measures, things were going well. I had real impact, worked with great people, and the product was growing fast. The day-to-day was wearing me down anyway: constant context switching, no time to think deeply, managing up, down, and sideways all at once. I started seriously thinking about leaving the company.

What changed my mind was a conversation with my partner. She listened to me vent, then said: “It seems like you’re still very passionate about the company and the team’s mission. You’re just in the wrong job.” She was right. I still cared about Airbnb, the team, and the mission. What had happened was that I’d drifted into a shape of work that didn’t fit me, and I hadn’t been honest with myself about it. I’d gotten pulled in by the feeling of having impact and stopped asking whether the work itself was right.

That conversation pushed me to advocate for a move back into a more technical role, one built for deep problem solving. It wasn’t a straightforward move on paper, giving up a PM role to start over as an IC in engineering. It was the right call for what I wanted my days to look like, and it set up the most fulfilling chapter of my time at Airbnb.

Nobody will advocate for the work you love more than you will. Your manager has other priorities to juggle, the org has its own gravity, and the career ladder keeps pulling you toward the next level whether or not that’s what you want. Knowing yourself well enough to push back, and having the courage to do it, matters more than any promotion I got.

Parting Thoughts

I never thought I’d stay at Airbnb this long.

Looking back, it felt like working three distinct jobs across three different companies. Careers are non-linear, and staying self-aware is what let me course correct along the way. In the process, I came to understand what I’m good at, what I enjoy, and what I want more of. I made a real impact, and I was lucky to work alongside some of the kindest and most talented people I know, some of whom I now consider dear friends.

I’ll miss Airbnb dearly. I’m excited to see what the next chapter looks like, and this decade has given me what I need to navigate it.

—

Thank you, Jason, for onboarding me to Airbnb, which led to a decade-long friendship. Thank you, Vaughn, for teaching me everything about Airbnb and what it means to do right by the business. Thank you, Aaron, for teaching me Data Engineering. Thank you, Ricardo and Cuky, for believing in me as a Data Scientist. Thank you, Jeff, for betting on me to lead the Minerva team when I had no PM experience. Thank you, Shao, Dave, and Vyl, for helping me transition to SWE when I asked for it. Thank you to the whole Minerva team for working alongside me all those years. Thank you, Philip, Krist, and Clark, for being constants on the Minerva team and going through the ups and downs together. Thank you, Toby, Chris, Barak, and Ginter, for showing me what it means to be an outstanding engineer.