How Airbnb Scaled by Moving Away From a Monolith

4 lessons from Airbnb's scaling strategy, with comparisons to other big tech companies

Engineer’s Codex is a newsletter about real-world engineering.

---

Airbnb started as simple Ruby on Rails monolith in 2008.

In 2018, Airbnb began its migration to a service-oriented architecture, as the Ruby on Rails “monorail” started becoming hard to maintain and was a single point of failure.

The main difference between SOA and microservices has to do with the architecture scope.

In an SOA model, services or modules are shared and reused enterprise-wide, whereas a microservice architecture is built on individual services that function independently. (Source)

While Airbnb didn’t necessarily need to move to a SOA, they chose to as it made sense for their organizational needs.

In a recent 2023 talk, they outlined four lessons:

• Invest in shared infrastructure early

• Simplify service dependencies

• Centralize data hydration (fetching and transformation)

• Separate UI logic from backend logic

In this article, I summarize the talk and make architecture comparisons with other large tech companies, like Meta, Google, and Uber.

Invest in shared infrastructure early

Airbnb built common infrastructure shared by teams across services.

1. In-house API framework built on Apache Thrift

   1. Allowed all Airbnb services to define clean APIs and communicate with each other.

   2. The framework handled inter-service communication, error propagation, observability metrics, and schema validation.

   3. Engineers could focus on implementing core business logic.

   4. Facebook/Meta uses a customized, internal version of Apache Thrift called fbthrift, which they have open-sourced.

   5. Google, Netflix, Square, and other companies use gRPC.

2. Powergrid: An in-house library for running tasks in parallel

   1. Helped organize code into a directed acyclic graph (DAG), enabling tasks to run concurrently.

   2. Useful for tasks like validation and data aggregation.

3. OneTouch: A framework built on Kubernetes for managing services and deploying them efficiently.

   1. Ensured that all service configurations were managed in one place.

   2. Provided a command-line tool for deploying services to different environments.

   3. Google uses Borg as a cluster manager. It was a predecessor to Kubernetes. Google and Google Cloud run on Borg.

   4. Facebook/Meta uses Twine for cluster management.

4. Spinnaker: An open-source continuous delivery platform that facilitated safe and repeatable workflows for deploying changes to production.

   1. Featured automated canary analysis to ensure smooth deployments at scale.

   2. Automated canary analysis lets you partially roll out a change then evaluate it against the current deployment to assess its performance.

   3. Spinnaker was created at Netflix.

   4. Uber has a fantastic talk about safe and fast deploys at planet scale.

Source

By investing in this common infrastructure, Airbnb was able to accelerate its migration to SOA and reap the benefits of higher reliability and business agility.

Simplify service dependencies

As Airbnb continued to scale its architecture, the team separated services into layers based on technical priorities. This approach helped reduce complexity and ensure that stable services did not depend on more volatile ones.

Service Tiers

A higher-tier service could call a lower-tier service, but not vice versa. To enforce this topology-driven layer architecture, Airbnb introduced service blocks at the platform layer, each containing related business functionalities.

Example of service blocks

These blocks encapsulated both data and business logic, offering a simple and consistent API to upstream clients. This approach simplified service dependencies, making it easier for engineers to discover and leverage core functionality.

Centralize data hydration (fetching and transformation)

Data hydration is the process of importing data into an object.

It’s a common task in presentation services, involving fetching and merging data from multiple downstream services.

To streamline this process and avoid duplication (which is expensive at scale!), Airbnb introduced a centralized data access layer. This layer provided a consolidated GraphQL schema that stitched together different entities, such as listings, users, and reservations, from various data sources.

By centralizing data fetching and hydration logic, engineers could focus on product innovation rather than writing the same data hydration code repeatedly.

The data access layer also introduced a declarative approach to data retrieval, enabling engineers to write queries based on the data they needed, reducing potential errors and improving efficiency.

Separate UI logic from backend logic

To ensure a consistent user experience and streamline development, Airbnb introduced the App Framework, a unified, service-driven UI system. This framework allowed Airbnb to present a standardized API to clients and stopped engineers from constantly re-inventing the wheel with their own custom APIs.

The backend drove the content and styling of the UI within each section of a rendered frontend page. The frontend was mostly responsible for rendering each section.

The team broke down the UI into standardized sections, each with its data model and UI component type. This approach provided clear patterns for reusability and customization, allowing product teams to launch new features across clients without mobile app versioning or frontend deploys.

UI data was sent in from the backend.

They also showed off some handy dev tooling.

Standardizing the client-facing API is a common move for companies as they scale. DoorDash had a great post about their experiences with separation too.

Takeaways

• Undoing work is normal. Throughout the entire process, Airbnb usually had to undo earlier work. While it can be painful to delete code, it’s important to stay detached from it.

• Splitting out services allow teams to move at different speeds. At Airbnb, frontend teams commonly moved faster than other areas. The above move allowed each team to move forward at their own pace rather than be held back or pulled forward by another team.

• Make the move only when you have to. Airbnb operated fine on a single Rails monolith for a decade! They only made the migration to a SOA after it was getting really painful to continue development on the monolith.

• Microservices are more for scaling organizations than applications.