Subscribe now

On March 31, 2026, Anthropic accidentally shipped a .map sourcemap file inside a Claude Code npm update. In minutes, this was found and was going viral. The 600k lines of code were mirrored, analyzed, ported to Python and other languages, and uploaded to decentralized servers.

Claude Code is known to be notoriously closed down. Their Agent SDKs provide almost no insight into the internals of Claude Code, and Anthropic themselves do their best to keep the source as closed as possible.

How It Happened

Anthropic accidentally included a .map sourcemap file in a Claude Code npm package. Boris Cherny, a Claude Code engineer at Anthropic, confirmed it was plain developer error, not a tooling bug. He followed up with more detail: “Mistakes happen. As a team, the important thing is to recognize it’s never an individual’s fault. It’s the process, the culture, or the infra.”

Blameless post-mortems, popularized by Google’s SRE culture, focus on fixing the system rather than finding someone to fault. The goal is an environment where engineers report mistakes honestly instead of hiding them, which produces better fixes and fewer repeat incidents.

Chaofan Shou (@Fried_rice) was first to notice and posted a public link to the source. Within minutes the race was on.

Chaos and Legality

The most popular fork was claw-code on GitHub, created by @realsigridjin. He cloned the repo, ported it to Python from scratch (using OpenAI’s Codex!) to prevent legal issues, and pushed it.

The legal theory is that a clean-room AI rewrite cannot be touched by DMCA. Claw-code currently sits at 75,000+ stars and 75,000+ forks at the time of writing.

The legal question nobody has an answer to yet: does a codegen clean-room rebuild violate copyright?

Traditionally: “A clean room build in software engineering is a methodology used to re-implement existing software or reverse-engineer a system without infringing on copyright. It involves two separate teams: one analyzes the original software to create specifications, while a second "clean" team creates the new product based only on those specifications, ensuring no proprietary code is copied.”

This takes months and costs real money. It used to be a meaningful barrier. Now anyone with a Claude Max plan can point an agent at a codebase’s tests and have the logic rebuilt overnight. The practice has never been challenged in court.

Gergely Orosz put it bluntly: even if Anthropic tries to assert copyright, do they want the PR battle of suing an open source project for rebuilding their own AI-written product? And could they even prove it?

Meanwhile, 4nzn uploaded a stripped version to IPFS with all telemetry removed, security guardrails removed, and all experimental features unlocked. Whether DMCA can reach content stored on IPFS is itself an unresolved legal question.

At the moment, claw-code is still up, but non-rewritten forks have been DMCA’d by Anthropic.

Here is an example DMCA request Anthropic sent to a source clone: https://github.com/github/dmca/blob/master/2026/03/2026-03-31-anthropic.md

Inside the Source Code

KAIROS: Anthropic’s Response to OpenClaw

Hidden behind feature flags called PROACTIVE and KAIROS, the codebase contains a fully built autonomous agent mode that has not been publicly announced.

KAIROS runs in the background, 24/7, without you asking. It’s similar to OpenClaw.

Every few seconds it receives a heartbeat prompt: “anything worth doing right now?” It evaluates what is happening and makes a call: act, or stay quiet. If it acts, it can fix errors, respond to messages, update files, run tasks. Everything Claude Code can already do, just without you initiating it.

It has three exclusive tools that regular Claude Code does not:

• Push notifications so it can reach you on your phone or desktop even when the terminal is closed

• File delivery so it can send you things it created without being asked

• Pull request subscriptions so it can watch your GitHub and react to code changes on its own

It keeps append-only daily logs of everything it noticed, decided, and did. It cannot erase its own history. At night it runs a process the code literally calls autoDream, where it consolidates what it learned during the day and reorganizes memory. It persists across sessions. Close your laptop Friday, open it Monday, KAIROS has been working the whole time.

The codebase also contains 44 hidden feature flags and 20+ unshipped features total: background agents running 24/7, one Claude orchestrating multiple worker Claudes, cron scheduling, full voice command mode, browser control via Playwright, and agents that sleep and self-resume.

The interesting architectural decision in all of this is the separation of initiative from execution.

Regular Claude Code is reactive: it only acts when you send a message. KAIROS introduces a proactive loop, which requires a fundamentally different trust model. The agent now needs to decide on its own what is worth doing, which means the quality of that judgment becomes far more important than it is in a simple request-response system.

That is a hard problem, and the fact that Anthropic has it fully built and gated suggests they have been working on it for some time.

Anti-Distillation: Actively Poisoning Copycats

Distillation is a process where a smaller, more efficient AI model is trained to mimic the behavior and knowledge of a larger, more powerful “teacher” model, allowing it to perform nearly as well while being much faster and cheaper to run.

Claude Code sends requests with anti_distillation: ['fake_tools'] flags that inject decoy tool definitions into system prompts. The goal is to poison training data for any competitor trying to distill Claude Code’s behavior into their own model.

There is a second layer via CONNECTOR_TEXT: a server-side mechanism that buffers the assistant’s text between tool calls, summarizes it with cryptographic signatures, and returns only those summaries to API traffic recorders. Even if a competitor intercepts API traffic, they only get summaries, not the full reasoning chains.

What makes the fake tools approach interesting is where it operates. It targets the distillation attack surface at data collection time rather than at inference time. If a competitor routes requests through Claude Code to collect training data, they get poisoned tool schemas baked into every prompt, making any model trained on that data less reliable.

The author of this breakdown estimates a determined team could bypass both mechanisms within an hour using a MITM proxy or the CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS environment variable. The stronger deterrent is likely legal rather than technical.

SWE Quiz: Did you know what distillation was?

Distillation is an important concept that comes up in interviews with any company working on their own AI model. Did you know what it was?

SWE Quiz is the best way to figure out what you do and don’t know. It’s like Anki for System Design and AI/ML, with structured courses, quizzes, and case studies.

Join SWE Quiz

Model Codenames

The codebase is littered with internal codenames that were never meant to be public.

• Capybara (also called Mythos): Anthropic is already on version 8. It has 1M context and a “fast mode.” The code notes it still has issues with over-commenting and making false claims.

• Numbat: tagged with a comment reading “@[MODEL LAUNCH]: Remove this section when we launch numbat”, an upcoming model with an announced launch window baked into the source.

• Fennec: speculated by multiple researchers to be Opus 4.6.

• Tengu: referenced in the undercover mode, which strips internal codenames from builds going to external repositories.

The undercover mode (undercover.ts, 90 lines) has a critical design detail: there is no force-OFF. It is a one-way door. Claude Code operating in external repos cannot reference any of these names, internal Slack channels, or even call itself “Claude Code.”

The reasoning is straightforward from a product security standpoint: you want to guarantee internal codenames never escape into the wild, so you make suppression automatic and irrevocable rather than opt-in. But a side effect is that Anthropic employees using Claude Code to contribute to open source projects were not disclosing AI authorship in commits, and the tool was built to make sure that stayed the case.

DRM Below the JavaScript Layer

API requests contain placeholder values (cch=ed1b0) that Bun’s native HTTP stack, written in Zig, overwrites with computed hashes before transmission. This cryptographically proves a request originated from a genuine Claude Code binary rather than a third-party tool wrapping the API.

The reason this lives in Zig rather than in JavaScript is deliberate. JavaScript can be patched, monkey-patched, or proxied at runtime. Zig code compiled into the Bun binary cannot be inspected or overridden without recompiling from source. By pushing attestation below the JS layer, Anthropic made it much harder for third-party clients to spoof.

This is likely why tools like OpenCode ran into friction with Anthropic beyond just legal notices: they were being blocked at the API level too. The mechanism is gated behind a compile-time flag, which suggests it may not be active everywhere, but the design intent is clear.

The Memory Architecture

Claude Code’s memory system is far more sophisticated than “save a CLAUDE.md file.” It operates as a 3-layer index:

1. Index (always loaded): just pointers, ~150 characters per line

2. Topic files (loaded on demand): actual knowledge

3. Transcripts (never read, only grep’d): never loaded into context directly

The write discipline: write to topic file first, then update the index. Never dump content into the index. If a fact can be re-derived directly from the codebase, it is not stored at all.

autoDream consolidates, deduplicates, and removes contradictions from memory in a forked subagent with limited tool access, which prevents it from corrupting the main context while it runs. Memory is treated as a hint, not as truth. The agent verifies before using it.

The core design insight here is bandwidth awareness. Most agent memory implementations load everything into context every turn, which is expensive and introduces noise. Claude Code treats the context window as a scarce resource: the index is cheap and always loaded, topic files are fetched only when relevant, and transcripts are never loaded directly. This keeps the model’s working memory clean without losing access to historical information. What they choose not to store matters as much as what they do store.

Magic Docs: Self-Updating Documentation

One pattern worth borrowing from the internal codebase: Anthropic employees can create files with a MAGIC DOC header. When an internal build of Claude Code is idle, it fires off a dedicated subagent that reads the file, updates the documentation for the specified feature, and writes it back. The subagent is restricted to editing that single file and nothing else.

The result is documentation that stays current automatically, without anyone having to remember to update it. The tool restriction is the key design choice: scoping the agent to one file prevents it from drifting into unrelated changes while still giving it enough access to do something useful.

Summarizing the Harness

Drop DeepSeek or Gemini into the same Claude Code harness with some optimization and you may get improved coding ability with those models too.

The harness includes:

• Live repo context loading on every single turn (git branch, recent commits, CLAUDE.md files reread every query)

• Aggressive prompt cache reuse via a stable/dynamic prompt boundary, so Anthropic is not paying full token costs every turn

• Dedicated Grep and Glob tools instead of shell commands, giving the model better-structured search results

• LSP tool access for call hierarchies, symbol definitions, and references

• 5 distinct context compaction strategies, because context overflow is a central engineering problem

• A 25+ event hook system letting you intercept and modify behavior at every stage of execution

• Three subagent execution models (fork, teammate, worktree) with a prompt caching optimization: forked subagents inherit parent context as byte-identical copies, so spawning 5 agents costs barely more than 1

One design decision worth understanding is the prompt cache boundary. Anthropic pays per token, and a long system prompt rebuilt from scratch every turn would be expensive at scale. By splitting the prompt into a stable front half and a dynamic back half via SYSTEM_PROMPT_DYNAMIC_BOUNDARY, they make sure the expensive static portions are cached and reused across sessions. Anything tagged DANGEROUS_uncachedSystemPromptSection is explicitly marked as cache-breaking, so engineers know the cost before they change it.

Much of this system prompting was embedded directly in the CLI package distributed via npm all along. It was always technically readable. The .map file just made it obvious.

The Claude Code team has talked about Prompt Caching in Claude Code before.

Easter Eggs and Human Touches

• buddy/companion.ts is an April Fools feature that generates a deterministic creature per user: 18 species, rarity tiers, 1% shiny rate, RPG stats deliberately encoded to evade grep checks

• There are exactly 187 spinner verbs (Source: Wes Bos)

• print.ts is 5,594 lines with a single function that is 3,167 lines long and 12 nesting levels deep

• The code has LLM-oriented comments throughout, written for AI agents working on the codebase rather than human readers

March 2026 Was a Security Disaster

The leak did not happen in isolation. It landed one of the worst months in AI developer security on record.

In the same 30-day window:

• Axios (100M weekly npm downloads): maintainer account hijacked, a remote access trojan deployed across macOS, Windows, and Linux. The malware self-destructs after execution and deletes itself from node_modules. Google suspects its from North Korean bad actors.

• LiteLLM (97M monthly PyPI installs): backdoored with a three-stage attack: credential harvester sweeping SSH keys, AWS/GCP/Azure credentials, Kubernetes configs, crypto wallets, and every LLM API key; then Kubernetes lateral movement; then a persistent systemd backdoor. Live for three hours before PyPI quarantined it

• Railway (2M users, 31% of Fortune 500 as customers): CDN misconfiguration leaked authenticated user data to wrong users for 52 minutes

• Delve: YC-backed compliance startup allegedly generating fraudulent SOC 2 audit reports with identical boilerplate across 494 reports, fabricated board meeting evidence, and audit conclusions written before any client submitted evidence

• Mercor AI: alleged LAPSUS$ breach of 939GB source code and 4TB of total data via their TailScale VPN

• OpenAI Codex: command injection via branch names discovered December 2025, patched February 2026, disclosed March 2026. Attackers could steal GitHub auth tokens via unsanitized branch name parameters

• GitHub Copilot: injected promotional ads into 1.5M+ pull requests as hidden HTML comments without developer consent. GitHub VP confirmed it and said it was “the wrong judgement call”

The Claude Code leak is, in some ways, the least technically dangerous item on that list. I’ll probably cover AI security next!

Subscribe now

What Engineers Building Agents Should Take Away

The source code itself will probably get taken down, reposted, and litigated.

Some things stand out as directly applicable:

Split your system prompt at a stable boundary: The SYSTEM_PROMPT_DYNAMIC_BOUNDARY pattern is a practical response to a real cost problem. Static instructions should never change between sessions. Dynamic context should live after the boundary. Getting this right means your prompt cache actually works and you are not paying to recompute the same tokens on every turn.

• Prompt caching might be obvious to anyone who works with agents daily, but if you don’t, know that it’s an extremely important concept for keeping costs low.

Magic Docs: Having self-updating documentation is one of those practices that can upgrade any codebase and isn’t that hard to do with agents today.

Lots of comments: Generally, too many comments were considered too verbose when humans were the only readers and writers of code. But with agents, comments are useful when detailed, especially when they include decisions + reasoning (basically the stuff that’s obvious to humans but not to agents).

Permission classification via side-query: Send the command to Claude as a separate query: “Is this command safe?” The model evaluates context, working directory, and user intent. This replaces brittle allowlists with adaptive, context-aware security. Cost is negligible vs. the risk of a false positive.

• This is known as the “critic” pattern.

Claude Code and other coding agents this year have taken a massive leap in skill and ability. As a result, the nature of work for engineering has changed drastically.

When execution becomes cheap, the bottleneck moves elsewhere. Skills that once distinguished senior and staff+ engineers, such as architectural judgment, context management, and system-level thinking, are increasingly moving “up” the stack and are becoming expected earlier in an engineer’s career.

Historically, individual contributors split their time between implementation and higher-level work like planning and design. As agents take on more of the implementation, the engineer’s role shifts toward planning, architecting, reviewing, and deciding how to best utilize and steer these systems. This is similar to how search didn’t eliminate thinking, but rewarded people who learned how to search well as Google matured.

Writing code was never the hard part. Knowing what code to write, where to put that code, and how to design it was harder.

Introducing Nucleate (Featured)

Nucleate is a platform to prototype, build, and ship beautiful mobile apps with AI.

Past just coding with AI, it can integrate with your design tokens, be used as a Expo Snack replacement, and be used to generate different variants of screens.

You can use code ENGCODEX for 50% off your first month. The team is rapidly shipping updates with backend + Supabase integration, Git integrations, Claude Code subscription usage, and more coming down the pipeline this month.

Visit Nucleate

Maintaining context across multiple domain and projects

When I first worked with any technical L7+ at FAANG, I was always surprised at how much knowledge they had of the stack of a huge product. They were deep in the details of how the product was built, yet still spent most of their time doing high-level architecting. The skill they had here was being able to maintain rich context across multiple domains and projects.

A senior staff engineer generally designs architectures, reviews docs and code, and generally doesn’t write as much code, as that’s delegated to junior and senior engineers. Now, AI becomes the junior engineer.

Reading code is harder than writing it, mostly because writing code fills in the “understanding” gap that reading doesn’t do as well. It’s the same reason students are recommended to write down notes rather than just read the textbook, as writing things down commits things to memory in a different way.

Engineers who can understand system context and also add to that understanding without having to write code themselves will thrive.

Consider an agent tasked with refactoring a backend service to simplify its data model. The change is clean, tests pass, and performance even improves.

What the agent doesn’t know is that another service relies on a subtle ordering guarantee in the original implementation. One that was never formally documented because it lived in shared team knowledge. After the refactor, everything still works in isolation, but downstream systems begin failing under load.

Nothing in the diff looks obviously wrong. Only someone holding context across services would recognize that a “correct” change violated an implicit contract. Yes, this is partly a human problem. But we live in a human world.

The engineer now needs to understand how components interact, track constraints across domains, and notice when a “correct” change has subtle side effects.

Maintaining focus in an asynchronous workflow

With AI agents becoming more and more long-running, it can be tough to maintain focus. The topic of “flow” has been talked about extensively in being important for programming. Flow is useful when working focusedly on one task at a time, constantly thinking, writing code, and reviewing. However, AI agents work autonomously and present changes to you at the end. It doesn’t make sense to constantly watch them as they work, leading to issues maintaining focus.

An agent may take 5 minutes to complete a task, which is enough to check your phone, email, and lose 10+ minutes to distracted tasks. Getting back into gear is contextually expensive, and thus, a real productivity decrease.

Strong engineers will need new habits:

• batching agent requests,

• planning follow-up work in advance,

• treating agent runtime as intentional gaps, not distractions.

Managing attention is already tough enough outside of work, with distractions like social media everywhere, but now it’s part of the job.

Being able to plan and steer AI agents effectively

As agents get better, being able to steer them is more and more important.

It’s generally accepted that the best way to find confidence in agentic programming is to have it generate a plan first, where you review it and update the plan as needed before implementation starts.

Part of being efficient is making this planning stage more efficient, and ideally, parallelizing multiple tasks at once where necessary. Another part of it is figuring out how to steer an agent correctly in the context you’re in. Building a new feature versus updating existing production infrastructure will require drastically different approaches.

As agents improve, the bottleneck isn’t execution speed. It’s clarity of intent. Engineers who can break work into parallelizable tasks and define clean boundaries will multiply their output.

This is also personal to everyone and requires you to understand how you work. Do you prefer planning extensively and using subagents for implementation? Do you prefer getting a prototype out first, then reviewing the code and doing a cleanup after?

Each person’s way of steering and working with AI is personal, and figuring out your own style is important to working effectively with AI.

Reviewing and reading code, over writing it

Reviewing and reading code is already a hard skill that takes practice to learn. Holding the model of the previous state of a system and the proposed state of a system through reading a PR isn’t easy. AI can make this easier by writing summaries, READMEs, and other plain language to explain its work - but it’s not always aware of side effects and subtle risks like humans are.

As code generation gets cheaper, reviewing becomes more expensive. Thus, reading and reviewing code is more important than ever.

It helps that AI can help review code - but it’s best if you act as the main line of defense and use AI as a safety net.

Engineering is moving up the stack

The shift is clear: engineering work is moving up the stack. Each new model release feels like a new unlock. The skills that once separated senior engineers from junior ones, such as architectural thinking, context management, effective delegation, will now become baseline requirements.

Junior engineers now operate at what used to be senior-level abstraction. Senior engineers architect at scale previously only accessible to staff engineers.

The engineers who will thrive aren’t necessarily the ones who can prompt AI best, but the ones who can manage their own contexts the best.

Start by examining your own workflow. When do you reach for AI? When do you context-switch? Where do you lose focus? Your answers will reveal where to build new habits.

The tooling will keep improving. Your working style needs to improve with it.

Subscribe now

Recently, Linus Torvalds, the creator of Git and Linux, berated a Meta engineer on a PR they made recently.

“No. This is garbage and it came in too late. I asked for early pull requests because I'm traveling, and if you can't follow that rule, at least make the pull requests good.

This adds various garbage that isn't RISC-V specific to generic header files.

And by "garbage" I really mean it. This is stuff that nobody should ever send me, never mind late in a merge window.

Like this crazy and pointless make_u32_from_two_u16() "helper".

That thing makes the world actively a worse place to live. It's useless garbage that makes any user incomprehensible, and actively WORSE than not using that stupid "helper".

If you write the code out as "(a << 16) + b", you know what it does and which is the high word. Maybe you need to add a cast to make sure that 'b' doesn't have high bits that pollutes the end result, so maybe it's not going to be exactly pretty, but it's not going to be wrong and incomprehensible either.

In contrast, if you write make_u32_from_two_u16(a,b) you have not a f^%$ing clue what the word order is. IOW, you just made things WORSE, and you added that "helper" to a generic non-RISC-V file where people are apparently supposed to use it to make other code worse too.

So no. Things like this need to get bent. It does not go into generic header files, and it damn well does not happen late in the merge window.”

(Disregarding the fact that this pull request was sent in way too late, I agree)

I think this is a good example of “Write Everything Twice” or “Yes, Please Repeat Yourself”, as I’ve written about before.

Now that we have coding agents that are much better, I think the PRY principle here actually is more important than before.

But the main point Linus makes here is that good code optimizes for reducing cognitive load.

Reducing Micro-Context Switches

Today, code now has multiple readers: computers, LLMs, and software engineers.

Humans and LLMs have a limited amount of context they can store in their “context window” at a time. Having to go to a new file or function to understand it requires a slight context switch and requires extra brainpower (and tokens) to absorb that new information. An LLM now has to understand a new file, a new path, a new extra function. So, a function that uses 0 helper functions throughout requires less context and brainpower than a function with 5 helper functions.

Neuroscience shows switching tasks incurs measurable brain energy costs. Jumping across files is a micro–context switch. The more helpers/abstractions, the more “switch costs” pile up.

Humans have limited working memory capacity - let’s say the human brain can only store 4-7 “chunks” at at time. Each abstraction or helper function costs a chunk slot. Each abstractions costs more tokens. Too many layers means cognitive overload. Too many tokens used means a higher chance of error.

This means that sometimes, duplication is actually reducing cognitive load, because the “chunk” is self-contained.

Sometimes, of course, it’s justified to have abstracted components and functions. For example, if you want to enforce a certain behavior across your codebase, a helper function is probably a good thing.

On the other hand, it’s fine to have reused code throughout for simple operations, even if reused. Do you really need that helper file or abstracted class?

In fact, it makes code clearer, easier to understand, and most importantly, easier to iterate on. It’s much easier to point Claude Code or Codex to one file and ask it to make changes, rather than needing to DFS through 3-4 different files to fully understand the context of the “parent” file.

In performance engineering, “locality of reference” (keeping data/code close together) is necessary for efficiency. PRY is like a cognitive locality of reference.

Last, but not least, the cost of premature optimization has never been higher, especially in a world where, with modern IDEs and LLMs, code is cheaper to refactor than ever. The cost of duplication has dropped and will continue dropping moving forward.

It doesn’t cost anything to be nice

One thing to not take from Linus’s point: the tone and general rudeness. Everyone works differently, obviously, but being nicer never hurts :).

Being rude makes committing code a risk. Nobody wants to write code and risk the chance of being berated publicly like above. Abhinav put it well here:

Optimize for reducing cognitive load.

Subscribe now

Cursor, the popular AI IDE that recently announced they hit $300M ARR, uses Merkle trees to index code fast. This post goes over exactly how.

Before diving into Cursor's implementation, let's first understand what a Merkle tree is.

Merkle Trees Explained Simply

A Merkle tree is a tree structure in which every "leaf" node is labeled with the cryptographic hash of a data block, and every non-leaf node is labeled with the cryptographic hash of the labels of its child nodes. This creates a hierarchical structure where changes at any level can be efficiently detected by comparing hash values.

Think of them as a fingerprinting system for data:

1. Each piece of data (like a file) gets its own unique fingerprint (hash)

2. Pairs of fingerprints are combined and given a new fingerprint

3. This process continues until you have just one master fingerprint (the root hash)

The root hash summarizes all data contained in the individual pieces, serving as a cryptographic commitment to the entire dataset. The beauty of this approach is that if any single piece of data changes, it will change all the fingerprints above it, ultimately changing the root hash.

SWE Quiz (Sponsored)

Top software engineers know a lot. But how do you know what you don’t know?

SWE Quiz is a platform with roadmaps on system design fundamentals, like API design, databases, and more. There are three roadmaps launching in June: distributed systems, LLM fundamentals, and a React interview roadmap.

Get lifetime access to SWE Quiz, used by devs who have used the platform to get into companies like Google, Meta, and Airbnb.

Get SWE Quiz

How Cursor Uses Merkle Trees for Codebase Indexing

Cursor uses Merkle trees as a core component of its codebase indexing feature. According to a post by Cursor's founder and the security documentation, here's how it works:

Step 1: Code Chunking and Processing

Cursor first chunks your codebase files locally, splitting code into semantically meaningful pieces before any processing occurs.

Step 2: Merkle Tree Construction and Synchronization

When codebase indexing is enabled, Cursor scans the folder opened in the editor and computes a Merkle tree of hashes of all valid files. This Merkle tree is then synchronized with Cursor's server, as detailed in Cursor's security documentation.

Step 3: Embedding Generation

After the chunks are sent to Cursor's server, embeddings are created using either OpenAI's embedding API or a custom embedding model (I couldn’t verify this). These vector representations capture the semantic meaning of the code chunks.

Step 4: Storage and Indexing

The embeddings, along with metadata like start/end line numbers and file paths, are stored in a remote vector database (Turbopuffer). To maintain privacy while still enabling path-based filtering, Cursor stores an obfuscated relative file path with each vector. Importantly, according to Cursor's founder, "None of your code is stored in our databases. It's gone after the life of the request."

Step 5: Periodic Updates Using Merkle Trees

Every 10 minutes, Cursor checks for hash mismatches, using the Merkle tree to identify which files have changed. Only the changed files need to be uploaded, significantly reducing bandwidth usage, as explained in Cursor's security documentation. This is where the Merkle tree structure provides its greatest value—enabling efficient incremental updates.

Code Chunking Strategies

The effectiveness of the codebase indexing largely depends on how code is chunked. While my previous explanation didn't go into detail about chunking methods, this blog post about building a Cursor-like codebase feature provides some insights:

While simple approaches split code by characters, words, or lines, they often miss semantic boundaries—resulting in degraded embedding quality.

• You can split code based on a fixed token count, but this can cut off code blocks like functions or classes mid-way.

• A more effective approach is to use an intelligent splitter that understands code structure, such as recursive text splitters that use high-level delimiters (e.g., class and function definitions) to split at appropriate semantic boundaries.

• An even more elegant solution is to split the code based on its Abstract Syntax Tree (AST) structure. By traversing the AST depth-first, it splits code into sub-trees that fit within token limits. To avoid creating too many small chunks, sibling nodes are merged into larger chunks as long as they stay under the token limit. Tools like tree-sitter can be used for this AST parsing, supporting a wide range of programming languages.

For a crash course on tokens, read this.

How Embeddings Are Used at Inference Time

After covering how Cursor creates and stores code embeddings, a natural question arises: how are these embeddings actually used once they've been generated? This section explains the practical application of these embeddings during normal usage.

Semantic Search and Context Retrieval

When you interact with Cursor's AI features like asking questions about your codebase (using @Codebase or ⌘ Enter), the following process occurs:

1. Query Embedding: Cursor computes an embedding for your question or the code context you're working with.

2. Vector Similarity Search: This query embedding is sent to Turbopuffer (Cursor's vector database), which performs a nearest-neighbor search to find code chunks semantically similar to your query.

3. Local File Access: Cursor's client receives the results, which include obfuscated file paths and line ranges of the most relevant code chunks. Importantly, the actual code content remains on your machine and is retrieved locally.

4. Context Assembly: The client reads these relevant code chunks from your local files and sends them as context to the server for the LLM to process alongside your question.

5. Informed Response: The LLM now has the necessary context from your codebase to provide a more informed and relevant response to your question or to generate appropriate code completions.

This embedding-powered retrieval allows for:

1. Contextual Code Generation: When writing new code, Cursor can reference similar implementations in your existing codebase, maintaining consistent patterns and styles.

2. Codebase Q&A: You can ask questions about your codebase and get answers informed by your actual code rather than generic responses.

3. Smart Code Completion: Code completions can be enhanced with awareness of your project's specific conventions and patterns.

4. Intelligent Refactoring: When refactoring code, the system can identify all related pieces across your codebase that might need similar changes.

Why Cursor Uses Merkle Trees

Many of these details are security-related, and thus can be found in Cursor’s security documentation.

1. Efficient Incremental Updates

By using a Merkle tree, Cursor can quickly identify exactly which files have changed since the last synchronization. Instead of re-uploading the entire codebase, it only needs to upload the specific files that have been modified. This is important for large codebases where re-indexing everything would be too expensive in terms of bandwidth and processing time.

2. Data Integrity Verification

The Merkle tree structure allows Cursor to efficiently verify that the files being indexed match what's stored on the server. The hierarchical hash structure makes it easy to detect any inconsistencies or corrupted data during transfer.

3. Optimized Caching

Cursor stores embeddings in a cache indexed by the hash of the chunk, ensuring that indexing the same codebase a second time is much faster. This is great for teams where multiple developers might be working with the same codebase.

4. Privacy-Preserving Indexing

To protect sensitive information in file paths, Cursor implements path obfuscation by splitting the path by '/' and '.' characters and encrypting each segment with a secret key stored on the client. While this still reveals some information about directory hierarchy, it hides most sensitive details.

5. Git History Integration

When codebase indexing is enabled in a Git repository, Cursor also indexes the Git history. It stores commit SHAs, parent information, and obfuscated file names. To enable sharing the data structure for users in the same Git repo and on the same team, the secret key for obfuscating file names is derived from hashes of recent commit contents.

Embedding Models and Considerations

The choice of embedding model significantly impacts the quality of code search and understanding. While some systems use open-source models like all-MiniLM-L6-v2, Cursor likely uses either OpenAI's embedding models or custom embedding models specifically tuned for code. For specialized code embeddings, models like Microsoft's unixcoder-base or Voyage AI's voyage-code-2 are good for code-specific semantic understanding.

The embedding challenge is made more complex because embedding models have token limits. OpenAI's text-embedding-3-small model, for example, has a token limit of 8192. Effective chunking helps stay within token limits while preserving semantic meaning.

The Handshake Process

A key aspect of Cursor's Merkle tree implementation is the handshake process that occurs during synchronization. Logs from the Cursor application reveal that when initializing codebase indexing, Cursor creates a "merkle client" and performs a "startup handshake" with the server. This handshake involves sending the root hash of the locally computed Merkle tree to the server, as seen in Issue #2209 on GitHub and Issue #981 on GitHub.

The handshake process allows the server to determine which parts of the codebase need to be synced. Based on the handshake logs, we can see that Cursor computes the initial hash of the codebase and sends it to the server for verification, as documented in Issue #2209 on GitHub.

Technical Implementation Challenges

While the Merkle tree approach offers many advantages, it's not without implementation challenges. Cursor's indexing feature often experiences heavy load, causing many requests to fail. This can result in files needing to be uploaded several times before they get fully indexed. Users might notice higher than expected network traffic to 'repo42.cursor.sh' as a result of these retry mechanisms, as mentioned in Cursor's security documentation.

Another challenge relates to embedding security. Academic research has shown that reversing embeddings is possible in some cases. While current attacks typically rely on having access to the embedding model and working with short strings, there is a potential risk that an adversary who gains access to Cursor's vector database could extract information about indexed codebases from the stored embeddings.

Subscribe now

Anthropic, in their release of Claude 3.7 and Claude Code, wrote:

"…in developing our reasoning models, we've optimized somewhat less for math and computer science competition problems, and instead shifted focus towards real-world tasks that better reflect how businesses actually use LLMs. Early testing demonstrated Claude's leadership in coding capabilities across the board: Cursor noted Claude is once again best-in-class for real-world coding tasks, with significant improvements in areas ranging from handling complex codebases to advanced tool use."

It immediately reminded me of a Jeff Bezos quote from an interview he had:

"When the data and the anecdotes disagree, the anecdotes are usually right. It's usually not that the data is being miscollected. It's usually that you're not measuring the right thing. If you have a bunch of customers complaining about something, and at the same time your metrics look like they shouldn't be complaining, you should doubt the metrics." The full quote is great.

Developers who have used all the popular AI tools (ChatGPT, Claude, etc), generally prefer Claude for coding over the rest. I recently ran a survey on developers wh

Though we have Google's Gemini and OpenAI's models topping the AI leaderboards in terms of benchmarks, Claude still remains the best coding tool in terms of the real world. (Yes, I know the benchmark for this is technically SWE-Bench). There's a reason why it's the default model for popular AI IDEs like Windsurf - no matter what the benchmarks say, there's a certain "vibe" to Claude that ends up passing for devs compared to the other models.

Devs prefer Claude in the real world

I recently ran a survey with 1370 devs who use AI often (at least once a day) during coding.

A vast majority of them preferred Claude 3.5 over any other:

• 78% preferred Claude 3.5 as their #1 choice

• 13% chose OpenAI's o3 models

• 9% used other models, like Gemini

This isn't isolated to AI tools. As one developer noted on Hacker News:

"Claude is the best example of benchmarks not being reflective of reality. All the AI labs are so focused on improving benchmark scores but when it comes to providing actual utility Claude has been the winner for quite some time. Which isn't to say that benchmarks aren't useful. They surely are. But labs are clearly both overtraining and overindexing on benchmarks."

SWE Quiz (Sponsored)

SWE Quiz is the easiest way to build your system design fundamentals through 500+ questions on concepts like authentication, databases, and more.

SWE Quiz has crash courses to get up to speed quickly, smart quizzes to fill in gaps of knowledge, interview readiness scores, and concept roadmaps. It is offering a lifetime payment for a limited time and new content is being added weekly.

Redeem SWE Quiz's Lifetime Offer

Metric-driven development can be a problem

When metrics become the goal rather than a tool for measurement, we enter dangerous territory. At its core, metric-driven development is meant to provide objective guidance for improvement. But too often, it creates a myopic focus that misses the forest for the trees. This is especially true for large FAANG-level companies, where the size of the company and incentives for rewards like promotion are usually numbers-based. Launching a half-baked feature/product to millions of pre-existing mobile DAUs lets you say “you gained millions of DAUs” on your promotion packet, regardless of whether the product is even liked by users or not. After that, it commonly becomes someone else’s problem to find a metric that correlates with how much users like the product and then work towards improving it.

“A Harvard Business Review article looked at Wells Fargo as a case study of how letting metrics replace strategy can harm a business. After identifying cross-selling as a measure of long-term customer relationships, Wells Fargo went overboard emphasizing the cross-selling metric: intense pressure on employees combined with an unethical sales culture led to 3.5 million fraudulent deposit and credit card accounts being opened without customers' consent. The metric of cross-selling is a much more short-term concern compared to the loftier goal of nurturing long-term customer relationships.” - Data.Blog

This pattern repeats in tech companies of all sizes. A developer from a FAANG company shared this revealing story:

"At <FAANG mega-corp>, I used to work on an internal team which basically did nothing important, certainly nothing users ever think about. The problem that we were trying to solve, well, we had already solved it about 3 years ago, yet the team was still expanding, and our product managers kept spitting out new (mostly useless) project ideas. One of the main areas of focus was metrics. Leadership was obsessed with metrics, we measured everything."

The contrast came when this developer moved to a smaller team with a clearer purpose:

"One of the first things I said to my new product manager was 'so, what metrics do you care about? how do you measure success?' He was taken aback by my question. He seemed genuinely perplexed. He paused for a few seconds, then said 'metrics? What do you mean by metrics? I look at revenue, when the line goes up I'm happy'."

This simplicity - focusing on actual outcomes rather than proxy measurements - cuts to the heart of the problem with pure metric-driven development.

Feels Over Figures, Vibes Equals Value

What's happening with Claude is something I’ve been casually calling “Feels over Figures” - when a tool's real-world feedback doesn’t necessarily match its overall benchmark performance. Yes, Claude does dominate SWE-bench, but in general, Claude has won over users because it delivers what they actually need in their workflow.

“The Claude Effect” here shows what’s missing - the human element, the "feel" of using a tool, and the general vibes of using a tool.

I suspect this is also why OpenAI chose to release GPT-4.5, which from a practical numbers perspective, doesn’t really make sense at all. It’s seemingly worse numbers wise on so many things, like reasoning and price. But they specifically denote the “EQ” of the model, which is presumably the optimization that they are hill climbing for here. If they’re going to take a note from DeepSeek, they’ll probably use GPT-4.5 to increase the EQ of the rest of their models.

If anything, especially when working on something as non-deterministic as LLMs, it makes sense to accept that there is extra human toil necessary to understand the “vibes” aspect when evaluating performance. This means having more human spot checks and collecting more valuable human qualitative feedback (and not necessarily grinding it down to a number or throwing it onto a Likert scale immediately). It also means prioritizing the non-technical aspects too - like design and user flows.

The gap between metrics and real-world value isn't going away, so rather, it's best to have methods to bridge them as close as possible. The Claude Effect teaches us that sometimes the most valuable aspects of a tool come from qualities that aren't easily quantified - the intuitiveness of its responses, how well it aligns with developer thinking patterns, or how seamlessly it fits into existing workflows.

One Hacker News commenter put it as the "gamedev mindset": "Yes data is good. But I think the industry would very often be better off trusting guts and not needing a big huge expensive UX study or benchmark to prove what you can plainly see."

Subscribe now

LLMs have been all the rage in tech since ChatGPT’s release in November 2022. It feels crazy to think that it was released almost 2 years ago! Yet, in that same timeframe, there have been many other exciting advancements in software engineering that haven’t received the hype they may have deserved because of the LLM hype.

It’s true that LLMs are revolutionary, and while I work with LLMs daily, there are so many other things that are fascinatingly progressing. I go over some topics below and provide a host of links for each topic for those interested in learning more. I see these topics as trends that are only growing as time goes on.

Local-first software

Local-first software has been around for decades, but it seems that the past few years have seen increased activity in making the developer experience around local-first software better. Local-first software prioritizes storing and processing data on a user’s local device rather than relying solely on remote servers. 

The best intro to the local-first software paradigm is by Ink and Switch: Local-first software: You own your data, in spite of the cloud. 

Companies and packages like React-Query, PouchDB, InstantDB, Legend-State, PowerSync, ElectricSQL, and more focus on making the sync process easier between client and server.

Local-first is the natural next step of user experience. Making things persist locally allows for near-zero latency for interactions, better user experiences, and more resilient clients in low-to-no internet connections. I predict this will become particularly important as user expectations for software continues to rise in the future.

Some interesting work in this space is conflict resolution. Conflict resolution is needed when there are local changes and server-side changes that conflict with each other, and thus, need resolution. I wrote up a quick summary of the most common conflict resolution methods below. (Note: Some systems combine multiple strategies, like using CRDTs for simple data types and three-way merging for more complex documents.)

CRDTs (Conflict-free Replicated Data Types)

CRDTs are data structures designed to be merged automatically and deterministically without conflicts. They use mathematical principles to ensure that no matter the order in which changes are applied, all replicas will eventually converge to the same state. CRDTs are great for collaborative applications like text editors, shared to-do lists, or any system where users make frequent updates offline.

The best articles I’ve seen for CRDTs are An Interactive Intro to CRDTs and A Gentle Introduction to CRDTs.

Another article that argues against CRDTs for certain experiences is You don't need CRDTs for collaborative experiences.

Operational Transforms (OT)

Operational Transforms are also commonly used in collaborative text editors like Google Docs. OT involves transforming operations (like insertions or deletions) on shared documents to ensure that concurrent edits are applied in a consistent way. OT allows for real-time collaboration while maintaining data consistency, even when multiple users are editing simultaneously across devices.

Last-Write-Wins (LWW)

This is a simpler approach where, in case of conflict, the most recent change (based on timestamp) is accepted as the final version. While this strategy is straightforward and easy to implement, it may lead to data loss if conflicting changes are made almost simultaneously. It's best suited for scenarios where the latest update is usually the correct one or the preferred one.

Three-Way Merging

Inspired by version control systems like Git, this method involves maintaining a base version of the data along with changes from different devices. If conflicts are detected, the system attempts to merge them or prompts the user to resolve them. This approach provides flexibility but may require user intervention, so it’s best used in cases where it’s okay to have input from the user to resolve data. Still, this merge does its best to auto-merge everything. There is a “simpler” non-merge solution where conflicts are simply shown to the user, who can then pick the correct side themselves.

Event Sourcing

Instead of storing the current state, the system stores a series of events that led to that state. When conflicts occur, the system can replay events to determine the most accurate or meaningful resolution. Event sourcing provides an audit trail and allows for complex conflict resolution strategies, but it can be more complex to implement.

—

Links to read:

• Ink and Switch

• Local First News

• Local-First Software 

• Local-First Web Development | Hacker News 

WebAssembly

WebAssembly is a binary instruction format that allows code to run directly in the browser with near-native speed. 

The possibilities of WASM are super exciting:

In-built code execution in the browser

WASM allows developers to run complex applications within the browser itself, bypassing the need for server-side processing. This is awesome for tools like code editors, development environments, and real-time simulations that can now operate entirely client-side. 

SQLite in the browser

By running a SQLite database directly in the browser with WASM, web apps now have an alternative to IndexedDB and LocalStorage to store and query data locally. This opens up new possibilities for offline functionality and data caching. This connects really well with the local-first movement along with SQLite growing in popularity in general. (I talk about this more below, but Notion uses WASM SQLite).

Faster user experiences

WASM supports caching of compiled code, resulting in faster load times on subsequent visits. This means faster, better user experiences. It also lets applications deliver sophisticated functionality on par with native apps, without the need to repeatedly re-download code. This is a great article on this: Code caching for WebAssembly developers · V8 

AI in the Browser 

Yes, I know that I said this was a list of exciting non-AI innovations, but this one is very relevant and particularly promising. WASM has made it possible to run machine learning models directly in the browser, without needing a backend server. This lets AI-driven features, such as image recognition or text analysis, to function quickly and privately on users’ devices without any cloud costs. Projects like TensorFlow.js with WASM backends enable running AI models with improved performance compared to pure JavaScript implementations. Google Chrome had a great talk on this: WebAssembly and WebGPU enhancements for faster Web AI, part 1.

SQLite’s Renaissance

Most developers, when thinking about what database to use, think about MySQL/Postgres (relational) or Mongo/Dynamo (NoSQL). For many production grade applications, it’s probably a good idea to use one of these for the foundation of your database.

But it’s not always necessary. Kent Dodds argues that you could probably just use SQLite in production. So does Stephen Margheim at the Ruby Conf. Regardless of whether you agree or not, the points they makes in favor of SQLite are valuable:

• zero latency

• simplified setup

• easy multi-instance replication

• can handle massive databases (more than you think)

• much easier to develop and test with.

But in reality, SQLite simply has tradeoffs. My favorite quote from the founder of SQlite is simply that:

"Think of SQLite not as a replacement for Oracle but as a replacement for fopen().".

SQLite has recently been seeing a lot of renewed activity in terms of hype too. Why? It’s probably because of cycles. Dev tools go through cycles of hype and usage, and now that there’s more commercial force behind SQLite’s usage, we’re seeing a lot more public writing pushing to use it. That’s not to say SQLite was some infrequently used database before.

Hacker News has some interesting discussion on people who have used SQLite as a primary database.

While I don’t see too much movement to SQLite as a primary database in general, I do see value in SQLite for local-first software (especially combined with WebAssembly advancements).

Expo (React Native) constantly releases new versions of expo-sqlite with cool new features and upgrades. Expo v52 introduces a new Storage drop-in replacement for React Native’s AsyncStorage, with added synchronous APIs for convenience. In local-first mobile apps, SQLite is a rock-solid local store and usually a better option than AsyncStorage.

Notion implemented SQLite caching in their desktop apps in 2021, resulting in:

• 50% faster initial page loads

• 50% faster navigation between pages

Inspired by this success, Notion's team decided to bring these performance gains to their web app using WebAssembly (WASM) SQLite. This led to Notion reducing their page navigation latency on the web by 20% using SQLite client-side. (Summary, Blog Post). 

They also go over some interesting problems they faced, like having to optimize WASM SQLite loading and fixing slow disk reads on slower devices.

Cross-Platform is getting better (React Native, Flutter)

Cross-platform is trudging along, making great strides in performance, dev experience, usability, and  more. For example, React Native recently introduced their New Architecture, which leads to more polished experiences.

One of the coolest recent examples of this is Shopify migrating their entire mobile app to React Native. (Source) Now, 86% of their code is shared between iOS and Android. Not only that, performance improved along the way! They improved screen load times by 59%, app launches are now 44% faster, and webviews are now 63% faster.

This doesn’t really take away the need for native development of course. Mustafa Ali gives some great lessons learned from the experience too:

1. “Native code and native devs are crucial. There is no replacing the experience and taste that comes from having built high-quality mobile apps.”

2. “100% React Native should be an anti-goal. Use native wherever it is the best tool for the job (widgets, Siri shortcuts, watch app and complications, etc), or in places where there are high performance requirements.”

3. “Achieving good performance takes work and should be a priority from the start. Measure everything and relentlessly optimize every layer. Add automated monitoring to catch regressions.”

Automated reasoning 

Automated reasoning is a bit more obscure, but also super exciting. Automated reasoning focuses on using logic and mathematical proofs to make sure that systems behave as intended. Unlike traditional testing, which validates system behavior in specific scenarios, automated reasoning allows engineers to verify correctness across all possible scenarios. Basically, it’s a shift from reactive testing to proactive verification, leading to (ideally) better reliability and security. Amazon has a gentle intro to automated reasoning published.

AWS also recently wrote a great blog post on how they’ve been using automated reasoning techniques in their infrastructure. Over a decade of applying these techniques, AWS has found that formally verified code often outperforms the unverified code it replaces. This is because the process of formal verification helps uncover bugs early, leading to optimizations that improve runtime performance.

For example, they found that by building a formal spec of IAM, they were able to optimize code that processes over 1.2 billion requests per second, resulting in a 50% performance boost. For S3, they used automated reasoning to find hidden bugs, which let them ship updates more frequently - from every quarter down to every 1-2 months.

Engineer’s Codex is a publication about real-world software engineering.

Subscribe now

Fireship is one of YouTube’s most popular programming channels. With over 3 million subscribers and 200 million total views, he’s garnered a fanbase with his fast-paced, humorous, meme-filled videos explaining programming and software trends.

Each Fireship video, with a combo of sarcastic humor combined with bite-sized learning, leaves you satisfyingly entertained and gladly informed.

What makes Fireship’s videos so good?

What is his secret to growing so fast on one of the hardest niches on YouTube?

And how exactly did he do it?

That’s what we’ll answer in this article.

Share

The Types of YouTube Programmers

Programming is a popular topic on YouTube, and most programming YouTubers fall into one of four categories. (Some YouTubers will fall into more than one or be in half a category).

1. Tutorials: Tutorial YouTubers create tutorials for various technologies, usually by walking through the creation of a sample project. However, the main issue here is that someone watching a “Firebase and Angular” tutorial may not want to watch a “React Native” tutorial, making one’s subscriber base fragmented. (Example: thenewboston)

2. Humor: Humor-focused programmers are extremely popular. Creators rise quickly from making comedy skits with niche tech and software engineering jokes. (Example: Joma Tech)

3. Coaches: YouTubers who generally work in tech as a software engineer and teach others how to get a job. (Example: Power Couple)

4. Commentary: YouTubers who generally work in tech as a software engineer and commentate on happenings. (Example: ThePrimeagen and his glorious moustache, Neetcode)

5. Indirect Entertainment: These are more “classic” YouTubers, where programming is in the background and entertainment is the first part. (Example: Fundy).

Which category does Fireship fall into?

Fireship’s Beginning

Fireship began by creating niche tutorials, specifically Angular + Firebase tutorials.

He eventually expanded his niche as his audience grew. 

He stuck with programming, but started generalizing the tutorials, going from “Angular and Firebase” to “Firebase Basics”.

Then, as he kept growing, he widened the niche of programming even more, going over more generalized concepts while adding more personality into his explanations. His videos got faster-paced and much more interesting, as he started adding his sarcastic humor, niche jokes, and funny memes into each one.

Fireship’s Secret Sauce: Humor

Let’s look a little deeper into Fireship’s sense of humor.

1. Niche Jokes

He commonly makes jokes that only people with some programming experience would understand, such as jokes about how every week there’s a new JavaScript framework coming out.

2. Funny Memes

Since he doesn’t show his face, he needs to put something on the screen that’s visually entertaining and he uses memes and hilarious scenes from various media.

3. Sarcasm/Self-deprecating humor

Lastly, his sense of humor is sarcastic and often self-deprecating. In his “How to get a 1M subscribers in 100 seconds video” he talks about how he has the personality of a carrot (he doesn’t) and his handsome face (he is).

All these humorous aspects combined make Fireship an approachable YouTuber. He feels like your friend or coworker. He has jokes that feel like inside jokes, he shares memes with you (like your friend that sends you Reels on Instagram), and his sarcastic, self-deprecating humor gives him a sense of humbleness that makes him feel like a peer rather than this “YouTuber”.

Fireship’s YouTube Strategy Explained

Thanks to these aspects, viewers are encouraged to comment and discuss the topic at hand in the comments, boosting engagement. Since he teaches things too, you’ll see many comments glad they learned something new in an entertaining way.

And the way Fireship structures his channel not only drives viewer engagement, but also encourages “binge sessions”.

Evergreen Videos

YouTube loves it when a viewer binges someone’s videos. It shows that the viewer is satisfied enough that they keep watching one’s content and as a result, will boost that YouTube channel in the hopes that other people will also binge that content (and spend more time on YouTube as a result).

The best way to create a binge library is through evergreen videos - videos that are part of a series, but not news or trend related.

Fireships’s “X in 100 Seconds” series fits this bill. He publishes quick, fast-paced explainers in his unique style of topics like languages, frontend frameworks, and more. 

When someone watches “C in 100 seconds” and they enjoy it, they’re hooked. They’ll watch another video, maybe “Java in 100 seconds”. Each video is satisfying to watch, filled with Fireship-style humor, and short, leading to a satisfying experience and the feeling of “let me just watch one more video.. It’s only 2 minutes..”.

Other longer evergreen videos also do well, like The God Tier Developer Roadmap (11 minutes long), which is currently the second most viewed video on his channel, with almost 7 million views at the time of writing.

Trending Videos

After publishing evergreen videos for a long time, Fireship started a new series, The Code Report, on Feb 1, 2022 (How a CS student tracks Elon’s Private Jet with Python). The Code Report was introduced as “a new series where I bring you breaking news every programmer should know.” 

This video did really well and it fit the same style as the rest of Fireship’s videos - fast-paced, snappy, humorous, and educational. 

Along the way, Fireship still continued uploading evergreen videos too.

Growth

This strategy of combining trendy topics along with evergreen videos was genius. Paddy Galloway, renowned YouTube strategist, has noted similar YouTubers do this, like Graham Stephan in the finance niche.

It’s genius because trendy topics are pushed immediately, so they get a large amount of views in the beginning and then eventually taper off.

Evergreen videos may not get as many views in the beginning, but they grow in views steadily over time.

Combined, you get a view graph that looks like this. It’s spiky but the trend is up over time.

Fireship’s growth really exploded when ChatGPT came out. At a time where there was tons of AI hype along with new AI announcements coming out daily, Fireship was uploading at a rapid pace. This pushed him to a new high of 16.9 MILLION views in March 2023, a time where ChatGPT plugins, GPT-4, and other AI advancements were all announced. He also gained a record high of 120K subscribers at this time too.

Of course, the tech world is chock-full of drama and new happenings, so Fireship beat this record in December 2023 with over 20M views that month and over 150k new subs.

Key Takeaways

• Keep videos fast-paced. It boosts retention and keeps viewers watching.

• Fireship excels at edutainment. Humor is a great way to educate and Fireship’s unique combination of humor scratches an itch for people watching.

• Trend-jacking: The Code Report capitalizes on trends, meaning his video about a timely topic will be searched for often and be pushed by YouTube more.

• Evergreen videos: Evergreen videos are necessary too, as they mean you have content and be discovered by new viewers at all times. A video like C in 100 seconds is timeless - C as a language is not going anywhere.

• There is no “limit” to growth: Looking at other programming YouTubers, you may think that growth can stall at some point. And it might! But programming is growing as a field and the younger generation is more YouTube-focused than most. The programming niche on YouTube (and other platforms!) can only grow, and by starting early, Fireship has set himself up as “Programming YouTube’s” top creator.

Subscribe now

My favorite takeaway is definitely “there’s no limit to growth.” It’s true of a lot of things, like companies. People never thought Apple would be a $1T company, yet it’s worth almost $3T today. It’s the same in the creator space. I’m sure that Gergely Orosz never expected the Pragmatic Engineer to one day have over 700,000 subscribers, yet it does!

Subscribe now

Using booleans is a trap. Let me explain.

Something I’ve run into more often in the past year is the “boolean trap.” Booleans are great - simple with only two states. They’re very easy to use.

However, the issue comes when using booleans as parameters in functions or APIs. It’s an issue that I’ve ran into enough that I try to actually refrain from using booleans if possible.

In my specific example, I was working on some LLM pipeline outputs. Initially, the LLM outputs were all in plaintext, so we had no flag. Suddenly, there was a need to also support JSON outputs because JSON is much easier to parse for further pipeline steps compared to plaintext.

So, we started off with the following output_plaintext flag:

def call_llm(prompt: str, output_plaintext: bool)

While premature optimization is the devil, this flag should already be setting off red flags in your head. Remember, the key of a good API is that it’s easy to understand when reading.

Imagine seeing this call in your Python code where types aren’t necessarily enforced, however:

output = call_llm("Subscribe!", True)

Anybody reading this will be confused. What is true referring to? What is the argument here?

Remember, well-written code is predictable. Don’t allow surprises.

Let’s say we add the argument in:

output = call_llm(prompt="Subscribe!", output_plaintext=True)

Now, this begs the question - what is the output when output_plaintext is False? HTML? CSV? LISP?

Now, let’s say we have 400 instances of call_llm throughout our codebase now, all using this boolean flag. Uh oh, we just got a request to have call_llm support XML output too! Now this is a problem because we can’t shove XML support into a binary state. Plus, we have to go around and update each call to support the new type. Gross.

You can see how this big ugly problem is very prone to human error and the greatest creator of bugs are… humans. Yes, code is written for and processed by computers, but at the same time, you don’t code in a blackbox. You’re also coding for other humans, including your own future self.

So, make your life easier and don’t use boolean flags.

In my opinion, it’s better to use enums, EVEN IF you only have a binary state to represent.

• Enums are much more readable - both in terms of function parameters and explicit typing.

• Enums are more maintainable because they are extendable.

• Enums are shareable across files.

From the above example, we can use the following enum:

class LLM_OUTPUT(Enum):
    TEXT = "text"
    JSON = "json"
    XML = "xml"
    MARKDOWN = "markdown"
    HTML = "html"
    CSV = "csv"
    YAML = "yaml"

This is much nicer. We’re able to easily extend functions that already take in the LLM_OUTPUT enum past just binary states into N-ary states without even needing to update callers of this enum.

Now, the codebase will look like this, where in cases of no explicit keyword arguments, the functionality of the call is still extremely clear. Plus, someone doesn’t have to look into the actual application code to see what the different outputs are - a simple hover over the enum solves that.

output = call_llm("Subscribe!", LLM_OUTPUT.JSON)
output = call_llm(prompt="Subscribe!", output_type=LLM_OUTPUT.JSON)

Now, as always, context is everything. If a function does one thing, and it’s literally called setIsModalOpen(bool isModalOpen), like for a modal in React, then booleans are perfect here. There is no chance of any possible extension in the future (what, you’re going to have it half open?) and seeing setIsModalOpen(true) is extremely obvious in what the function is doing. (Of course, there’s another argument here that instead of having a function like that, we could just have setModalOpen() and setModalClosed() instead (which I agree is much nicer)).

In general though - coding is as much science as it is art, and art is easier to have opinions on. Using booleans can be a trap, but also aren’t necessarily wrong. In my own experience, though, booleans usually are less maintainable than enums for API/function parameters over time.

Subscribe now

I do want to note (else Hacker News will kill me) that I’m not the first, nor will I be the last person to write about this:

• Is it wrong to use a boolean parameter to determine behavior?

• Flag Argument

• And many more, I’m sure

Subscribe now

I recently came across a quote by Linus Torvalds (creator of Linux and Git) in this great Stack Overflow post. (This post reviews many quotes from that post.)

It succinctly described a problem I’ve been working on recently:

"Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

Right before the above quote, Linus said:

git actually has a simple design, with stable and reasonably well-documented data structures. In fact, I'm a huge proponent of designing your code around the data, rather than the other way around, and I think it's one of the reasons git has been fairly successful […] I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important.

Good data structures make code easier to design and maintain. It makes software more reliable, systems more understandable, and code more readable. When designing any software, application logic often follows the data model. Treating the data model as an afterthought results in more work down the line. The inverse is true also - having a well-thought out data model makes migrations and building on top of complex systems easier down the line.

When I read this quote, I actually was able to recognize countless examples in the past of this. I once worked on a project where we spent quite a while optimizing complex algorithms, only to realize that by restructuring our data, we could eliminate entire classes of problems. We replaced a 500-line function with a 50-line function and a well-designed data structure. Not only was the new code faster, but it was also much easier to understand and maintain. (Of course, then the problem also shifted “down the stack” to where the majority of toil was in restructuring existing data.)

Another relevant quote here is in The Art of Unix Programming: 

Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.

Even the simplest procedural logic is hard for humans to verify, but quite complex data structures are fairly easy to model and reason about. To see this, compare the expressiveness and explanatory power of a diagram of (say) a fifty-node pointer tree with a flowchart of a fifty-line program. Or, compare an array initializer expressing a conversion table with an equivalent switch statement. The difference in transparency and clarity is dramatic. See Rob Pike's Rule 5.

Data is more tractable than program logic. It follows that where you see a choice between complexity in data structures and complexity in code, choose the former. More: in evolving a design, you should actively seek ways to shift complexity from code to data.

The Unix community did not originate this insight, but a lot of Unix code displays its influence. The C language's facility at manipulating pointers, in particular, has encouraged the use of dynamically-modified reference structures at all levels of coding from the kernel upward. Simple pointer chases in such structures frequently do duties that implementations in other languages would instead have to embody in more elaborate procedures.

The actionable tip here is to start with the data. Try to reduce code complexity through stricter types on your interfaces or databases. Spend extra time thinking through the data structures ahead of time.

That’s not to say the code isn’t important. Obviously, everything is important - but it’s useful to have a strong high-level approach of how data will flow and how different components interact before going deeper into code-related details. 

It’s why one of the Senior Engineer (L5) requirements (at least for FAANG) generally involves writing higher-level design docs for more complex systems (which includes driving team planning and building good roadmaps for medium-to-large features).

from wrote a great piece on the higher level of influence L5s generally have here:

The Developing Dev

FAANG Career Ladder: Mid-level (L4) vs Senior (L5)

👋 Hi, this is Ryan with this week’s newsletter. I write about software engineering, big tech/startups and career growth. Since last week’s article was well-received, we’ll continue our “FAANG Career Ladder” series, comparing IC levels from Junior (L3) to Principal (L8) at FAANG-like companies…

Read more

3 years ago · 95 likes · 6 comments · Ryan Peterman

from also wrote a great piece about getting to Senior Engineer:

High Growth Engineer

Fast-track your career: From Junior -> Senior engineer in 2 years

Hi everyone, Jordan here. In this issue, you’ll learn a wealth of actions you can do right now to grow in your career, regardless of your level. By doing these things, I was able to go from an L1 (Junior Engineer) at Gusto to L3 (Senior Engineer) through 2 back-to-back promotions…

Read more

3 years ago · 184 likes · 34 comments · Jordan Cutler

Subscribe now

I recently built and designed a massive service that (finally) launched successfully last month. During the design and implementation process, I found that the following list of “rules” kept coming back up over and over in various scenarios.

These rules are common enough that I daresay that at least one of them will be useful for a project that any software engineers reading this are currently working on. But if you can’t apply it directly now, I hope that these principles are a useful thought exercise that you are free to comment on below or challenge directly too.

One thing I will note here is that of course - each “principle” has a time and place. Nuance is necessary, as always. These are conclusions that I find myself erring towards in general because oftentimes, the opposite that is the default that I see when I’m reviewing code or reading design docs.

1. Maintain one source of truth.

If there’s two sources of truth, one is probably wrong. If it’s not wrong, it’s not wrong… yet. 

Basically, if you’re trying to maintain a piece of state in two different locations within the same service… just don’t. It’s better to try to just reference the same state wherever you can. For example, if you’re maintaining a frontend application and have a bank balance that is from the server, I’ve seen enough sync bugs in my time that I always want to get that balance from the server. If there is some balance that is derived from that, such as “spendable balance” versus “total” (for example, some banks make you keep a minimum balance), then that “spendable balance” should be derived on-the-fly rather than stored separately. Else, you’ll now have to update both balances whenever a transaction happens.

In general, if there is a piece of data that is derived from another value, then that value should be derived rather than stored. Storing that value leads to synchronization bugs. (Yes, I know this isn’t always possible. There will always be other factors in play, like the expense of the derivation. At the end of the day, it’s a tradeoff.)

2. Yes, please repeat yourself.

We’ve heard of DRY (Don’t Repeat Yourself) and now I present to you PRY (Please Repeat Yourself).

Far too many times I’ve seen code that looks mostly the same try to get abstracted out into a “re-usable” class. The problem is, this “re-usable” class gets one method added to it, then a special constructor, then a few more methods, until it’s this giant Frankenstein of code that serves multiple different purposes and the original purpose of the abstraction no longer exists.

A pentagon may be similar-looking to a hexagon, but there is still enough of a difference that they are absolutely not the same. 

I’m also guilty of spending way too much time trying to make things reusable, when a bit of code duplication works perfectly fine. (Yes, you have to write more tests and it doesn’t scratch the “refactoring” itch, but oh well.)

3. Don’t overuse mocks.

Mocks. I have a love-hate relationship with mocks. My favorite one-liner from a Reddit discussion about this post was “with mocks, we sell test fidelity for ease of testing.”

Mocks are great when I have to write unit tests to test something quickly and don’t want to mess with “prod-level” code. Mocks are not great when prod breaks because as it turns out - something you mocked broke deeper down the stack, even though that “deeper down the stack” is owned by another team. It doesn’t matter because it was your service that broke so it's your responsibility to fix it.

Writing tests is hard. The line between unit tests and integration tests is blurrier than you think. Knowing what to mock and not mock is subjective.

It’s much nicer to find things while developing rather than in prod. As I continue writing software, I try to stay away from mocks if possible. Tests being a bit more heavyweight is completely worthwhile when it comes to a much higher reliability. If mocks are really required by my code reviewer, I’d rather write more (and maybe even redundant) tests rather than skip out on tests. Even if I can’t use a real dependency in a test, I will still try to use other options first before mocks, like a local server.

Google’s “Testing on the Toilet” has a good note on this from 2013. They note that overusing mocks causes:

• Tests can be harder to understand because now you have this extra code someone has to understand along with the actual production code.

• Tests can be harder to maintain because you have to tell a mock how to behave, which means you leak implementation details into your test.

• Tests overall provide less assurance now because the reliability of your software now is only guaranteed IF your mocks behave exactly like your real implementations (which is hard to guarantee and often ends up out of sync).

4. Minimize mutable state.

Computers are VERY fast. In the optimization game, it’s super popular to instantly throw caching and store everything in a database immediately. I think this is probably the end state of most successful software products and services. Of course, most services will need some sort of state, but it’s important to figure out what is truly necessary storage-wise versus what can be derived on-the-fly. 

In the “v1” of something, I’ve found that minimizing as much mutable state as possible gets you pretty far. It lets you develop faster because you don’t have to worry about sync bugs, conflicting data, and stale state. It also lets you develop functionality piece-by-piece, rather than introducing too much at once. Machines are fast enough today where doing a few redundant calculations is totally fine. If machines are supposedly “replacing us” soon, then they can handle a few extra work units of calculations.

Subscribe now

Meta recently released a paper called “Automated Unit Test Improvement using Large Language Models at Meta”.

A major win of this paper is that while it integrates LLMs into a developer’s workflow, it also recommends fully-formed software improvements that are verified to be both correct and an improvement to current code coverage. This is not a magic pill, but it’s a good start on making LLMs more useful. Compare this to ChatGPT, where suggestions still have to be manually verified to work - and we all know that debugging code is twice as hard as writing it.

Meta claims that “this is the first paper to report on LLM-generated code that has been developed independent of human intervention (other than final review sign off), and landed into large scale industrial production systems with guaranteed assurances for improvement over the existing code base.” (Bleh! That was a mouthful.)

Furthermore, there are solid principles that developers can take away in order to use AI effectively themselves.

Table of Contents:

• Key Points

• Stats

• Actionable Takeaways ← if you’re short on time, just read this!

• How TestGen-LLM Works

Key Points

TestGen-LLM uses an approach called ‘Assured LLM-based Software Engineering’ (Assured LLMSE), using private, internal LLMs that are probably fine-tuned with Meta’s codebase. This means that it uses LLMs to generate code improvements that are backed by “verifiable guarantees”of improvement and non-regression.

TestGen-LLM uses an ensemble approach to generate code improvements. This means that it uses “LLMs, prompts, and hyper-parameters” to generate a set of candidate improvements, and then selects the best one. This approach can help to improve the quality of the generated improvements.

TestGen-LLM is specifically designed to improve existing human-written tests rather than generate code from scratch. A good way to think about this LLM is: it's a junior dev with the task of creating more comprehensive tests for existing code. Other devs have more important things to work on, so this LLM gets the fun task of improving unit tests.

The tests that it creates in its pull requests are often good, and sometimes trivial or pointless. Occasionally, a test it produces is really good or uncovers a bug inadvertently. Regardless, this work wouldn't have been done by humans anyways due to priorities. All of its pull requests require a human reviewer before pushed into the codebase.

TestGen-LLM has been integrated into Meta's software engineering workflows. This means that it can be used to automatically improve tests as part of the development process. It would be cool to see some screenshots of how exactly it’s integrated, but the paper doesn’t provide any.

Stats

These stats are either direct quotes or paraphrased quotes from the paper. A short reminder that all tests that were landed required a final sign-off by human reviewers.

• The image above shows that: in an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM test cases that were generated built correctly, 57% passed reliably, and 25% increased coverage.

• TestGen-LLM was able to improve 10% of all classes to which it was applied and 73% of its test improvements were accepted by developers, and landed into production.

• In a “test-a-thon” between engineers, where various Meta engineers created tests in order to increase Instagram’s test coverage, “the median number of lines of code added by a TestGen-LLM test was 2.5.”

• However, one test case “hit the jackpot” and covered 1,326 lines.

• All improved cases generated during the “test-a-thon” did “cover at least one additional valid corner case, such as an early return and/or special processing for special values such as null and empty list.”

Actionable Takeaways

TestGen-LLM is a good example of how LLMs can be used to improve dev productivity and software reliability in a time-efficient manner. (Note: many of these are my own personal opinions that I’ve taken away from the paper.)

Incremental, integrated improvements for specialized use cases

Small context windows and scattered dependencies make LLMs nearly unusable for non-boilerplate solutions in large codebases. Aside from any privacy concerns, it’s not feasible to paste in multiple files of code into an LLM when there could be 20+ dependencies from across a codebase in a C++ header file (as an example). Even if you do paste in multiple files, there is a time and cognitive cost to actually using and trying the code outputted by an LLM in a chat window or even in the code editor by GitHub Copilot.

The price of extra cognitive load cannot be understated. Hacker News commenters find the inaccuracies of GPT-based tooling exhausting and unreliable. This is where the verification of outputs being both valid and non-regressive is extremely important. Each test by TestGen-LLM requires a human sign-off anyways, so any programmatic guarantees are useful here.

This means that for a long-term productivity boost in large codebases, improvements will probably come in incremental, specialized use cases, like test generation and automatic suggestions during code reviews. These are also low risk ways to save cumulative developer time. Basically, “GPT wrappers” will continue to be useful 🙂.

Finding and catching edge cases

The real value of LLMs here are displayed through the edge cases. The paradox of writing good code is that nobody ever gets credit for fixing problems that never happened.

Will Wilson writes:

“The fundamental problem of software testing… is that software has to handle many situations that the developer has never thought of or will never anticipate. This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too. This makes conventional testing great for catching regressions, but really terrible at catching all the “unknown unknowns” that life, the universe, and your endlessly creative users will throw at you.”

LLMs can’t really “think outside the box” because they only really know their training data. However, their box can be larger than a human’s, so they have the potential to think about cases that a human might miss.

Most of the test cases created by Meta’s TestGen-LLM only covered an extra 2.5 lines. However, one test case covered 1326 lines! There are two ways to think about this case:

• The optimistic case is that the LLM possibly caught an important edge case that was missed by humans when testing.

• The more realistic case is that the LLM caught a codepath that was just missed altogether, rather than catching an edge case.

More code coverage != better code. We don’t want to use lines covered as a chaseable metric, though it would be any bad engineering manager’s dream. More code coverage is gameable, but the downsides of rote coverage created by LLMs is that bugs are also codified, which means they can go unnoticed until they appear in production.

The value here is mostly bringing edge cases that lie outside of the developer’s immediate context, but still within the LLM’s training data, to the developer’s attention.

Like I mentioned earlier, a good way to think about this LLM is that it's a junior dev with the task of creating more comprehensive tests for existing code. LLMs may not generate perfect code on the first try, but it at least provides options that otherwise may not have been thought of. Other devs have more important things to work on, so this LLM gets the fun task of improving unit tests. The tests that it creates in its pull requests are often good, and sometimes trivial or pointless. Occasionally, a test it produces is really good or uncovers a bug inadvertently.

In fact, it’s so high that the creator of FoundationDB’s startup, Antithesis, is entirely based on the fact that software testing edge cases are best found by continuously searching software for problems. For reference, FoundationDB was acquired by Apple and is the basis for Apple iCloud’s billions of databases.

Orchestration, pipelines, and processing are required

Base model LLMs aren’t “plug-n-play" and shouldn’t reasonably ever be expected to. Sure, they might output pristine React and Tailwind CSS code, but that’s a narrow use case in most production codebases. They need a fair amount of processing and filtering for code generation tasks that require correctness. Part of this processing means grounding LLMs with examples. Google and Meta both make suggestions based on existing code, where the results are much, much better than raw generation. LLMs used in production should take ideas from how Meta processes and filters LLM outputs, and most outputs should be expected to be discarded.

Integrations win

LLMs do work best integrated into workflows. This is a reason why GitHub Copilot is so popular and another reason why Google’s Workspace integrations are a great idea. Asking a chatbot works great for certain use cases, like boilerplate code generation, but chatbots can often fail at more complex use cases.

How TestGen-LLM Works

TestGen-LLM applies a series of semantic filters to candidate solutions generated by Meta’s internal LLMs, making sure that only the most valuable tests are preserved. Here’s how it works:

Filter 1: Buildability: Initially, TestGen-LLM checks if the generated code can be built within the app's existing infrastructure. Any code that fails to build is immediately discarded.

Filter 2: Execution (does the test pass?): Next, the system runs the tests that passed the buildability filter. Any test that doesn't pass is discarded. This step is crucial because, without a way to automatically determine the validity of a failing test (whether it's due to a bug or an incorrect assertion), TestGen-LLM opts to keep only those tests that can be used for regression testing (aka making sure they can protect current code against future regressions). 

Filter 3: Flakiness: To address the issue of flakiness (tests that pass or fail inconsistently under the same conditions), TestGen-LLM employs repeated execution. A test must pass consistently across multiple (five) executions to be considered non-flaky.

Filter 3: Coverage Improvement: Finally, to ensure that new tests actually add value, TestGen-LLM evaluates them for their contribution to test coverage. Tests that do not enhance coverage by exploring new code paths or conditions are discarded. Only tests that provide new insights or protect against regressions are kept.

These processing filters are pretty important as they guarantee improvements to a test suite. It also shows that LLMs are very far from being “plug-and-play.”

The tests that successfully pass through all these filters are guaranteed to enhance the existing test suite, offering reliable regression testing capabilities without duplicating effort or wasting resources. Pre- and post-processing steps in TestGen-LLM facilitate the extraction and reconstruction of test classes, streamlining the integration of new tests into the software development workflow.

Conclusion

This paper is a good formalization of an use case that many devs probably already use LLMs like ChatGPT, Gemini, and Mistral/LLaMA for. Keeping it in writing is a good way of tracking the progress of future improvements on LLMs in the software reliability space. So far, we see that LLMs work best as an extension of the human, best thought of as a junior dev that needs assistance and prodding. As time goes on, we’ll definitely see LLMs be able to catch and test for bugs in increasingly complex software systems.

The question is - will that make software easier to develop in the long run or will it lead to a proliferation of software complexity in the future?

Author’s Note: These are my personal notes on publicly available papers regarding iCloud. All sources are linked at the bottom.

Subscribe now

Throughout the last few months, I’ve written about various tech “behind-the-scenes” in Big Tech, such as Meta's internal serverless platform, that we are lucky to see written about publicly.

Apple, on the other hand, is not as public with their infrastructure. I wanted to find out how Apple built iCloud and in this article, I cover everything I know.

Apple uses FoundationDB and Cassandra for iCloud and CloudKit, their cloud backend service. And yes, the title is not incorrect: Apple really does store billions of databases in their extreme multi-tenant architecture.

Timeless Real-World Lessons

Before reading, these are the applicable lessons and guidelines.

I found that a lot of the lessons from the paper and from Apple track very closely to the lessons of Meta’s architecture of their serverless platform.

• Both use asynchronous processing smartly in order to make user functionalities smoother. Meta uses their serverless stack for non-user-facing functions. Apple uses async processing for almost all of the Record Layer’s functionalities (explained in-depth down below) in order to hide latency from the user.

• Both utilize stateless architecture liberally, knowing that they have intense scalability needs.

• Both isolate resources logically for reliability and availability.

• Both handle diverse ranges of needs simply. Apple mentions how it’s “tempting

  to provision and operate separate systems to store ‘small data’ and ‘big data.’” However, this increases operational complexity, and instead, they handle all types of data needs with one abstraction. Meta does the same with their serverless platform providing one abstraction for all kinds of function loads.

• Both build layers of abstractions in order to make the developer experience better. Application developers shouldn’t have to worry about scalability needs - that’s handled deeper down the stack by distributed system engineers.

• Know your user. Each layer, API, and design decision provided by Meta and Apple is guided by clear knowledge on who the user of the specific technology is, whether it be an application development team or an observability team.

Cassandra

Cassandra is a wide-column NoSQL database management system. It was originally developed at Facebook to power the Facebook inbox search feature. Interestingly, Meta themselves have replaced much of their Cassandra usage with ZippyDB instead.

iCloud is partly powered by Cassandra. Apple runs one of the largest Cassandra deployments in the world, according to DataStax.

They report:

• Over 300k instances/nodes

• Hundreds of petabytes of data (if not exabytes)

• Over 2 PB per cluster, with thousands of clusters

• Millions of queries per second

• Thousands of applications

Other snippets of Cassandra in iCloud show that there are exabytes of data under management. There are multiple Cassandra nodes per server, and the teams at Apple are smart about blast radius and sharding. This makes sure iCloud data availability is near 100%.

Cassandra is still actively improved upon at Apple. Scott Andreas of Apple gave a talk about the future of Cassandra last month. On Apple’s jobs page, they commonly mention Cassandra when hiring for distributed systems engineers.

However, CloudKit + Cassandra ran into two scalability limitations, which led to their adoption of FoundationDB.

1. Within a single zone, only one operation can happen at a time, even if different records are being edited. This can be problematic for apps where multiple users or devices need to work on shared data simultaneously.

2. When updating multiple records simultaneously in an atomic operation, the updates are limited to a single Cassandra partition. These partitions have a maximum size they can handle, and as a partition's size increases, Cassandra tends to slow down.

FoundationDB and the Record Layer solved both of these issues.

FoundationDB

Apple is much more public about FoundationDB. They acquired FoundationDB in 2015 and have since published various papers detailing their use of FoundationDB.

FoundationDB is an open-source, distributed, transactional key-value store. It’s designed to handle large volumes of data and works well for both read/write workloads and write-heavy workloads. It’s also ACID-compliant. 

Apple uses the FoundationDB Record Layer extensively for CloudKit (Apple’s cloud backend service). 

From their GitHub:

The Record Layer is a Java API providing a record-oriented store on top of FoundationDB, (very) roughly equivalent to a simple relational database, featuring:

• Structured types - Records are defined and stored in terms of protobuf (Protocol Buffer) messages. Protocol Buffers were first designed by Google.

• Indexes - The Record Layer supports a variety of different index types including value indexes (the kind provided by most databases), rank indexes, and aggregate indexes. Indexes and primary keys can be defined either via protobuf options or programmatically.

• Complex types - Support for complex types, such as lists and nested records, including the ability to define indexes against such nested structures.

• Queries - The Record Layer does not provide a query language, however it provides query APIs with the ability to scan, filter, and sort across one or more record types, and a query planner capable of automatic selection of indexes.

• Many record stores, shared schema - The Record Layer provides the ability to support many discrete record store instances, all with a shared (and evolving) schema. For example, rather than modeling a single database in which to store all users' data, each user can be given their own record store, perhaps sharded across different FDB cluster instances.

• Very light weight - The Record layer is designed to be used in a large, distributed, stateless environment. The time between opening a store and the first query is intended to be measured in milliseconds.

• Extensible - New index types and custom index key expressions may be dynamically incorporated into a record store.

From the FoundationDB Record Layer paper, they write:

“[FoundationDB Record Layer is used] to provide powerful abstractions to applications serving hundreds of millions of users. CloudKit uses the Record Layer to host billions of independent databases, many with a common schema.”

Why is the FoundationDB Record Layer used?

The structure of FoundationDB, Record Layer, and CloudKit looks like this:

• FoundationDB does all the distributed systems and concurrency control work.

• The Record Layer serves as a relational database to make FoundationDB easier to work with.

• CloudKit is the highest layer on top and provides features and APIs for application developers. CloudKit isn’t the only thing built on top of the Record Layer, but there are also other layers on top built internally for things that need structured storage, like a JSON document store.

The Record Layer allows Apple to support multi-tenancy at scale. 

Actually, that’s somewhat underselling it.

The Record Layer is used for extreme multi-tenancy, where each user of each application gets independent record stores. This means the Record Layer hosts billions of independent databases sharing thousands of schemata.

That’s better! And much, much more impressive.

The Record Layer is engineered to handle multi-tenancy at such a large scale thanks to two fundamental architectural decisions. 

1. The layer operates statelessly allowing for effortless scaling of the compute resources by simply adding more of these stateless instances.

   1. This stateless architecture simplifies the task for load balancers and routers since they only need to focus on the location of the data rather than the capabilities of the compute servers. Additionally, stateless servers have a reduced set of resources to distribute amongst clients.

2. The layer uses record store abstraction to manage resource allocation and scalability effectively. This abstraction represents the entirety of a logical database, comprising serialized data, indexes, and the operational state.

   1. Each record store is allocated a specific key range, which guarantees logical separation of data for different tenants. Transferring a tenant's data, if necessary, becomes a straightforward process of relocating the assigned key range to a new cluster, as all the information required to manage and use the record store is contained within this range.

This is a good stopping point for a cursory glance of how Apple built iCloud.

For those who are interested in the technical aspects of CloudKit, FoundationDB, and Record Layer, read on.

How CloudKit uses FoundationDB and Record Layer

In CloudKit, an application is represented by a 'logical container' which follows a defined schema. This schema outlines the necessary record types, fields, and indexes to enable efficient data retrieval and queries. Applications organize their data into 'zones' within CloudKit, which allows for grouping records logically for selective synchronization with client devices.

For each user, CloudKit designates a unique subspace in FoundationDB. Within this subspace, it creates a record store for each application a user interacts with. In essence, CloudKit manages a massive number of logical databases—multiplying the number of users by the number of applications—each containing its own set of records, indexes, and metadata, amounting to billions of databases.

When CloudKit receives a request from a client device, it directs this request to an available CloudKit Service process via load balancing. This process then interacts with the specific Record Layer record store to fulfill the request.

CloudKit converts the defined application schema into a metadata definition within the Record Layer, which is stored in a separate metadata store. This metadata is augmented by CloudKit-specific system fields that track the record's creation, modification times, and the zone where the record is stored. Zone names are prefixed to primary keys to enable efficient access to records within each zone. Alongside indexes defined by the user, CloudKit also manages 'system indexes' for internal purposes like managing storage quotas by keeping an index that tracks the size of records by their type.

FoundationDB and the Record Layer together solve 4 key problems for Apple that either Cassandra alone or FoundationDB alone could not.

Problem Solved: Personalized full-text search

FoundationDB helps solve personalized full-text search for users to quickly access their data.

Their system utilizes FoundationDB's key order to allow for quick searches on text beginnings (prefix matching) and also for more complex searches (like finding words that are close together or in a specific order - proximity and phrase search) without no extra overhead.

In traditional search systems, you often need extra processes running in the background to keep the search index up to date, but Apple’s system does everything in real-time, which means as soon as data changes, the search index is updated instantly—no extra steps needed. 

Problem solved: High-concurrency zones

With FoundationDB, CloudKit handles many updates happening simultaneously in a smooth way.

Previously, with Cassandra, CloudKit used to rely on a special index that kept track of changes in each zone to synchronize data across devices. When a device needed to update its data, it would check this index to see what's new. But this system had a drawback: it could cause conflicts when multiple updates happened at the same time.

But with FoundationDB, CloudKit uses a special kind of index that keeps track of the exact sequence of each change without causing conflicts. This is done by assigning a unique 'version' to every change, and when CloudKit needs to sync, it looks at these versions to figure out what updates a device has missed.

However, when CloudKit needs to move data across different storage clusters—perhaps to distribute the load more evenly—things get tricky because each cluster has its own version numbers that don't match up. To solve this, CloudKit gives each user's data a 'move count' (called the “incarnation”), which increases every time their data is transferred to a new cluster. Each record update includes the user's current “incarnation” number, ensuring that even after a move, CloudKit can still figure out the correct order of updates by looking at both the incarnation and version numbers.

When they switched to this new system, CloudKit faced the challenge of dealing with old data that didn't have these version numbers. They cleverly overcame this by using a special function that sorts old updates using the previous system before the new ones. This meant no complicated changes to the app or leaving behind outdated code. This function took into account the incarnation, version, and the old update-counter values to maintain the correct order of records.

Problem solved: High latency queries

FoundationDB is designed for high-concurrency, not low latency. This means it can handle a lot of tasks at the same time rather than focus on the speed of individual tasks.

To make the most of this design, the Record Layer does a lot of its work 'asynchronously'— it queues up tasks to be completed in the future, allowing for other work to be done in the meantime. This approach helps to cover up any delays that might occur during these tasks.

However, the tool that FoundationDB uses to communicate with its database is designed to do one thing at a time using a single thread for networking. In earlier versions, this setup caused a traffic jam in the system because everything was waiting for its turn in this network thread. The Record Layer had been using this single-threaded approach, which led to a bottleneck.

To improve this, Apple reduced the workload on this network thread. Now, complex tasks seem faster because the system is working with the database on several fronts at the same time, rather than forming a queue. This way, the latency, or the apparent slowness, is hidden because the system doesn't wait for one task to finish before starting another.

Problem solved: Conflicting transactions

In FoundationDB, if one transaction is reading certain keys and another transaction modifies those same keys at the same time, it causes a 'transaction conflict'. FoundationDB allows precise management of these conflicts by providing control over the sets of keys that can cause these conflicts when read or written to. 

A common method to avoid unnecessary conflicts is to perform a special kind of read that doesn't cause conflicts, known as a 'snapshot' read, on a range of keys. If this read finds important keys, the transaction will only flag those specific keys for potential conflict, rather than the entire range. This ensures the transaction is only affected by changes that actually matter to its outcome.

The Record Layer uses this strategy to efficiently manage a structure known as a skip list, which is part of its ranking index system. However, manually setting these conflict ranges can be tricky and can lead to bugs that are difficult to identify, especially when they're mixed in with the main logic of the application. Therefore, it's recommended that systems built on top of FoundationDB create higher-level tools, like custom indexes, to handle these patterns. This approach helps to avoid leaving the responsibility to relax conflict rules up to each client application, which could lead to mistakes and inconsistencies.

Thanks for reading! If you’re interested in reading the full FoundationDB Record Layer paper, it’s here: FoundationDB Record Layer: A Multi-Tenant Structured Datastore.

As I became a better software engineer, I noticed 4 key habits in my daily workflow that had made me much more productive.

Dev Starter Packs (Sponsored)

Dev Starter Packs has 100+ pre-built, interactive, beautiful React Native components to help you build your mobile app faster. The components are used in mobile apps today with thousands of users and are constantly being updated weekly with new ones.

Similar to Tailwind UI, these components are lifetime access with more 200+ more components planned to be released. Dev Starter Packs also has a full-featured Next.js boilerplate and a fully-integrated React Native boilerplate, both offering easy payments, auth, logging, and error monitoring setup. With a holiday sale ongoing, don’t miss out - build your ideas in 2025.

1. Leave work slightly unfinished for easier flow the next day

“Flow” is the root of productivity when programming. 

Since software engineering is a “maker” activity where I’m producing something, I generally perform best when I have a large block of uninterrupted “flow” time to work on a project.

However, it can often be really hard to get into flow if you’re stuck scrambling on what tasks your project goals entail. Ambiguity is difficult to deal with. Not even knowing where to start can make reaching that “flow state” much harder.

Each successful action snowballs into more.

There are a few techniques I use to do this:

• Stop right before a “sticking point.” A sticking point is a task that’s part of a project where I know the steps to do to complete it, but I don’t know if there are hidden costs. 

  • For example, if my sticking point is deploying my ML model and HTTP server to a dev instance and verifying that it processes requests properly, then the hidden costs are deployment errors, authentication errors, resource constraints, etc.

• Write down the next steps extremely clearly. Writing down steps makes regaining context and the state of mind from the day before easier.

  • Make them actionable and unambiguous.

Subscribe now

2. Get better at keyboard and mouse shortcuts

My first experience with a real “shortcut ninja” was actually not with a software engineer. It was with my investment banker friend, who sped around his Excel sheets without ever touching the mouse. 

After that, I took the time to learn keyboard shortcuts, to the point where I grab my mouse ~60% less than I used to. (Yes, I tracked this.)

It’s not just me! Sam Altman has a quote on this in his Productivity essay:

I also made an effort to learn to type really fast and the keyboard shortcuts that help with my workflow.

Every editor and tool in my workflow has keyboard shortcuts for pretty much any action you can think of. This doesn’t just apply to your IDE, but also your version control systems, your web browser, and your docs.

For example, my IDE has a linter/formatter/cleaner all in one shortcut, which I use often as I write code to make sure lines stay neat.

I commonly use Command/Ctrl + Shift + V to paste in text without formatting in docs and chats.

Pressing . (period) on a GitHub repo page will automatically open up the repo in a VSCode Web instance.

When I do need to touch my mouse, it’s configured with shortcuts also. I’m lucky enough to have a mouse with buttons on the sides. I’ve programmed these buttons to switch between Spaces on my Mac, though you can program them for whatever feels intuitive for you. (You can even program them to be different per program.)

The best way to learn shortcuts? Introduce the most common parts of a program that you use, one a time.

For example, if you find yourself right-clicking to format your code often, that can be the first one you “practice.” Every so often, add a new shortcut to your repertoire and use it naturally as you code throughout the weeks. Over time, the shortcuts will be muscle memory.

3. Keep a list of searchable commands and links handy

I commonly have to run a set of common commands on my terminal.

I have certain pages that I always visit and some notes about various languages that I always come back to. For example, I simultaneously use templates too rarely and yet too often when writing C++, meaning I usually need to reference the docs when using them.

Instead of digging around documentation pages or constantly looking through my terminal history, I keep commands and common doc lookups in a giant doc with one word describing the command. I call it my Big Book of Commands, which is around ~10 pages long now. I’m easily able to find any command I need with a quick Ctrl+F. Then, a Shift + Command + ➡️ is a full line-select for an easy copy-paste.

I also have a few common macros programmed into my keyboard for the commands and terms, like hard-to-remember ACL groups. Sometimes, I utilize Terminal aliases.

My friend wrote a great article that dives into his own workflow tips, which starts off with a great primer into aliases, keyboard shortcuts, and tools: The top 7 software engineering workflow tips I wish I knew earlier 🧰

4. Say “no” more

This is less directly programming related, but I learned to say “no” to things.

I said no to novel technology when boring technology would do the job.

I said no to automating something when it only needed to be done once manually.

I said no to more tasks when I knew I was already overloaded with work (even though my people-pleasing mind pleaded to take them on).

I said no to scope creep suggested by our designers.

I said no to low-impact tasks.

Learning to say no was harder than I expected, yet one of the most valuable skills I’ve applied in both the workplace and in my personal life.

Sometimes, it’s painful to say no to things. In both my career, hobbies, and personal life, there are times I say no to things I really want to do. But I don’t because I know my time and energy is better spent on what I’m currently focusing on.

It’s a cliche to quote Steve Jobs, but I remind myself of his famous quote “focus is about saying no” often.

My friend also has a fantastic article about saying no, which I highly recommend: The Software Engineer's guide to saying "no"

Subscribe now

A lot of ex-Google engineers talk about how much they miss Critique, Google’s code review tool, out of all the internal tools they leave behind.

Another Reddit comment laments about how they “miss [Critique] so bad”, listing off various features they miss, like its “attention set,” which is explained below.

This tracks with Google’s own findings. Internally, 97% of Google software engineers are satisfied with Critique.1

But what is Critique exactly and what makes it so good? 

How does this pair with Google’s actual process of code review?

In this article, I dive into:

• Google’s guidelines for efficient code review

• Critique, their code review tooling, and AI-powered improvements

• Internal statistics on Google code reviews

• Why Critique seems to be so loved by Googlers

While the SWE Book covers the basics of Critique, new blog posts and research by Google show new developments in their code review processes, such as using AI to make automatic suggestions and improvements to code changes.

Dev Starter Packs (Sponsored)

Dev Starter Packs has 100+ pre-built, interactive, beautiful React Native components to help you build your mobile app faster. The components are used in mobile apps today with thousands of users and are constantly being updated weekly with new ones.

Similar to Tailwind UI, these components are lifetime access with more 200+ more components planned to be released. Dev Starter Packs also has a full-featured Next.js boilerplate and a fully-integrated React Native boilerplate, both offering easy payments, auth, logging, and error monitoring setup. With a holiday sale ongoing, don’t miss out - build your ideas in 2025.

Google’s Code Review Guidelines

To understand how Google writes clean, maintainable code, I wrote about Google’s readability process here.

Google’s guidelines for a good code review include:

• Continuous improvement over perfection: While a more seasoned developer may find a less experienced developer’s code to not be up to their personal standard, Google encourages continuous improvement over perfection. Developers need to make progress; overly difficult reviews can discourage future improvements. 

• Maintain or improve the health of the codebase

• Follow the Style Guide: When code style comes into question, Google’s style guides are followed and referenced to a tee.

• Always Share Knowledge: Reviewers are encouraged to share knowledge about language features, the codebase, and other relevant artifacts through code review. Usually, these guidelines are sent with “supporting documentation,” such as links to Google/Abseil’s C++ Tips of the Week. 

  • At Google, the educational aspects of a code review are highly emphasized.

• Write Small Changes: Keep changes limited to about 200 lines of code if possible.

• Strict Standards for Lightweightness: Google expects reviewers to review a code change in less than 24 hours and encourages reviews to only have one reviewer if possible, which saves time for everyone involved.

  • “The majority of changes at Google are small, have one reviewer and no comments other than the authorization to commit. During the week, 70% of changes are committed less than 24 hours after they are mailed out for an initial review.”2

• Politeness and Professionalism: Maintaining a culture of trust and respect is crucial. Feedback should be professional, avoiding personal criticism. Reviewers should be open to the author's approach, offering alternatives only if necessary, and treating each comment as a learning opportunity.

  • I wanted to comment on this one specifically, as while it seems obvious, it’s not as easy to do in practice. It’s common for people to get attached to the code they’ve written or for reviewers to come off as harsher than they meant to. The written word leads for less nuanced communication compared to face-to-face.

  • Google has done a lot of research into how code comments can affect developer productivity and motivation.

    • Predicting Developers’ Negative Feelings about Code Review

    • Destructive Criticism in Software Code Review Impacts Inclusion

    • Detecting Interpersonal Conflict in Issues and Code Review: Cross Pollinating Open- and Closed-Source Approaches 

    • Using research to make code review more equitable

For a changelist (CLs are Google’s version of pull requests, or PRs) to go through, it must have 0 unresolved comments, a LGTM (Looks Good To Me) from at least one reviewer, and 2 types of approval:

• people who “own” the part of the codebase the files go into

• Readability approvals, who approve the “readability” and style of the code

One person can be the LGTM-er and approver all at once.

Subscribe now

Critique: Google’s Code Review Tool

Critique is Google’s code review tool, allowing engineers to efficiently review and submit code changes.

They also have a diff view between the current codebase and the proposed changes:

Importantly, Google’s recent publications in 2023 show they have comprehensive AI-powered code review tools in-house now (as shown above). 

When reviewers leave comments on code, Critique will show suggested ML-powered edits, which mean that the author of the code review just has to click one button to address the comment in entirety.

Recent developments, based on Google research papers, tell us that Google is improving developer productivity with AI-powered code review tools where possible.

The Code Review Flow

The SWE Book covers Critique in detail with over 5000 words, but I summarize the key points here and have provided a link for those who want to dive deeper into it. 

Stage 1: Create a Change

A CL (or Pull Request) is created in Google’s in-house code editor Cider, which is “tightly integrated with Critique and other in-house Google tools”, leading to higher developer productivity.

• Prereview Tools: Critique assists in polishing changes before review, showing diffs, build and test results, and style checks.

• Diffing and Visualization: Enhanced with syntax highlighting, cross-references, intraline diffing, whitespace ignoring, and move detection.

• Analysis Results: Displays results from static analyzers, highlighting important findings and offering fix suggestions. These include “presubmits,” which are automated tests that are run in Critique that enforce project-specific invariants.

Critique integrates feedback channels for analysis writers.

Reviewers have the option to click “Please fix” on an analysis generated comment as a signal that the author should fix the issue.

Either authors or reviewers can click “Not useful” in order to flag an analysis result that is not helpful in the review process. (Source)

Stage 2: Request Review

When a pull request is ready to be reviewed, the code author adds reviewers and sends it to them officially for review.

When a PR/CL is sent in for review, “presubmits” are run if they haven’t already on the current snapshot of code. This means that everyone involved in the review knows whether or not the code breaks anything.

Code reviews can also be anonymized, where the code author can be kept anonymous from the reviewer. However, Google didn’t find much of an useful difference between anonymous code reviews and “real” code reviews.3

Stages 3 and 4: Understanding and Commenting on a Change

• Anyone can comment on a change and there are features to track review progress and resolve comments.

  • Unresolved comments represent action items for the change author to definitely address. These can be marked as “resolved” by the code author themselves when they reply to the comment.

  • Resolved comments include optional or informational comments that may not require any action by a change author

There’s a dashboard for review statuses and an “attention set” which lets people involved in a certain code review who is the current person being waited upon.

The attention set is commonly a highly-regarded feature by Google engineers.

Stage 5: Change Approvals

As noted above, for a review to go in, it needs an LGTM from at least one reviewer. It needs 0 unresolved comments, though a code author can mark a comment resolved themselves when they reply. It also needs approvals from owners of the part of the codebase the code is going into, along with a readability approval.

As mentioned before, this can all be done by one reviewer.

Stage 6: Committing a Change

Changes are submitted and committed within Critique itself.

Critique is valuable even after changes are submitted.

“Google researchers found strong evidence that Critique’s uses extend beyond reviewing code. Change authors use Critique to examine diffs and browse analysis tool results. In some cases, code review is part of the development process of a change: a reviewer may send out an unfinished change in order to decide how to finish the implementation. Moreover, developers also use Critique to examine the history of submitted changes long after those changes have been approved.”

Modern Code Review Stats at Google

Google conducted a study on code review at the company specifically. I’ve listed some interesting stats from their paper.

Change Authoring Frequency:

• Median: 3 changes per week.

• 80% of authors make fewer than 7 changes weekly.

Review Frequency:

• Median: 4 changes reviewed per week.

• 80% of reviewers handle fewer than 10 changes weekly.

Time Spent Reviewing Per Week:

• Average: 3.2 hours per week

• Median: 2.6 hours per week

Initial Feedback Waiting Time:

• Small changes: Median time under 1 hour.

• Very large changes: About 5 hours.

Overall Review Process Time:

• Median latency for all code sizes: Under 4 hours.

• Comparison with Other Companies:

  • AMD: 17.5 hours (median time to approval).

  • Chrome OS: 15.7 hours.

  • Microsoft Projects: 14.7, 19.8, and 18.9 hours.

  • Microsoft (another study): 24 hours.

“Previous studies have found that the number of useful comments decreases and the review latency increases as the size of the change increases. Size also influences developers’ perception of the code review process; a survey of Mozilla contributors found that developers feel that size-related factors have the greatest effect on review latency.” (Source)

Furthermore, the number of comments received per change decreases with seniority.

Why is Critique so loved by Googlers?

Most Googlers and ex-Googlers are big fans of Critique.

Internally, 97% of Google software engineers are satisfied with Critique.4

I asked 7 Googlers about why they prefer Critique over tools they may have used in the past, like GitHub.

They noted:

• Static analysis: Google has a full-featured suite of static analysis tools that provide actionable feedback on code automatically. This saves both code authors and reviewers time as reviewers then don’t have to nitpick at obvious items.

• Focus on only the latest changed files: There’s a focus on just the latest “snapshot” of code. Previous snapshots, commits, and code changes are not really a focus, allowing for a cleaner user interface.

• A familiar, side-by-side diffing interface: It shows "diff from the last review" by default.

• ML-powered suggestions: Google’s new ML-powered suggestions speed up code review immensely.

• Tight integration with other Google tooling: Critique is integrated extremely well with Google’s IDE and other internal tools, like their bug tracker. When everything is connected together, productivity is better.

  • This includes easy linking of code, comments, and tickets.

• “Action set” tracking: This lets people know who is supposed to be taking the next action.

• Satisfying gamification: While Critique isn’t built to be gamified, Googlers reported how they enjoyed it when Critique “went green,” which meant a PR was ready to submit (all tests passed, reviewers LGTM-ed and approved).

A Reddit comment lists more:

A great quote on static analysis:

A qualitative study of 88 Mozilla developers found that static analysis integration was the most commonly-requested feature for code review.

Automated analyses allow reviewers to focus on the understandability and maintainability of changes, instead of getting distracted by trivial comments (e.g., about formatting).

Thoughts and Takeaways

While many of these features are available in other tools today, I believe it is the tight integration and extreme “personalization” of the tooling towards a Google-specific workflow and codebase that makes it so loved.

At the same time, that means it’s not realistic for every company to replicate Critique and related tools exactly. For example, some of their tooling seems specific to challenges created by their monorepo structure.

While Critique itself will never be open-sourced, Gerrit is a similar tool to Critique. It’s an open-source code review tool also created and maintained by Google.

However, I think Google does put a lot of effort and thought into developer productivity. They publish their research freely and there are useful takeaways from their work.

Subscribe now

When I was an undergrad, Leetcode broke my brain. I would see top solutions of esoteric one-liners and wrongly think “how do I ever get this good?”

This is commonly called “code golfing”. It’s a fun hobby, but very far from “good code.”

Everybody (including those on Leetcode) knows this isn’t good code. In the industry, it’s the worst code one could write.

Okay, I admit - this is a pretty bad example of clever code, because it is so obviously bad. Here’s an example of a code snippet I came across in some old code that I found annoying:

However, on the other end of the spectrum, I realized eventually that the clearest code was actually the hardest to write.

It made sense retrospectively. Reviewing the code of a senior staff software engineer was much easier to follow and review compared to the code of an entry-level L3 engineer.

Nucleate (Featured)

Nucleate is an online platform to build and deploy beautiful web and mobile apps with AI. It takes away the set-up headaches so you can focus on building your product.

You can use the code ENGCODEX for 50% off your first month.

Clear code: the good and the bad

"Debugging code is twice as hard as writing the code in the first place. Therefore, if you write code as cleverly as possible, you are, by definition not smart enough to debug it."

- Brian W. Kernighan

The “power” of clear code, for better or for worse, was made fully clear to me after a certain incident at work.

I once wrote a module in C++, a language that is a bit harder to read compared to other languages simply due to its verbosity.

I started with just two files (.h/.cpp) and all the implementation code went into just these two files.

The result was this giant, disgusting piece of spaghetti on the inside, but a perfectly working program on the outside.

This would never get past code review.

I split the implementation into 8-10 diffs. Each diff was a neat, containerized piece of code, with convenient placeholders for dependencies that would arrive in a later diff. It had code neatly split out into helper functions and helper files when necessary.

Each diff had reasonable unit test coverage - the basics and some obvious edge cases were covered, but I didn’t go wastefully overboard with it.

Each diff also took me quite a few iterations of “code cleaning,” refactoring, and more. It took a lot more effort than I expected to achieve “clear code,” especially for such a large program.

The result? A beautiful landing of the module, with easy to read, clear code.

While I was proud of it, there was suddenly a problem when I talked to my manager about it.

“While I understand how complex this was, when it comes to performance reviews, this code looks trivial. It looks too easy, too simple.

I would recommend writing an implementation doc of this module just so we can demonstrate that this was actually quite complex.”

I was shocked - this wasn’t some startup. This was one of the biggest companies in the world, known for their engineering culture.

I now understood why Big Tech seemingly had so many docs — half of the docs I wrote didn’t need to be written, except they did… because I wanted to get raises and be promoted.

While promotion culture in Big Tech is a story for another article (subscribe to see it in your inbox soon 🙂), the main point here is that great code is very clear and readable.

Subscribe now

There’s a popular saying that debugging code is twice as hard as writing it. It’s the reason why when ChatGPT outputs some hogwash, it’s easier just to re-prompt it or write it from scratch yourself instead of trying to figure out the errors in its buggy code.

Clever code is harder to read and looks esoteric.

Clear code is harder to write and looks easy.

Some other thoughts about clear code

• The only way I got better at writing clear, readable code was just writing a lot of code while strictly following a clear style guide.

  • Also, having more experienced devs review my code with a magnifying glass.

    • It was agony to get tons of comments and “nits” about seemingly pointless style in the beginning, but it paid off in the end.

• Coding style is more important than I expected in the beginning. My start to software engineering started from being on the product-minded end of the spectrum and moved towards the “technical-minded” side of the spectrum. 

  • I had started coding solely to start a business, so I initially only cared about code as a tool, resulting in crappy, unmaintainable code.

  • It’s only through more experience with writing code and working within teams that the importance of clear, readable code became more obvious.

  • It’s not just me. This is an obvious revelation to anybody who has been writing code in the industry for more than a year.

• John Carmack once wrote a long email about coding style in 2007, which is an interesting read.

• Google probably has the most public style guide. Vercel also recently released their style guide, and pretty much every company uses some sort of linter and prettifier.

Subscribe now

On January 15th, 1990, AT&T's New Jersey operations center detected a widespread system malfunction, shown by a plethora of red warnings on their network display.

Despite attempts to rectify the situation, the network remained compromised for 9 hours, leading to a 50% failure rate in call connections.

AT&T lost over $60 million as a result with over 60,000 of Americans left with fully disconnected phones.

Furthermore, 500 airline flights were delayed, affecting 85,000 people.

AT&T's long-distance network was supposedly a paragon of efficiency, handling a substantial portion of the nation's calls with its advanced electronic switches and signaling system. This system usually completed call routing within seconds.

However, on this day, a fault originating in a New York switch cascaded through the network. This was due to a software bug in a recent update that contained a critical bug affecting the network's 114 switches. When the New York switch reset itself and sent out signals, this bug caused a domino effect, leading to widespread network disruption.

This software patch had already gone through layers of testing without being caught. This incident was especially surprising because AT&T was known for their rigorous testing.

SWE Quiz

To be a great software engineer, you need to know a lot. But how do you figure out what you don’t know?

SWE Quiz is a resource of 500+ questions to reveal your gaps in software domain knowledge, like databases, caching, and networking. Many of these questions are verified concepts that have been asked in interviews with Meta, Apple, Airbnb, Google, and more.

SWE Quiz also has structured roadmaps for learning important software concepts.

Get Lifetime Access to SWE Quiz

The Problem

The root cause was traced back to a coding error in a software update implemented across the network's switches.

The error, within a C program, involved a misplaced break statement within nested conditional statements, leading to data overwrites and system resets.

The pseudocode:

1  while (ring receive buffer not empty 
          and side buffer not empty):

2    Initialize pointer to first message in side buffer
     or ring receive buffer

3    get copy of buffer

4    switch (message):

5       case (incoming_message):

6             if (sending switch is out of service):

7                 if (ring write buffer is empty):

8                     send "in service" to status map

9                 else:

10                    break // The error was here!

                  END IF

11           process incoming message, set up pointers to
             optional parameters

12           break
       END SWITCH


13   do optional parameter work

The problem:

• If the ring write buffer is NOT empty, then the `if` statement on line 7 is skipped and the break statement on line 10 is hit instead.

• However, for the program to function properly, line 11 should have been hit instead.

• When the break statement is hit instead of the incoming message being processed and pointers being set up to optional parameters, then data (the pointers that should’ve been held) is overwritten

• The error correction software identified the data overwrite and initiated a shutdown of the switch for a reset. This issue was compounded because this flawed software was present in all switches across the network, leading to a chain reaction of resets that ultimately crippled the entire network system.

Despite having a network designed for resilience, one line of code was able to bring down half the country’s main line of communication.

The Fix

It took engineers 9 hours to get AT&T’s system fully back online. They did so mostly by rolling back the switches to a previous, working version of code.

It actually took software engineers two weeks of rigorous code reading, testing, and replication to actually understand where the bug was.

Subscribe now

Conclusion

For AT&T, unfortunately, this wasn’t even their biggest system crash of the 90s. They encountered many more issues later in the decade.

In reality, it wasn’t one line of code that brought down a system. It was a failure in processes.

Today’s companies have even better processes in place, and even then, bugs slip through. Google wrote a great retrospective on 20 years of Site Reliability Engineering, where they reflect on YouTube’s first global outage in 2016.

The scale of an outage for companies is huge and there are lessons to be learned from each outage. For most, however, outages come down to human error and gaps in processes.

My favorite resources for reliability engineering

Google SRE

GitLab’s SRE Role

Site Reliability Engineering Resources

Sources:

https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

https://telephoneworld.org/landline-telephone-history/the-crash-of-the-att-network-in-1990/

https://www.mit.edu/hacker/part1.html

https://prezi.com/qxeu8iayvwpu/1990-att-long-distance-network-crash/

Subscribe now

John Carmack, previous CTO of Oculus and creator of id Software (Doom, Quake, etc), was asked how AI will affect software developer jobs in the future.

He had only one piece of advice: “build full product skills.”

In fact, he’s beat the drum about this trait over and over, in tweets, podcasts, and his writing: being product-minded makes you an extremely valuable engineer.

The best engineers I know are not only extremely sharp technically, but also are able to apply their technical skills in a way that provides the most impact to move their product forward.

Some companies have formalized this mindset into the role of “product engineers,” where they are responsible for the overall direction of the product, not just the engineering. This is surprisingly not common in a lot of tech companies today. 

In this article, I’ll refer to “stack engineers” as basically all software engineers.

When I was doing research on product engineering, I found that PostHog has written a lot of blog posts about it, so I reached out to them to chat about:

• What’s the main difference between “product engineers” and “stack engineers”?

• What does “product engineering” look like in the real world?

• Tips for engineers to build full product skills

• Recommendations for engineers who prefer to be tech-oriented instead of product-oriented

• What else does PostHog do uniquely?

Product for Engineers

Helping engineers and founders flex their product muscles

By Andy Vandervell

What’s the main difference between “product engineers” and “stack engineers”?

Neil Kakkar, a product engineer at PostHog, writes: 

“Probably the biggest shift becoming a product engineer was with thinking from the eyes of the user. What’s technically sound takes a back seat to what the user wants.

When I was a stack engineer, I’d approach problems from the lens of “what’s technically possible?” and choose a compromise that somewhat works, negotiating with the product manager.

Now, it’s more like: “Ok, what we ideally want is technically impossible. But we still need the user to have a good experience, so maybe we can relax some other constraints such that it sparks joy for the user, and becomes technically possible as well”.

I still don’t always do it, and sometimes fall into the default of thinking through the tech side, where my lovely team mates help prod me back on the right track.

Neil linked to an example of this from PostHog’s GitHub, which is public.

One thing I want to emphasize here is that being product-minded and code-oriented are not mutually exclusive.

In fact, they go together hand-in-hand.

What does “product engineering” look like in the real world?

Generally, product engineering is similar to how most projects are planned.

Neil wrote a detailed blog post about how he owns projects as a software engineer:

Gather Context: Understand the problem thoroughly, focusing on use cases.

Figure out a Solution: Start with ideal solutions and tease out first principles.

Build: Prioritize tasks effectively, set clear priorities, and avoid distractions.

Gather Feedback: Conduct user interviews to confirm principles and validate solutions.

Align Metrics with Feedback: Ensure that feedback aligns with actual product usage.

These steps are similar to a software project being planned in a “normal company.”

However, there are a few things that stand out to me:

Gathering context

Gathering context in a normal tech company generally means using data that’s already there or interfacing with product managers and UX teams.

In a product engineering role, gathering context means talking directly to the customer and having a more hands-on approach. This is rare! 

Most engineers in companies are silo’d away from users and interface with product managers instead. This is usually done because having clear “roles” allows for higher focus.

On the other hand, product engineers get to wear many hats, doing engineering, product, and UX work. The interface of a product manager is removed.

Instead, a product manager goes deeper into aspects of the product when necessary, supplementing surface level insights and taking over investigations that product engineers don’t have the bandwidth for.

You can also see it in real life, since PostHog makes their sprint planning for each team public.

Subscribe now

Feedback Loops

An interesting note mentioned was feedback loops.

There’s a huge focus on generating faster feedback loops, allowing engineers to make sure they’re going in the right direction. Checking for feedback often allows for micro-adjustments towards a better product.

A good question that Neil mentioned here was him asking himself “How can I verify my thinking?", actively forcing him to seek feedback.

Metrics are always user-focused

This seems obvious, but having worked in big tech, it’s not always the case.

At a startup like PostHog, it’s a necessity for metrics to be user focused. The name “product engineer” keeps this focus top of mind for all engineers.

On the other hand, results vary greatly at larger tech companies. Metrics often become another thing to game to show impact for promotions. Of course, there are many cases where metrics align with a better user experience, but this is much more common at a “product engineering” company.

For example, at Big Tech companies like Meta and Google, there are countless stories littered across the web where making a better user experience is often implicitly discouraged, leading engineers to simply work on “hard problems”, regardless of whether they bring user value or not. Engineers must often pick between doing what’s best for users or what’s best for their careers (though this is not always the case).

This is very obviously not the case for product engineering-focused companies.

I think that there is a non-engineering benefit here where working on things that have impact on the end-user leads to better outcomes for both engineers and the company. It leads to less burnout and less time wasted on company politics.

Software isn’t always the solution

Owning a project provides a holistic view from talking to customers to getting feedback.

This perspective helps in understanding the entire flow and when software might not be the right solution.

When the product comes first, this is more encouraged. Compare this to other tech companies, where volume of code and velocity of pull requests are common metrics tracked as a gauge for productivity, regardless of the real value provided.

Subscribe now

Tips for engineers to build full-product skills

• Most companies would like for engineers to be more product-minded. While it’s made waves on “engineering” social media, it’s not really that common for engineers to think about the product that deeply.

• Work alongside data analysts, implementation specialists, and product managers more. Talk to them more.

• Ask questions about the user experience. What metrics matter for the end result? What user experience is being aimed for?

• This also means really paying attention during all-hands. It’s easy to breeze through this, but it can be important to try to come out of an all-hands or earning call with questions.

  • Pretend you’re leading your company and/or org.What would you focus on and what moves would you make for both the company and the org you’re in?

  • If your predictions align with what direction the org is going in, great!

  • But if they don’t. Why don’t they?

  • It’s actually quite common that they don’t. When visions differ enough and somebody feels strong enough to feel frustrated by leadership’s vision (or lack thereof), startups are created. 

    • It’s very common for big tech companies to flounder through projects that took years and eventually shut down. I’ve seen it first hand! VPs and Directors will be assigned to projects that they have no business leading, and when those orgs get shut down due to failures a couple years later, it’s usually employees who suffer.

• Build more prototypes. As Neil said, making an MVP is super valuable.

• Try to get a good mix of 0-1 projects along with “improvement” projects. At companies, it’s common to work on feature improvements or changes to existing products. But, ideally, there are opportunities to work on projects that start from “ground-up.”

  • At startups, these opportunities might be easier to find. 

  • At big tech, it’s definitely harder, but it’s possible.

What if I prefer being tech-oriented instead of product-first?

Being product-minded and code-oriented are not mutually exclusive.

Great engineers are able to steer both the product and technical direction the right way at the same time!

This isn’t easy, but it’s a skill that can be developed over time.

There are many engineers who prefer being deep in the weeds of code and don’t want to talk to users or be product-facing. I think that’s okay. In fact, I’d say this is actually encouraged in some Big Techs.

At Google, for example, it’s only at L6 (Staff Engineer) that one becomes responsible for the product direction. Before that, “you can be largely successful by building good technical solutions.” (Adam Bender, Principal Software Engineer at Google) 

An easy way to work on the “tech-first” is to simply work on distributed systems and infrastructure. In these cases, your users are now other teams!

What does PostHog do uniquely?

I wanted to thank PostHog for being willing to reach out and chat about product engineering. It’s not everyday that a startup is willing to take time out to chat transparently about their processes.

Product for Engineers

Helping engineers and founders flex their product muscles

By Andy Vandervell

However, this is something that PostHog does right - transparency. I read a lot of PostHog’s articles while writing this one, and I was impressed by the level of transparency they provide. 

At the same time, I think it makes sense. It builds trust and works as a marketing tool. Furthermore, it is notoriously hard to market products to developers (versus sales, managers, etc). However, I do think PostHog’s transparent culture and open-source codebase is a mark of transparency that is to be appreciated.

They also have a key focus on product engineering and growth engineering. These are twists to the normal “stack engineers” we see - but I think simply having those titles sets the tone of the roles and company. Engineers come in knowing they are going to have to wear more hats. They aren’t here to just work on improving the backend or frontend - they are here to improve the product and grow it.

This seems like a small detail, but it’s really easy for people to lose sight of this a few months into the job.

Other great articles about product engineering

• has a great article about being product-minded.

• Turning your engineers into Product People (PostHog)

• Developer vs. Engineer

Author’s Note: These are my personal notes of reading the paper regarding Meta’s XFaaS.

Subscribe now

Meta’s XFaaS is their serverless platform that “processes trillions of function calls per day on more than 100,000 servers spread across tens of datacenter regions.”1

XFaaS is Meta’s internal version of public Function-as-a-Service (FaaS) options, such as AWS Lambda, Google Cloud Functions, and Azure Functions.

I was lucky enough to get an early look at the paper they wrote about it.

Computer scientists aren’t really the most creative at naming, as there are two papers this year introducing “XFaaS.” One is by Meta, the other by IISC Bangalore scientists.

Both are completely different systems, so if you Google “XFaaS,” just know that this paper is talking about Meta’s version of the system.

In this article, I start off with the high-level takeaways and lessons, for those who want a quick summary. (estimated read time: 3 minutes)

Then, I do a more detailed walkthrough of the paper for those who want to understand the architecture behind XFaaS. (estimated read time: 5 minutes)

Interesting Stats and Takeaways

• A key point of the paper is that hardware usage can be optimized by software to improve serverless performance.

• Meta recognizes the wastage of startup overhead for serverless functions and aims to emulate a universal worker, i.e., any worker can execute any function instantly without startup overhead.

  • Hardware costs at scale are huge, and every percentage matters. 

• XFaaS is only used for non-user-facing functions. Serverless functions have too much variable latency to be consistently used for user-facing functions. 

• The clients of XFaaS submit function calls in a highly spiky manner. The peak demand is 4.3 times higher than the off-peak demand.

  • One example of load they demonstrate has 20 million function calls submitted to XFaaS within 15 minutes.

  • Meta found that even their spiky functions had patterns, which they leveraged to make spiky functions more predictable on their workload.

How efficient is XFaaS?

XFaaS has achieved a daily average CPU utilization of 66%, much better than the industry average.

It efficiently spreads the load using time (by delaying functions) and space (by sending it to a less-loaded datacenter).

Meta is continuing to transition as many of their functions to being scheduled during off-peak hours, where the load and cost is more predictable.

Since it’s an internal cloud, Meta is able to perform a lot of unique optimizations, such as running multiple functions from different users in the same process.

Most functions execute in less than a second, but not all.

SWE Quiz - Lifetime Access to 450+ system design questions

SWE Quiz is a compilation of 450+ software engineering and system design questions covering databases, authentication, caching, etc.

They’ve been created by engineers from Google, Meta, Apple, and more.

It’s helped many of my peers (including myself) pass the “software trivia questions” during interviews and feel more confident at work.

Check out SWE Quiz

Problems Solved by XFaaS

Problem: Lengthy cold start time.

• If a container shuts down too early, the entire container must be initialized again for the next call.

• If a container shuts down too late, it sits idle, wasting valuable compute resources.

• Solution: XFaaS uses methods like just-in-time compilation to approximate that every worker can instantly execute any function.

Problem: High variance of load.

• Results in extra hardware costs due to over-provisioning or in a slower system when under-provisioned

• Solution: XFaaS postpones running delay-tolerant functions until off-peak times and distributes function calls worldwide across datacenter regions.

Problem: Overloading downstream services.

• Example: Once, a spike in calls from non-user-facing-functions caused an outage for user-facing online services.

• Solution: XFaaS employs a mechanism similar to TCP congestion control to regulate the execution of functions.

Comparisons with Public FaaS

• Public cloud FaaS confines function executions to a single datacenter region, while XFaaS can dispatch function calls globally, allowing for better load balancing.

• FaaS platforms focus on reducing latency first, overlooking hardware utilization. XFaaS focuses on hardware utilization and throughput of function calls.

Lessons for Public FaaS

XFaaS techniques that could benefit public clouds include:

• Allowing callers to specify function execution start times.

• Enabling function owners to set Service Level Objectives (SLOs) regarding completion deadlines. (Low SLOs can be delayed for a better time.)

• Permitting function owners to assign criticality levels to functions.

While public clouds might not run functions from different users in the same process like XFaaS, large cloud customers can adopt the XFaaS approach in their virtual private clouds.

A small number of Meta teams consume a significant portion of XFaaS’s capacity. Similar large customers in public clouds might also benefit from XFaaS’s strategies.

Background: Assumptions and Requirements

This is where the detailed overview of XFaaS begins. Estimated read time: 5 minutes

As mentioned before, XFaaS handles very spikey loads. At peak demand, just one function can receive 1.3 million calls per minute.

Assumptions

A key insight is that most XFaaS functions are triggered by automated workflows, which are okay with delays. Again, this allows XFaaS to balance load across time (by delaying functions) and space (by sending it to a less-loaded datacenter).

XFaaS supports runtimes for PHP, Python, Erlang, Haskell, and a generic container-based runtime for any language.

A function has several attributes that developers can set, such as function name, arguments, runtime, criticality, execution start time, execution completion deadline, resource quota, concurrency limit, and retry policy. The execution completion deadline can range from seconds to 24 hours.

Lastly, XFaaS has varying hardware capacity across regions, so load balancing must take this into account.

Types of Workloads

Meta has three types of workloads on XFaaS: queue-triggered functions, event-triggered functions (from data-change events in data warehouse and data-stream systems), and timer-triggered functions (auto-fired on a pre-set timing).

XFaaS is used for non-user-facing functions, such as async recommendation systems, logging, productivity bots, notifications, and more.

Overall architecture

Clients initiate function calls by sending a request to a submitter, which applies rate limits before passing it to the QueueLB (queue load balancer).

The QueueLB directs the function calls to a DurableQ for storage until completion.

Periodically, the scheduler moves these calls to its FuncBuffer (in-memory function buffer), sorting them by criticality, deadline, and quota.

Ready-to-execute calls move to the RunQ, and the WorkerLB (worker load balancer) then dispatches them to suitable workers.

The submitter, QueueLB, scheduler, and WorkerLB are all stateless, unsharded, and replicated without a designated leader, so their replicas play equal roles.

DurableQ is stateful and has a sharded, highly available database.

Scaling a stateless component is easy by just adding more replicas. If a stateless component fails, it’s work can just be taken over by a peer.

Central Controllers and Fault Tolerance

Central controllers are separate from the function execution path. They continuously optimize the system by updating key configuration parameters, which are consumed by critical-path components like workers and schedulers.

 These configs are cached in the components themselves, so if the central controllers fail, the system keeps running just fine (but it cannot be reconfigured).

This allows for central controller downtime for tens of minutes. It’s an example of how Meta has built resilience into this system.

Data Isolation

Functions that require strong isolation for security or performance are assigned to different namespaces, each using distinct worker pools to achieve physical isolation.

Multiple functions can operate in a single Linux process due to trust within a private cloud, mandatory peer reviews, and existing security measures. Data can only flow from lower to higher classification levels.

Submitter

The client sends calls to the submitter. The submitter improves efficiency by batching calls and writing them to DurableQ as one operation.

The submitter manages large argument storage through a distributed key-value store and has built-in rate-limiting policies. To manage the variation in client submission rates, regions have two submitter sets for regular and high-frequency clients. 

XFaaS actively monitors high-frequency clients, and human intervention is required for throttling or Service Level Objective (SLO) adjustments.

QueueLB and DurableQ

The submitter then sends function calls to the QueueLB. The Configerator (Configuration Management System central controller) provides routing policies to the QueueLB to balance loads across different regions, as DurableQ hardware capacities vary. 

Calls to DurableQs are sharded using UUIDs for equal distribution. Each DurableQ categorizes and stores function calls by their scheduled execution time, set by the caller. 

The scheduler continuously queries DurableQs for stored function calls that are due. When a function call is handed over to a scheduler by a DurableQ, it is exclusive to that scheduler unless there's a failure in execution.

The scheduler communicates with the DurableQ:

• Sends an ACK message for successful execution. Then the function call is permanently deleted from DurableQ.

• NACK for unsuccessful attempts. The function call is made available again in the DurableQ for another scheduler to process.

Scheduler

A scheduler's primary role is to prioritize function calls based on their importance, deadline, and capacity quota.

It fetches function calls from DurableQs, consolidates them into FuncBuffers (ordered by importance, then execution deadline), and then arranges them in a RunQ for execution. Both FuncBuffers and RunQ are in-memory data structures.

To manage efficient execution and prevent system lags, the RunQ controls the flow of function calls. The system also ensures load balancing, with the Global Traffic Conductor (central controller) directing function call traffic across regions based on demand and capacity, optimizing worker utilization across the board.

WorkerLB and Workers

The RunQ from the scheduler sends functions to the WorkerLB (worker load balancer), which has functions run by workers in a worker pool.

In the XFaaS system, functions that use the same programming language have isolation, with a dedicated runtime and worker pool.

The system's design aims to allow immediate function execution by any worker without any initial delays. XFaaS maintains an always-active runtime and keeps functions’ code updated on local SSDs.

Cooperative Just-In-Time (JIT) Compilation to Reduce Overhead

XFaaS introduces cooperative JIT compilation, periodically bundling and pushing function code to workers. Since code is frequently updated, this is necessary.

There are three execution phases to JIT compilation:

• A few workers test the new code.

• 2% of workers further test the code; some perform profiling for JIT compilation.

• JIT compilation is done right before receiving function calls, eliminating delays.

With cooperative JIT compilation, a worker reached its maximum requests-per-second in only 3 minutes after a function code update, compared to 21 minutes without JIT compilation.

Locality Groups for Efficient Memory Consumption

Memory constraints make storing every function's JIT code infeasible.

The Locality Optimizer (central controller) partitions functions and workers into groups, ensuring memory-hungry functions are distributed.

Using locality groups reduced memory consumption by 11-12% and allowed for efficient, consistent memory utilization by workers.

How XFaaS Efficiently Handles Load Spikes

• Quota for Functions: Every function has a quota, set by its owner, which defines its CPU cycles per second. This quota is transformed into a requests-per-second (RPS) rate limit. The Central Rate Limiter determines whether to throttle function calls based on the function's RPS limit.

• Time-Shifting Computing: XFaaS provides reserved (for prompt execution) and opportunistic quotas (for execution within 24 hours during low-demand periods). The Utilization Controller dynamically adjusts the rate of function invocations based on worker utilization.

• Dynamic RPS Limits for Functions: To prevent overburdening downstream services, XFaaS utilizes a TCP-like AIMD (Additive Increase, Multiplicative Decrease) method to adjust the RPS limits dynamically. It can set concurrency levels and manage RPS shifts using a slow start approach.

Past challenges, like an XFaaS function overloading the TAO database causing a service cascade failure, highlighted the need for these safeguards. Now, XFaaS has proven its adeptness in shielding downstream services, as seen in real-world incidents where its back-pressure mechanism preemptively curbed potential overloads, ensuring service stability.

Subscribe now

My Opinion

This is a fantastic paper. Meta gives us a deep, honest look into their own serverless platform while providing clear lessons for developers and companies who want to optimize their own serverless function usage. The paper can be found here, though you might need institutional access to read it for free.

Furthermore, hardware optimization using software, such as efficient CPU usage, isn’t focused on enough in the industry. It’s not talked about much compared to software optimizations.

In the meantime, if you’re interested in other real-world distributed systems, I also wrote about how Meta handled billions of requests a second by creating the largest Memcached system in the world. Read it here.

If you have any questions about any parts of the paper or system, please do leave a comment below. I will answer them.

Subscribe now

If you’re a manager who wants to burnout your best engineers and destroy their faith in your leadership skills, I can help you.

I’ve watched as smart engineers around me have burnt out before in both Big Tech and at a small startup. These teams follow the same “burnout playbook,” step-by-step.

Step 1: Don’t trust your engineers

The first step is to micromanage your engineers. Sure, they’re smart, but do they truly understand what you want the product to look like? Probably not.

When I was leading engineering at the startup, I worked alongside the CEO. He would call me daily, for hours, nitpicking at little implementation details that had no effect on the actual user experience. Are you sure we should use DynamoDB? Why does this Lambda function use Python instead of Node? 

It was exhausting to get pinged so much.

At my team in Big Tech, I also was given a limited amount of autonomy. Not necessarily because of my manager, but because of the system.

I remember working on a design for a system I was building.

I was subtly told: “We need to implement this project in this way. I know it will take a month longer, but I need it in this way [so that it looks better on my promo packet].

Subscribe now

Step 2: Introduce unnecessary, time-wasting processes.

When I worked at the startup, I was suddenly forced one day to start writing out huge design docs before I ever implemented anything. Every single little API endpoint created had to be discussed extensively beforehand. Suddenly, shipping a feature went from a few days to a few weeks. It was frustrating.

The worst part? We hadn’t even shipped our product yet! There was no point of all these processes before we even had an inkling of revenue.

Look, Google has a culture of many, many design docs and one-pagers. But, at a startup, you are not Google. Google needs a writing culture because they are massive and people are joining and leaving all the time. 

This can go overboard. At a certain team in Big Tech, the amount of processes often took too long. To access certain types of data, I had to file a request that was manually approved by another team, leaving me blocked for days. To launch my feature, I had to get approval from security, product, engineering, legal, compliance, and the CEO’s dog.

Some days, all I would do is edit documents, send email, read documentation, and reply to people. I did things that had 0 impact, and that’s not because I wanted to.

Of course, these processes are needed… but there needs to be a balance. Not every project an engineer works on has to go through these processes, and they shouldn’t. Luckily, not every team at Big Tech is like this. In fact, most aren’t.

Leave a comment

Step 3: Don’t ship code to customers.

There’s nothing worse than working on something for 8 months that nobody will ever see. Especially if those 8 months were full of long hours.

Even worse: those 8 months gets pushed to 12, 16, 20. And then, the product gets cut and what you worked on is never shipped to anybody to use.

Working on projects that never ship is, anecdotally, one of the largest causes of burnout.

At the startup I worked at, we toiled away at the product for over a year without ever shipping to a customer once. Each time we would think the product was ready to ship, our CEO would ask for “just a few more features.” After a while, the team lost faith in the CEO’s ability to execute. It was hard to care deeply about something where the only feedback we got was from our CEO’s ideas and not actual users.

Products should be bug-free and have a nice user experience when they ship. But they don’t have to be perfect. If he kept inching towards perfection, then we were in for a very bad time. Especially since we hadn’t reached product-market fit yet.

Perfectionism is often an excuse for procrastination.

Looking back, I think our CEO was just procrastinating actually selling the product and receiving critical feedback.

At the Big Tech team I was in, we had constant re-orgs as Directors and VPs came and went. Each re-org came with a new executive, who had a new vision for the organization. This meant requirements changed constantly, many projects we worked on for months or years were scrapped, and we never got to see the real-world impact of our projects.

Not shipping to customers happens because of a lack of focus. When there’s no strong vision leading a product, that product is doomed to fail.

Leave a comment

Bonus step: Overpromise and underdeliver

This step is a culmination of all the above ones. The nail in the coffin, if you will.

Most engineers on these teams were over-promised benefits that never came.

At my startup, during the year of working on a product that never shipped, the CEO spent time fundraising. While I initially had a significant equity stake in the company, fundraising before product-market fit meant my stake was getting diluted. A lot. Suddenly, the financial incentive of the startup disappeared for me - I would make just as much working at Big Tech over the span of a decade even if the startup exited for $1B. Except Big Tech compensation was fully liquid and guaranteed, and a $1B exit is increasingly rare in this environment.

At Big Tech, every engineer that left had joined because of the following reasons:

• Product vision: they were passionate about the product and the problems it solved

• Scope/visibility: they felt like they would have opportunities for more scope, and therefore, more growth

• Better promotion chances: more scope and visibility → promotion

Unfortunately, every year, each engineer lost a little bit more hope in their goals being achieved.

Hopelessness accelerates burnout.

The reason they had joined simply wasn’t happening. When their hope dipped below a certain threshold, they left.

And eventually, burnout caught up to me too. So I left.

Subscribe now

Key Lessons on Burnout

Burnout doesn’t necessarily come from hours worked.

Today, I work more than I did on the teams where I burnt out. Yet, I feel amazing. I’m having fun, growing, and I visibly see the impact my work is having on users.

I want to note that burnout CAN come from working too much. Remote work has enabled more opportunities for burnout due to the reduced separation of work and life.

It’s easy to feel as if you’re on “permanent on-call.”

Burnout happens when my work doesn’t matter.

If I’m putting all this time and effort into something, I want some sort of outcome from it. 

There is a sense of accomplishment in being given autonomy over a project and shipping it.

I’d rather have my project fail and be a bust over not shipping at all!

I need to be attached to the mission. 

These experiences made me realize that I need to be attached to the mission of the team. If I don’t care about the product area or what the team can teach me, then that’s a recipe for burnout.

This is different for everyone. While my satisfaction comes from the mission of my work, for others, it may come from working with other people, getting a paycheck that supports their family, or growing their skills.

Burnout is more common than I thought.

I’ve read some other great articles about burnout.

My friend wrote some fantastic articles about burnout that are worth reading.

Musings Of A Caring Techie

10 Anti-burnout strategies for success in an early-stage startup

Anybody who joins an early-stage startup is out of their mind. That’s what I used to believe. I mean, how can they not be? The intense pressure and uncertainty of early-stage companies can easily lead to burnout and derail even the most driven individuals. And burnout sucks. I learned it the hard way (I talk about my burnout experience at length…

Read more

3 years ago · 14 likes · 7 comments · Irina Stanescu

Musings Of A Caring Techie

Burnout in Tech - Part 1: Declaring War

People working in Tech are privileged. We don’t deal with occupational hazards; we are paid well, have access to wonderful perks; and most importantly: we get to build things with talented like-minded individuals. The appeal is obvious! Behind the scenes, however, sometimes things include long hours, performance reviews graded on a curve, very tight dea…

Read more

5 years ago · 22 likes · 1 comment · Irina Stanescu

Plus a nice little comic I found:

Subscribe now

Airbnb started as simple Ruby on Rails monolith in 2008.

In 2018, Airbnb began its migration to a service-oriented architecture, as the Ruby on Rails “monorail” started becoming hard to maintain and was a single point of failure.

The main difference between SOA and microservices has to do with the architecture scope.

In an SOA model, services or modules are shared and reused enterprise-wide, whereas a microservice architecture is built on individual services that function independently. (Source)

While Airbnb didn’t necessarily need to move to a SOA, they chose to as it made sense for their organizational needs.

In a recent 2023 talk, they outlined four lessons:

• Invest in shared infrastructure early

• Simplify service dependencies

• Centralize data hydration (fetching and transformation)

• Separate UI logic from backend logic

In this article, I summarize the talk and make architecture comparisons with other large tech companies, like Meta, Google, and Uber.

Subscribe now

Invest in shared infrastructure early

Airbnb built common infrastructure shared by teams across services.

1. In-house API framework built on Apache Thrift

   1. Allowed all Airbnb services to define clean APIs and communicate with each other.

   2. The framework handled inter-service communication, error propagation, observability metrics, and schema validation.

   3. Engineers could focus on implementing core business logic.

   4. Facebook/Meta uses a customized, internal version of Apache Thrift called fbthrift, which they have open-sourced.

   5. Google, Netflix, Square, and other companies use gRPC.

2. Powergrid: An in-house library for running tasks in parallel

   1. Helped organize code into a directed acyclic graph (DAG), enabling tasks to run concurrently.

   2. Useful for tasks like validation and data aggregation.

3. OneTouch: A framework built on Kubernetes for managing services and deploying them efficiently.

   1. Ensured that all service configurations were managed in one place.

   2. Provided a command-line tool for deploying services to different environments.

   3. Google uses Borg as a cluster manager. It was a predecessor to Kubernetes. Google and Google Cloud run on Borg.

   4. Facebook/Meta uses Twine for cluster management.

4. Spinnaker: An open-source continuous delivery platform that facilitated safe and repeatable workflows for deploying changes to production.

   1. Featured automated canary analysis to ensure smooth deployments at scale.

   2. Automated canary analysis lets you partially roll out a change then evaluate it against the current deployment to assess its performance.

   3. Spinnaker was created at Netflix.

   4. Uber has a fantastic talk about safe and fast deploys at planet scale.

By investing in this common infrastructure, Airbnb was able to accelerate its migration to SOA and reap the benefits of higher reliability and business agility.

Subscribe now

Simplify service dependencies

As Airbnb continued to scale its architecture, the team separated services into layers based on technical priorities. This approach helped reduce complexity and ensure that stable services did not depend on more volatile ones.

A higher-tier service could call a lower-tier service, but not vice versa. To enforce this topology-driven layer architecture, Airbnb introduced service blocks at the platform layer, each containing related business functionalities.

These blocks encapsulated both data and business logic, offering a simple and consistent API to upstream clients. This approach simplified service dependencies, making it easier for engineers to discover and leverage core functionality.

Subscribe now

Centralize data hydration (fetching and transformation)

Data hydration is the process of importing data into an object.

It’s a common task in presentation services, involving fetching and merging data from multiple downstream services.

To streamline this process and avoid duplication (which is expensive at scale!), Airbnb introduced a centralized data access layer. This layer provided a consolidated GraphQL schema that stitched together different entities, such as listings, users, and reservations, from various data sources.

By centralizing data fetching and hydration logic, engineers could focus on product innovation rather than writing the same data hydration code repeatedly.

The data access layer also introduced a declarative approach to data retrieval, enabling engineers to write queries based on the data they needed, reducing potential errors and improving efficiency.

Subscribe now

Separate UI logic from backend logic

To ensure a consistent user experience and streamline development, Airbnb introduced the App Framework, a unified, service-driven UI system. This framework allowed Airbnb to present a standardized API to clients and stopped engineers from constantly re-inventing the wheel with their own custom APIs.

The backend drove the content and styling of the UI within each section of a rendered frontend page. The frontend was mostly responsible for rendering each section.

The team broke down the UI into standardized sections, each with its data model and UI component type. This approach provided clear patterns for reusability and customization, allowing product teams to launch new features across clients without mobile app versioning or frontend deploys.

UI data was sent in from the backend.

Standardizing the client-facing API is a common move for companies as they scale. DoorDash had a great post about their experiences with separation too.

Subscribe now

Takeaways

• Undoing work is normal. Throughout the entire process, Airbnb usually had to undo earlier work. While it can be painful to delete code, it’s important to stay detached from it.

• Splitting out services allow teams to move at different speeds. At Airbnb, frontend teams commonly moved faster than other areas. The above move allowed each team to move forward at their own pace rather than be held back or pulled forward by another team.

• Make the move only when you have to. Airbnb operated fine on a single Rails monolith for a decade! They only made the migration to a SOA after it was getting really painful to continue development on the monolith.

• Microservices are more for scaling organizations than applications.

Share Engineer’s Codex