Databricks LTAP Explained: Turning Postgres WAL into Lakehouse Storage
Databricks recently published a technical blog on Lakebase and LTAP: From monolith to Lakebase to LTAP: rethinking database storage. The post is interesting not because it introduces another acronym, but because it reframes a familiar data infrastructure problem:
Why do operational databases and analytical systems still feel like two different worlds?
Most production systems begin with an operational database: PostgreSQL, MySQL, Oracle, SQL Server, or a cloud-native variant. These systems handle user-facing writes: orders, payments, inventory updates, account changes, workflow state, application metadata.
But the moment a team wants analytics—dashboards, long-range aggregations, ML features, customer segmentation, agent context, fraud detection—the data is usually copied somewhere else: a warehouse, a data lake, or a lakehouse.
That copy is not free. It creates pipelines, lag, schema drift, validation jobs, replay logic, failure handling, and a permanent question: which system has the freshest and most correct view?
LTAP is Databricks’ attempt to move part of that boundary. Instead of treating operational data as something that must be exported into the lakehouse later, LTAP asks:
Can the storage layer of an operational database directly produce lakehouse-native columnar data?
That is the core idea. The rest of the architecture is a way to make that idea plausible.
1. The old split: OLTP wants rows, OLAP wants columns
Operational databases and analytical systems optimize for different access patterns.
An OLTP database serves short, high-concurrency transactions:
create an order
update a payment status
look up one user
decrement inventory
insert an event
fetch a workflow state record
These operations usually touch a small number of rows and must be fast, isolated, and durable.
An OLAP system serves large analytical queries:
revenue by region over 12 months
cohort retention
product-level conversion rates
feature generation for ML
historical context for AI agents
joins across large business tables
These queries often scan many rows but only a subset of columns.
That difference leads to different storage layouts.
A row-oriented layout stores one record together:
order_id | user_id | product_id | price | status | created_at
This is good when the application frequently reads or updates one full record.
A column-oriented layout stores values from the same column together:
order_id: 1, 2, 3, 4, ...
user_id: 9, 9, 5, 8, ...
price: 12.0, 25.0, 7.5, ...
status: paid, pending, paid, ...
created_at: ...
This is better for analytical scans. If a query only needs price, status, and created_at, the engine can avoid reading the rest. Apache Parquet, for example, is explicitly designed as an open-source column-oriented file format for efficient storage and retrieval in analytical workloads.1
So the basic tension is simple:
Applications prefer row-oriented transactional storage. Analytics prefers column-oriented scan storage.
Most data stacks resolve this by copying data from one world to the other.
This works, but it also means the analytics stack is downstream from the operational database. It has to catch up.
2. The key database primitive: WAL
To understand LTAP, we need one database concept: WAL, or Write-Ahead Log.
A database does not simply modify data files and hope for the best. Before it makes a durable change, it first records the change in a log.
That log is WAL.
WAL matters for several reasons:
Durability: once a transaction commits, the database can recover it even if the machine crashes.
Recovery: after restart, the database can replay WAL to reconstruct a correct state.
Replication: replicas can follow the primary by consuming the WAL stream.
Change capture: downstream systems can use WAL as a chronological stream of data changes.
For LTAP, the fourth point is the important one.
If WAL already contains the ordered history of database changes, then it is a natural place to derive other representations of the data.
Traditional replication uses WAL to maintain another database copy. CDC uses WAL to feed another system. Neon-style storage uses WAL to reconstruct Postgres pages. LTAP pushes the idea further: use WAL to produce lakehouse-native columnar data.
3. Lakebase starts by disaggregating Postgres
Lakebase builds on the architecture popularized by Neon: separate Postgres compute from the storage layer.
In a conventional Postgres deployment, the compute process, WAL, buffer cache, data pages, indexes, checkpoints, and local storage are tightly coupled around the database instance.
In a Neon-style architecture, the system is split into distinct services:
Postgres compute executes SQL and talks to applications.
Safekeepers durably replicate WAL and define the commit boundary.
Pageservers consume WAL and reconstruct Postgres data pages.
Object storage holds long-term immutable history.
Neon’s own architecture docs describe Safekeepers as the component that replicates WAL, the Pageserver as the component that turns WAL into queryable data pages, and object storage as the place for long-term immutable history.2
This is already a meaningful architectural change.
Postgres compute becomes closer to stateless execution. It can be started, stopped, scaled, or replaced more easily because the durable state is no longer just local files owned by that compute process.
But disaggregated Postgres alone is not LTAP. It is the foundation.
The LTAP step is to ask: if the Pageserver already consumes WAL and materializes database state, why should it only materialize Postgres pages?
4. The LTAP move: materialize WAL into lakehouse files
Databricks’ LTAP idea is to use the database storage layer to produce open columnar files—Parquet with table formats such as Delta or Iceberg—that analytical engines can read directly.
In the original Lakebase / Neon model, WAL is used to reconstruct row/page-oriented Postgres data.
In LTAP, WAL can also feed a columnar materialization path:
This is the architectural hinge.
LTAP is not just “Postgres plus a CDC connector.” It is an attempt to make the columnar lakehouse representation part of the database storage design.
Databricks’ blog states the stronger version of this claim: instead of two copies in two formats, there is one durable copy in open columnar formats such as Delta and Iceberg, stored as Parquet, with caches on both sides for performance.3
That claim needs careful interpretation, but it explains the ambition.
The goal is not to make Postgres run huge analytical scans. The goal is to let Postgres handle transactions while Spark / Photon / Databricks SQL read a columnar representation generated from the same underlying change stream.
5. Why this is not just CDC
It is fair to ask: isn’t this just CDC with better branding?
CDC also reads WAL and sends changes downstream. A typical CDC pipeline looks like this:
The difference is not that one uses WAL and the other does not. Both can use WAL.
The difference is where the system boundary is drawn.
CDC treats the operational database and the analytical store as separate systems. The CDC pipeline is responsible for moving changes between them. That pipeline must handle ordering, retries, schema changes, deduplication, backfill, lag monitoring, and consistency validation.
LTAP tries to absorb more of that responsibility into the storage layer of the database itself.
A simplified comparison:
That last row is the hardest one.
If the columnar files are merely an asynchronously maintained mirror, LTAP behaves like a deeply integrated zero-ETL system.
If the columnar files are truly the durable representation from which the database’s row/page structures can be reconstructed, then LTAP is closer to a new database storage architecture.
That distinction matters.
6. How can analytics read fresh data?
A lakehouse file is usually not updated row by row. Analytical systems prefer large immutable files, metadata snapshots, and compaction. Operational systems produce small, frequent changes.
So how can an analytical query see fresh data?
The useful mental model is:
Analytical snapshot at LSN N =
stable columnar files already materialized up to N + recent changes not yet compacted into those filesAn LSN is a position in the WAL stream. It identifies a point in the ordered history of database changes.
When an analytical query starts, the system can choose a database snapshot by LSN. Most data may already be available in compacted columnar files. Very recent changes may still live in the Pageserver or a delta layer and need to be merged at query time.
This is where the implementation becomes difficult.
The system must preserve transaction visibility. It must handle updates and deletes. It must avoid showing half of a transaction. It must cope with schema changes. It must keep recent deltas from growing too large. It must compact data without breaking snapshot reads.
The concept is clean. The engineering is not.
7. Why object storage is attractive—and awkward
At first glance, putting database state in object storage sounds strange. Object storage is not a low-latency database disk.
But it has properties analytical systems love:
low cost
elastic capacity
durability
separation of compute and storage
open file access
compatibility with multiple engines
natural fit for Parquet, Delta, Iceberg, and similar formats
Apache Iceberg, for example, is a table format for large analytic tables and supports capabilities such as schema evolution without rewriting the entire table.4
The awkward part is that operational writes are small and frequent, while lakehouse files want to be large and scan-friendly.
So LTAP needs intermediate structures:
WAL durability path
Pageserver state
hot caches
recent delta representation
columnar file materialization
compaction
table metadata snapshots
Object storage is not serving every OLTP request directly. It is the durable, low-cost, lakehouse-readable layer underneath a set of caches and materialized structures.
A more accurate picture is:
This is why “single copy” should be read carefully.
A practical system will still have multiple physical representations: pages, caches, deltas, indexes, columnar files, metadata, and compaction outputs. The stronger claim is about reducing the need for a separate user-managed database-to-lake pipeline, not eliminating all internal representations.
A better phrase may be:
single logical source of truth, multiple physical access structures.
8. A small comparison: similar idea, different boundary
The idea of deriving columnar data from a transactional log is not entirely new.
Some HTAP systems maintain a row-oriented write path and a column-oriented analytical representation. TiDB / TiFlash is one example: TiFlash maintains columnar replicas for analytical queries, using replication from TiKV; TiDB’s documentation describes TiFlash columnar replicas as being replicated asynchronously through Raft Learner and using MVCC / Raft index validation for snapshot isolation.5
That comparison is useful, but only up to a point.
The important difference is where the system draws the boundary.
In a database-native HTAP system, the row store, column store, optimizer, and execution layer are usually part of one database abstraction.
In Databricks LTAP, Postgres remains the transactional interface, while Spark / Photon remains the analytical interface. The unification happens lower, in the storage layer.
So the comparison should not be framed as “which one is the same.” They are different answers to a similar pressure:
How do we serve operational and analytical workloads without asking users to manage fragile data movement?
9. The hard parts
LTAP is compelling because the data flow is easy to explain. It is hard because databases are full of edge cases.
Transactional consistency
An analytical query cannot see half a transaction.
If an order write updates orders, payments, and inventory, the analytical snapshot must either include all of those changes or none of them.
This requires the lakehouse-readable representation to preserve database snapshot semantics, not merely file-level freshness.
Updates and deletes
Columnar files are excellent for scans, but operational databases update and delete individual rows all the time.
On object storage, files are usually immutable. An update often becomes some combination of new files, delete markers, metadata changes, and later compaction.
That is manageable, but it is not free.
Recent delta size
If materialization falls behind, more recent changes must be merged during query execution. That can turn a clean columnar scan into a hybrid scan-plus-delta problem.
The cost depends on write rate, query shape, lag, indexing, and how deltas are organized.
Small files and compaction
OLTP produces small changes. Lakehouse analytics prefers large, well-clustered files.
A practical LTAP system must continuously convert many small changes into fewer scan-efficient files. Compaction has to balance freshness, write amplification, object storage cost, and query performance.
Schema evolution
Operational schemas change: columns are added, renamed, removed, widened, or retyped.
Lakehouse table formats support schema evolution, but preserving Postgres semantics across schema versions is harder than simply adding a Parquet column.
PostgreSQL compatibility
Postgres is not just tables and rows. It has MVCC, TOAST, indexes, collations, extensions, custom types, DDL behavior, vacuum, isolation levels, and many subtle semantics.
Supporting common cases is one thing. Supporting arbitrary production Postgres workloads is another.
10. How to read LTAP
The most useful way to read LTAP is not as a new acronym that replaces OLTP, OLAP, or HTAP.
Read it as a storage architecture proposal:
Use the transactional log as the source of change, and materialize that change into a representation that both the operational side and the analytical side can use.
For Databricks, the analytical side is the lakehouse: object storage, open columnar formats, Spark / Photon, SQL analytics, governance, ML, and AI pipelines.
The architectural bet is that the storage layer can absorb complexity that today leaks into user-managed CDC and ETL pipelines.
That does not make the complexity disappear. It moves it.
Instead of every customer operating a fragile chain like this:
Postgres → CDC → stream → lake ingestion → table maintenance → analyticsLTAP tries to move more of the work into the database storage system:
Postgres WAL → storage-layer materialization → lakehouse-readable tablesThat is a real idea, and it is worth understanding.
The open questions are equally real:
How fresh can analytical reads be under heavy write load?
How expensive is query-time merging of recent changes?
How complete is Postgres semantic compatibility?
How does the system handle schema changes and long-running snapshots?
Are columnar files truly the durable source of truth, or a tightly integrated replica?
What operational tools exist for validation, recovery, and debugging?
Those answers will determine whether LTAP becomes a new database storage pattern or a better-packaged form of zero-ETL.
Either way, the direction is important.
The long-standing split between operational databases and analytical systems has created enormous operational overhead. LTAP is interesting because it attacks that split at a lower layer than a connector or pipeline.
The simplest summary is this:
Lakebase keeps Postgres as the transactional interface. LTAP tries to turn Postgres WAL into lakehouse-native storage. If it works, the lakehouse gets much closer to fresh operational data—not by pulling data out of the database later, but by participating in how that data is materialized in the first place.
References
Databricks, From monolith to Lakebase to LTAP: rethinking database storage, https://www.databricks.com/blog/lakebase-ltap-rethinking-database-storage
Neon Docs, Neon’s lakebase architecture, https://neon.com/docs/introduction/architecture-overview
Apache Parquet, official project page, https://parquet.apache.org/
Apache Iceberg, official project page and specification, https://iceberg.apache.org/ and https://iceberg.apache.org/spec/
PingCAP Docs, TiFlash overview, https://docs.pingcap.com/tidb/stable/tiflash-overview