Figure 3: Deployed System
Figure 3 shows system components of OpenHouse deployed at LinkedIn. Each component is numbered and its purpose is as follows:
-
Table service: This is a RESTful web service that exposes tables REST resources. This service is deployed on a Kubernetes cluster with a fronting Envoy Network Proxy.
-
REST clients: A variety of applications use REST clients to call into table service (#1). Clients include but are not limited to compliance apps, replication apps, data discovery apps like Datahub and IaC, Terraform providers, and data quality checkers. Some of the apps that work on all the tables in OpenHouse are assigned higher privileges.
-
Metastore Catalog: Spark,Trino, andFlink engines are a special flavor of REST clients. An OpenHouse specific metastore catalog implementation allows engines to integrate with OpenHouse tables.
-
House database service: This is an internal service to store table service and data service metadata. This service exposes a key-value interface that is designed to use a NoSQL DB for scale and cost optimization. However the deployed system is currently backed by a MySQL instance, for ease of development and deployment.
-
Managed namespace: This is a managed HDFS namespace where tables are persisted in Iceberg table format. Table service is responsible for setting up the table directory structure with appropriate FileSystem permissioning. OpenHouse has a novel HDFS permissioning scheme that makes it possible for any ETL flow to publish directly to Iceberg tables and securely into a managed HDFS namespace.
-
Data services: This is a set of data services that reconciles the user / system declared configuration with the system observed configuration. This includes use cases such as retention, restatement, and Iceberg-specific maintenance. Each maintenance activity is scheduled as a Spark job per table. A Kubernetes cronjob is run periodically on a schedule to trigger a maintenance activity. All the bookkeeping of jobs is done in House Database Service using a jobs metadata table for ease of debugging and monitoring.
Architecturally, OpenHouse is built to run in any cloud environment, using blob stores, managed compute, and cloud databases. Both the table service and data service are packaged as containers that should make it easy to deploy in a diverse environment. We are working on Terraform recipes that would automate deployment of the entire stack in minutes.
Managed Iceberg Lakehouse
At LinkedIn, OpenHouse tables are persisted on HDFS in Iceberg table format. Compared to Hive table format, Iceberg allows us to improve the reliability of tables on HDFS by providing features like incremental data processing, snapshot isolation, ACID transactions, and reproducible data flows through time travel queries.
Building a functional, scalable and easy to use lakehouse architecture with Iceberg as the table format required us to make new foundational investments. We invested in various data services that can work with Iceberg table format.
-
To keep the tables optimal, we automated orchestration of Iceberg maintenance jobs such as snapshot expiration, orphan file deletion, quarantine zones for deleted files, and manifest compaction.
-
To keep tables compliant, we have built data services that can delete data based on user requested purging and time partition expiration.
-
To provide data disaster recovery, we have built a data service that can replicate Iceberg snapshots efficiently across data centers.
Finally all our data services can be triggered almost instantaneously as Iceberg snapshots are committed.
Impact
In LinkedIn’s data lakes, two distinct categories of tables have emerged: centrally managed tables and self-managed tables. Centrally managed tables offer public sharing capabilities and robust table management support, including compaction and replication. On the other hand, self-managed tables are private to end-users and lack consistent management practices. Surprisingly, 65% of tables fall under the self-managed category, indicating a need for a more streamlined approach.
Our central managed platform imposes a laborious onboarding process, burdened by human intervention, resulting in significant time investment. It takes 2 to 3 weeks to onboard tables, and the ingestion is eventually consistent, creating operational complexities for both Site Reliability Engineers (SREs) and end-users.
With OpenHouse, end-users can self-serve creation of centrally managed, publicly shareable, and compliant tables in seconds. By eliminating the friction and operational complexities of traditional onboarding processes, OpenHouse empowers end-users to collaborate effectively while ensuring granular table sharing and adherence to compliance requirements, thereby transforming the way data lakes are operated.
Roadmap
OpenHouse has been deployed since late 2022 and serves a portion of our production traffic from LinkedIn’s GoToMarket systems that support LinkedIn Sales and Marketing. Our data engineers and data scientists who use dBT to create ETL flows were among the first to utilize this new system. Over the coming quarters, we will ramp production to serve the entirety of LinkedIn’s data lakehouse tables. We expect to share more details in future posts as well as any further plans to open source this technology early next year.
Acknowledgements
Big thanks to team members who have relentlessly shipped incremental milestones and delivered customer impact for this multi-year initiative: Lei Sun (founding engineer), Sushant Raikar, Stanislav Pak, Abhishek Nath, Malini Venkatachari, Rohit Kumar, Levi Jiang, Manisha Kamal, Swathi Koundinya, Vishal Saxena, and Naveen Selvaraj.
Over a year and half ago, OpenHouse was incubated under the leadership of Sumitha Poornachandran and she remains our unwavering pillar of support. Also, huge thanks to continuous support to our executive leadership Renu Tewari, Kartik Paramasivam, and Raghu Hiremagalur for believing in OpenHouse.
Many thanks to the thought leadership of Eric Baldeschwieler, Owen O’ Malley, Sriram Rao, Vasanth Rajamani, and Kapil Surlaker, who helped shape the product value proposition. Also we are grateful to peer reviewers for the blog, Erik Krogen, Daniel Meredith, and Diego Buthay.
Finally, OpenHouse is a product of many passionate discussions with leads across LinkedIn: Walaa Eldin Moustafa, Bhupendra Jain, Ratandeep Ratti, Kip Kohn, Issac Buenros and Maneesh Varshney.
