This session looks at how adoption of open table formats by data warehouse database management vendors and advances in SQL are making it possible to merge siloed analytical systems into a new federated data architecture supporting multiple analytical workloads.

Target Audience: Data architect, enterprise architect, CDO, data engineer
Prerequisites: Basic understanding of data architecture & databases
Level: Advanced

Extended Abstract:
In the last 12-18 months we have seen many different architectures emerge from many different vendors who claim to be offering 'the modern data architecture solution' for the data-driven enterprise. These range from streaming data platforms to data lakes, to cloud data warehouses supporting structured, semi-structured and unstructured data, cloud data warehouses supporting external tables and federated query processing, lakehouses, data fabric, and federated query platforms offering virtual views of data and virtual data products on data in data lakes and lakehouses. In addition, all of these vendor architectures are claiming to support the building of data products in a data mesh. It's not surprising therefore, that customers are confused as to which option to choose.  

However, in 2023, key changes have emerged including much broader support for open table formats such as Apache Iceberg, Apache Hudi and Delta Lake in many other vendor data platforms. In addition, we have seen significant new milestones in extending the ISO SQL Standard to support new kinds of analytics in general purpose SQL. Also, AI has also advanced to work across any type of data. 

The key question is what does this all mean for data management? What is the impact of this on analytical data platforms and what does it mean for customers? What opportunities does this evolution open up for tools vendors whose data foundation is reliant on other vendor database management systems and data platforms? This session looks at this evolution and helps vendors realise the potential of what's now possible and how they can exploit it for competitive advantage.

• The demand for data and AI
• The need for a data foundation to underpin data and AI initiatives
• The emergence of data mesh and data products
• The challenge of a distributed data estate
• Data fabric and how can they help build data products
• Data architecture options for building data products
• The impact of open table formats and query language extensions on architecture modernisation
• Is the convergence of analytical workloads possible?