This is welcomed by many organisations where SQL is often the ubiquitous language of the data analyst, and legacy transforms already written in SQL and can be reused.

In the lake house model Redshift becomes the transformation engine of choice. It also means that effort is not wasted transforming data feeds that will never be used. This enables you to onboard new data fast, without the constraint of teams building ETL pipelines. Data is loaded into S3 in the raw format, and typically only when requested by the customer is data transformed into a derived format for further use. What does the lake house concept mean for my AWS data warehouse?īy virtue of its architecture, the lake house follows an ‘ELT’ paradigm (extract, load, transform). Amazon predict that 97% of customers will get the benefits of concurrency scaling without any additional charges. When enabled, the cluster will accumulate one hour of ‘concurrency scaling credits’ for every 24 hours that it runs. This means the Redshift cluster can be sized for average workloads and not the high-water mark of peak requests. Amazon are advertising a two-times performance improvement on the previous generation Redshift instances.Īs of March 2019, Redshift can handle peaks in workload by automatically scaling additional transient capacity to process the concurrent queries and maintain consistent performance. These instances are built on the new AWS Nitro controllers, delivering near bare metal performance, and supporting high bandwidth network connectivity. The new RA3 instance type allows you to scale Redshift data storage and compute needs independently, meaning you pay the lowest possible price for managing your combination of data and workload. Spectrum is also fully managed and serverless, charging only for the resources consumed on the query. This is achieved by distributing the query across many spectrum nodes to achieve highly performant MPP (massively parallel processing) that scales to petabytes. It allows standard SQL queries to be submitted against data stored in S3 via the Redshift cluster. The parquet format speeds up offload by two-times and consumes up to six-times less storage when compared to the same data in text format.Īlthough not new this year, Redshift Spectrum is a key part of the lake house architecture. Previously the offload was only possible to text files.

This unlocks the use of Redshift as a transient SQL transform engine, with the ability to offload the results back to S3 in an analytics optimised format for consumption by SageMaker (machine learning), Athena (ad-hoc querying) and EMR/Glue (ETL). READ NEXT: AWS re:Invent 2019: Andy Jassy's keynote speechĪndy Jassy announced in his keynote that Redshift supports the offloading of data to S3 in parquet format with partitioning based on column data supported. The above has only really become possible with the latest set of Re:Invent announcements.

Performant SQL querying regardless of concurrency.

Independently scalable compute, capable of massively paralleled processing (MPP).

In reality, this means allowing S3 and Redshift to interact and share data in such a way that you expose the advantages of each product.Īn AWS lake house allows you to achieve the holy grail of: Data Lake + Data Warehouse = Lake HouseĪ new pattern is emerging from those running data warehouse and data lake operations in AWS, coined the ‘lake house’. Here we explore the latest set of AWS announcements to see how a combination of the cloud and data can help solve organisations' biggest data challenges. Indeed, many organisations have already made the jump to cloud-native data solutions, albeit with varying levels of success. The promise of both elastic and scalable compute & storage ought to make this a much easier problem to solve. These products are not only eye-wateringly expensive, but do not measure up in the petabyte scale world we now live in.Įnter the cloud. In the past, they have typically relied on expensive and proprietary on-premise data warehouse solutions to store and analyse their data.

Indeed, 30 of the 77 announcements were data, analytics or machine learning related.ĭata continues to be a focus area for all large organisations, many of whom have struggled to scale their operations in line with the growth of data. Read next: Managing data in the lakehouse: An introduction to the data lakehouseĪt this year’s Re:Invent, there was a considerable focus on data. We investigate some of the recent announcements from Re:Invent and how they are influencing the architecture patterns of data warehouses in the cloud.

Lakehouse architecture