何謂資料湖

Data lakes are larger data repositories than data warehouses, which provide the greatest ease and largest capacity for storing nearly any type of data format. Data lakes are often the first repository in a data stack, receiving the influx of all raw, semi-structured, and structured data that applications and infrastructure produce by a company, and acting as the organization's central data repository.

Because the speed and volume of data growth have only been accelerating in our digital world and are expected to accelerate more as IoT connects more devices than ever to the Internet, data lakes were created to solve the massive job of rapidly ingesting and storing a diverse collection of Big Data sets.

Functionally, data lakes operate by storing data differently than other repositories, foregoing the added step of data analysis that data warehouses perform. Because data lakes do not perform data analysis (not true in emerging cases, as newer technology is available that enables data lakes advanced analytics features), they do not bother with structuring data before storing, rather they simply store the data in its native format, speeding up ingestion.

Data lakes use flat architectures and Object Storage rather than hierarchical file systems found in data warehouses. Object Storage tags data with metatags and unique identifiers making it possible to easily retrieve data later. This is considered a schema-read principle, where data is stored with no pre-defined data schema. Data lakes can then be used by data warehouse analytics systems, dipping into the lake and pulling out the desired data that is then parsed and adapted to a data schema and moved to the data warehouse, analyzed, refined, and combined with other data sets.

Data warehouses, unlike data lakes, are considered scheme-write systems, meaning that when data is stored in a data warehouse, it is fitted into a predefined data scheme which helps in cataloging and organizing. This process alludes to the fact that data warehouses are designed to carefully prepare data before storage so that analysis can quickly follow.

Though data warehouses cannot store the same volume as data lakes, to try would be exceptionally cost-prohibitive, they are helpful in processing immediate, critical data metrics helpful to real-time business operations. Oftentimes, enterprises use data lakes as a base in their data stack, connecting it to data warehouses, or other AI and machine learning analytics through their data pipeline.

Data lakes are broader data repository systems with data ingestion as a primary concern over data analysis. Though analytics is developing around data lakes, data lakes are highly inclusive, accepting all data types, supporting all users, and easy to adapt. Because of these characteristics, data lakes potentially hold the deepest business insights. The challenge in drawing out those insights is defined by the very data lake characteristics that enable deep insights, so much data and the breadth of diversity requires time to process and analyze.

In contrast, data warehouses standardize data formats at ingestion so that insights can be quickly delivered about domain-specific channels on time, such as marketing insights, or account billings. Conceptually, data warehouses represent an increase in data refinement at the sacrifice of data scope over data lakes.

Data lakes have the potential of becoming a fundamentally critical piece of many enterprises’ IT makeup. Despite the advantages that drive companies to use data lakes, they are still emerging as a technology and therefore have challenges yet to be overcome. Most of these challenges stem from the fact that data resides in a single morass of data types and sets that muddy reliability, performance, security, and data governance. This is referred to as a data swamp, and results from:

• Reliability and Visibility Issues — Because data lakes are first repositories with little or no content oversight, the data needs to be made usable. Data lakes are heterogeneous, and without proper tools difficu lt to categorize. Depending on the setup, syncing the main repository to local data sources can result in inconsistent data.
• Slow Performance — Data lakes intend to grow, and fast. As that happens, system performance will decrease. Data duplication will occur as more analysis is applied to the lake, replicating data while searching for insights. Partitions and metadata management help establish “road signs” that can help in data management.
• Data Security — The data swamp is a difficult terrain to secure. Data lake architecture inherently lacks the fine-grained access controls that are found in other enterprise systems because using Object Storage does not allow clear data segmentation. One file object may contain huge amounts of unstructured or raw data, giving access to a single file object could expose sensitive data to unsanctioned users. Approaches to secure data lakes have tried built-in IAM controls, which are difficult to implement, or partitioning certain data into staging areas within the data lake and providing access to those staging areas, and using high-level data tools with access controls and analytics.
• Governance Compliance — The latest challenge to implementing data lakes is data governance as laid out in the EU’s General Data Protection Regulation (GDPR). And while there is no broad law like GDPR in the United States, there are several that regulate data within specific industries like healthcare, and state-level data protection laws, see California Consumer Privacy Act (CCPA). The trend means greater data regulations, and in short, data lakes are poor repositories for sensitive data like personally identifiable information PII, or protected health information (PHI). Rather, these laws may require sensitive data to be stored on-premises, separate from the data lake.

Data is ubiquitous, and how we choose to use it makes it valuable or simply cluttered. The main use case of data lakes is to rapidly ingest and store real-time streaming data flow and batch processing data, in any format, and then secondarily perform analytics on sets of diverse data. To that end, large multinationals, manufacturing, municipalities, and other companies have leveraged data lakes for many businesses uses:

• Advanced Analytics Support
• Application Support
• Archival and Historical Data Storage
• Augment Data Warehouse Storage
• Business Analysis
• Distributed Processing
• Experimental Analysis
• Lambda Architecture Support
• Preparation for Data Warehousing