什么是数据虚拟化？

Data virtualization is an approach to data analysis that overcomes the challenges of drawing on data stores in various physical locations by creating a virtualized logical data layer that can integrate data sources of multiple types from many global sources without the need to draw in and manipulate data into an additional data store as in data warehouses. This approach beneficially eliminates the need for error prone data replication or data migrations that can lead to data corruption. For the end data users, data is presented in a single unified view, often with advanced visualizations.

Data virtualization was born as a solution to the challenges of data lakes. When data lakes began to pool up as a result of massive data producing technologies like social media, IoT enabled devices, and mobile devices, organizations began searching for a way to integrate these data sources, often in unstructured formats, with their traditional structured business data.

Data warehouses take unstructured data lake data and through a data extraction, transformation, and loading (ETL) process structure it to be analyzed later. It's a process that requires extra data replication and storage. This approach has a high potential for data inconsistencies as numerous ETL processes are necessary. Also repositories must be kept in sync. Data replication, further, raises data security and governance concerns that impact sensitive data, namely where to store users' sensitive data. The solution to these challenges is data virtualization with features that allow data customers to view, access, and perform analytics on disparate data sources regardless of location without having to move that data.

Two factors are pressuring enterprise data strategies, the massive volume of data generated by devices and systems used in day-to-day operations, and the incapability of on-premise infrastructure to cost-effectively keep pace with those demands. While this has technologically cornered some organizations who have stalled in the efforts to migrate to the cloud, others have learned of the benefits of data virtualization to corral disparate data sources and incorporate them into a unified real-time data view.

The main features of data virtualization middleware are:

• The ability to abstract data using a virtualized layer.
• To integrate unstructured and structured data from disparate sources.
• To facilitate the retrieval and manipulation of that data.

Deploying a data virtualization architecture awards several advantages to organizations.

• Real-time Access — Data virtualizations provide real-time data access to disparate data sources, geographically separate.
• Reduced Errors / Increased Data Accuracy — AI and automation as well as the elimination of data replication improves data accuracy.
• Less Resource Consumption / Cost Savings — Data virtualization reduces the steps traditional data warehouses must go through, i.e. ETL, directly reducing costs.
• Less Disk Requirements — Because data remains at the source, disk requirements are not duplicated to store processed data.
• Enforce Data Governance — DV tools can integrate with data governance tools and create a stable endpoint for sensitive data stores to ensure data governance remains intact.
• Logical Abstraction and Decoupling of Data — Data virtualization allows for heterogeneous data sources to interact.
• Incorporates Structured and Unstructured Data — Data virtualizations bridge semantic gaps between different types of structured and unstructured data.
• Share Data Easily / Eliminate Data Silos — Data virtualization can overcome departmental data silos

Data warehouses and data lakes are two elements within a larger data storage ecosystem. While they both have massive storage features, each stores data differently which gives each their advantages. Data lakes store data in unstructured, native, raw formats. Data warehouses hold their data in schema structured formats which requires first processing to align data to the schema.

For data lakes, the purpose is to be optimized to store data in a manner that allows it to pile up. Because data today is being produced in such volumes as to require data analysis when resources allow or on-demand when cases arise, data must be allowed to easily accumulate. In this way data is never disposed of, just saved until needed. Data lakes use a system called object storage, which treats huge tracts of unstructured data as individual objects rather than file systems which treat data as files. Object storage facilitates massive data storage, usually in the petabyte range.

Data warehouses on the other hand can store huge amounts of data only after it’s been fitted into a data schema that structures the data warehouse (data warehouses are more selective of their data, so they store much less data than data lakes). This data manipulation requires resources, and often presents a necessary extra step for drawing insights out of data. Because the function of the data warehouse is migrating to the cloud, being replaced with data virtualization and data federations in combination with AI and data lakes, the use of on-premise data warehouses will decline sharply in the coming years.

Data virtualization tools all aim to create a unified view for users through a logical collection of separate data sources. The following are identified by Gartner and G2 as the leaders in data virtualization tools.

• AWS Glue — A fully managed solution to unburden clients of their data virtualization administration.
• Denodo Platform — Highly touted as user friendly with a quick learning curve.
• Oracle Virtualization — A popular brand name built for a Linux Kernel-based Virtual Machine (KVM) environment.
• SAP HANA — SAP HANA promotes itself as a single In-Memory architecture platform, and is highly-rated for disaster recovery, access control, and authentication.
• TIBCO Data Virtualization — TIBCO devoured the Cisco data virtualization package through acquisition in 2017, and touts one of the widest data source connectivity capabilities.