数据发现

Data discovery is the process of identifying data sources within an organization's changing data environment. By leveraging automation, and often cloud-based systems to support data environments, data discovery becomes a foundational aspect of agile businesses. A data discovery platform makes business operations transparent, and sets up future success by creating a hub for other data innovations.

Data classification is a step following data discovery where data is identified and categorized by type using pattern and keyword rules that apply labels to identified data. In one instance of this, In the health industry, medical ID patterns are used to find patterns of categorization.

Technically, data discovery is the process of consolidating raw data from multiple sources, of which each may be fundamentally different, like combining structured and unstructured data. Because of significant volumes of data, organizations rely on smart data discovery tools to digest operational information and visualize it. Popular data visualization includes graphs, charts, tables, maps, infographics, dashboards, etc.

Generally, there is a 5-step data process that intakes raw data and produces valuable insight.

• Connect And Blend Data Sources — Enumerating all data sources is the starting point. This includes listing and understanding necessary measurements and metrics to be collected. Typically, these data are ingested and stored within a data warehouse—a place where it is possible for disparate data types to merge. This stage is where discovery occurs.
• Clean And Prepare Raw Data — Raw data is typically unreadable, and processing prepares data for analytical use. Raw data in this stage is cleaned, standardized or normalized, in order to detect and remove data errors, distortions, and corruptions. Also data is put in alignment, such as using correct units of measure.
• Data Sharing — Data sharing among data stakeholders is a key benefit of data discovery. At this level, analysis has not yet occurred, and data can be categorized into informational domains which benefit different levels in the organization. Sometimes this comes in the form of a data mart, something like a data warehouse except for a specific singular data domain. Teams can then access the data most relevant to their day-to-day, and planning.
• Analyze And Develop Business Insights — At this point, management teams and data scientists can access and analyze data sets relevant to their needs. To analyze, teams deploy data discovery tools that can perform distributional analysis, and predictive analytics or market basket analysis. Depending on the need, custom analysis is often performed.
• Visualize Insights — What typically comes hand in hand with data analysis is data visualizations, which supports the main aim of data discovery, to deliver insights as clearly and quickly as possible. From this point, teams can develop action plans and react to these insights with confidence.

Smart data discovery is a popular term for AI and machine learning advancements in data discovery. Before machines could perform data discovery, these tasks were conducted manually by data stewards. AI functions within the data discovery domain reached a tipping point and Gartner identified a new category of business intelligence software capable of dramatically organizing company data, and discovering sensitive data that can now be secured and made compliant to regulations.

Gartner defines smart data discovery: “Automatically finds, visualizes and narrates important findings such as correlations, exceptions, clusters, links and predictions in data that are relevant to users without requiring them to build models or write algorithms. Users explore data via visualizations, natural-language-generated narration, search and natural-language query technologies.”