O que é DataOps? Todos os aspectos das operações de dados explicados.

Data Pipelines are a key concept in DataOps and encompass all data processing steps to rapidly deliver comprehensive and curated data to data consumers. Analogously, these are sometimes referred to as data factories to impart the feeling of tangibility to data assets and the thinking that data is a raw material to be processed.

Data engineers may architect multiple data pipelines in their DataOps designs, making pipelines between data providers and data preparers, and pipelines to serve data consumers. For example, one data pipeline may flow application data into a data warehouse, another from a data lake to an analytics platform, and further pipelines may feedback into themselves simply for processing, like sales data back into a sales platform.

DataOps has foundations stemming from many processes historically grounded in DevOps. Today three schools of thought comprise the main foundational principles of DataOps: Agile Development, DevOps, and Lean Manufacturing.

• ·Agile Development —A popular software development process, agile development methods in DataOps are founded on rapid development cycle times, and highly collaborative environments. Agile teams work in short cycles, called 'sprints', which may last just a day to weeks. Quick cycle times allow the team to reevaluate their priorities often and make course corrections to their application as necessary based on the needs of users.
  In practice, teams can hunt down bugs and roll out fixes faster (potentially within hours of discovery), as well as receive user feedback and deliver improved application features as they are imagined, instead of bundling updates and fixes to be released in a single version late to market.
  The flexibility and quickness of agile development methods provide developers an ideal framework for rapid development around growing DataOps assets.
• DevOps (Development Operations) — DevOps, a building block of DataOps, relies on the automation of repetitive tasks, like testing code, to produce an environment of continuous development and accelerate the build lifecycle. DevOps practices allow teams to easily communicate and collaborate, release fixes and updates faster and reduce time-to-resolution.
  Although DataOps borrows liberally from the ideas of DevOps, DevOps is a process amongst technical roles on development teams, whereas DataOps serves not only technical but non-technical roles, like data consumers, inside and outside the organization.
• Lean Manufacturing — Lean Manufacturing is a set of principles and methods that aim to reduce wastage while maintaining productivity in manufacturing settings. These principles have proved to be portable, and DataOps borrows methods that suit its needs well, like statistical process control (SPC), which has been applied to data factories to great effect.
  In the case of data errors, SPC allows the testing of data that flows through the DataOps pipeline to ensure its validity at every stage: at the input stage tests catch data supplier inconsistencies, tests verify data integrity throughout processing, and output tests can catch final data errors before passing data downstream to other data consumers.
  To maximize data as an enterprise asset, DataOps takes a holistic approach, focused on improving communications, integrations, and automation. Fundamentally, DataOps is founded on people, processes, and technology.

At small levels, companies can improve their DataOps processes and accuracy using specialized data tools that improve their data pipelines. The overarching mission of DataOps, however, is to achieve a full organization-wide culture change that appreciates data first and drives to maximize all data assets. The following DataOps framework elements help guide organizations in thinking holistically about their people, processes, and technology.

1. Enabling Technologies — Use technologies such as IT automation, and data management tools.
2. Adaptive Architecture — Deploy systems that allow for continuous integration (CI) and continuous deployment (CD).
3. Intelligent Metadata — Technology that automatically enriches incoming data.
4. DataOps Methodology — Game plan for deploying analytics and data pipelines, and adhering to data governance policies.
5. Culture and People — Cultivation of organizational ethos that appreciates and utilizes data and aims to maximize data assets.

A defining characteristic of DataOps is the numerous roles that interact and contribute to the accumulation, processing, and use of a company's data assets. Towards the extreme, companies whose data assets are their main value proposition have the most immediate need to understand the people engaging with proprietary information. These DataOps roles can be generally classified as data consumers, data preparers, and data suppliers.

Data Suppliers — Data suppliers are the end data owners, like database administrators, responsible for data management, processing, and user access control of a company’s DataOps.
Data Preparers — Due to the ever-complicating nature of DataOps, a middle ground of roles is developing between data engineers, data suppliers, and data consumers. Data engineers build the pipelines that refine raw data into new usable, valuable, and monetizable data. Data curators is a developing role that begins with the needs of consumers to optimize accordingly DataOps content to businesses the needed context for enhancing final assets. Another developing role due to heightened requirements around data governance is the data steward which is responsible for developing company data governance policies and ensuring compliance.
Data Consumers — Data Consumers receive the final data output and the largest group that interacts with DataOps assets. Many roles have emerged: data scientists apply data to solve business problems, data citizens are frontline workers in need of real-time information, and data developers need accurate DataOps as they build business applications that use those pipelines.