Data Reliability Supports Real-time Reporting & Analytics

Overview

Challenge

• Give retailer access to structured, reliable and timely data for critical reporting and business insights.

Solution

• Pentaho Data Integration blends complex set of retail chain’s data sources for analytics and modernization.

Outcome

• Reduced data processing time for ETLs from 3-4 hours to 1 hour.
  Improved stability of environment, giving nearly 100 users access to reliable data for hundreds of reports.
• Successful blend of data from old and new systems during extensive digital transformation project.
• Complex digital transformation supported with minimal disruption.
  Effective staged approach moves dozens of data processing changes through testing to production, weekly.

Challenge

Integrate business data during complex digital transformation project

Bing Lee is a consumer electronics retail chain with more than 35 stores across New South Wales, Australia. Family-owned and founded in 1957, Bing Lee is known for affordability and a radio jingle that has become part of the local pop culture. Its annual turnover is almost 500 million AUD.

Though a traditional family business, Bing Lee recognized a need to change with the times and began a technology transformation.

Its legacy ERP had become a highly customized system, dependent on a single point of failure. Its data was unstructured, and problematic processes were making the data generated by the system unreliable.

The retailer invested in a new cloud-based ERP and developed their own Point of Sale (POS) system. A new reporting tool was also selected to improve user experience for employees.

The plan was to get the new ERP in place and switch off the old. However, when the Covid-19 pandemic hit, lockdowns and remote work slowed the transition to the new ERP. Bing Lee’s sales and number of transactions substantially grew, increasing the volume of data through its systems.

“Our management team including buyers, shop managers, senior managers needed reliable reporting on a daily basis during the transformation project,” explained former GM of 16 years, now Head of Modernization, Phil Moujaes. “Data management became more complicated because we now needed a merged view of the purchase and sales data coming from both ERP systems, for reporting and analysis.”

Solution

Pentaho Data Integration and Data Management-as-a-Service

Bing Lee had been working with Hitachi Vantara partner, BizCubed, for years. It decided to use BizCubed’s Data Engineering and Enablement Platform (DEEP) for data management, with Hitachi Vantara’s Pentaho Data Integration (part of the Pentaho Platform Suite) as the engine for ETL design and orchestration. This solution is provided as-a-service, which allows Bing Lee to continually update its data processing at the pace of business.

“Our data warehouse team needed a way to engineer the necessary Extract, Transform and Load (ETL) procedures to support the evolving reporting needs of the entire organization,” said Rebecca Main, project manager for Bing Lee’s modernization efforts.

“We’ve been using the intuitive drag and drop Pentaho tools, to update or create business logic during this project,” said Main. “It will allow us to easily add or change ETLs on an ongoing basis.”

Integrating data from two ERPs can be tricky. For example, a new ERP may introduce new data fields that don’t align with historical data sets. Even if the data fields match, their contents might be different data types or formatted differently. Pentaho gives Bing Lee the tools to overcome these barriers.

The project team uses Pentaho to generate ETL processes that successfully merge and blend daily reporting data from two ERP solutions and more than 45 different API sources, including online purchases, inventory purchases and movements, customer information, plus data from all the stores such as unit sales and shop traffic.

BizCubed’s methodology provides a framework for rapid data acquisition, monitoring, recovery, and governance. This means that Bing Lee can have confidence in the reliability and accessibility of its data, every single day.

Bing Lee now has a low-risk data management platform that is reliable and accessible for its analysts and managers.

Outcome

Stable ETLs and reliable, aggregated data

Rather than waiting for all of the required ETL logic to be engineered so it could all “go live” at once, individual ETL processes were built, put through user acceptance testing and moved to production as soon as they were ready. Up to two dozen complete, new processes went live each week, with new reporting capabilities accessible to users every day.

“The business is changing the way it does things, and we’re only partway through the overall transformation,” said Moujaes. “But the fact that we decided to go with a staged approach has meant that the disruption to our staff who rely on these reports has been minimized

“The reports are critical to track sales, inventory, purchasing and margins right down to the performance KPIs of each salesperson and store, so the value of this is far-reaching, across the whole business.”

According to Main, most data warehousing and ETL processes rely on a daisy chain methodology. If one fails, the rest of the processes have to be re-run.

“With our old systems, recovering data took at least 4 hours, if not longer. Users would not get updated data until at least the afternoon.” said Main.   “We’re creating more stable ETLs and making sure our data is more reliable. This means fewer issues for the nearly 100 users of the hundreds of reports required by the business,” said Main. “Recovery is quicker, and the process takes less time overall.”