Automating data analysis is a must for midsize businesses
Executives of midsize companies are right to be excited about the opportunities to harness the value of their large data sets. But data in midsize businesses tends to be messy – spreadsheets and plain text files, many different formats, are difficult (if not impossible) to integrate. It takes a lot of time and money to clean it to make it useful. Poor, disintegrated data can sabotage even the best initiatives, including AI designed to increase value and efficiency. HdL Companies, a government services company based in Brea, Calif., Has used its data strategically and experienced significant efficiency gains. The author offers three lessons for executives to consider as they begin to automate data analysis.
As midsize businesses grow, they develop data streams and data lakes (repositories for structured and unstructured data) that are too large for a single person or even a team. can handle and use them effectively. And while a business is currently deriving value from its data, the people doing the work may move on, leaving the business tasked with finding, attracting, and hiring expensive data analysts in a rush.
Having a well-functioning and up-to-date enterprise resource planning (ERP) system will not solve the problem or relieve the pressure. Most midsize businesses start with finance-focused ERPs and end up with systems to store other data, such as customer activity and manufacturing throughput, more operational than strategic.
Therefore, automating the analysis of data as the business grows is a very, very good idea. Automation is often where programmers write algorithms that perform previously manual tasks according to instructions. This quickly pays dividends, spurs innovation and growth, and paves the way for the implementation of artificial intelligence, which makes just about everything easier, more efficient and more profitable. AI is coded to learn how to perform a task, by inventing and sort of writing its own algorithms.
But data in midsize businesses tends to be messy. Spreadsheets and plain text files, many different formats, are difficult, if not impossible, to integrate. It takes a lot of time and money to clean them to make them useful. Poor, disintegrated data can sabotage even the best initiatives, including AI designed to increase value and efficiency.
As Joe Pucciarelli, group vice president and executive IT advisor for market research firm International Data Corporation (IDC), said during a recent Channel Company webinar: âMost of the data sets in organizations are not in great shape. We talk about data and analytics as a strategy and a priority, but data is not ready to back it up.â¦ Most organizations, when trying to solve a problem, the analyst who working there typically spends more than 75% of the timeâ¦ just prepare the data.
As you can imagine, the ROI on the time spent doing this is not good. Let’s take a look at how a midsize business has harnessed the value of their data, and explore three steps midsize business leaders can take to do the same.
How a midsize business handled their data
One of my clients, HdL Companies – a government services company headquartered in Brea, California – is engaged by municipalities in California, Texas and other states to analyze the distribution of tax revenues in their states. to ensure that their city or town gets its fair share. HDL looks for allocation errors and discrepancies that municipalities may report when seeking redress from the state. The heart of this work is to compare different databases to expose the discrepancies that affect who should collect sales tax revenue. For example, in one database a business might be listed in Dublin, California, but in two other databases it might be listed in the nearby town of Pleasanton. This makes a tax imputation error very likely; HDL’s job is to find it.
California’s 40 million residents purchase taxable products from 5.9 million authorized resellers, creating a massive dataset of nearly 46 million tax records in 2020. For years, HDL has employed analysts to examine this data quarterly, looking for errors. HdL’s IT group created software to help, but over the years its analysis team adopted a lot of idiosyncratic manual techniques, and the IT group had a long backlog of work to continue building the codebase for it. include these techniques. Dealing with the backlog delayed HdL’s automation projects and the development of new techniques to more effectively highlight tax gaps. At the same time, the state of California was making its own improvements, leaving less of the discrepancies than could be found with older HdL tools. âOur team is always finding new analytical techniques to identify hard-to-find assignment errors,â says Matt Hinderliter, director of audit services at HdL. “However, we have been heavily reliant on manual exports and data manipulation in Excel, as well as the need to have top-level analysts to manually review spreadsheets that often exceed 70,000 or 80,000 rows of data. . “
To deal with external stressors (California improvements) and internal stressors (IT department overloaded with HdL and laborious manual analysis), HdL – a mid-size company with a mid-size budget – hired a talented intern who was earning her full-time master’s degree in data analysis. . She was able to transform some of the analytical processes that team members used to identify potential allocation errors into algorithms that could generate more tax revenue reallocation opportunities in a fraction of the time.
Given this efficiency gain, one could assume that HdL would consider layoffs. Instead, its audit department is recruiting to exploit any opportunities that automated analysis has revealed. And HdL has moved closer to implementing and deploying AI.
Improving operational efficiency is almost always a top priority for midsize businesses. In a Channel Company survey of mid-market IT leaders, where 75% of companies have revenues of $ 50 million to $ 1 billion, 58% of those surveyed said their top priority was ” improve operational efficiency. This far exceeded their second priority, increasing new income (36%). Both goals can be supported by automating data analysis, as they were at HdL.
Midsize businesses cannot seize every opportunity. Their budgets and staff and the hustle and bustle of day-to-day operations will not allow it. (They’re not Google, after all.) Midsize businesses should therefore start automating their data analytics processes by focusing on areas where critical operations are either inefficient, too dependent on one person, or of a handful of people. Before automation, HDL had 15 people who spent a lot of their time doing what algorithms do today.
HDL was already working on the data; many companies (printers, plumbing suppliers, etc.) are not. But these companies continue to accumulate data, and they can benefit from its strategic use. It is important to start with a solid foundation. Here are three things to consider when executives start automating data analysis.
Prioritize cleaning. Data in a midsize business is usually messy and needs a lot of tidying up before it can become useful. Another fundamental activity is to identify important data and then clean it up. It can be slow work at first, and it doesn’t come cheap, so find areas where the business can get a return on investment in the first year. It will turn skeptics into believers.
Hire the right people. Leaders are not analysts. They lack the time, patience and skills to perform data analysis in addition to their daily tasks. Business analysts are both programmers and business people. HdL started with an intern and hired her as a full time business analyst.
Prepare the data. It’s only when your data is carefully prepared that you can start thinking about AI. AI creates its own logic from an analysis of the patterns it discovers in the data. While AI and machine learning are useful and exciting, both technologies need large datasets to train on, with both positive and negative results confirmed. After sufficient data cleansing and a few algorithm-based scans, most midsize businesses will have a sufficiently large and useful data set on which to train an AI model.
Executives of midsize companies are right to be excited about the opportunities to harness the value of large data sets. Now is the time to embark on this multi-year journey and commit to hiring the right talent while taking incremental action to generate value from data automation and other types of advanced analytics.