What Is Big Data & How Will It Affect Consulting

“Big data” is one of those terms, like “the cloud,” that seems a lot more complex than it actually is. Almost all companies these days are collecting data on their users and customers. Sure, companies have been doing that for a long time now, but today it’s a bit different. A telecom company could be collecting not only call histories, but geo data, cell browsing history, a profile of the products you like, and websites you frequent. But how are they using all of that data? Most importantly, what is big data doing for consulting?

In this article, we’ll break down “big data” into some more meaningful, fundamental concepts. Then we’ll explore how these concepts are influencing the consulting industry.

What Is Big Data?

Let’s start by looking at some commonly accepted definitions of big data:

SAS, a large database and information management company, says “Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis.”

Wikipedia: “Big data” is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software.

As far as a working big data definition goes, either of these works well.

Our professional lives began when Microsoft Excel already existed. So we don’t remember a time when you couldn’t use Excel to analyze a data set. Let’s start by saying that the introduction of Excel into modern business life was phase 0 of “big data.” It paved the way for the average businessperson to engage with and analyze large amounts of data. It went far beyond what they could handle manually in their head or on paper.

One simple but pretty accurate way to think about big data today is that businesses increasingly find that they have more interesting data to be analyzed than can be handled by Microsoft Excel. The structure of the data isn’t more complex, there are just more rows & columns of data collected than can fit into Excel. If you’ve worked with a large data set in Excel, and scrolled over a hundred plus columns to see a data point, you can understand how cumbersome this could get!

What Is Big Data Analytics?

Big data analytics is simply the process of harnessing and analyzing large amounts of data to identify insights. The term includes processes, methodologies, and programming tools, and the definition should be kept broad. Given that Excel can now handle millions of rows of data, it’s probably not even fair to say that if you are using Excel, you aren’t engaging in “big data” analytics. But big data applications like Apache Hadoop, SQL, R, Knime, Tableau, and many others take various elements of data analysis to levels far beyond what Excel is capable of. From data storage to processing power to statistical insight to data visualization, common big data analytical tools help the user “make sense” of large amounts of information.

Benefits Of Big Data

Effective analysis of big data reveals insights that may previously have been hidden. Discovering and cataloguing those insights, and acting on them to improve business performance, is what big data is all about. Insights might relate to products that are selling well, to whom, and as a result – which marketing strategy to use. Everything from pricing to predictive maintenance can be influenced by analysis of large data sets.

A “stage two” benefit of big data starts to consider insights and adjustments to operations without human intervention. These adjustments are based on what is learned from the data in real-time. This is effectively an early step in the direction of artificial intelligence. The way that prices are set on Amazon is by integrating real-time info on competitive website pricing and user activity. When you are buying your plane ticket, big data and AI are in play via dynamic pricing. There is a competitive lens combined with real-time views of supply and demand being used. This illustrates the potential of artificial intelligence to help improve business performance.

Most of us now have become accustomed to popups on a website or promotional emails in our inboxes. This is happening because a program is analyzing the data associated with our shopping habits. It sees what we’re interested in and then takes action without human intervention.

Uses Of Big Data: A Case Study

We’re ex-consultants, and one of our former clients is a helpful real-life and relatively simple example of “big data analytics” and how it varies from traditional analytics. Our former client is a B2B distributor of products that cost between $5 and $20. Historically (~20 years ago), a large volume of purchases was still made via catalog. Customers would open the catalog, find what they needed, and call a salesperson to discuss and place their order. If you wanted to be diligent about tracking who you sent catalogs to, you could analyze almost everything you’d want to analyze about this business model in a simple spreadsheet. You could track number of catalogs sent and where, who purchased, how much they purchased, etc. If you asked the salesperson to document when someone called and did or didn’t purchase, you could analyze conversion from catalog vs. not, and more.

But if you fast forward to today, the situation is far more complex. The company has made dozens of acquisitions. At the same time, new competitors with a far more online oriented business model have popped up. The company has extended its product line, and now sells far more SKUs. It also does a lot of business over the internet and has a direct electronic data interface with many clients. Yet, they still take phone orders from others. People check prices on multiple sites before ordering and make smaller orders.

Back In Time

Imagine I went back in time to 20 years ago and, in each year, I asked myself the question, “could I take all of the available data this company can provide to me, put it in Excel, and analyze its business?” 20 years ago, hands down, the answer would have been yes. 15 years ago, the answer was probably still “yes.” 10 years go? The answer is probably yes, but I’d have to accept that maybe some information couldn’t fit properly. Starting about 5 years ago, Excel became a sub-optimal tool for analyzing the business. Now, we could extract data on how many people visited the website in the previous year, how long they browsed, whether they purchased, and how much they purchased. But all of that data just doesn’t fit in Excel.

Big Data In Consulting

SAS rightly notes that “it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.”

There has traditionally been a split between management consulting and IT consulting. A management consultant would use a lot of data and analysis to recommend a new business model, pricing strategy, or marketing mix. They would deliver the recommendation and action plan and the project would end. An IT consulting firm, on the other hand, might implement a new Enterprise Resource Planning (ERP) system. The costs and benefits of such a system are clear. The business users just need to be trained on the system.

Big data analytics, however, doesn’t really fit into either model. Successful big data analytics is an ongoing process of discovery and usage of new tools and techniques. This may result in the client needing ongoing support or training on how to harness the power of the analytics. Big data innovation is an ongoing process that clients may need to explore with a consulting partner over time. In this regard, big data is a great partner for consultants.

Implications For Consultants

In the near term, as many projects remain short term in nature and focused on insight identification, consulting analysts need to move from only using Excel to gaining comfort with tools like R, Knime, or Tableau. In other words, “big data” is changing the nature of regular consulting work. But the change goes deeper and farther than that. We see a premier strategy consulting firm like Bain setting up a separate Advanced Analytics Group with specialized expertise in extracting useful insights from client’s data. It’s not hard to imagine that over time, the influence and importance of a “specialized” group like that will grow.

Historically, Bain, BCG, or McKinsey might have helped a large client segment their customer base based on needs and then develop customized marketing and pricing strategies for each group. The client would be left to implement, and because of the potential value-at-stake, the consulting fees would be very high. Today, implementation work can be done with support from a firm like www.zilliant.com, a SaaS company powered by predictive software and supported by data scientists. They offer clients the ability to monitor data on an ongoing basis, and to test and implement different marketing or pricing strategies. No wonder McKinsey, Bain, BCG and other top consulting firms are building out their implementation practices and Big Data analytics groups.

Does Big Data Threaten To Replace Consultants?

In the end, big data does not threaten to unseat a consultant’s role, although it could make it possible for there to be three analysts on a team instead of four. In fact, with salaries rising and utilization rates capped, firms are probably hoping Big Data will help them reduce the size of their engagement teams.

Overall, big data applications can crunch numbers better than an army of interns, but somebody still has to make sense of the results and attribute meaning to that data and results. AI or machines will have a hard time replacing humans in that regard. So rest easy – for now, big data is a helpful tool rather than something to be feared by consultants.

Conclusion

Big data is a way to analyze and draw insights from large data sets. It has the potential to impact a business through increased effectiveness in campaigns and strategies drawn from those insights. Consulting firms are increasingly embracing big data for consulting, and even creating their own implementation departments to cash in on this growing business sector. While there is great potential with big data in consulting and other industries, humans are still needed to assign meaning to what data is important, and develop hypotheses to ensure data analysis is as efficient as possible. Because of that, big data and AI will not replace consultants all together, but consultants do need to be familiar with the uses of big data!

More Content:

• Problem Solving & Decision Making
• Deloitte Strategy & Operations (S&O) Consulting
• Accenture vs Deloitte
• McKinsey Retail Store