A Conversation with Nick Caldwell on the State of the Business Intelligence Landscape

Looks like the summer is heating up in the Business Intelligence (BI) industry! Two major acquisitions have been announced in June 2019 within a week of each other. The first: Google’s acquisition of Looker for $2.6B. This deal is expected to be completed later this year. Thomas Kurian, Google Cloud CEO released an announcement on his intention to acquire the unified platform for BI, data apps, and embedded analytics. Kurian views the acquisition of Looker as an extension of Google’s business analytics offerings. He states, “For any business that is looking for a partner to help drive digital transformation, the combination of Google Cloud and Looker will offer an incredible data management and analytics platform.” While Google is looking to beef up its cloud offerings, Salesforce recently bought Tableau for $15.7B. This was announced the week following the Looker acquisition. Their strategy, according to Salesforce CEO Marc Benioff, is to merge the top CRM platform and the top analytics platform to help speed customer digital transformations.

I had an opportunity to chat with Looker’s Chief Product Officer, Nick Caldwell about the state of business intelligence and where the industry is heading. He leads the company’s Engineering and Product Management and Design teams. Prior to Looker, he was VP of Engineering at Reddit and General Manager of Power BI for Microsoft. I wanted to get his take on the data visualization landscape and how it’s impacting the data-driven workforce.

Nick acknowledged that there is a lot of consolidation in the BI space right now as big platform players are looking for BI to be “the sizzle on the steak” and give an extra oomph to their portfolios.

this is a BI world that will call for more than just dashboards, data prep and advanced analytics

Nick views both the Looker and Tableau acquisitions as examples of big platforms highlighting their collective power versus the individual capabilities of their product suites. The amount of data at our fingertips is more prevalent in our lives than it has ever been. He explained that this is a BI world that will call for more than just dashboards, data prep, and advanced analytics — A trend he’s excited about. We discussed the trends influencing the BI market and how Looker has been riding the data wave! Nick sees three major trends:

Trend 1. The Arrival of a Data-Driven Workforce

Data is embedded in all facets of our lives, especially in the workplace. I definitely agree with him on this point. He went on to say that everyone needs to get the job done — Not just traditional analysts or quantitative marketers. I’m counting the days to when Looker or Tableau is a standard installation on office laptops but think standalone apps are the future.

“Of course we still need great BI tools for our analysts and data jockeys. But we must also be aware that there are a rapidly increasing number of data-enabled workers who view the idea of using dashboards the same way that you or I might view using a rotary phone. And that’s okay.”

Trend 2. A proliferation of SaaS Apps

Nick explained that untraditional data folks expect the data to come to them in an interface designed specifically for the task at hand, ideally integrated into a tool they are already familiar with. He speculates that the future of BI will not look much like a BI application at all and believes we will not need to have everyone become familiar with analyst tools to have an insight-driven business. Software as a Service (SaaS) apps is “polished user interfaces on top of rows of data”.

“People aren’t going to go to BI, BI has to go to the people. This is already happening in a big way.” ~Nick Caldwell

While Nick was explaining how the adoption of SaaS apps was skyrocketing, my mind went to dashboards and how they were on the top of every department’s wish list. This usually caused a bottleneck at the analyst as they looked to gather business requirements and find the data set needed to build the views. Then once the dataset was identified, performance was the next problem to deal with. Nick helped me to understand that SaaS apps put the power of data in the hands of non-traditional data folks from factory workers to pizza delivery drivers. I questioned the processing power that would be needed for all of these individual apps. Like dashboards, I figured this would call for numerous data silos and countless meetings with IT to make sure each app ran efficiently. This takes us to trend #3.

Trend 3. Modern Data Infrastructure

Nick explained that Looker took a huge bet and invested heavily in being able to natively support massively parallel processing (MPP) data warehouses like Google Cloud Big Query, Snowflake, Amazon Redshift Spectrum. This infrastructure supports the world of SaaS apps. “MPP data warehouses can hold immense amounts of data, query it all in seconds, and even do advanced analytics directly in the database — all this at a cost which is bafflingly cheap compared to last generation technologies,” says Caldwell. He went on to point out this infrastructure improves the lives of data engineers because they can spend less of their time building and maintaining data pipelines and more time doing what they really want: empowering end-users with data-driven experiences.

In a world where we can dump as much data as we want into one place, query it fast, and pay pennies for the privilege, whole steps in the traditional data engineering workflow can be simplified. — Nick Caldwell on modern data structures.

The Future of Business Intelligence

Nick’s insights broadened my perspective on the BI space. He has a vision: “businesses need a data platform with the openness to talk to any source and the flexibility to build data-driven applications for any need.” He pointed out that Looker uses 140 SaaS apps to run their business. Looker’s platform is customer-centric; its customer value proposition is to provide tailored data experiences that sit nicely on top of the customer’s data infrastructure.

As organizations become more data-driven, data literacy will become an inhibitor to growth.

The future of BI is provocative — are we ready for it? There is a gap in the level of data literacy that is required of employees and the level of literacy that they currently have. As organizations become more data-driven, data literacy will become an inhibitor to growth. According to Gartner, they expect 80% of organizations will initiate data literacy programs to overcome extreme deficiencies by 2020. They also estimate that 50% of organizations will lack sufficient AI and data literacy skills to achieve business goals.

Does the proliferation of SaaS mitigate the data literacy gap organizations are observing with employees? If BI is moving in a way where we’re going to have app layers that take away the need for coding and querying, will this narrow the data literacy gap and a person’s need to analyze and communicate with data?

Nick believes the introduction of more SaaS apps will close the data literacy gap because more people will be able to access data without specialized knowledge. At the same time, he speculates there will be an increasing need for people to actually build those apps and understand the data. That being said, he feels our current approach to using dashboards as a way to reach the masses is changing.

“This is a great thing, it means your pizza delivery drivers now get better because they have apps to tell them where to go powered by predictive analytics.” ~Nick Caldwell on how the proliferation of SaaS apps will begin to narrow the data literacy gap.

Getting Comfortable Getting Uncomfortable

As Chief Product Officer, Nick played a major part in developing the Looker platform to the point of acquisition. I asked him what life lessons or mantras helped him navigate this $2.6B acquisition. Nick shared that he leaned on having to get comfortable getting uncomfortable. After spending 16 years at Microsoft, he moved to Silicon Valley. This was definitely a culture shift let alone added risk working for startups. Nick realized you have to get uncomfortable because that’s where the growth is.

As analysts, we may begin to feel a little uncomfortable with the shifts we’re seeing in the BI space. I’ve observed that there are a lot of self-serve BI tools out there making it easier for non-technical business users to develop insights on their own with user interfaces that are making it simple on the surface while complex querying is happening under the hood. We should welcome these innovations because there is an influx of data out there! According to a Forbes article, there are 2.5 quintillion bytes of data created every day.

With the current demand, analysts cannot keep up with the multitude of requests that come across our desks. Dashboards become obsolete as we build them with all the different insights our business stakeholders are asking for. We have to relinquish some of the power. While data can be dangerous in the wrong hands, it is still our job to be the data champion or whisperer. Data whisperers’ domain expertise and analytical skills help them ask the right questions to flesh out the facts needed to provide a holistic and unbiased view of the analysis. This ensures that the story is presented in a manner that allows the business owner to make an informed decision.

Spreading the BI love also frees us up to work on analytics that goes beyond metrics reporting. We will have more bandwidth to mine deeper-level insights. The proliferation of SaaS apps will also present opportunities to learn how to develop them. I agree with Nick wholeheartedly when he states that you learn a lot from situations that take you out of your comfort zone.

“One thing remains constant: the analyst is still the hero. It is their knowledge and passion for data that will provide the deepest insights to businesses.” ~ Nick Caldwell on the future of BI.