Recently I was asked to speak on a panel to college students about technical career paths. It’s always fun doing tech conferences because while our community is pretty small, we have a ton of events of this type. Inevitably you end up running into friends and friends of friends.
On this occasion one of my co-panelists was a friend of (multiple) friends who had just started a new job at a startup I had other connections to. The startup was growing rather quickly and as we swapped war stories over a free conference lunch, friend of friend remarked, “Our next couple of hires better be data engineers or else I’m going to lose my mind.”
Interest piqued, I started asking her what she was looking for.
She ended up describing exactly the types of problems Exversion was designed to solve: their most critical data was kept on Excel with nothing keeping people from making changes that corrupt the data or deleting it completely. Even when the data is in a traditional database, there’s no accountability. Things can be changed, altered, deleted with very little record of why they were changed, when or by whom. Everyone on the dev team knew this was a potential vulnerability and the product managers were increasingly uncomfortable with sales projections being built on top of data no one was sure was even accurate.
What’s especially weird about this scenario is that I know the cofounder of this company. We’ve been introduced a couple of times, one of her early investors is a valuable mentor of mine and at no point was it ever suggested that they might have a need for exactly what I was building.
Was the CEO simply not aware of the problem? Did she not care or not understand the seriousness of the issue? Or was it more that she knows and understands but is not interested in purchasing a solution from an outside company?
If you’ve read this blog before, you will know that I’ve been struggling with the reality that the problems we face now in data have existed for twenty, thirty years. Everyone knows they exist, everyone knows they cost businesses billions of dollars, yet seemingly no one is investing in their solutions. Not ANY KIND of solutions. I understand why my individual company can, and will in fact most likely fail … but I do not understand why no one else has risen to the task.
It seems to me to be the lowest of all the low hanging fruits, but evidently not.
Sometimes insight comes not from having all the facts in front of you, but having all the facts in front of you in the right context. This little career conference turned out to be exactly the right context. We spent the day talking about the overwhelming pressure to be perfect you must ignore in order to learn, and it occurred to me that the reason why VC dollars go to “Big Data” companies and visualization generators is that those purchases are indicators of a company’s success. To generate enough activity to need a “Big Data” solution is a good problem to have. To have the audience that will eagerly consume and share pretty visualizations is an good problem to have.
In consumer studies there’s something called conspicuous consumption, that is products bought purely to convey status. When we talk about conspicuous consumption– particularly the tendency to avoid necessary purchases while indulging in luxuries– we’re generally talking about the behavior of people. But maybe the same impulses govern the decision making of companies as well.
Let’s say theoretically there are three basic types of problems a company might need solved:
– Problems that confirm your success
– Problems that block new business
– Problems that come from perceived internal failures
My fellow panelist obviously thought the best way to solve their data problems was to hire an expert to fix the malfunctioning internal system and police everyone to behave themselves. She kept implying that as an organization they wouldn’t have this problem if everyone on the team did things the right way. But the reality is many companies have these problems. They don’t exist because people are lazy or stupid, but because of limitations in the existing technology.
The perception is buying a product to solve this problem is admitting incompetency, while buying a “Big Data” solution your company doesn’t need is to broadcast success.
So it seems to me that companies trying to solve these type of problems have to focus on rebranding the problem so that the companies they hope to bring on as clients do not see it as something they wouldn’t have to deal with if their dev team was perfect or if they made better decisions.