As AI becomes more entrenched, data is becoming more important – not less.
Data is the fastest-growing commodity, and is today’s “wild west” and the battlefield of today’s tech titans. We talk about AI as the new gold rush, but data is the commodity everyone is mining—and the real advantage comes from how you refine it, not just how much you collect.
According to IDC, the volume of data stored globally is doubling roughly every four years — from 33 zettabytes in 2018 toward a projected 175 zettabytes by 2025.
A staggering 402 million terabytes of data are created daily, which means around 130 zettabytes of data will be generated this year. But those numbers are vastly understated because AI and agents are poised to create and consume data on a scale we’ve never seen before.
Video is still growing rapidly, and so is IoT, with about 14% annual growth. There are now over 21.1 billion connected devices. Of course, AI is driving growth even higher.
Alphabet, Amazon, Apple, Facebook, and Microsoft all have unprecedented amounts of data (and power). And the new generation of giants like OpenAI and Anthropic (along with current trends in generative AI content creation, LLM usage, data center growth, etc.) tip the scales further towards almost unimaginable quantities of data, knowledge, and insights.
Rapid growth means little time to create adequate rules (or tools). Everyone’s jumping to own more data than the next person and to protect it from prying eyes.
Collecting basic data and using basic analytics were enough … but not anymore. The game is changing.
For example, traders used to focus on price data … but there has been an influx of firms using alternative data sets and making extraordinary investments in hardware and software to find an edge. If you’re using the same data sources as your competitors and competing on the same set of beliefs, it’s hard to find a sustainable edge.
Understanding the game others are playing (and its rules) is important. However, that’s only table stakes.
Figuring out where you can find extra insight — or where you can make the invisible visible — is what actually separates you from the field. But like the flywheel this week’s other piece [link] describes, it’s a separation you have to keep re-earning, not something you bank once.
Here is a quick high-level video recorded back in 2019 on data as fuel for your business — it holds up remarkably well. Check it out.
It is interesting to think about what’s driving the new world (of trading, technology, AI, etc.), which often involves identifying what drove the old world.
History has a way of repeating itself. Even when it doesn’t repeat itself, it often rhymes.
With that said, the key to unlocking the pathway to the new world often comes from a new or alternative data set that lets you approach the problem, challenge, or opportunity from a different perspective.
Before e-mails, fax machines were amazing. Before cars, people were happy with horses and buggies. Now, let’s talk about how technological improvements like dashboards and reporting seem old-world compared to firms that use data to re-architect their business models, create whole new opportunities … or even new industries.
These comparisons help explain the importance of data in today’s new-world economics.
via gapingvoid
Data as the New Oil
You’ve heard “data is the new oil” before — it’s been the opening line of tech keynotes for over a decade. Clichés survive because something true is under them. Worth pushing on where this one holds and where it breaks.
Petroleum has played a pivotal role in human advancement since the Industrial Revolution. It fueled (and still fuels) our creativity, technology advancements, and a variety of derivative byproducts. There are direct competitors to fossil fuels gaining steam, but I think it’s more interesting to compare petroleum to data because of their parallels in their effects on innovation.
Pumping crude oil out of the ground and transforming it into a finished product is not a simple process. Yet, it is relatively easy for someone to understand the process at a high level. You have to locate a reservoir, drill, capture the resource, and then refine it to the desired product – heating oil, gasoline, asphalt, plastics, etc.
We discussed this in the video, thinking through what actually makes data usable:
You’ve got to figure out what data you might have, how it might be useful, you have to figure out how to refine it, clean it, fix it, curate it, transform it into something useful, and then how to deliver it to the people that need it in their business. And even if you’ve done this, you then have to make people aware that it’s there, that it’s changing, or how they might use it. For people who do it well, it’s an incredible edge. – Howard Getson
In a sense, data fuels the information economy much like oil fuels the industrial economy. The amount of power someone has can be correlated to their control of and access to these resources. Likewise, things that diminish or constrain access or use of these resources can lead to extreme consequences.
Why Data Is Better Than Oil
The analogy works, but it’s just that, an analogy, and the more you analyze it, the more it falls apart. Unlike the finite resource that is oil, data is all around us and increasing at an exponential rate, so the game is a little different:
- Data is a renewable resource. It’s durable, it’s reusable, and it’s being produced faster than we can process it.
- Because it’s not a scarce resource, there’s no need to hoard it – you can use it, transform it, and share it, knowing it won’t diminish.
- Data becomes more valuable the more you use it.
- As the world’s oil reserves dwindle, and renewable resources grow in popularity and effectiveness, the relative value of oil drops. It’s unlikely that will happen to data.
- Also, while data transport is important, it’s not expensive the way it was with oil. Here is an example difference that dramatically changes the implications… Data can be transported, replicated, and transformed at light speed.
The cheapest crude you’ll ever refine is the data you’re already generating and throwing away.
Another high-value data concept is that alternative data gives traders an advantage, but it doesn’t always require confidential or hard-to-find information.
For example, Traders now have access to vast amounts of structured and unstructured data. A significant source that many overlook is the data produced through their own process or the metadata from their own trades or transactions.
The video highlights a prediction about where this goes next:
In the very near future, I expect these systems to be able to go out and search for different sources of information. It’s almost like the algorithm becomes an omnivore. Instead of simply looking at market data or transactional data, or even metadata, it starts to look for connections or feedback loops that are profitable in sources of data that the human would never have thought of. – Howard Getson
A word of caution: there are two common mistakes people make when making data-driven decisions.
First, people often become slaves to the data, losing sight of the bigger picture. It’s a mistake that’s become even more common in the age of AI. Both data and AI are extraordinary tools, but neither should replace critical thinking, experience, or judgment. AI can summarize, analyze, and recommend at incredible speed, but it still requires humans to ask the right questions, validate the answers, and decide what truly matters.
Second, even the most sophisticated models can’t predict black swan events. AI excels at identifying patterns in what has happened before, but history doesn’t always repeat itself. The unexpected still happens. Resilience, adaptability, and preparation remain just as important as prediction.
The future of data has never been brighter, but the challenges have grown just as quickly. Privacy concerns, data ownership, misinformation, and synthetic content are no longer theoretical debates—they’re everyday realities. Likewise, AI has dramatically lowered the cost of creating convincing text, images, audio, and video, making it easier than ever to blur the line between fact and fiction. At the same time, organizations are collecting and generating more information than ever before, making the ability to distinguish signal from noise one of the defining skills of the modern era. And all that doesn’t begin to unpack the risks from data quality, model risk, and how to know when you’re approaching the point of diminishing returns.
I believe one of the greatest challenges facing our youth—and, increasingly, all of us—isn’t a lack of information. It’s an overabundance of it.
No previous generation has had access to this much knowledge, or been bombarded by this much content. Ironically, more information doesn’t always produce greater understanding. Algorithms reward engagement over nuance. Headlines replace deep reading. AI can generate answers in seconds, but it can also create the illusion of expertise without the substance to back it up. The bottleneck is no longer access to information; it’s discernment.
The winners won’t simply be the people or organizations with the most data or the most powerful AI. They’ll be the ones who know what information to trust, what to ignore, and how to systematically combine technology with sound judgment.
In an age when intelligence is increasingly abundant, wisdom becomes increasingly scarce.
The question is no longer how to collect more data.
It’s how to use it without becoming a victim of it.
Leave a Reply