I’ve given a few speeches recently and have new subscribers to our weekly commentary (click here to sign up), so I thought it was a good time to write about the importance of data.
I revisit this topic about once a year because it’s important.
The Hidden Engine: Why Data Fuels Innovation
Technology and innovation are popular topics, but people often ignore what makes it all possible ... the hidden foundation, data.
Data is the lifeblood of modern businesses and the fastest-growing resource we have.
The quest to find and use data has created a modern-day “Wild West.” While AI is often positioned as a “Gold Rush,” data is the precious resource powering the race.
Another way to look at it is that data is the ammunition used by today’s tech titans in their battle for dominance.
In either case, it is easy to see that data is a scarce and valuable resource.
The Data Deluge: Finding Signal in the Noise
We’re living in an age of data explosion. Every day, a staggering 328.77 million terabytes of data are created, amounting to an estimated 120 zettabytes of new data by year’s end.
This rapid growth presents a challenge. Tech giants like Alphabet, Amazon, Apple, Facebook, and Microsoft all hold unprecedented data troves, creating a race for ownership and control. Regulations struggle to keep pace with this digital stampede.
Rapid growth means little time to create adequate rules. Everyone’s jumping to own more data than the next and to protect that data from prying eyes.
As a great example of this, I often warn people to keep their intellectual property off of ChatGPT or other hosted language models.
But here’s the real concern: Are we losing sight of the signal in all this noise? Just having vast amounts of data isn’t enough. The true value comes from extracting meaningful insights – the nuggets of gold buried within the data avalanche.
To Do The Impossible, Make The Invisible Visible
Collecting basic data and using basic analytics used to be enough, but it is not anymore. The game is changing.
I also see it trading, but it’s pervasive in every industry and our personal lives as well.
For example, traders used to focus on price data ... but there has been an influx of firms using alternative data sets and extraordinary hardware and software investments to find an edge. If you’re using the same data sources as your competitors and competing on the same set of beliefs, it’s hard to find a sustainable edge.
Understanding the game others are playing (and its rules) is important. However, that’s only table stakes.
Figuring out where you can find extra insight or where you can make the invisible visible creates a moat between you and your competition and lets you play your own game.
Here is a quick high-level video about Data as fuel for your business. Check it out.
It is interesting to think about what’s driving the new world (of trading, technology, AI, etc.), which often involves identifying what drove the old world.
Decoding the New World: Data as the Catalyst
Understanding today’s driving forces – like AI – often involves examining what propelled past eras.
History has a way of repeating itself. Even when it doesn’t repeat itself, it often rhymes.
Before e-mails, fax machines were amazing. Before cars, people were happy with horses and buggies.
The key to unlocking new economic realities lies in fresh perspectives.
In this new world, new or better data is often the game-changer. It’s the alternative dataset that allows us to approach challenges and opportunities from entirely new angles.
Before data analytics, businesses relied on intuition and limited information. Now, data empowers us to see patterns and make data-driven decisions, propelling innovation at an unprecedented pace.
These comparisons help explain the importance of data in today’s new world economics.
One of the more recent shifts is in the value of synthetic data.
Synthetic data can mimic the statistical properties of real-world data, making it useful for a variety of purposes.
For example, synthetic data can be used to train machine learning models when collecting traditional data is impractical or presents privacy concerns. It is also used in various other applications, such as data privacy, testing and development, data augmentation, simulation and modeling, risk assessment and management, and enhancing data quality.
You don’t have a competitive advantage if you use the same data and the same process as other people. That’s why understanding how to recognize and capture synthetic data is important. It can shift your perspective, add dimensionality, help you solve different problems, and create transformative results.
In the very near future, I expect these systems to be able to go out and search for different sources of information. It's almost like the algorithm becomes an omnivore. Instead of simply looking at market data or transactional data, or even metadata, it starts to look for connections or feedback loops that are profitable in sources of data that the human would never have thought of. – Howard Getson
While data is the foundation, it’s about transforming your data into actionable insights.
By identifying your real business, the KPIs of success, and what data you’re underutilizing, you can massively improve the efficiency and effectiveness of your business and create new products that transform your future.
In a word of caution, there are two common mistakes people make when making data-driven decisions. First, people often become slaves to the data, losing focus on the bigger picture. This is the same mistake people make with AI. Both are tools, not the end goal. Second, even the most insightful data can’t predict black swans. It’s essential to exercise caution and prepare for the unexpected.
The future of data is bright, but it’s also littered with potential challenges. Privacy concerns and data misuse are hot-button topics, as are fake news and the ability of systems to generate misleading data. In addition, as we gain access to more data, our ability to separate signal from noise becomes more important.
One of the biggest problems facing our youth—and really all of us—is how much information is thrust at us every waking moment of the day. No previous generation has had this much access to data. As a result, many are actually less informed than in the past. Soundbites become the entire news story, and nuance gets lost in the echo chambers.
The question becomes, how do you capitalize on data without becoming a victim of it?
This is obviously an approximation, but it's an interesting one nonetheless.
Social Security is the largest draw, but it is also one of the public services at the highest risk of failure due to an increasingly large aging population (with fewer active workers contributing to the system).
Unsurprisingly, Health and National Defense are the next biggest draws from your taxes. Medicare and Medicaid are expensive, and we do have the largest military force, by a large margin. We spend more on defense than the following ten countries combined.
It's interesting to see ... but I might add a cent to my tax dollar if it meant they'd fix all these potholes.
In fact, I hit five million "butt-in-seat" miles on American Airlines in 2019 (back when frequent flyer programs were about flying frequently rather than credit card spending).
It is 2024, and I am now just below 5.5M. That means I averaged a little over 100,000 miles per year, even through the COVID shutdown.
Yes, I expect that my travel will slow down. But as I traveled, I didn't expect it to continue at the pace it did.
Nonetheless, it has been good for me, and the time spent traveling has been productive.
I have a different workflow when I travel, and it works for me.
Ultimately, I believe that good things happen when you are in motion!
Many people, however, are focused on the hassle.
The practical realities of travel mean I spend some time thinking about the things airlines do well or poorly. Nonetheless, I appreciate the benefits more than the frustrations.
As you probably noticed, Airline Status means much less today than it used to (which is why it feels even more important to get). Every week, the airlines seem to make the space between seats smaller while the time it takes to find overhead luggage space gets shorter. It seems like most airlines could change its slogan to "We are not happy until you're not happy."
Yet the planes themselves are getting better. Here, for example, is what an empty 787 looks like.
It looks more like a set from Star Trek than the hellscape passengers complain about regularly.
If you really don't like commercial flying, you can fly on any of the "economical" private options like JetSmarter or WheelsUp. Or, better yet, you could be like this guy and buy the world's only private Boeing 787 Dreamliner.
This lighthearted post has something to do with artificial ... but nothing to do with artificial intelligence.
While doing my weekly reading and web browsing (which is how I pick those links you probably think an algorithm selects), I happened upon a post about Michael Jackson on Twitter (now called X), and I enjoyed it (or at least was drawn to click and watch it).
It's funny to look back on, but Zach used to dance to Michael Jackson's songs on stage at his Elementary School talent shows or at random restaurants. There was no choreography ... but lots of movement. I still smile when I think about it.
You might smile (or shake your head) while watching this short video chronicling the evolution of Michael Jackson's face changes from birth to death.
It's a staggering difference. I won't pretend to know what led him to make the changes, but they're substantial.
That being said, his music is both timely and timeless - which is very rare. He managed to make music in each era that fit in with the times but still felt very Michael Jackson.
March Madness is in full swing and will have the world's attention for a few more days. As you can guess, almost no one has a perfect bracket anymore. Yale beat Auburn, James Madison beat Wisconsin, Michigan State beat Mississippi State, and by the end of day 1, only 2,000 brackets remained intact. That's .008% of all brackets submitted.
Before 24/7 sports channels, people watched the weekly show "The Wide World of Sports." Its opening theme promised "The thrill of victory and the agony of defeat!" and "The human drama of athletic competition." That defines March Madness.
The holy grail is mighty elusive in March Madness (as in most things). For example, the odds of getting the perfect bracket are 1 in 9,223,372,036,854,775,808 (2.4 trillion based on a Duke Mathematician's formula that takes into account rank). It's easier to win back-to-back lotteries than picking a perfect bracket. Nonetheless, I bet you felt pretty good when you filled out your bracket.
In 2018, it was estimated that March Madness generated $10 Billion in gambling (twice as much as the Super Bowl)
Feeding the Madness
"Not only is there more to life than basketball, there's a lot more to basketball than basketball." - Phil Jackson
In 2017, I highlighted three people who were (semi) successful at predicting March Madness: a 13-year-old who used a mix of guesswork and preferences, a 47-year-old English woman who used algorithms and data science (despite not knowing the game), and a 70-year-old bookie who had his finger on the pulse of the betting world. None of them had the same success even a year later.
Finding an edge is hard - Maintaining an edge is even harder.
That's not to say there aren't edges to be found.
Bracket-choosing mimics the way investors pick trades or allocate assets. Some people use gut feelings, some base their decisions on current and historical performance, and some use predictive models. You've got different inputs, weights, and miscellaneous factors influencing your decision. That makes you feel powerful. But knowing the history, their ranks, etc., can help make an educated guess, and they can also lead you astray.
The allure of March Madness is the same as gambling or trading. As sports fans, it's easy to believe we know something the layman doesn't. We want the bragging rights of that sleeper pick, of our alma mater winning, of the big upset.
You'd think an NCAA analyst might have a better shot at a perfect bracket than your grandma or musical-loving co-worker.
In reality, several of the highest-ranked brackets every year are guesses.
The commonality in all decisions is that we are biased. Bias is inherent to the process because there isn't a clear-cut answer. We don't know who will win or what makes a perfect prediction.
Think about it from a market efficiency standpoint. People make decisions based on many factors — sometimes irrational ones — which can create inefficiencies and complexities. It can be hard to find those inefficiencies and capitalize on them, but they're there to be found.
In trading, AI and advanced math help remove biases and identify inefficiencies humans miss.
Can machine learning also help in March Madness?
“The greater the uncertainty, the bigger the gap between what you can measure and what matters, the more you should watch out for overfitting - that is, the more you should prefer simplicity” - Tom Griffiths
That being said, people have tried before with mediocre success. It's hard to overcome the intangibles of sports - hustle, the crowd, momentum - and it's hard to overcome 1 in 9.2 quintillion odds.
Two lessons can be learned from this:
People aren't as good at prediction as they predict they are.
Machine Learning isn't a one-size-fits-all answer to all your problems.
In January, Elon took to Twitter and announced that the first human recipient had received an implant and was showing promising neuron spike detection.
Neuralink designed PRIME to record and transmit neural data to interpret brain activity into movement intention. The PRIME Brain-Computer Interface empowers disabled individuals by enabling them to communicate and engage with the world in innovative and impactful ways, such as regaining the ability to speak and interact with others. In the future, advancements in the PRIME Brain-Computer Interface could even assist individuals with spinal cord injuries learn to walk again.
The first patient was 29-year-old Noland Arbaugh, a complete quadriplegic who had lost sensation and suffered paralysis from below the shoulders after sustaining a spinal injury during a diving accident eight years ago.
When we first began receiving updates about him, we were excited to hear that he could use a computer cursor. That was a big step ... and the start of many others. Now, we're being told that he recently used the technology to stay up all night playing a video game called Civilization 6.
Similarly, in 2022, a completely paralyzed man used his brand-new brain implant to ask his caregivers for a beer.
It sounds like a joke, but these are the types of stories that make me optimistic. Both examples highlight a new capability ... but also a deeper purpose, freeing the human to enjoy being human and enhance the quality of their life.
This is a great reminder. Media coverage often focuses on the fear of an increasingly tech-driven world, and what it means for humanity ... but the best uses of technology allow us to be more human.
What used to be science fiction is becoming reality, and possibilities are becoming inevitabilities.
In context, large impact refers to full automation or significant alteration. Small impact refers to less disruptive changes.
IT and finance have the highest share of tasks expected to be "largely" impacted by AI ... which is unsurprising.
We've also already seen the impact of LLM and generative AI on customer service and customer care. As these tools improve, more cases will be able to be fully handled by AI.
This chart isn't meant to make you feel afraid that your industry will be automated—it's meant to help you understand what tasks you should consider automating.
It's a common theme in entrepreneurial discussion these days ... AI is coming for your jobs.
The more nuanced statement is that AI isn't going to take your job - but someone using AI better might.
Recently, Andrej Karpathy, ex-director of AI at Tesla and founding member of Open AI, posted a great tweet about how software engineering will be automated. He compared it to automated driving.
With automated driving:
first, the human performs all driving actions manually
then, the AI helps keep the lane
then, it slows for the car ahead
then, it also does lane changes and takes forks
then, it also stops at signs/lights and takes turns
eventually, you take a feature-complete solution and grind on the quality until you achieve full self-driving.
The progression is similar for software engineering (and, you guessed it, your business as well)
first, the human writes the code manually
then, GitHub Copilot autocompletes a few lines
then, ChatGPT writes chunks of code
then, you move to larger and larger code diffs
then, a tool starts coordinating other tools (a terminal, browser, code editor, etc.)
You get the point. Human oversight begins to move towards increasingly higher levels of abstraction and management.
If you think about it, this parallels a pretty generic path that a typical employee might take in your business. A junior employee can't handle any ambiguity. As they move up, a mid-level employee can probably handle some mild ambiguity ... they need to know where they're headed, but they don't need hand-holding on how to implement it. A senior employee needs to know what problems they need to tackle, and then you get to entrepreneurs, and they don't even need to know what problems to tackle ... they'll find some.
This suggests a pretty solid modus operandi for the coming years. If you're worried about being replaceable, focus on higher-level behaviors.
AI empowers businesses to do more with less. Early adopters of AI will gain a significant competitive advantage by automating tasks, enhancing customer experiences with personalized recommendations, and making data-driven decisions that lead to cost savings and increased revenue. Integrating AI into your business will propel your organization forward by unlocking new levels of efficiency, effectiveness, and certainty. If you're steering the ship, you don't need to be as afraid of the waves.
Here is a framework I created to identify the path to some not-so-easy wins that lead to sustainable business growth and progress:
Create Process Playbooks that leverage automation and AI to help businesses exceed standards both front-stage and backstage. This class of solutions improves practical and business outcomes and helps avoid errors, omissions, and discretionary mistakes.
Use Outcome Integrity Trackers to log decisions, actions, and results, hopefully improving and standardizing processes and outcomes. This capability will evolve into the ability to measure the difference between skill and luck reliably and to the creation of accurate recommendation engines with real-time expectancy scoring.
Capture, Calculate, and Curate Custom Metrics. Much of what happens each day is lost. Finding a way to save this data creates, expands, and augments a valuable new asset that is valuable itself, helps solve complex problems, and leads to new products, services, and solutions.
Curate a Single Integrated Source of Trusted Data that is accurate, complete, and up-to-date. Together, that data becomes the foundation for building new models, metrics, validations, certification, and compliance solutions.
Developing a Comprehensive AI Strategy is Crucial for Business Success
Businesses that don't adapt to changing landscapes fail. Having a roadmap, centered on what doesn't change is a reliable life support. Change doesn't have to be dramatic to be valuable. Just by taking these little steps and asking the right questions, you can make a big impact. I hope you're finding way to reap the rewards of these transformations, not just surviving them.
Just because something is overhyped doesn’t mean it’s bad.
Gartner’s Hype Cycle is a great example of this concept. It highlights the likely cycle of inflated expectations, disillusionment, and, ultimately, utility.
The key takeaway from the Hype Cycle model is that much of what happens is predictable ... and that a significant portion of the extreme swings are based on human nature rather than technical merit.
Haters are going to hate, and sometimes a fad is more than a fad. For example, here is a front-page article from the New York Times in 1879. It questions the utility of electric lights as a replacement for gas-powered lighting. In case you were wondering, that one might have been a bright idea.
The point is that humans have proven themselves to be pretty bad at exponential thinking. We’re not bad at recognizing periods of inflection, but we often have trouble recognizing the consequences of the change (and the consequences of those consequences) and predicting who the winners and losers will be as a result of those regime changes.
There are countless examples. Here’s a funny one from Maximum PC Magazine in 2008. It shows that hype isn’t always a sign of mistaken excess. This list purported to show things that were getting too much attention in 2008. Instead of being a list of has-beens and failures, many of these things rightfully deserved the attention and hype they were getting.
It’s been over 15 years since this came out. How did the predictions hold up?
Apple has become one of the world’s biggest and most successful companies (with a market cap approaching 3 Trillion dollars). The iPhone has sold over 2.2 billion phones and accounts for over half of Apple’s total revenue. Meanwhile, Facebook has become Meta and is also one of the biggest and most successful companies in the world (with a market cap of well over a Trillion dollars). And the list keeps going: HD video, 64-bit computing, downloading movies from the internet, and multiple GPU video cards.
Take just that last one. Nvidia has been the primary beneficiary of GPU growth, and it is one of the highest-performing stocks of the past few decades (with a market cap of well over 2 trillion dollars).
It’s hard to believe how poorly this image aged.
Remember that the trend is your friend while it continues.
Just because something is overhyped - doesn’t mean you shouldn’t be excited about it.
The key is to stop thinking about the thing that’s being hyped and, instead, to start thinking about how to use things like that to create what you really want.
Recently, I've been thinking a lot about how businesses scale and technology adoption accelerates.
For example, consider how fast AI is improving and transforming business.
Last year, I shared a short video called Speed Matters. It includes some thought-provoking ideas. You can view it by clicking here.
While speed matters, faster is not always better.
As you focus on doing more things faster, it becomes more essential to give people room to do important things slower.
You've almost certainly heard the phrase, "It's better to measure twice and cut once." It's much easier to do something the right way from the beginning rather than trying to fix it after you mess it up.
Activity does not create progress if it doesn't move you in the right direction.
This reminds me of a distinction my friend Nic Peterson makes. What you want in your business is velocity rather than speed. Velocity implies a vector (and preferably only one vector). You want to move fast in the desired direction, not fast towards distractions, mistakes, or money down the drain.
To add one more layer to this, there's an "almost true" axiom in technology: you can only have two out of these three things: a project done fast, done right, and done cheaply.
But it's only almost true for a big reason. If you've already built the team and put in the work, replicating that work for new projects can be done fast, right, and comparatively cheaper.
Are you just moving fast, or do you have velocity in your business?
The Real Business We're All In ...
I’ve given a few speeches recently and have new subscribers to our weekly commentary (click here to sign up), so I thought it was a good time to write about the importance of data.
I revisit this topic about once a year because it’s important.
The Hidden Engine: Why Data Fuels Innovation
Technology and innovation are popular topics, but people often ignore what makes it all possible ... the hidden foundation, data.
Data is the lifeblood of modern businesses and the fastest-growing resource we have.
The quest to find and use data has created a modern-day “Wild West.” While AI is often positioned as a “Gold Rush,” data is the precious resource powering the race.
Another way to look at it is that data is the ammunition used by today’s tech titans in their battle for dominance.
In either case, it is easy to see that data is a scarce and valuable resource.
The Data Deluge: Finding Signal in the Noise
We’re living in an age of data explosion. Every day, a staggering 328.77 million terabytes of data are created, amounting to an estimated 120 zettabytes of new data by year’s end.
Video is a significant driver, but so is the Internet of Things, which is growing more than 15% annually. There are now almost 20 billion connected devices, and that number will continue to grow.
This rapid growth presents a challenge. Tech giants like Alphabet, Amazon, Apple, Facebook, and Microsoft all hold unprecedented data troves, creating a race for ownership and control. Regulations struggle to keep pace with this digital stampede.
Rapid growth means little time to create adequate rules. Everyone’s jumping to own more data than the next and to protect that data from prying eyes.
As a great example of this, I often warn people to keep their intellectual property off of ChatGPT or other hosted language models.
But here’s the real concern: Are we losing sight of the signal in all this noise? Just having vast amounts of data isn’t enough. The true value comes from extracting meaningful insights – the nuggets of gold buried within the data avalanche.
To Do The Impossible, Make The Invisible Visible
Collecting basic data and using basic analytics used to be enough, but it is not anymore. The game is changing.
I also see it trading, but it’s pervasive in every industry and our personal lives as well.
For example, traders used to focus on price data ... but there has been an influx of firms using alternative data sets and extraordinary hardware and software investments to find an edge. If you’re using the same data sources as your competitors and competing on the same set of beliefs, it’s hard to find a sustainable edge.
Understanding the game others are playing (and its rules) is important. However, that’s only table stakes.
Figuring out where you can find extra insight or where you can make the invisible visible creates a moat between you and your competition and lets you play your own game.
Here is a quick high-level video about Data as fuel for your business. Check it out.
It is interesting to think about what’s driving the new world (of trading, technology, AI, etc.), which often involves identifying what drove the old world.
Decoding the New World: Data as the Catalyst
Understanding today’s driving forces – like AI – often involves examining what propelled past eras.
History has a way of repeating itself. Even when it doesn’t repeat itself, it often rhymes.
Before e-mails, fax machines were amazing. Before cars, people were happy with horses and buggies.
The key to unlocking new economic realities lies in fresh perspectives.
In this new world, new or better data is often the game-changer. It’s the alternative dataset that allows us to approach challenges and opportunities from entirely new angles.
Before data analytics, businesses relied on intuition and limited information. Now, data empowers us to see patterns and make data-driven decisions, propelling innovation at an unprecedented pace.
These comparisons help explain the importance of data in today’s new world economics.
One of the more recent shifts is in the value of synthetic data.
via Gartner
Synthetic data can mimic the statistical properties of real-world data, making it useful for a variety of purposes.
For example, synthetic data can be used to train machine learning models when collecting traditional data is impractical or presents privacy concerns. It is also used in various other applications, such as data privacy, testing and development, data augmentation, simulation and modeling, risk assessment and management, and enhancing data quality.
You don’t have a competitive advantage if you use the same data and the same process as other people. That’s why understanding how to recognize and capture synthetic data is important. It can shift your perspective, add dimensionality, help you solve different problems, and create transformative results.
While data is the foundation, it’s about transforming your data into actionable insights.
By identifying your real business, the KPIs of success, and what data you’re underutilizing, you can massively improve the efficiency and effectiveness of your business and create new products that transform your future.
In a word of caution, there are two common mistakes people make when making data-driven decisions. First, people often become slaves to the data, losing focus on the bigger picture. This is the same mistake people make with AI. Both are tools, not the end goal. Second, even the most insightful data can’t predict black swans. It’s essential to exercise caution and prepare for the unexpected.
The future of data is bright, but it’s also littered with potential challenges. Privacy concerns and data misuse are hot-button topics, as are fake news and the ability of systems to generate misleading data. In addition, as we gain access to more data, our ability to separate signal from noise becomes more important.
One of the biggest problems facing our youth—and really all of us—is how much information is thrust at us every waking moment of the day. No previous generation has had this much access to data. As a result, many are actually less informed than in the past. Soundbites become the entire news story, and nuance gets lost in the echo chambers.
The question becomes, how do you capitalize on data without becoming a victim of it?
Food for thought!
Posted at 09:28 PM in Business, Current Affairs, Gadgets, Ideas, Market Commentary, Science, Trading Tools, Web/Tech | Permalink | Comments (0)
Reblog (0)