Avoiding Survivorship Bias in On-Chain Models

Definition

Survivorship bias is a cognitive shortcut and a statistical error that occurs when a dataset or analysis focuses exclusively on successful outcomes or entities, while ignoring those that failed, disappeared, or were otherwise excluded from the observation. In financial markets, this typically means evaluating the performance of existing investments without considering those that have ceased to exist, leading to an overestimation of historical performance and an overly optimistic view of market health or strategy efficacy.

Survivorship bias is the tendency to evaluate the performance of existing investments without considering those that have failed, leading to an overestimation of historical performance.

Key Takeaway

Ignoring the full historical universe of assets, including those that have failed or been delisted, inevitably leads to an inflated perception of past performance for any trading strategy or investment thesis. This is particularly dangerous in the rapidly evolving and often unforgiving cryptocurrency market, where projects can emerge and disappear with unprecedented speed. A strategy backtested solely on currently active assets will almost certainly show better results than it would have achieved in reality, as it implicitly removes all the 'losers' from its historical sample.

Mechanics

Survivorship bias manifests in on-chain models primarily through the selective construction of the tradable universe. When researchers or traders build a dataset for backtesting or analysis, they often inadvertently include only assets, protocols, or addresses that are still active or relevant at the time of the analysis. This creates a distorted historical view where:

1. Token Universe: If a backtest considers only the top 100 cryptocurrencies by market capitalization today, it automatically excludes countless tokens that were once prominent but have since failed, been delisted from exchanges, or become entirely inactive. A strategy that might have traded a now-defunct token like LUNA (pre-collapse) or various early ICOs would appear to have avoided these failures if the backtest only includes current survivors.
2. Protocol Selection: On-chain models often analyze specific DeFi protocols, NFTs, or DApps. If the analysis focuses solely on protocols that are currently successful and liquid, it ignores the numerous projects that launched, gained initial traction, and then faded into obscurity or suffered critical exploits. For instance, a model analyzing liquidity pools might only consider those on established platforms like Uniswap or Curve, overlooking the many smaller, riskier pools that collapsed or were rug-pulled.
3. Address Activity: When analyzing wallet behavior or smart contract interactions, focusing only on currently active or 'whale' addresses can also introduce bias. Historical analysis of address clusters might miss the patterns of addresses that were once significant but have since become inactive due to loss of funds, project abandonment, or simply exiting the market. This can lead to an incomplete understanding of historical market dynamics and participant behavior.

The core issue is that the market universe used for historical analysis does not accurately reflect the market universe that existed at each specific point in time in the past. This leads to an artificial inflation of strategy performance because all the negative outcomes associated with failed assets are systematically removed from the historical simulation.

Trading Relevance

For quantitative traders and researchers developing on-chain models, understanding and mitigating survivorship bias is paramount. Its presence can lead to severely misleading conclusions about a strategy's viability and profitability. A strategy that appears highly profitable in a backtest might perform poorly, or even catastrophically, in live trading due to this bias.

Firstly, inflated backtest results are the most direct consequence. A strategy might show impressive returns and low drawdowns historically, but these figures are artificially boosted because the strategy is implicitly assumed to have avoided all the failed assets. This creates a false sense of security and can lead to overconfidence in a flawed model. Secondly, misleading risk assessment is a significant danger. If a strategy's historical volatility and maximum drawdown are calculated based on a universe of only surviving assets, the true risk profile is underestimated. The actual market, with its inherent failures and black swan events, is far riskier than the biased historical data suggests. This can result in traders taking on more risk than intended, leading to substantial capital losses.

Furthermore, survivorship bias can hinder the generalization of a strategy to future market conditions. A model trained on a biased dataset may identify patterns that are specific to successful assets, rather than robust market dynamics. When deployed in a live environment, where new projects emerge and old ones fail constantly, the strategy may struggle to adapt or perform as expected. Ultimately, the goal of on-chain modeling is to identify robust, repeatable edges. Survivorship bias undermines this goal by presenting an idealized, rather than realistic, historical landscape.

Risks

The risks associated with survivorship bias in on-chain models extend beyond mere statistical inaccuracies, impacting capital allocation, psychological confidence, and overall trading success.

1. Overestimation of Alpha and Returns: The most immediate risk is believing a strategy generates more alpha or higher returns than it actually does. This can lead to unrealistic profit expectations and poor investment decisions, as capital is allocated based on an unachievable historical performance.
2. Underestimation of Drawdown and Volatility: By excluding assets that experienced significant price declines or complete failure, the historical drawdown and volatility metrics of a strategy will appear lower than they truly were. This can lead to insufficient risk management, inadequate position sizing, and a higher probability of catastrophic losses during adverse market conditions.
3. False Confidence and Psychological Impact: Traders and investors may develop a false sense of confidence in their models, leading them to take on excessive leverage or make larger bets. When the strategy inevitably underperforms in live trading, it can lead to significant psychological distress, doubt, and potentially impulsive decisions to abandon a potentially sound strategy prematurely or, conversely, to double down on a flawed one.
4. Suboptimal Capital Allocation: Resources, whether computational power for data processing or actual trading capital, might be misallocated to strategies that are not genuinely robust. This diverts capital from potentially more effective strategies or safer investments, hindering overall portfolio growth.
5. Systemic Risk in Ecosystems: If a significant portion of the market relies on models affected by survivorship bias, it could contribute to systemic instability. A collective misjudgment of risk across many participants could lead to cascading failures when market conditions deviate from the biased historical expectations.

In the crypto space, these risks are amplified by the market's inherent volatility, rapid innovation cycles, and the prevalence of projects with short lifespans. Rug pulls, smart contract exploits, project abandonment, and exchange delistings are common occurrences that, if ignored in historical data, paint an unrealistically rosy picture of potential returns.

History and Examples

Survivorship bias is not unique to crypto; it has a long history in traditional finance. A classic example involves mutual funds: if one only analyzes the performance of funds that exist today, the average returns will appear significantly higher than if one includes the returns of funds that have closed or merged due to poor performance. Similarly, stock market indices that periodically remove underperforming companies and add new, successful ones can also exhibit a form of survivorship bias if not properly adjusted.

In the context of on-chain models, specific examples abound:

• The ICO Boom and Bust (2017-2018): Many on-chain models developed during or after this period might focus on the few ICOs that survived and thrived (e.g., Ethereum, Binance Coin) while ignoring the thousands that failed, were scams, or simply faded away. A backtest that only includes the 'winners' would drastically overstate the profitability of an ICO investment strategy.
• DeFi Summer and Subsequent Crashes (2020-2022): The rapid growth of decentralized finance saw countless new protocols emerge. Many offered high yields and innovative features, but a significant number suffered exploits, rug pulls, or simply couldn't sustain their models. If an on-chain model for yield farming or liquidity provision only considers currently successful protocols like Aave or Compound, it ignores the historical risks associated with projects like Iron Finance (TITAN) or various algorithmic stablecoins that de-pegged and collapsed.
• NFT Projects: The NFT market has seen explosive growth and subsequent corrections. An on-chain analysis of NFT trading strategies that only includes successful collections like CryptoPunks or Bored Ape Yacht Club, while omitting the vast majority of projects that saw little to no trading volume or value, would be heavily biased. The true historical performance of an NFT trading strategy must account for the numerous failed or illiquid collections.
• Exchange Delistings: Tokens are frequently delisted from exchanges due to low volume, regulatory concerns, or project failure. If a backtest uses data from a major exchange but doesn't account for tokens that were delisted during the historical period, it implicitly assumes the strategy would have continued to hold or trade these tokens without issue, which is unrealistic.

These examples highlight that to accurately assess historical performance, one must reconstruct the tradable universe as it existed at each point in time, including all assets that were available for trading, regardless of their eventual fate.

Common Misunderstandings

Several misconceptions often prevent traders and researchers from fully addressing survivorship bias in their on-chain models.

One common misunderstanding is the belief that survivorship bias only applies to tokens that went to zero. While tokens collapsing to zero are the most dramatic examples, the bias also arises from excluding assets that were merely delisted, became illiquid, or simply underperformed to the point of being ignored in current analyses. A token doesn't need to be completely worthless to introduce bias; its removal from the active universe is enough to skew historical performance metrics. Similarly, protocols that are still technically operational but have lost significant user base or TVL (Total Value Locked) might be implicitly excluded from a 'top protocols' list, leading to bias.

Another frequent misconception is that data providers automatically handle survivorship bias. While some high-quality data providers offer historical universe construction, it's not a universal feature, especially for the granular, rapidly changing data required for on-chain analysis. Many data feeds simply provide current active symbols or historical data for currently listed assets. It is the responsibility of the researcher to explicitly request or construct a comprehensive historical universe that includes delisted or failed entities. Relying solely on a data provider without understanding their methodology can perpetuate the bias.

Finally, some believe that survivorship bias is negligible in short-term trading strategies or when dealing with highly liquid assets. While its impact might be less pronounced over very short timeframes, it still affects the historical context and the selection of the asset universe. Even for highly liquid assets, the composition of the top-tier assets changes over time. A short-term strategy backtested on the current top 10 assets might still miss the historical volatility and eventual decline of an asset that was once in the top 10 but has since fallen out of favor or failed. The principle remains: any historical analysis that does not account for the full, time-varying universe of tradable assets is susceptible to this bias.

Summary

Survivorship bias represents a significant challenge in the development and backtesting of on-chain trading models, particularly within the dynamic and volatile cryptocurrency landscape. It arises from the tendency to focus solely on successful or currently existing assets, protocols, or addresses, thereby ignoring those that have failed, been delisted, or become inactive. This selective observation leads to an overestimation of historical strategy performance, an underestimation of true risk, and ultimately, the creation of models that are not robust enough for live trading environments.

To effectively mitigate survivorship bias, it is imperative for quantitative traders and researchers to meticulously reconstruct the historical tradable universe. This involves actively seeking out and incorporating data for all assets that were available at any given point in time, regardless of their subsequent fate. By including the 'losers' alongside the 'winners' in backtests, models can achieve a far more realistic and reliable assessment of their historical efficacy, leading to more informed decision-making and more resilient trading strategies in the complex world of on-chain finance. Ignoring this bias is akin to navigating a minefield while only looking at the paths where others have successfully walked, oblivious to the many paths where they failed.