I’ll admit, I’ve said it before: SEO is simple (content, links, technical). <\/p>\n\n\n\n
But it\u2019s also a deeply complex system of interactions between publishers and their competition, users and their evolving behavior, and an intermediary directory with an evolving set of algorithms, machine learning, personalization, and AI features.<\/p>\n\n\n\n
What we want to achieve is straightforward: we want to be visible in SERPs to connect with our target customers and convince them to consider our product or solution. <\/p>\n\n\n\n
To do that, we must predict and approximate many factors (probably an endless amount), from who is searching for a given keyword, how many people, if that query is growing in popularity or shrinking, how we can best position ourselves to answer that qwerty, which factors Google (and now Perplexity and ChatGPT) deem most important to award us with that visibility. <\/p>\n\n\n\n
It\u2019s a complex system, but paradoxically, if we get bogged down in the complexity<\/a>, we risk missing the forest for the millions of potential trees.<\/p>\n\n\n\n Let\u2019s look at a familiar example: nutrition science. <\/p>\n\n\n\n We\u2019ve got mechanistic knowledge\u2014understanding how certain nutrients should work in the body. But what happens when we study them in the real world? The body\u2019s complexity, with its checks and balances, interaction effects, and unknowns, makes it hard to translate this knowledge into measurable outcomes. Still, we lean on correlations from observational studies and randomized controlled trials (RCTs) to make decisions. We also have cultural wisdom from our grandmothers that is usually on point. <\/p>\n\n\n\n Every few years a supplement seems to come out promising longevity (notoriously difficult to study in humans) because of its mechanistic and theoretical application. Resveratrol comes to mind. Lots of promise, but little evidence when it comes to human data<\/a> or randomized controlled trials. <\/p>\n\n\n\n As they say, \u201cIn theory, theory and practice are the same. In practice, they are not.\u201d<\/p>\n\n\n\n In SEO, we face similar challenges. Take backlinks, for example. The original Google patent emphasized<\/a> the quantity and quality of links to a site, but how do you measure the causal impact of a single backlink today? How much does one link<\/em> actually contribute to your rankings, and how much of it is just background noise in the larger system?<\/p>\n\n\n\n We know some things work because we\u2019ve seen patterns over time (think of these as our observational studies), and we\u2019ve got some micro-optimizations we can test directly (like A\/B tests on title tags). But even with all this, we often rely on faith in the system\u2014understanding mechanistically<\/em> how backlinks or technical SEO should factor into rankings, but lacking precise numbers to back up each action.<\/p>\n\n\n\n So, how do you know what matters?<\/p>\n\n\n\n In both science and SEO, not all evidence is created equal.<\/p>\n\n\n\n When you\u2019re making decisions\u2014especially in areas where the exact causal impact is hard to measure\u2014you need to evaluate the strength of your evidence. <\/p>\n\n\n\n This is where the hierarchy of evidence, <\/strong>which I learned about from Ronny Kohavi<\/a>, comes in.<\/p>\n\n\n\n It\u2019s a pyramid that ranks types of evidence based on their reliability, from the strongest forms at the top to the weakest at the bottom.<\/p>\n\n\n\n Image Source<\/a><\/p>\n\n\n\n At the top of this pyramid, you have randomized controlled trials (RCTs)<\/strong> and meta-analyses<\/strong>.<\/p>\n\n\n\n These are the gold standard for establishing causality because they control for variables and aggregate results across multiple studies to find patterns.<\/p>\n\n\n\n Something I deeply miss from my career in growth and experimentation was the relatively cleanability to parse out elements in A\/B tests and quantify their effects. While we can do this in some ways with SEO (A\/B testing title tags, for example), there\u2019s a lot more noise.\u00a0<\/p>\n\n\n\n There are also non-randomized experiments we can lean on. <\/p>\n\n\n\n I\u2019ll give you a counterintuitive example from our client portfolio. <\/p>\n\n\n\n We worked with an extremely large global brand with billions in revenue and the highest domain authority you could ask for. <\/p>\n\n\n\n One of the projects they asked us to help them with was to generate a large amount of high quality backlinks to a few key pages around generative AI. <\/p>\n\n\n\n I was skeptical, if not outright doubtful, about the impact of this engagement.<\/p>\n\n\n\n But they had budget, we had the resources and team in place, and their executives, for some odd reason, deeply cared about backlinks as a success metrics for their organic growth team. <\/p>\n\n\n\n So we took it on, driving ~80 links to 4 key pages from November 2023 to March 2024. It was an isolated effort \u2013 no technical pages, nothing on-page done, no new content or internal links, and it does seem like it moved the needle.<\/p>\n\n\n\n Again, this is non-randomized, so we have many underlying invisible variables that could have impacted this (user behavior signals, non-SEO campaigns, or merely time itself). But it\u2019s a decent if not relatively weak signal that links matter, even at the highest peaks of brand and domain authority. <\/p>\n\n\n\n Next are observational studies<\/strong>, which, while not as robust as RCTs, help identify correlations by examining real-world data. In SEO, this might be the equivalent of analyzing broad industry trends to understand how tactics like backlinking, site structure, or keyword placement influence rankings. Luckily our industry is passionate about collecting this sort of industry data and sharing patterns with the rest of us (e.g. Lily Ray<\/a>, Kevin Indig<\/a>, etc.) <\/p>\n\n\n\n Lower on the pyramid are case studies<\/strong> and anecdotal evidence<\/strong>. These might include single-site SEO experiments or anecdotal reports from industry professionals. While valuable for generating hypotheses, these forms of evidence aren\u2019t enough to bet the farm on. Just like in nutrition science, where a single success story about a diet might be inspiring, it doesn\u2019t mean the results are replicable for everyone.<\/p>\n\n\n\n For instance, my hyperscaling of affiliate-based content<\/a> on my personal website resulting in a big crash is an interesting data point. But it\u2019s much more interesting when we can detect patterns from many sites who have done the same thing (which it does appear is starting to emerge with regards to mass-generated unhelpful content).<\/p>\n\n\n\n The key takeaway: The stronger the evidence, the more confidently you can move forward with your strategy.<\/strong> Weaker forms of evidence can be useful for guiding hypotheses and decisions, but you should weigh them against the risk and potential upside. <\/p>\n\n\n\n For instance, while case studies and anecdotal reports about link-building strategies might point in a certain direction, they need to be backed up by more reliable data before you pour resources into them.<\/p>\n\n\n\n When operating in a complex system like SEO, the goal is often to balance risk and reward<\/strong>. If you\u2019re unsure about the value of a specific action, ask yourself two key questions: 1) How strong is the evidence? and 2) What\u2019s the downside risk if the action<\/a> doesn\u2019t pan out?<\/p>\n\n\n\n Take something like a coldplunge in health and wellness. While there isn\u2019t conclusive evidence that it offers huge benefits<\/a>, the downside is minimal and northern Europeans have been doing this forever, so the \u201cdo no harm\u201d principle suggests it\u2019s worth trying if you\u2019re interested. But if you don\u2019t like the feeling of an ice bath, you don\u2019t need to feel any FOMO just because every health influencer posts selfies doing them. <\/p>\n\n\n\n The same can be applied to SEO strategies. Testing creative copywriting in your title tags may not yield dramatic results, but it also won\u2019t \u201charm\u201d your site in any meaningful way. It’s also an easily reversible decision – a two-way door.<\/a> It\u2019s a low-risk, medium-reward scenario that\u2019s worth experimenting with.<\/p>\n\n\n\n On the other hand, let\u2019s consider more risky bets, like buying cheap backlinks from a questionable source<\/strong>. Sure, there\u2019s anecdotal evidence that buying links can lead to short-term boosts in rankings, but there\u2019s also significant downside risk\u2014a potential sitewide penalty from Google.<\/p>\n\n\n\n Here, the weak evidence combined with the high downside means this is a risky bet you\u2019re better off avoiding.<\/p>\n\n\n\n In our running nutrition science analogy, all too often I see health influencers making potential risky suggestions with little likely upside. <\/p>\n\n\n\n That\u2019s been my contention with Andrew Huberman, who has recommended taking Fadogia Agrestis (which\u00a0has been linked<\/a>\u00a0to testicular toxicity) and Tongkat Ali, which appears to activates Luteinizing Hormones (LH) that stimulates the Leydig cells to produce testosterone.<\/p>\n\n\n\n However, the few human studies available have shown these to be, for the most part, clinically meaningless<\/a>. Not only could they have potential side effects (always possible when you mess with complex systems like hormones), but quality control for these supplements is notoriously terrible, so you have no idea the quality of the supplement you\u2019re taking. <\/p>\n\n\n\n So little tangible upside, with mostly downside potential. Count me out. Not a risk vs reward tradeoff that I like. I’ll stick with diet and exercise and the occasional sauna. <\/p>\n\n\n\n In essence, “do no harm” bets are those where the upside is uncertain, but the downside is limited.<\/strong> These are often worth pursuing in SEO because the potential reward outweighs the minimal risk. Conversely, when you encounter tactics with weak evidence and substantial risks, it’s often best to steer clear.<\/p>\n\n\n\n One of the biggest traps in SEO\u2014and in any complex system\u2014is trying to achieve perfect precision in your models. <\/p>\n\n\n\n But here\u2019s the truth: All models are approximations of reality.<\/strong> The point isn\u2019t to model every single factor with 100% accuracy; the point is to make the model useful enough to guide your decisions.<\/p>\n\n\n\n For example, in SEO, we know, pretty broadly, that \u201clinks matter.\u201d<\/strong> <\/p>\n\n\n\n But do we need to know the exact<\/em> causal value of every single backlink? Not really.<\/p>\n\n\n\n It\u2019s more productive to establish key principles based on the available evidence\u2014things like the importance of relevant topic matter, the strength of a site\u2019s domain rating, and the traffic to the linked page.<\/p>\n\n\n\n Notably, Google is likely looking at signals that are \u201chard to fake\u201d or game, meaning it\u2019s unlikely driving a ton of unnatural backlinks will have a large impact without associated brand equity and \u201corganic\u201d link acquisition.<\/p>\n\n\n\n They\u2019re also likely more useful to establish baseline credibility and have decreasing marginal utility after you pass a certain threshold effect. <\/a><\/p>\n\n\n\n These broad strokes – a mixture of mechanistic knowledge with natural experiments and observational studies and a little bit of critical thinking – give us a good enough model to make informed decisions without getting bogged down in minutiae.<\/p>\n\n\n\n Chasing too much precision can actually reduce the utility<\/strong> of your model. The time and resources spent trying to measure the precise impact of every single link would be better spent executing strategies based on well-established principles.<\/p>\n\n\n\n In this sense, the cost of increasing precision<\/strong> can lower the expected value of your model. The goal should be to achieve a balance between accuracy and practicality<\/strong>\u2014enough accuracy to be confident in your decisions, but not so much that you get lost in unnecessary details.<\/p>\n\n\n\n Not to digress too far, but that\u2019s what frustrates me the most about the never-ending attribution modeling debate. It is both obvious and well-known<\/strong> that attribution models are flawed to some extent, but that doesn\u2019t mean they are useless. It simply means you must acknowledge you\u2019ll never have 100% precision and move forward with a model that is both flawed and useful in improving decisions. <\/p>\n\n\n\n In short, you don\u2019t need to model every single element of a system to make it work for you. Use first principles like \u201clinks matter\u201d or \u201ccontent should solve searcher problems\u201d or \u201cour content should map back to our core product\u201d and then move forward.<\/p>\n\n\n\n Perfect precision isn\u2019t necessary for effective decision-making<\/strong>\u2014what matters is having a model that\u2019s useful<\/em> for driving results.<\/p>\n\n\n\n We\u2019re always operating under conditions of uncertainty, and while we have strong evidence of the efficacy of some variables, there are many variables we\u2019re likely not even aware of. That shouldn\u2019t prevent us from taking action and driving outcomes. <\/p>\n\n\n\n This is where I advise companies to establish a well-balanced portfolio of bets, and put more weight behind the actions you have the most confidence in. We use the “barbell strategy,”<\/a> with a high portion of bets being safe and predictable and a small portion being lower evidence and higher potential upside. <\/p>\n\n\n\n SEO is like investing: some actions have a higher expected return than others, but the risk is always there. You can never be sure of the exact causal impact, but you can model your decisions based on the highest confidence predictions.<\/p>\n\n\n\n Here\u2019s a good rule of thumb: if something clearly<\/em> leads to an outcome\u2014like publishing content clearly leading to traffic\u2014it\u2019s a high-confidence bet and more easily modelable. Without content, there\u2019s no traffic, just purely from a logical perspective. <\/p>\n\n\n\n For other actions, like technical SEO changes or backlink strategies, you may have to triangulate industry patterns and observational data, mechanistic understanding of search engines and retrieval, and your own critical thinking to identify the top 5% of actions that will move the needle, ignoring the rest, as you build a high output organic growth system.<\/p>\n\n\n\n <\/p>\n\n\n\n Want more insights like this?\u00a0Subscribe to our\u00a0<\/a>Field\u00a0<\/a>Notes<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" In some sense, this essay is exploring epistemological questions, like…<\/p>\n","protected":false},"author":3,"featured_media":6651,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[184,25],"tags":[],"wf_post_folders":[],"class_list":{"0":"post-5107","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-field-notes","8":"category-seo"},"yoast_head":"\nYou don\u2019t need to understand cell biology to eat well<\/strong><\/h2>\n\n\n\n
The hierarchy of evidence<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/beomniscient.com\/wp-content\/uploads\/image-94.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/beomniscient.com\/wp-content\/uploads\/image-95-1024x423.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/beomniscient.com\/wp-content\/uploads\/image-96-1024x437.png\")
<\/figure>\n\n\n\nDo no harm bets and what\u2019s worth risking<\/strong><\/h2>\n\n\n\n
Utility versus precision tradeoffs<\/strong><\/h2>\n\n\n\n
Building a risk-adjusted portfolio of bets<\/strong><\/h2>\n\n\n\n