Screaming Frog for Competitor Research

What Are Our Competitors Doing/Targeting/Investing In?

At UpBuild, we get asked this a LOT, and it’s not without basis. In order to successfully compete in organic search, we need to ensure we’re outperforming our search results competitors.

Crawl Competitor Sites with Screaming Frog

If you’re an SEO, you’re likely super accustomed to throwing your own site in Screaming Frog, crawling it, checking on site health, optimization, all that good stuff, but have you ever taken a deeper look under the hood of your competitors’ sites? Much of the same benefit you gain from crawling your own site, you can achieve by crawling your competitors’.

What Are Our Competitors Targeting? (Keyword Research)

Export H1 titles, page titles, and meta keywords

Clean them up and identify trends.

This can be the most time-consuming part, but I’ve found huge benefits from it, answering so many questions that a sales/product team has about competitors.

I like using pretty, pretty word clouds to help me simply identify overarching themes within a site’s or multiple sites’ keyword targeting:

Make things extra simple on yourself by assigning topics to specific keyword phrases. Here’s an example of what that may look like:

Keyword Topics Keywords





Chocolate chip cookies

Cupcake recipes from scratch

How to make cupcakes

Buttercream frosting recipe

Cookies recipe for kids

Chocolate cupcake recipes

Easy cupcake recipes

Best chocolate chip cookies recipe

Vanilla cupcake frosting


What Are Our Competitors Writing About? (Content Strategy)

Export h1 tags

The h1 tags should, of course, be the name of the piece of content. It’s as simple as that. However, it is worth noting that this may not always be the case if your competitor has no SEO at all, or is working from an odd template. While a client may be completely familiar with a competitor’s core pages, there are often ‘hidden’ areas of a site that require a bit more digging. I’ve found that these can often be extremely successful organic pages, delving deep into specific topics and providing value to the end user.

If you identify a topic a competitor is writing about, and you think ‘Hey! That sounds something we do!’, ask yourself, ‘Can we do a better job writing about the topic?’ (For heaven’s sake, no plagiarizing.) If the answer is a resounding ‘yes!’ then by all means write! Make it ten times better! Make it more visual! Add a video (with a transcript)! Go above and beyond to make it the best gosh darn piece of content about that topic the internet has ever seen.

Quick Tip: We get asked about content volume and word count pretty frequently. You can see how your competitors are doing for individual pages by going to Internal > Word Count in Screaming Frog. Aim to create a piece of content that’s a better resource than your competitors’; typically, that will mean it’s longer, but if you can say it better in fewer words, go for it.

What Content Types Are Our Competitors Investing In?

This question can be broken down with the above keyword analysis and a look into the page type breakdown.

If they (smartly) have a hierarchical URL structure you can easily take a look at the page breakdown in chunks:

  • Are they investing a ton in resources?
    • /resources/
  • Are they producing more product/demo videos?
    • /resources/videos/
  • How many blog posts a week are they producing?
    • /blog/
  • What new product features are they announcing?
    • /press-releases/

By getting a clearer understanding of what portion of the total site these individual sections make up, we can see where our own site may be lacking, then work to strategize investing more time/money/energy into that vertical.

I must say, in all the times I’ve done it, an in-depth competitor research strategy has always been extremely valuable. No matter how many times I may have visited and clicked around a client’s competitor’s site, I never get more familiar with it than when I’m able to look under the hood with the steadfast help of Screaming Frog.

Written by
A strategic leader on the UpBuild team, Laura’s career includes 10+ years of agency experience working on everything from SEO and the semantic web to analytics and CRO.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *