Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Save big on Moz Pro!
      Limited time offer

      Save big on Moz Pro!

      See pricing
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      Access the Moz API for less
      Save on Moz Data

      Access the Moz API for less

      Hurry - ends Dec 6th!
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Save 20% on all Moz Academy courses
      Limited time offer

      Save 20% on all Moz Academy courses

      Level up your SEO
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Explore Moz updates & save up to 40%
      Big Savings!

      Explore Moz updates & save up to 40%

      See the latest
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Moz Tools
    3. Moz Pro
    4. The Moz.com bot is overloading my server

    Unsolved The Moz.com bot is overloading my server

    Moz Pro
    rogerbot moz crawler
    6
    8
    404
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • paulavervo
      paulavervo Subscriber last edited by

      The Moz.com bot is overloading my server. How to solve it?

      1 Reply Last reply Reply Quote 0
      • alyxbond
        alyxbond last edited by

        For a step-by-step guide on setting up the tool, check out the solara executor download tutorial.

        1 Reply Last reply Reply Quote 0
        • ChotiGolpo
          ChotiGolpo last edited by

          maybe crawl delay will help.

          1 Reply Last reply Reply Quote 0
          • KerryDuggan
            KerryDuggan Staff @paulavervo last edited by

            @paulavervo
            Hi,
            We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
            Please contact us via help@moz.com or https://moz.com/help/contact
            We will be happy to help.
            Cheers,
            Kerry.

            1 Reply Last reply Reply Quote 0
            • Abdul9865
              Abdul9865 last edited by

              very nice brother i like it very good keep it up !

              1 Reply Last reply Reply Quote 0
              • Abdul9865
                Abdul9865 last edited by

                very nice !

                1 Reply Last reply Reply Quote 0
                • paulavervo
                  paulavervo Subscriber last edited by

                  does the moz team even monitor this forum?

                  KerryDuggan 1 Reply Last reply Reply Quote 0
                  • Alex2211
                    Alex2211 Banned last edited by

                    If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your robots.txt file by specifying a crawl delay for the Moz bot using directives like User-agent: rogerbot and User-agent: dotbot, followed by Crawl-delay: 10 to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in your robots.txt file. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your .htaccess file to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.

                    1 Reply Last reply Reply Quote 0
                    • 1 / 1
                    • First post
                      Last post

                    Got a burning SEO question?

                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                    Start my free trial


                    Browse Questions

                    Explore more categories

                    • Moz Tools

                      Chat with the community about the Moz tools.

                    • SEO Tactics

                      Discuss the SEO process with fellow marketers

                    • Community

                      Discuss industry events, jobs, and news!

                    • Digital Marketing

                      Chat about tactics outside of SEO

                    • Research & Trends

                      Dive into research and trends in the search industry.

                    • Support

                      Connect on product support and feature requests.

                    • See all categories

                    Related Questions

                    • awilliams_kingston

                      Unsolved 503 Service Unavailable (temporary?) Rogerbot takes a break

                      rogerbot 503 ideas

                      A lot of my Moz duties seem to be setting hundreds of issues to ignore because my site was getting crawled while under maintenance. Why can't Rogerbot take a break after running into a few of these and then try again later? Is there an official code for Temporary Service Unavailability that can smart bots pause crawls so that they are not wasting compute, bandwidth, crawl budget and my time?

                      Product Support | | awilliams_kingston
                      0
                    • Anderik

                      Unsolved Why is'nt there an update yet!

                      moz crawler

                      ff2a317a-cd2e-4394-9e5a-5c832ff8c13a-image.png

                      Moz Tools | | Anderik
                      0
                    • MOZ.info

                      Meta Tag Descriptions not being found in Moz Crawls

                      Hey guys, I have been managing a few websites and have input them into Moz for crawl reports, etc. For a while I have noticed that we were getting a gratuitous amount of errors when it came to the number of missing meta tags.  It was numbering in the 200's.  The sites were in place before I got here and a lot of the older posts no one had even attempted to include tags, links of the page or anything. As they are all Wordpress Sites and they all already had the Yoast/Wordpress SEO plug-in installed on them, I decided I would go through each post and media file one at a time and update their meta tags via the plug in.  I personally did this so I know that I added and saved each one, however the Moz crawl reports continue to show that we are missing roughly 200 meta tags.  I've seen a huge drop off in 404 errors and stuff since I went through and double checked everything on the sites, however the meta tag errors persist. Is this the case that Moz is not recognizing the tags when it crawls because I used the Yoast Plugin?  Or would you say that the plugin is the issue and I should find another way to add meta tags to the pages and posts on the site?  My main concern is that if Moz is having issues crawling the sites, is Google also seeing the same thing? The URLS include:
                      sundancevacationsblog.com
                      sundancevacationsnews.com
                      sundancevacationscharities.com Any help would be appreciated!

                      Moz Pro | | MOZ.info
                      0
                    • G2W

                      Aren't domain.com/page and domain.com/page/ the same thing?

                      Hi All, A recent Moz scan has turned up quite a few duplicate content notifications, all of which have the same issue. For instance: domain.com/page and domain.com/page/ are listed as duplicates, but I was under the impression that these pages would, in fact, be the same page. Is this even something to bother fixing or a fluke scan? If I should fix it does anyone know of an .htaccess modification that might be used? Thanks!

                      Moz Pro | | G2W
                      0
                    • ChristieC

                      How do retrieve historical traffic data in Moz?

                      I need to find traffic data from the start of my campaign in Moz, how do I do that?  thanks!

                      Moz Pro | | ChristieC
                      0
                    • imageworks-261290

                      SEO moz poor rankings reporting

                      How are we supposed to send ranking reports to clients if the rankings are constantly reporting false metrics This is not the first time this has happened, but the rankings are WAAAAAAY off. What is the point of reporting rankings that are not accurate? What is the possible cause of this. Is there a solution? What is SEOmoz doing to improve their ranking programs. Our client actually ranked #2 for "northern Virginia HVAC contractors" when moz had us not in the top 50! How are we supposed to bank on information that is so consistently unreliable? northernvirginiahvaccontractors.png northernvirginiahvaccontractorsrank.png

                      Moz Pro | | imageworks-261290
                      0
                    • eseyo

                      The value of .uk.com domains

                      It was a long night last night (working, not playing), therefore forgive the potential stupidness of this question... Running an OSE report on a website earlier their top linking domain was *.uk.com. This is a valid TLD, and I know for sure that the website has not acquired links direct from the uk.com website but from some-other-registered-domain.uk.com. What I am asking is does the Moz Calculation and/or the pagerank calculation give more value to these links? *.uk.com has a Domain Authority of 71 and 156,497 linking root domains. If so i'm gonna go buy me a bunch of .uk.com domains..... Opinions please.

                      Moz Pro | | eseyo
                      2
                    • StephenALee

                      SEOMoz site crawlers created an issue for our servers

                      I have set up a number of campaigns with your pro tool. Unfortunately we have 7 sites on our server and our IT dept have said that we had an issue when your site crawlers visited for several sites at the same time - is there any way that I can retain the campaigns but have the sites crawled on request rather than automatically?

                      Moz Pro | | StephenALee
                      0
                    Moz logo
                    • Contact
                    • Community
                    • Free Trial
                    • Terms & Privacy
                    • Accessibility
                    • Jobs
                    • Help
                    • What's New
                    • News & Press
                    • MozCon
                    © 2021 - 2024 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.

                    Looks like your connection to Moz was lost, please wait while we try to reconnect.