Navigation
Archive / robots.txt in a section
This is archived content. Visit our new forum.

Tagged: 

  • Author
    Posts
  • July 2, 2016 at 1:00 pm #1069735
    Scott M
    Spectator

    Hello, How can I make a certain section/content of a page to be NOT CRAWLED by Google?

    For instance, If I have a one page Nav site, I only want certain content to be used for my keywords and such. I want to leave out certain sections that don’t pertain to my keywords.

    Please take your time on this one, as I believe this will be really helpful to have in the forum. I’m assuming I would use a raw content element and add some code, but please be specific on the best practice for this task.

    thanks!

    July 2, 2016 at 10:30 pm #1070023
    Rue Nel
    Moderator

    Hi There,

    Thanks for writing in! Crawlers and bots index the whole page. There is no available directive yet that you can skip one particular section of the page. For more information on how Robots.txt and how exclusion works, please check it out here:
    http://www.robotstxt.org/orig.html
    http://www.robotstxt.org/robotstxt.html
    https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag#abstract

    Hope this explains it.

    July 3, 2016 at 2:26 pm #1070606
    Scott M
    Spectator

    I see, thanks!

    Could you possibly elaborate on how to avoid duplicate content penalties with your one page nav sites?

    For instance, if I start my “about” section with 200 lines of text on my home page, and then have a READ MORE link which takes visitors to a separate About page to continue reading, how shall I avoid duplicate content when I have some of the same text on both pages?

    thanks!

    July 3, 2016 at 8:15 pm #1070902
    Rue Nel
    Moderator

    Hello There,

    Thanks for the updates! Regretfully, this particular inquiry is outside the scope of our support as this is not related to an issue with the theme and instead has to do with your site optimization. As such, you will need to investigate this particular issue on your own or seek help from SEO specialist. If you have any further questions about the theme, we are more than happy to provide you with assistance on these inquiries.

    For further reading about SEO and duplicate contents, these links might help you:
    https://support.google.com/webmasters/answer/66359?hl=en
    https://moz.com/learn/seo/duplicate-content
    https://blog.kissmetrics.com/myths-about-duplicate-content/

    Google’s Matt Cutts: Duplicate Content Won’t Hurt You, Unless It Is Spammy

    Thank you for your understanding.

  • Author
    Posts
  • <script> jQuery(function($){ $("#no-reply-1069735 .bbp-template-notice, .bbp-no-topic .bbp-template-notice").removeClass('bbp-template-notice'); }); </script>