/pro/ is being indexed

[magikweb]

It might be because we use WordPress Multisite, but the /pro/ is being indexed by Google.
What would be the cleanest right way to prevent this? (the modification would edit multiple websites)

It’s weird, because that page requires to be logged in to be accessed.
https://www.magikweb.ca/pro/

Thank you for any suggestion!

[friech]

Hi @magikweb,

Thank you for writing in, If you want to block a specific page from website then you must need to set robots.txt file as below.

User-agent: *
Disallow: /page-URL

Please follow this thread for more details.

Hope it helps,
Cheers!

[magikweb]

Hello @friech,
WordPress Multisite does not have a physical robots.txt you can edit. That’s why we wondered if there was a cleaner way than virtualize / hook into it.

Here is the solution if anyone else comes around:

/* SEO - Adds the sitemap URL in the virtual robots.txt */
add_filter( 'robots_txt', 'mw_robots_mod', 10, 2 );
function mw_robots_mod( $output, $public ) {
    $output .= "Disallow: /pro/*\n";
    $output .= "\nSitemap: " . get_site_url() . "/sitemap_index.xml";
    return $output;
}

Add that to your functions.php and it should be good to go.

[rubin]

Thank you for sharing this information with our community!