Customizing the List of Ignored URLs

10 11/22/2024

The SEO Booster plugin includes a feature that monitors 404 errors on your website to help you maintain a healthy SEO profile. However, not all URLs need to be tracked for 404 errors. Certain URLs, such as those used for site configurations or specific files requested by browsers, are commonly ignored. This article explains how the SEO Booster plugin determines which URLs to ignore and how you can customize this list to better suit your website’s needs.

Default Ignored URLs

By default, the SEO Booster plugin ignores a predefined list of well-known URLs that are often requested by browsers, search engines, or other automated systems. These URLs typically do not impact user experience or SEO and include:

  • ads.txt: A file used for digital advertising transparency.
  • app-ads.txt: Similar to ads.txt, but specifically for apps.
  • sellers.json: A file used by ad exchanges to identify sellers.
  • /.well-known/: A directory for well-known URIs as defined by the IETF.
  • robots.txt: A file that instructs search engine crawlers which pages to crawl or ignore.
  • humans.txt: A file often used to include information about the site’s creators.
  • favicon.ico: The favicon file, often requested by browsers and crawlers.
  • browserconfig.xml: A file used by Microsoft for pinned site configurations.
  • apple-touch-icon.png: Used by iOS devices as a touch icon.
  • crossdomain.xml: A site policy file used by Flash or Silverlight.
  • sitemap.xml: The standard XML Sitemap for search engines.
  • sitemap_index.xml: A sitemap index file, often generated by WordPress or other CMSs.
  • sitemap.xml.gz: A compressed sitemap file.
  • index.php: The default index file for WordPress.
  • *.js.map: Any URL ending with .js.map for JavaScript source map files.
  • manifest.json: A file used for configuring progressive web apps (PWAs).
  • service-worker.js: A JavaScript file for handling background processes in PWAs.
  • sw.js: A file for service worker configuration.
  • site.webmanifest: A JSON file that defines a web app’s metadata.

Customizing the Ignored URLs List

While the default list of ignored URLs covers many common cases, your website might have additional URLs that you want to exclude from 404 monitoring. The SEO Booster plugin allows you to customize this list using the seo_booster_get_ignored_urls_for_404 function. Here’s how you can modify the ignored URLs:

Example 1: Adding a Custom URL to the Ignored List

To add a new URL to the list of ignored URLs, you can use the following code:


function custom_seo_booster_ignored_urls_for_404( $ignored_urls ) {
    // Add a custom URL to be ignored
    $ignored_urls[] = 'my-custom-url.txt';

    return $ignored_urls;
}
add_filter('seo_booster_ignored_urls_for_404', 'custom_seo_booster_ignored_urls_for_404');

Example 2: Removing an Existing URL from the Ignored List

If you want to stop ignoring a particular URL, you can remove it from the list:


function custom_seo_booster_ignored_urls_for_404( $ignored_urls ) {
    // Remove an existing URL from the ignored list
    $key = array_search('favicon.ico', $ignored_urls);
    if (false !== $key) {
        unset($ignored_urls[$key]);
    }

    return $ignored_urls;
}
add_filter('seo_booster_ignored_urls_for_404', 'custom_seo_booster_ignored_urls_for_404');

Example 3: Completely Replacing the Ignored URLs List

In some cases, you might want to replace the entire list with your custom URLs. Here’s how you can do that:


function custom_seo_booster_ignored_urls_for_404( $ignored_urls ) {
    // Replace the ignored URLs list with a custom one
    $ignored_urls = array(
        'custom-url-1.txt',
        'custom-url-2.txt',
        '/custom-path/',
    );

    return $ignored_urls;
}
add_filter('seo_booster_ignored_urls_for_404', 'custom_seo_booster_ignored_urls_for_404');

Example 4: Clearing All Ignored URLs

If you prefer not to ignore any URLs and want the plugin to monitor every 404 error, you can clear the entire list:


function clear_seo_booster_ignored_urls_for_404( $ignored_urls ) {
    // Clear the ignored URLs list
    return array();
}
add_filter('seo_booster_ignored_urls_for_404', 'clear_seo_booster_ignored_urls_for_404');

Submit Your Ignored URL Suggestions

We’re always looking to improve the SEO Booster plugin to better serve our users. If you encounter URLs that are commonly accessed by crawlers or browsers and believe they should be ignored by default, please let us know. You can submit your suggestions, and we’ll consider adding them to the default ignored URLs list in future updates.

The ability to customize the list of ignored URLs in the SEO Booster plugin gives you greater control over how 404 errors are monitored on your website. By tailoring this list to your specific needs, you can ensure that the plugin focuses on the most important pages, helping you maintain a healthy SEO profile and user experience.