The following documents my journey while trying to fix the Yoast attachment URL bug. The issue is described, along with a formal apology, in this blog post.
It all began when one of our SEO client sites became a victim of the Yoast attachment URL bug with an update done to the Yoast SEO plugin. This was an appropriate test because the client is Digiceed, a provider of SEO services in London.
The first step taken was to check and ensure the Yoast setting (Redirect attachment URLs to the attachment itself?) was set to yes. This was set to “No”, and we had not made that change so we updated the setting to “Yes”.
A quick look in the Google Search console confirmed a list of URLs that had been indexed, but not submitted in the sitemap.
It should be noted that Yoast have published a search index purge plugin to fix the attachment URL issue. You can download and install the plugin from here. I did not use the plugin due to the large volume of support issues in the WordPress plugin repository. After reading some of them, I decided to try and develop a solution on my own.
Attempted Fix Using .htaccess File
The next step I took was to look for a quick fix. The Google Search Console offers a “Remove URL” tool, but you can only add URLs one at a time. I needed a fix that would tell Google the pages were gone forever, never to return. I needed to assign an Error 410 code response when the Google Crawler came to the website. My first choice was to use the .htaccess file.
I went back to Google and performed a quick search for “Yoast Attachment URL bug using .htaccess”. Many of the results returned directed back to the Yoast website, with the top result being the Yoast issue description and apology. No good.
So I decided to use my knowledge and try the .htaccess fix myself. I opened up the .htaccess file located in the root folder of the website and added a few lines below the line with “# END WordPress” Please note that the original URL has been replaced with “testurl” as a placeholder.
# Respond with 410 for these URLs RewriteEngine On RewriteBase / RewriteRule ^test-url/?$ - [G, NC]# END Respond with 410
A quick save a test, the website crashed. The website was returning an “Error 500 — Internal server error, configuration”. This means there was a problem with my lines of code added to the .htaccess file. So I removed my test lines and decided to move them before the WordPress bits, but in the same section. So my revised codes looked like this:
# BEGIN WordPress <IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^testurl/?$ - [G, NC] RewriteRule ^index\.php$ - [L]
...Again, crash! Back to Google.
Getting the Indexed URLs to Respond with a 410
A search for my problem showed I was entering in the Rewrite Conditions properly, it just was not working on this site for some reason. I thought there must be an easy way, so I tried another Google Search. But this time for “WordPress 410” to see what came back.
Bingo, I see a WordPress plugin, “410 for WordPress” by Samir Shah.
I installed and activated the plugin, and added my list of URLs I wanted to respond with a 410 response. I checked one of the URLs and I got the same result. I tried looking at the source of the URL returned to see if I could determine the response but I could not get the source to display. Keep in mind that a 410 response is NOT like a 404 response. There is nothing displayed unless you create an error template in your theme directory. But that is only if you are getting a proper 410 response.
My next step was to visit the HTTP Header Status Checker and make sure a 410 response was being returned. Again, another success.
Removing URLs from the Google Search Console
I still wanted Google to know about the change and still wanted to try and get the URLs inserted into the Removal Tool. While discussing the issue with a colleague, this URL was provided to me: https://george.bg/fix-yoast-image-pages-problem-fast/
This is a step by step guide to using the Google Console to add URLs for removal in bulk. It requires you to install a Chrome extension, which made me cautious. I proceeded and followed the steps including the install. The extension hooks into the Google Console page and provides a file upload button to upload a file with a list of the URLs to remove. Once uploaded, the extension acts as a macro and adds each URL, one at a time, to the Remove URLs page.
So far, this is the end of my journey chasing the Yoast attachment URL bug. The next few days should tell me if this was all successful. But I happy with the results so far. I will attempt to update my findings as they happen.
Originally published at siamcomm.com on July 3, 2018.
