Update robots.txt file for edX
Issue Key: CR-200
Reporter: Greg Kristan
Currently, we are blocking Google / Bing from looking at our Spanish learn pages on our site. Search engines won't see all URL's that include /es/aprende/ as of now, so we are not driving organic traffic to them.
To fix this issue, I would like a new line added to the robots.txt file to include this command below. https://www.edx.org/robots.txt
I included a picture in this ticket to show where the line should be and how it will allow Google to see the content.
This ticket is for SEO (search engine optimization) and will drive traffic to these pages once they are updated.
Let me know if there are any questions.
File updated and successfully deployed to production.
That sounds good to me!
This is in the queue to be triaged. It will be picked up as soon as we have someone available.
Just checking in on the status of this request. Do you have a timetable on when this fix might be pushed? I'm out Thursday and Friday of this week, so I wanted to verify the change if possible before being out the rest of the week.