Update robots.txt file for edX


Issue Key: CR‌-200
Reporter: Greg Kristan

Currently, we are blocking Google / Bing from looking at our Spanish learn pages on our site. Search engines won't see all URL's that include /es/aprende/ as of now, so we are not driving organic traffic to them.

To fix this issue, I would like a new line added to the robots.txt file to include this command below. https://www.edx.org/robots.txt

Allow: /es/aprende/

I included a picture in this ticket to show where the line should be and how it will allow Google to see the content.

This ticket is for SEO (search engine optimization) and will drive traffic to these pages once they are updated.

Let me know if there are any questions.



Epic Link



Tasawer Nawaz
September 14, 2018, 5:51 AM

File updated and successfully deployed to production.

Greg Kristan
September 12, 2018, 7:32 PM

That sounds good to me!

Albert (AJ) St. Aubin
September 12, 2018, 6:20 PM

This is in the queue to be triaged. It will be picked up as soon as we have someone available.

Greg Kristan
September 12, 2018, 1:40 PM

Just checking in on the status of this request. Do you have a timetable on when this fix might be pushed? I'm out Thursday and Friday of this week, so I wanted to verify the change if possible before being out the rest of the week.


Your pinned fields
Click on the next to a field label to start pinning.




Tasawer Nawaz


Greg Kristan