There is always remaining tasks you have to do in order to finalize a website. Out of this series I'd like to share my simple Meta Tag and Open Graph implementation in XM Cloud using Headless SXA, JSS and NextJs.
There are those features that you always have to implement when coming towards the end of the project and think about going live. robots.txt is one of them. In general the robots.txt is responsible for telling Search Engines what parts of the website to index and which ones not. That does not mean that Search Engines care. If you want to read more: https://developers.google.com/search/docs/crawling-indexing/robots/intro Luckily Headless SXA provides that feature so you don't have to implement something yourself.
