Robot. txt file is just a text file placed within the website, It allows Google bot to Crawl the listed URL's , Otherwise the bot not crawled.