This is done making use of a file called robots.txt (crawling) and a meta tag called meta robots (indexing).

Feel free to surf to my web page :: do it yourself organic seo