Do you know robots.txt？
百度百科