in search results
is not here to talk about the "prohibited collected", but "no index", the above situation is included the ban, but the search engine is still on the web page index, when the user needs to return that reliable information. After has been added to the robots.txt as a rule, in order to completely remove these pages from search engine results in the list, we can through the following ways.
How to realize the
< meta name=" ROBOTS" content=">
this is the search engine, although not found recorded this page, but because there are many links to this page, so it is valuable that this page, there is a close relationship between the possible and user search behavior in this place, so displayed, but does not show respect for the owners, detailed information on URL.
robots.txt from the captured file search engine will not access, do not grab. It is important to note that robots.txt has banned URL grab or may appear in the search results, as long as there are import links to this URL, the search engine knows the existence of URL, although not crawl the page content, but may take the following form displayed in search results:
displays only URL, no title, description.
1, using meta robots label
display into the anchor text as a title and description.
is included in this page (as indexed) head plus the following code
robots.txt banned page why
some time, you will be surprised to find that, when the ban included page in robots.txt, but appeared in the search results list, especially the most easy to find at the time of site. This is you don’t panic, to doubt whether robots.txt grammar rules wrong.
display development directory or YAHOO and other important directory included title and description.
pictured above, Google will still be a jump and link the display in the search results, and in the URL below the prompt "due to the robots.txt system does not provide…" As a result, this is not the title page Jump < title>, but the link to the anchor text page Jump, you can try the address above, verify the effect.
An important reason why