Disallow:
Excluding robot-specific protocol tags
URI, REP tags are applied to certain indexer tasks, and in some cases to nosnippet, noarchive and noodpquery engines or to a search query. For resources tagged with exclusion tags, search engines like Bing SERP australia mobile number list free listings display these external links as prohibited URLs. In addition to crawl directives, specific search engines will interpret REP tags differently. An example of this can be seen in how Bing sometimes lists external references in their SERPs as prohibited. Google takes the same listings and deletes the URL and ODP references in their SERPs. The idea is that X-Robots override directives that conflict with META elements.
Microformats
Particular HTML factors will override the page's settings in microformatted index directives. This programming method requires skills and a very keen knowledge of web servers and the HTTP protocol. An example of this protocol would be an X-Robot tag page with a particular link element that says follow and then rel-nofollow. Robots.txt indexers typically lack directives, but it is possible to set up URI group indexers that have a server-side scripting at the site level.
Pattern Matching
Webmasters can still use two different expressions to denote page exclusion. The two characters are the asterisk and the dollar sign. The asterisk denotes that it can represent any combination of characters. The dollar sign is to denote the end of the URL.