Log Analyzer will allow you to download and save log files with information on spider visits, as well as export it into different file formats. Using Log Analyzer you can track spider visits, identify malicious and unwanted spiders and exclude them from your site and create spider visits reports in HTML, Microsoft Excel CSV and XML formats.
Robots.txt Editor is an easy-to-navigate visual editor that will enable you to specify different directives for selected spiders in specific areas of the site and generate the robots.txt file quickly and easily. Using Robots.txt Editor you can generate industry standard robots.txt files, direct search engine spiders to the appropriate content for multilingual sites, keep spider our of sensitive and private areas of your web site.
These two modules, along with FTP Uploader and a number of other useful features, make up a powerful and handy tool for controlling spiders on your web site. |
