Rosalia: An Experimental Web Crawler

What is Rosalia?
Rosalia is an experimental web crawling robot, used to collect documents and other information from the internet.

How do I prevent Rosalia from crawling my site or parts of my site?
Rosalia obeys the Robot Exclusion Standard. Robots.txt is a standard document that tells Rosalia not to download some or all of the information from your web servers.

Why does Rosalia download the same page multiple times?
Rosalia requires only one copy of each file from your site during a given crawl, however, if the crawler is stopped and restarted it may re-crawl a recently crawled page.

What kinds of links does Rosalia follow?
Rosalia follows HREF links.

For what you will use the information collected by Rosalia?
Currently this information is used for our private research interests, but there is some idea to publish some data that may be of general interest.

If you have additional questions please contact us at antirez (at) gmail (dot) com.