Introduction
As you probably know, Qwant uses a web crawler to enhance its index and provide the best possible service.
From now on, our crawler is named Bleriot, in reference to the pioneer Louis Bleriot who was the first man to fly over the Channel.
This bot is essentially based on the opensource project of Qwant Research named Mermoz.
Technical specifications
During the crawls we announce ourselves as:
Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
The crawler respects the robots rules that you defined in your path //hostname/robots.txt, we use a homemade library URLFactory.
Thus we take into account the crawl delay that you defined in the file, otherwise we set it by default to 2 secs.
Troubles
If something went wrong when we visited your website using our crawler, we are sincerely sorry for this inconvenience.
Actually, Bleriot is developed by humans. And humans still propagate errors within numerical codes, even if one tests all features before pushing it to production.
Also, as Bleriot is constantly being improved, new problems might appear during a testing phase.
Thus, please report us any problem caused by Bleriot by sending a message to this address: bleriot [at] qwant [dot] com