What is Crawling?

Crawling is a method of scanning a selected website by creating a dedicated robot that visits every link on the page and collects selected data, which can then be used in our system. 

Crawling will not overload your server because it only retrieves the text data (without downloading any images or JS scripts). You can also specify the parameters of crawling by limiting the number of queries per second to the website's server. Crawling a website of a store offering up to 5,000 products shouldn’t take longer than a few minutes. We can also crawl websites that have millions of products, but in such a case the update process can take up to several hours - still, it’s not a problem for us.