Crawler-Lib Engine

The Crawler-Lib Engine is a high performance and high throughput task processor for the Microsoft .NET Framework and Mono. It has workflow capabilities. Tasks like scrape website, send email, twitter something, .. can be easily implemented with it.

The Crawler-Lib Engine is a fully generalized task processor. Although it is way beyond a crawler it can be used to develop high performance bots (robots) for web crawling and web scraping. But also data extraction, publishing and data distribution applications can use it in C#, Visual Basic and any other .NET programming language. It is optimized to perform thousands of requests per second. It has a great support for the async pattern (IAsync implementations) and the Task Parallel Library (TPL) to perform async and parallel requests of any kind. Due to its workflow capabilities it can be used to perform complex request sequences with dependencies. For example a web crawling workflow: Establish a cookie, fill a form, post it and process the results.

Get Started

Download and Install

Please download and install the Crawler-Lib Engine NuGet Package in your project with the NuGet Package Manager. A license file for the free "Anonymous Edition" is included in the packages so the Crawler-Lib Engine can be used and tested without registering as customer. All packages are licensed under the Crawler-Lib Framework End User License Agreement. You accept the terms and conditions of this agreement by downloading and installing the package. To generate a license file for your free or purchased edition go to the Downloads Page and click the "Generate License" link. Follow the installation instructions there.

Code Samples

Take a look at the code samples section to see the available samples and to get an overview over the capabilities. Please download also the Code Samples (Visual Studio Solution with all C# Samples) and watch the Tutorials and C# Sample Videos on YouTube.


Compare Editions

Processing Threds 1 2 int.MaxValue
Max Working Tasks 2 20 int.MaxValue
Max Tasks per Minute 60 600 int.MaxValue
Max Tasks per Minute
Check Interval (s)
60 120 TimeSpan.MaxValue
Crawler-Lib Engine Free Community Edition
Registered users get the Community Edition for free.
Crawler-Lib Engine Unrestricted Editon - Single Developer
This license allows a single person to develop software with the Crawler-Lib Engine and distribute the software without additional fees
$380.00 incl tax