How to set User Agent for HttpRequest?

Posted: 3 years ago Quote #118
Is it possible to set/change User Agent string when I do the HttpRequest?
I found no property about this. Is there something, I'm not seeing?

Thanks.
Posted: 3 years ago Quote #120
This can be done by adding it to the headers collection of the HttpRequestConfig. But it is not very nice, I will introduce properties for this.
Crawler-Lib Developer
Posted: 3 years ago Quote #122
Pending a new version with Headers properties headers, I do this

Before :
HttpRequest request = await new HttpRequest(new Uri(new Uri("http://localhost:10080/), 
                                                                   new HttpRequestQuota { MaxDownloadSize = 100000, OperationTimeoutMilliseconds = 10000, ResponseTimeoutMilliseconds = 5000 },
                                                                   AwaitProcessingEnum.All);


I replace by :
HttpRequest request = await new HttpRequest( new HttpRequestConfig
                                                    {
                                                        Url = new Uri("http://localhost:10080/"),
                                                        Quota = new HttpRequestQuota { MaxDownloadSize = 100000, OperationTimeoutMilliseconds = 10000, ResponseTimeoutMilliseconds = 5000 },
                                                        Headers = new List<KeyValuePair<string, string>>() { new KeyValuePair<string, string> ("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0") },
                                                        AwaitProcessing = AwaitProcessingEnum.All
                                                    });


It's OK?
Posted: 3 years ago Quote #123
Yes that is Ok. I have taken the header issue on priority.
Crawler-Lib Developer
Posted: 3 years ago Quote #124
I've been assigned to that issue. We are currently implementing the common header properties. Have you additional suggestions for this? I will merge it with main and trigger the build process. When everything is ok, (unit tests, test deploy, …) i will trigger the release process. I think we can do this this eveneing.
  
Crawler-Lib Developer
Build and Deployment
Posted: 3 years ago Quote #125
Good new 🙂
Tell me when release version will be available on Nuget. I test it with my current devellopement.
Posted: 3 years ago Quote #126
We have finished the releasing of version 2.3.5529.650. nuget packages and zip files. you can update your project.
Crawler-Lib Developer
Build and Deployment
Posted: 3 years ago Quote #127
Can you explain what has been added in this version?



I design my crawler with your example Website Crawler. With this new version, I encounter an error on this line

this.engine.AddLimiter("PageRequestLimiter",  new QuotaLimiter( new QuotaLimiterConfig { MaxRunningLimited = 20 }));

It seems that AddLimiter method no longer accepts 2 parameters but one of type ILimiter.
But I found no example, to add limiter to the engine with type of ILimiter.

Documentation about this is no up to date : CrawlerEngine.AddLimiter Method

EDIT : I found !
this.engine.AddLimiter(new QuotaLimiter(new QuotaLimiterConfig { Name = "PageRequestLimiter", MaxRunningLimited = 20 }));

[b][/b]
Posted: 3 years ago Quote #128
the change was in version 2.3.5513.24819 not in the latest version.
Breaking: AddLimiter() takes name form LimiterConfig now.

https://www.nuget.org/packages/CrawlerLib.Engine

it makes no sens that the AddLimiter() method and the configuration has a Name property / parameter.
set name in LimiterConfig and all is ok.
Crawler-Lib Developer
Build and Deployment
Posted: 3 years ago Quote #129
OK.

an example on how to use header properties in HttpRequest?