Web Crawler using FetchAndProcess Framework.

 This Project shows  you how to crawl the Web and Process those web documents.

 This was built using Fetch and process frame work, code can be found at   Fetch And Process.

 

   WebDUPlugin.cs : Crawler Class (fetcher)

  WebDUProcess.cs :: Processor Class

 

  Configuration: <>\WebDU\WebDU\WebDUSample.xml file.

  Fetcher and Processor classes instances will be created dynamically based on configuration by fetch and process framework. 

Class instances will be called by frame work threads based on work items in queue ( framw work takes care of it , as developer only you need to worry about functionality ).

 

Last edited Oct 22, 2013 at 12:35 AM by nspreddy, version 5