AAH – Application-Aware Helper for Web archive crawling
Introduction
The Application-Aware Helper (AAH) is a Java crawler for Web applications (Web forums, blogs, content management systems…), developed by Muhammad Faheem during his PhD, within the ARCOMEM project. It detects the kind of Web application a site is running, and uses a knowledge base of crawling actions specific to that kind of application to crawl it intelligently and to extract structured content from it. It is described in the following publications:
- Intelligent and Adaptive Crawling of Web Applications for Web Archiving (ICWE 2013);
- Demonstrating Intelligent Crawling and Archiving of Web Applications (CIKM 2013 demonstration);
- Adaptive Web Crawling Through Structure-Based Link Classification (ICADL 2015), for the ACE adaptive crawler, which learns crawling strategies from the structure of pages.
Download
The code is provided as is, as it was left at the end of the project (2013–2015), under the GNU General Public License, version 3. It was originally distributed from Muhammad Faheem's Web page at Télécom ParisTech, which no longer exists.
- AAH: sources, Javadoc, third-party libraries, tester, and prototype sources (ZIP)
- ACE adaptive crawler: sources (ZIP)
- Crawl of the Rock am Ring Web forum (ZIP): the results of the AAH, ACE and wget crawls of www.rockamring-blog.de used in the experiments of the papers above.