hiddenweb – Probing and wrapper induction for hidden-Web sources
Introduction
This Java system discovers how to query a hidden-Web source, that is, a database accessible only through an HTML form. It analyzes the form, probes it with instances drawn from a domain of knowledge, clusters the result pages, induces a wrapper for the records they contain, and finally publishes the source as a Web service described in WSDL. It was developed at Télécom ParisTech and Inria Lille, and is described in the following publications:
- Automatic Wrapper Induction from Hidden-Web Sources with Domain Knowledge (WIDM 2008);
- Understanding the Hidden Web (PhD thesis, 2007), which describes the approach as a whole.
Download
The code is provided as is, as it was left in 2008, for research purposes.
- Java sources, WSDL service descriptions, and the documentation of each component (ZIP), together with the material of the experiments: the bibliographic domain knowledge used to probe the forms (author, conference, journal and title instances with their frequencies), the list of hidden-Web sources probed, and the annotated result pages, evaluation scripts and measurements of the wrapper induction experiments.
Two third-party components the system depends on are not included: the WordNet lexical database, used to relate the concepts of a domain to the labels of a form, and the XCRF library of conditional random fields for XML, developed by the Mostrare team at Inria Lille, used to annotate result pages.