Probing the Hidden Web
======================

1) External entities to the system: 
	The URL list is taken as input through the file named "URLs". The URLs_backup contains all the URLs that have been discovered. 
	The file named "stop_words.txt" contains the stop words that are used during the Pre-processing of the textual context.
	The file named "domain_concepts" contains the concept names that have to be used for mapping the input fields.
	The directory named "domain_knowledge" contains the domain instances with which to probe the forms. The instances also contain the cumulative frequencies associated, so that we can probe with probability proportional to the frequency.
	The directory named "wordnetdata" contains the wordnet synsets that are used by the package edu.* and the corrresponding functions are used by HiddenWeb.Ontology in order to get the related words.
	
2) Files created by the system
	The XML files that are created after analysing each URL are placed in the directory named "Database".
	The WSDL documents generated after probing are placed in "WSDL/"
	The list of related words to the concepts are placed in the files of the same names.

3) Probing the Hidden Web - System Description
	Cluster.java - This file contains the methods for the clustering of the feature vectors
	Domain.java - All the domain related tasks such as getting probe terms, getting related words etc. are done through functions in this file.
	FormProber.java - Probes the form with instances drwan from the domain. Hnadles connecting to the web site, submitting the form, getting the response page(s), analysing the pages, and writing the modifed XML file
	Ontology.java - Handles the ontology related tasks. Conatins a method writeRelatedWords, which would write the related words to a given concept in a target file. This method is the only interface of the system with the ontology, and hence if at any time we need to change the ontology, we have to rewrite this function only.
	pathDOM.java - Hnadles the feature vector construction from the DOM tree for a given HTML page.
	Stemmer.java - Porter's stemmer
	TagTree.java - Handles the construction of DOM tree from the HTML page
	URLanalyzer.java - Handles the syntactic analysis of the form and finally creates the XML representation of the form
	VectorSpace.java - Handles the cosine similarity, tf-idf type methods for creating the feature vector
	WSDLgenerator.java - Generates the WSDL document given the XML representation
	XMLdatabase.java - Java API for accessing the XML representation of the form

	myService.InputWrapper - Java Servlet which is the web service described by the WSDL document generated. It handles the incoming SOAP request, and sends a SOAP response to the client accessing the WSDL.

   