focusedrl – Focused crawling with reinforcement learning
Introduction
focusedrl is a C++ focused crawler developed by Miyoung Han during her PhD, which learns, through reinforcement learning and experience replay, which links are worth following to reach pages relevant to a given topic. It is described in the following publication:
- Focused Crawling through Reinforcement Learning (ICWE 2018).
Download
The code is provided as is, as it was left in 2018, for research purposes.
Topic relevance is computed from a taxonomy derived from Wikidata (a CC0-licensed public database) and from the public GloVe word embeddings (6B tokens, 50 dimensions), neither of which is bundled here.