Proceedings of the 21st Paral- lel and Distributed Processing Symposium
Optimized inverted list assignment in distri- buted search engine architectures
作者:
J. Zhang， T. Suel.
关键词:
data structures；peer-to-peer computing；query processing；search engines；distributed Web search engine architecture；index organization；inverted list assignment optimization；peer-to-peer architecture；query processing；Bandwidth
摘要:
We study efficient query processing in distributed web search engines with global index organization. The main performance bottleneck in this case is due to the large amount of index data that is exchanged between nodes during the processing of a query, and previous work has proposed several techniques for significantly reducing this cost. We describe an approach that provides substantial additional improvement over previous techniques. In particular, we analyze search engine query traces in order to optimize the assignment of index data to the nodes in the system, such that terms frequently occurring together in queries are also often collocated on the same node. Our experiments show that in return for a modest factor increase in storage space, we can achieve a reduction in communication cost of an order of magnitude over the previous best techniques.
在线下载