基于HBase的数据高效读技术研究

Research on Efficient Data Reading Technology Based on HBase

摘要: 在大数据时代，关系型数据库面临着海量数据存储的挑战。HBase是一种基于列存储的NoSQL数据库，广泛应用于大数据存储。HBase在数据的检索方面仍然存在着不足之处，本文对HBase的数据检索技术进行分析和研究，针对目前存在的问题提出了相应的改进和优化。针对HBase在查询数据时需要访问磁盘，查询速度慢的问题，本文提出使用Redis索引HBase的热点数据，并综合考虑数据的查询频率、更新频率和历史积热对缓存的影响，设计了一种基于数据查询频率和更新频率的热值缓存驱逐策略，提高了Redis的缓存命中率。针对HBase在检索非行键字段时需要全表扫描，检索效率低的问题，本文提出了为非行键字段建立二级索引的策略，设计了一种基于协处理器和Redis的二级索引方案。实验结果表明改进后的缓存驱逐策略的命中率高于LRU策略，在查询模块引入Redis缓存热点数据并且为非行键字段建立二级索引后，改进后的查询模块的数据检索性能提升显著，极大地提高了查询速度。

Abstract: In the era of big data, relational databases face the challenge of massive data storage. HBase is a NoSQL database based on column storage, which is widely used in big data storage. HBase still has shortcomings in data retrieval. This paper analyzes and studies the data retrieval technology of HBase, and puts forward corresponding improvements and optimization for the existing problems. In view of the problem that HBase needs to access the disk when querying data and the query speed is slow, this paper proposes to use Redis to index the hot data of HBase, and comprehensively considers the impact of data query frequency, update frequency and historical heat accumulation on the cache, and designs a calorific cache eviction strategy based on data query frequency and update frequency, which improves Redis cache hit rate. Aiming at the problem that HBase needs full table scanning when retrieving non-row key fields, and the retrieval efficiency is low, this paper proposes a strategy of establishing secondary index for non-row key fields, and designs a secondary index scheme based on coprocessor and Redis. The experimental results show that the hit rate of the im-proved cache eviction strategy is higher than that of the LRU strategy. After the Redis cache hot da-ta is introduced into the query module and the secondary index is established for non-row key fields, the data retrieval performance of the improved query module is significantly improved, and the query speed is greatly improved.