| 基于NoSQL数据库的农田物联网云存储系统设计与实现 |
| |
| 引用本文： | 许鑫,时雷,何龙,张浩,马新明.基于NoSQL数据库的农田物联网云存储系统设计与实现[J].农业工程学报,2019,35(1):172-179. |
| |
| 作者姓名： | 许鑫 时雷 何龙 张浩 马新明 |
| |
| 作者单位： | 1. 河南农业大学信息与管理科学学院,郑州 450002; 2. 河南粮食作物协同创新中心,郑州 450002;,1. 河南农业大学信息与管理科学学院,郑州 450002;,1. 河南农业大学信息与管理科学学院,郑州 450002;,1. 河南农业大学信息与管理科学学院,郑州 450002; 2. 河南粮食作物协同创新中心,郑州 450002;,1. 河南农业大学信息与管理科学学院,郑州 450002; 2. 河南粮食作物协同创新中心,郑州 450002; 3．农业部黄淮海农业信息技术科学观测实验站,郑州 450002; |
| |
| 基金项目： | 河南省科技创新杰出人才(184200510008);河南省现代农业产业技术体系（S2010-01-G04）;十三五国家重点研发计划（2016YFD0300609）;河南省重大科技专项（171100110600-01） |
| |
| 摘 要： | 为了解决农田物联网大量图像、视频和传感器等结构化和非结构化数据实时处理与写入问题,该文基于分布式存储与NoSQL(NotOnlySQL)技术,结合农田物联网数据特征,利用HDFS(HadoopDistributedFileSystem)和HBase(Hadoop Database)存储非结构化和结构化数据,基于Redis缓存服务,设计了三层物联网数据云存储框架,实现了海量农田物联网数据存储中的业务处理、事务处理、图片打包与索引、负载均衡等关键技术。面对复杂业务下的事务数据一致性,该文采用基于HLock的乐观锁机制,实现了HBase对强事务性的支持,经过与传统MySQL集群事务对比测试,当数据量级在500万时,数据读取效率提升达35.75%。为了提高农田物联网中大量的小图片和小文件处理效率,基于图片打包合并策略,利用SequenceFile技术实现物联图片的快速索引读写技术,与原生HDFS存储效率相比,读写效率提升30%以上。该研究可以为海量农业物联网数据的存储和管理提供技术参考和理论支撑。
|
| 关 键 词： | 农田 数据存储系统 管理 物联网 NoSQL Hadoop HBase 云存储 |
| 收稿时间： | 2018/9/25 0:00:00 |
| 修稿时间： | 2018/11/23 0:00:00 |
Design and implementation of cloud storage system for farmland internet of things based on NoSQL database |
| |
| Xu Xin,Shi Lei,He Long,Zhang Hao and Ma Xinming.Design and implementation of cloud storage system for farmland internet of things based on NoSQL database[J].Transactions of the Chinese Society of Agricultural Engineering,2019,35(1):172-179. |
| |
| Authors: | Xu Xin Shi Lei He Long Zhang Hao and Ma Xinming |
| |
| Institution: | 1. College of Information and Management Science, Henan Agricultural University, Zhengzhou 450002, China;2. Henan Grain Crops Collaborative Innovation Center, Zhengzhou 450002, China;,1. College of Information and Management Science, Henan Agricultural University, Zhengzhou 450002, China;,1. College of Information and Management Science, Henan Agricultural University, Zhengzhou 450002, China;,1. College of Information and Management Science, Henan Agricultural University, Zhengzhou 450002, China; 2. Henan Grain Crops Collaborative Innovation Center, Zhengzhou 450002, China; and 1. College of Information and Management Science, Henan Agricultural University, Zhengzhou 450002, China; 2. Henan Grain Crops Collaborative Innovation Center, Zhengzhou 450002, China; 3. HHH Science Observation and Experiment Station of Agricultural Information Technology, Ministry of Agriculture, Zhengzhou 450002, China |
| |
| Abstract: | Abstract: In order to solve the problems of storing large amounts of structured and unstructured data, such as images, video and sensors and so on, and real-time processing and writing, a data cloud storage system for farmland Internet of Things (IOT) with mass storage, high performance and easy expansion is constructed. Based on Hadoop platform, in this paper, we constructed a massive farmland IOT data cloud storage system by combining the characteristics of farmland IOT data, using distributed storage and NoSQL(Not Only SQL) technology. From the security, reliability, efficient reading and writing, data conversion, transaction processing, small file processing, cache strategy, load balancing and other issues of the system, HDFS was used to store unstructured data such as pictures and videos in the farmland IOT system, HBase was used to store structured data such as meteorology and moisture in the farmland IOT system, Redis was used in cache servers. Three layers of data cloud storage architecture for IOT were designed. The system classifies and processes video, image, text and structured data. For large video block storage, small file image packaging and merging storage, text classification and conversion strategy, unstructured data were written to HDFS, structured data were written to HBase, and Redis was used as the system cache to realize the data of the IOT writing and reading business. In distributed cluster environment, the reliability of cross-line transaction and long transaction processing was restricted. It was difficult to process cross-line transaction and long transaction accurately and orderly, and it was difficult to ensure data consistency in complex services such as massive data analysis. In this paper, a distributed transaction mechanism based on optimistic lock was designed. The transaction processing module cooperates with the HLock(optimistic lock) structure to control the state of the transaction. The NTP server guarantees the uniqueness of the transaction timestamp. The transaction ACID features, including reading and writing data, were solved. HBase''s strong transactional support has been tested to improve query efficiency by 35.75% compared with traditional MySQL clusters when the data level was 5 million. Thus, NoSQL-based structured data storage scheme was feasible in dealing with high concurrent massive data scenarios. In order to solve the problem of a large number of small pictures and small files in the farmland IOT, the sampled pictures were packaged and measured. The "SequenceFile" technology was used to merge multiple pictures into a "Block" to realize the strategy of merging and storing small files. Fast index reading, compared with the original HDFS storage reading and writing efficiency, image file storage reading and writing efficiency improved by more than 30%. Therefore, based on the "SequenceFile" file merging technology, image file name design and index optimization strategy, it was suitable for large-scale image storage scene in the farmland IOT. The system had been applied to the monitoring system of farmland IOT in China Henan Province. It was distributed in more than 60 monitoring stations in Changge, Huaxian, Luohe and Fangcheng counties and cities, providing real-time data for storage, management and visualization, and considering the incorporation of more sensors and monitoring stations, the system was in good working order. In summary, based on Hadoop platform and NoSQL technology, we designed a massive farmland IOT data storage model, designed and implements the key technologies such as data reading and writing, transaction, picture packaging, index, load balancing module, and develops a massive farmland IOT data storage, management system. Based on NoSQL massive farmland IOT data storage scheme suitable for the storage and management needs of the IOT massive, real-time data, for farmland IOT storage transaction consistency, small file processing and other issues, for massive agricultural IOT data storage solutions. It can combine distributed computing and machine learning technology to compute the data of IOT in real time and provide real-time operation and decision-making services for agricultural production. |
| |
| Keywords: | farms data storage equipment management IOT NoSQL Hadoop HBase cloud storage |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《农业工程学报》浏览原始摘要信息 |
| 点击此处可从《农业工程学报》下载免费的PDF全文 |
|
