一種基于Spark的語義推理引擎實現(xiàn)及應(yīng)用
發(fā)布時間：2018-12-09 13:15
【摘要】：近些年在知識圖譜蓬勃發(fā)展的大背景下,與之相關(guān)的語義Web的數(shù)據(jù)規(guī)模也呈現(xiàn)爆發(fā)態(tài)勢。如何在大規(guī)模語義Web數(shù)據(jù)上有效地進(jìn)行語義推理是研究者們面臨的棘手問題。具體來說,在大規(guī)模語義Web數(shù)據(jù)上實施語義推理時,計算量巨大、消耗時間長都是突出的問題,特別是當(dāng)應(yīng)用復(fù)雜規(guī)則邏輯進(jìn)行推理時,情況更是如此。傳統(tǒng)單機(jī)環(huán)境下的語義推理引擎無法應(yīng)對大規(guī)模知識圖譜下的推理,缺乏可擴(kuò)展性方面的考慮,難以滿足在數(shù)據(jù)規(guī)模上日益增長的語義關(guān)聯(lián)數(shù)據(jù)的推理需求。從分布式角度來看,已有的基于Hadoop MapReduce實現(xiàn)的語義推理框架由于欠缺推理算法相關(guān)的網(wǎng)絡(luò)通信和磁盤I/O等的優(yōu)化,推理效率依然較低。本文針對上述問題,圍繞分布式內(nèi)存計算平臺Spark,研究以下幾個方面的內(nèi)容:首先設(shè)計一個良好模塊化且推理規(guī)則可配置的完整分布式推理引擎架構(gòu)。接著研究現(xiàn)有的單機(jī)和分布式語義推理算法,基于Spark框架對相關(guān)算法進(jìn)行分布式的實現(xiàn),并針對Spark的原理和特點做相應(yīng)的優(yōu)化。將基于Spark實現(xiàn)的推理引擎與現(xiàn)有的傳統(tǒng)分布式推理引擎在推理效率上進(jìn)行對比實驗。實驗結(jié)果表明,本文設(shè)計的基于Spark的語義推理引擎在推理效率上要遠(yuǎn)好于以Hadoop MapReduce為代表的推理實現(xiàn),同時兼具了高可擴(kuò)展性。最終將本系統(tǒng)應(yīng)用到物聯(lián)網(wǎng)領(lǐng)域,適應(yīng)實時和流式的語義數(shù)據(jù)流處理和推理場景。
[Abstract]:In recent years, with the rapid development of knowledge map, the data scale of semantic Web, which is related to it, has also taken on an explosive trend. How to effectively perform semantic reasoning on large scale semantic Web data is a difficult problem for researchers. Specifically, when implementing semantic reasoning on large scale semantic Web data, it is an outstanding problem that the computation is huge and the time is long, especially when the reasoning is based on the logic of complex rules. The traditional semantic reasoning engine in single machine environment can not cope with the reasoning under large-scale knowledge atlas, and it is difficult to meet the reasoning needs of the increasing data scale of semantic association data due to the lack of scalability considerations. From a distributed point of view, the existing semantic reasoning framework based on Hadoop MapReduce is still inefficient due to the lack of network communication related to reasoning algorithm and optimization of disk I / O. Aiming at the above problems, this paper studies the following aspects around the distributed memory computing platform Spark,: firstly, a complete distributed reasoning engine architecture with good modularization and configurable reasoning rules is designed. Then the existing single machine and distributed semantic reasoning algorithms are studied. The distributed implementation of the related algorithms based on the Spark framework is carried out and the corresponding optimization is made according to the principle and characteristics of Spark. The reasoning engine based on Spark is compared with the traditional distributed reasoning engine in reasoning efficiency. The experimental results show that the semantic reasoning engine based on Spark is much more efficient than the reasoning implementation represented by Hadoop MapReduce, and it also has high scalability. Finally, the system is applied to the field of Internet of things, which adapts to real-time and streaming semantic data flow processing and reasoning scenarios.
【學(xué)位授予單位】：浙江大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP311.52
本文編號：2369416
[Abstract]:In recent years, with the rapid development of knowledge map, the data scale of semantic Web, which is related to it, has also taken on an explosive trend. How to effectively perform semantic reasoning on large scale semantic Web data is a difficult problem for researchers. Specifically, when implementing semantic reasoning on large scale semantic Web data, it is an outstanding problem that the computation is huge and the time is long, especially when the reasoning is based on the logic of complex rules. The traditional semantic reasoning engine in single machine environment can not cope with the reasoning under large-scale knowledge atlas, and it is difficult to meet the reasoning needs of the increasing data scale of semantic association data due to the lack of scalability considerations. From a distributed point of view, the existing semantic reasoning framework based on Hadoop MapReduce is still inefficient due to the lack of network communication related to reasoning algorithm and optimization of disk I / O. Aiming at the above problems, this paper studies the following aspects around the distributed memory computing platform Spark,: firstly, a complete distributed reasoning engine architecture with good modularization and configurable reasoning rules is designed. Then the existing single machine and distributed semantic reasoning algorithms are studied. The distributed implementation of the related algorithms based on the Spark framework is carried out and the corresponding optimization is made according to the principle and characteristics of Spark. The reasoning engine based on Spark is compared with the traditional distributed reasoning engine in reasoning efficiency. The experimental results show that the semantic reasoning engine based on Spark is much more efficient than the reasoning implementation represented by Hadoop MapReduce, and it also has high scalability. Finally, the system is applied to the field of Internet of things, which adapts to real-time and streaming semantic data flow processing and reasoning scenarios.
【學(xué)位授予單位】：浙江大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP311.52
【參考文獻(xiàn)】
相關(guān)期刊論文 前1條
1 劉嶠;李楊;段宏;劉瑤;秦志光;;知識圖譜構(gòu)建技術(shù)綜述[J];計算機(jī)研究與發(fā)展;2016年03期
相關(guān)博士學(xué)位論文 前1條
1 李韌;基于Hadoop的大規(guī)模語義Web本體數(shù)據(jù)查詢與推理關(guān)鍵技術(shù)研究[D];重慶大學(xué);2013年
，本文編號：2369416
本文鏈接：http://www.lk138.cn/shoufeilunwen/xixikjs/2369416.html
最近更新
教材專著
