發(fā)布時(shí)間：2018-07-17 19:12

【摘要】：作為用戶共享信息的平臺(tái),文庫系統(tǒng)為用戶帶來了效率和方便,然而,隨著用戶數(shù)據(jù)的增長(zhǎng),使用量的加大,文庫資源的形式和種類也越來越多,成指數(shù)倍增長(zhǎng)的海量數(shù)據(jù)資源給存儲(chǔ)系統(tǒng)帶來了難題,如何高效存儲(chǔ)和管理這些數(shù)據(jù)成為急待解決的問題。 云存儲(chǔ)技術(shù)的出現(xiàn),使高效存儲(chǔ)和管理這些海量數(shù)據(jù)成為可能。本文選取了當(dāng)前非常流行的云平臺(tái)Hadoop做為在線文庫系統(tǒng)的存儲(chǔ)平臺(tái),利用Hadoop下的云存儲(chǔ)文件系統(tǒng)HDFS存儲(chǔ)和管理在線文庫系統(tǒng)的文檔文件。HDFS只是為了解決一般性的數(shù)據(jù)存儲(chǔ)和管理難題,簡(jiǎn)單將其應(yīng)用于在線文庫系統(tǒng)不能投入實(shí)際的應(yīng)用,必須作必要的改進(jìn)。在線文庫系統(tǒng)的文檔資料一般為word、pdf、txt之類的文本文件,這些類型的文件都比較小,90%以上的文檔大小在32KB到20MB之間。HDFS的元數(shù)據(jù)信息是存儲(chǔ)在元數(shù)據(jù)節(jié)點(diǎn)的內(nèi)存中,因此在存儲(chǔ)海量的小文件時(shí),會(huì)導(dǎo)致HDFS元數(shù)據(jù)節(jié)點(diǎn)(NameNode)內(nèi)存的過量消耗,進(jìn)而降低整個(gè)HDFS系統(tǒng)的存儲(chǔ)容量,因此本文提出了一種將小文件合并成大文件的存儲(chǔ)優(yōu)化方案,有效地減少了元數(shù)據(jù)節(jié)點(diǎn)的內(nèi)存損耗。另一方面,考慮到合并后存取速度的折損,本文還提出了一種數(shù)據(jù)預(yù)取機(jī)制,該機(jī)制包括兩級(jí)緩存,通過這兩級(jí)緩存可以大大提升用戶文件讀取的流暢度,緩解云存儲(chǔ)元數(shù)據(jù)管理節(jié)點(diǎn)的壓力。圖22幅,表3個(gè),參考文獻(xiàn)60篇。
[Abstract]:As a platform for users to share information, library system brings users efficiency and convenience. However, with the increase of user data and the increase of usage, the forms and types of library resources become more and more. The massive data resources which increase exponentially bring problems to storage system. How to store and manage these data efficiently becomes an urgent problem to be solved. Cloud storage technology makes it possible to store and manage these massive data efficiently. This paper selects Hadoop, a very popular cloud platform, as the storage platform of online library system. Using the cloud storage file system HDFS under Hadoop to store and manage the document files of the online library system. HDFS is just to solve the general problem of data storage and management, and simply applying it to the online library system can not be put into practical application. Necessary improvements must be made. The document material of the online library system is usually a text file such as word _ _ _. Therefore, when storing large amount of small files, it will lead to excessive consumption of memory in HDFS metadata node (name Node), and then reduce the storage capacity of the whole HDFS system. Therefore, this paper proposes a storage optimization scheme to merge small files into large files. The memory consumption of metadata nodes is reduced effectively. On the other hand, considering the loss of access speed after merging, this paper also proposes a data prefetching mechanism, which includes two levels of cache, which can greatly improve the fluency of user file reading. Ease cloud storage metadata management node pressure. There are 22 figures, 3 tables and 60 references.
【學(xué)位授予單位】：中南大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 孔亞楠;余躍;;云計(jì)算下的虛擬存儲(chǔ)研究及應(yīng)用[J];電腦知識(shí)與技術(shù);2010年30期

2 鐘偉彬;周梁月;潘軍彪;文錦軍;;云計(jì)算終端的現(xiàn)狀和發(fā)展趨勢(shì)[J];電信科學(xué);2010年03期

3 趙吉志;李金;姚萃南;;云計(jì)算數(shù)據(jù)中心及標(biāo)準(zhǔn)化發(fā)展[J];信息技術(shù)與標(biāo)準(zhǔn)化;2011年03期

4 吳吉義;傅建慶;平玲娣;謝琪;;一種對(duì)等結(jié)構(gòu)的云存儲(chǔ)系統(tǒng)研究[J];電子學(xué)報(bào);2011年05期

5 朱偉;;網(wǎng)絡(luò)虛擬化典型技術(shù)探討[J];廣東通信技術(shù);2011年01期

6 朝樂門;;云計(jì)算環(huán)境下的電子文件遷移模型研究[J];檔案學(xué)通訊;2013年01期

7 余慶;;分布式文件系統(tǒng)FastDFS架構(gòu)剖析[J];程序員;2010年11期

8 劉田甜;李超;胡慶成;張桂剛;;云環(huán)境下多副本管理綜述[J];計(jì)算機(jī)研究與發(fā)展;2011年S3期

9 張敬亮;張軍偉;張建剛;許魯;;藍(lán)鯨文件系統(tǒng)中元數(shù)據(jù)與數(shù)據(jù)隔離技術(shù)[J];計(jì)算機(jī)工程;2010年02期

10 許春聰;黃小猛;吳諾;孫寧偉;楊廣文;;分布式文件系統(tǒng)存儲(chǔ)介質(zhì)評(píng)測(cè)與分析[J];計(jì)算機(jī)學(xué)報(bào);2010年10期

，

本文編號(hào)：2130658