發(fā)布時(shí)間：2018-11-08 16:09

【摘要】：為了在保證數(shù)據(jù)訪問(wèn)性能的同時(shí)降低系統(tǒng)的冗余存儲(chǔ)開(kāi)銷,分布式存儲(chǔ)系統(tǒng)通常會(huì)采用異步編碼技術(shù)。在新數(shù)據(jù)被寫(xiě)入時(shí),系統(tǒng)使用多副本機(jī)制對(duì)這些數(shù)據(jù)進(jìn)行存儲(chǔ),并在數(shù)據(jù)訪問(wèn)變少后,在后臺(tái)將這些數(shù)據(jù)轉(zhuǎn)化為糾刪碼方式存儲(chǔ)。由于分布式系統(tǒng)通常采用隨機(jī)分布的數(shù)據(jù)塊放置方法,邏輯地址連續(xù)的數(shù)據(jù)塊通常會(huì)分散在系統(tǒng)的所有節(jié)點(diǎn)中。因此在執(zhí)行編碼操作時(shí),編碼進(jìn)程需要通過(guò)跨機(jī)架下載來(lái)獲取數(shù)據(jù)塊。而在編碼完成后,又需要跨機(jī)架的數(shù)據(jù)塊重新分布來(lái)保證數(shù)據(jù)的可靠性。這種方法即降低了異步編碼操作的執(zhí)行效率,也影響了系統(tǒng)中前臺(tái)任務(wù)進(jìn)程的性能。為了提高異步編碼的執(zhí)行效率并降低其對(duì)前臺(tái)任務(wù)性能的影響,本文提出了一種新型的編碼條帶構(gòu)建方式,我們稱之為動(dòng)態(tài)條帶構(gòu)建技術(shù)(Dynamic Stripe Con-structiom,DSC)。DSC根據(jù)當(dāng)前系統(tǒng)中數(shù)據(jù)塊的放置信息來(lái)組建編碼條帶。放入同一編碼條帶中的數(shù)據(jù)塊需要滿足以下兩種性質(zhì):(1)這些數(shù)據(jù)塊存在副本存儲(chǔ)于同一機(jī)架中,以保證在編碼時(shí)不會(huì)引起跨機(jī)架的數(shù)據(jù)塊下載;(2)這些數(shù)據(jù)塊存在副本分散在其他獨(dú)立的機(jī)架中,以保證編碼完成后不會(huì)引起跨機(jī)架的數(shù)據(jù)塊重新分布。為了在龐大的選擇空間中有效地組建編碼條帶,我們?cè)O(shè)計(jì)了一種管理數(shù)據(jù)塊放置信息的數(shù)據(jù)結(jié)構(gòu),并基于這一數(shù)據(jù)結(jié)構(gòu)提出了一種線性時(shí)間復(fù)雜度的動(dòng)態(tài)條帶構(gòu)建算法。該算法可以以熱插拔的方式應(yīng)用于使用任何數(shù)據(jù)放置方式與糾刪碼配置的分布式集群。為了驗(yàn)證動(dòng)態(tài)條帶構(gòu)建技術(shù)的有效性,我們將DSC實(shí)現(xiàn)在HDFS系統(tǒng)上。在真實(shí)集群的測(cè)試實(shí)驗(yàn)中,DSC可以顯著的提高異步編碼的執(zhí)行效率(實(shí)驗(yàn)中最高改進(jìn)可達(dá)81%),并降低其對(duì)前臺(tái)任務(wù)進(jìn)程的影響。在系統(tǒng)集成的過(guò)程中,我們首先探討了異步編碼中節(jié)點(diǎn)上數(shù)據(jù)局部性與負(fù)載均衡的問(wèn)題,隨后設(shè)計(jì)了文件間編碼與迭代編碼技術(shù)來(lái)優(yōu)化異步編碼在小文件與追加文件場(chǎng)景下的應(yīng)用。為了適應(yīng)分布式集群中不斷變化的數(shù)據(jù)訪問(wèn)負(fù)載,我們還提出了一種將動(dòng)態(tài)副本與糾刪碼結(jié)合的新型數(shù)據(jù)塊管理架構(gòu)。這種架構(gòu)模式使得我們可以對(duì)系統(tǒng)中的數(shù)據(jù)塊進(jìn)行動(dòng)態(tài)的管理,以在提高數(shù)據(jù)可靠性與訪問(wèn)性能的同時(shí)最小化系統(tǒng)的存儲(chǔ)開(kāi)銷。
[Abstract]:In order to ensure the performance of data access and reduce the redundant storage overhead, asynchronous coding is usually used in distributed storage systems. When the new data is written, the system uses multi-replica mechanism to store the data, and after the data access becomes less, the data is converted into erasure code storage in the background. Since distributed systems usually use randomly distributed data blocks, logical address blocks are usually scattered across all nodes of the system. Therefore, when performing encoding operations, the encoding process needs to obtain blocks of data through cross-rack downloads. After the coding is completed, the data blocks across the rack need to be redistributed to ensure the reliability of the data. This method not only reduces the efficiency of asynchronous coding operation, but also affects the performance of foreground task process in the system. In order to improve the efficiency of asynchronous coding and reduce its impact on the performance of foreground tasks, this paper proposes a new coding band construction method, which we call dynamic stripe construction technology (Dynamic Stripe Con-structiom,). DSC). DSC constructs coding bands based on the placement information of data blocks in the current system. The data blocks placed in the same coding strip need to satisfy the following two properties: (1) the data blocks are stored in the same rack in order to ensure that the data blocks across the frame will not be downloaded; (2) the existing copies of these blocks are scattered in other independent frames to ensure that the data blocks across the rack will not be redistributed after the coding is completed. In order to construct coding bands effectively in a large selection space, we design a data structure that manages the information placed in blocks of data. Based on this data structure, we propose a dynamic stripe construction algorithm with linear time complexity. The algorithm can be applied to distributed clusters using any data placement and erasure code configuration. In order to verify the effectiveness of dynamic stripe construction technology, we implement DSC on HDFS system. In the real cluster test, DSC can significantly improve the efficiency of asynchronous coding (up to 81% in the experiment) and reduce its impact on the foreground task process. In the process of system integration, we first discuss the problem of data locality and load balancing in asynchronous coding. Then we design inter-file coding and iterative coding techniques to optimize the application of asynchronous coding in small file and append file scenarios. In order to adapt to the changing data access load in distributed cluster, we also propose a new data block management architecture which combines dynamic replica with erasure code. This architecture pattern enables us to dynamically manage the data blocks in the system to minimize the storage overhead while improving the data reliability and access performance.
【學(xué)位授予單位】：中國(guó)科學(xué)技術(shù)大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP333

【相似文獻(xiàn)】

相關(guān)期刊論文 前10條

1 ;廉價(jià)、高效、穩(wěn)定 微軟新一代分布式存儲(chǔ)系統(tǒng)[J];新電腦;2006年06期

2 何公明;張?jiān)獫?;面向數(shù)字媒體的高性能分布式存儲(chǔ)系統(tǒng)的研究與應(yīng)用[J];廣播電視信息;2009年10期

3 范劍波,郭建康;分布式存儲(chǔ)系統(tǒng)性能模型的建立與應(yīng)用[J];計(jì)算機(jī)工程與應(yīng)用;2001年13期

4 范劍波,徐利浩;分布式存儲(chǔ)系統(tǒng)可靠性的研究[J];計(jì)算機(jī)工程;2001年06期

5 吳英;謝廣軍;劉t，

本文編號(hào)：2318997