本文關鍵詞：基于Hadoop的云轉碼系統(tǒng)研究及性能優(yōu)化　出處：《北京交通大學》2014年碩士論文　論文類型：學位論文

  更多相關文章： 云計算 Hadoop 云轉碼 HDFS 負載均衡

【摘要】：摘要：目前,視頻流量已經成為互聯網的主要流量,各種視頻應用層出不窮,從數字高清電視到IPTV�；ヂ摼W用戶使用視頻應用的終端也日益多樣化,從PC到手機。然而,不同的網絡視頻平臺和終端支持的視頻內容和格式,如編碼格式、分辨率、幀率等參數不盡相同。為了滿足不同平臺和用戶的視頻服務需求,往往需要對視頻進行轉碼,即進行相應的編碼格式、分辨率和幀率等格式轉換。視頻轉碼是一項非常耗時耗資源的工作,隨著視頻數量的急劇增長,傳統(tǒng)的單機或者集中式轉碼已經不能滿足人們對效率和質量的要求。而云計算通過集中、分配資源可以提供強大的計算能力,并且有良好的擴展性和較高的容錯能力。所以可以將視頻轉碼工作轉移到云計算平臺上。采用云平臺進行視頻轉碼,不僅可以承受海量視頻數據的存儲、轉碼需求,同時由于云計算本身具有的資源聚集特性,取用方便,費用低廉。在眾多的云計算平臺中,Hadoop由于其開源特性,是目前應用最為廣泛的云計算平臺。 本論文首先設計和實現了基于Hadoop的云轉碼系統(tǒng)。該系統(tǒng)利用MapReduce分布式機制進行視頻轉碼。系統(tǒng)包括代理服務器,視頻轉碼模塊,Cache模塊三大組件。代理服務器負責處理用戶的視頻服務請求,視頻轉碼模塊負責視頻處理工作,Cache模塊負責管理原視頻和轉碼后的視頻文件。 接著,論文對所實現的轉碼系統(tǒng)的性能進行了測試和分析。比較該系統(tǒng)與單機的視頻轉碼性能,測試分析了分段數量和分段大小對系統(tǒng)轉碼性能的影響,分析了各個階段在系統(tǒng)執(zhí)行過程中所占的時間比例。 在系統(tǒng)的執(zhí)行過程中,視頻文件需要進行多次對HDFS進行讀寫,當前HDFS讀數據時副本選擇策略是選擇離客戶端網絡拓撲距離最近的節(jié)點,當熱門副本集中在同一節(jié)點或者一個機架內時,用戶就會對有限的資源進行激烈的競爭,造成該節(jié)點或者該機架的負載大大增加,從而影響整個集群的性能。為了克服該不足,論文提出了基于負載均衡的副本選擇策略,使用線性加權法定量描述節(jié)點的負載量,選擇負載量最輕的節(jié)點作為讀取節(jié)點。仿真實驗表明,改進的算法有效減少了副本傳輸時間,增加了HDFS集群的吞吐率。
[Abstract]:Absrtact: at present, video traffic has become the main flow of the Internet, a variety of video applications emerge in endlessly, from digital HDTV to IPTV.Internet users using video applications are increasingly diverse. However, different network video platforms and terminals support video content and formats, such as encoding formats, resolution. Frame rate and other parameters are different. In order to meet the needs of different platforms and users, it is often necessary to transcode the video, that is, the corresponding coding format. Video transcoding is a very time-consuming and resource-intensive task, with the rapid growth of the number of videos. Traditional single-machine or centralized transcoding can not meet the requirements of efficiency and quality. Cloud computing can provide powerful computing power through centralized allocation of resources. And has good expansibility and high fault-tolerant ability, so we can transfer the work of video transcoding to cloud computing platform. Using cloud platform for video transcoding, not only can withstand the massive storage of video data. Transcoding requirements, at the same time due to cloud computing itself has the characteristics of resource aggregation, easy to use, low cost. Hadoop in many cloud computing platforms due to its open source features. Is the most widely used cloud computing platform. This paper first designs and implements a cloud transcoding system based on Hadoop. The system uses MapReduce distributed mechanism to transcode video. The system includes proxy server and video transcoding module. The proxy server is responsible for handling the user's video service request and the video transcoding module is responsible for the video processing. The Cache module is responsible for managing the original video and the video files after transcoding. Then, the performance of the transcoding system is tested and analyzed. The video transcoding performance of the system is compared with that of a single machine. The effects of the number of segments and the size of segments on the transcoding performance of the system are tested and analyzed. The time ratio of each stage in the process of system execution is analyzed. During the execution of the system, the video file needs to read and write the HDFS several times. The current replica selection strategy when HDFS reads the data is to select the node nearest to the client network topology. When the hot copy is concentrated in the same node or a rack, the user will compete for the limited resources, resulting in the load of the node or rack increased greatly. Therefore, the performance of the whole cluster is affected. In order to overcome this deficiency, a replica selection strategy based on load balancing is proposed, which uses linear weighted quantification to describe the load of nodes. The simulation results show that the improved algorithm can effectively reduce the copy transfer time and increase the throughput of the HDFS cluster.
【學位授予單位】：北京交通大學
【學位級別】：碩士
【學位授予年份】：2014
【分類號】：TN919.81

【參考文獻】

相關期刊論文 前4條

1 朱文武;;多媒體云計算[J];電子產品世界;2011年09期

2 唐丹,金海,張永坤;集群動態(tài)負載平衡系統(tǒng)的性能評價[J];計算機學報;2004年06期

3 苗秀;俞俊生;劉紹華;陳曉東;;基于云計算平臺的移動IPTV系統(tǒng)設計及負載均衡技術研究[J];軟件;2011年01期

4 胡旭邁,任金昌;一種基于GOP的MPEG-2媒體流切割與合并方法[J];微型電腦應用;2005年07期

，

本文編號：1431546