LAMOST科學(xué)計算云平臺系統(tǒng)的構(gòu)建與應(yīng)用
發(fā)布時間：2019-01-23 20:04
【摘要】：隨著探測器和空間技術(shù)的發(fā)展,天文觀測從可見光、射電波段擴展到包括紅外、紫外、X射線和γ射線在內(nèi)的電磁波各個波段,形成了全波段天文學(xué),現(xiàn)發(fā)展到了一個全新的階段,即全波段-大樣本-巨信息量時期。天文學(xué)已然成為各學(xué)科中擁有海量數(shù)據(jù)的龍頭老大,由于天文數(shù)據(jù)量的龐大和增長速度的迅猛,這些巡天項目產(chǎn)生的數(shù)據(jù)量通�？梢赃_(dá)到TB甚至PB級。如斯隆數(shù)字巡天SDSS,用了十年時間來覆蓋8000平方度的天空,得到大約108個恒星、星系及類星體的大約40TB的成像及光譜數(shù)據(jù)。 隨著LAMOST巡天計劃的開展,要完成對1000萬個星系、100萬個類星體及1000萬顆恒星光譜的觀測,將產(chǎn)生的數(shù)據(jù)將會是SDSS的十倍之多,對海量數(shù)據(jù)的存儲和處理將會是一個極大的挑戰(zhàn),本文針對LAMOST的需求,對海量光譜的數(shù)據(jù)存儲和處理構(gòu)建了一套適合天文數(shù)據(jù)處理的科學(xué)計算平臺并設(shè)計并實現(xiàn)了可定制的云儲存系統(tǒng)。 本文主要工作如下: 1、在LAMOST數(shù)據(jù)處理中心的24臺服務(wù)器上構(gòu)建了一套基于Hadoop開源框架并適合天文數(shù)據(jù)處理的科學(xué)計算平臺,其中包含NumPy、SciPy、PyFITS等常用的工具包。使用Python和Shell完成自動部署的程序包,以方便快捷地添加刪除物理節(jié)點以及設(shè)置負(fù)載均衡。 2、基于Hadoop核心組件HDFS,設(shè)計并實現(xiàn)了多用戶的云存儲系統(tǒng),為用戶提供了新建文件夾、文件上傳、下載文件/文件夾、刪除文件/文件夾、回收站、記事本及個人信息管理等功能。另外,管理員角色擁有賬號管理(包括新增、修改、配額、刪除等操作)、單位管理及系統(tǒng)信息查詢功能等。用戶利用該平臺可以方便地存儲相關(guān)數(shù)據(jù)和處理結(jié)果等。 3、研究了科學(xué)計算平臺的核心組件MapReduce編程模型。在目前較完善的模板匹配算法基礎(chǔ)上,使用MapReduce編程規(guī)范完成模板匹配,使用KNN和卡方最小化算法對數(shù)據(jù)進(jìn)行了測試來驗證改進(jìn)之后的算法,并分別在單機和集群環(huán)境下進(jìn)行了性能對比分析。
[Abstract]:With the development of detectors and space technology, astronomical observation extends from visible light, radio wave band to electromagnetic wave band including infrared, ultraviolet, X ray and 緯 ray, forming full band astronomy. Now it has reached a new stage, that is, the period of full-band-large sample-huge information. Astronomy has become the leader in the field of science with huge amounts of data. Due to the large amount of astronomical data and the rapid growth of astronomical data, the amount of data generated by these survey projects can usually reach TB or even PB level. For example, the Sloan Digital Sky Survey (SDSS,) took 10 years to cover 8000 square degrees of sky, obtaining about 108 stars, galaxies and quasars about 40TB imaging and spectral data. With the launch of the LAMOST survey program, the spectral observations of 10 million galaxies, 1 million quasars and 10 million stars will produce ten times as much data as SDSS, which will pose a great challenge to the storage and processing of massive data. In order to meet the requirements of LAMOST, a scientific computing platform for astronomical data processing is constructed and a customizable cloud storage system is designed and implemented. The main work of this paper is as follows: 1. A set of scientific computing platform based on Hadoop open source framework and suitable for astronomical data processing is built on 24 servers of LAMOST data processing center, which includes NumPy,SciPy,PyFITS and other commonly used toolkits. Use Python and Shell to complete automatic deployment packages to add and delete physical nodes and set load balancing quickly. 2. Based on Hadoop core component HDFS, a multi-user cloud storage system is designed and implemented, which provides users with new folder, file upload, download file / folder, delete file / folder, recycle bin, etc. Notepad and personal information management functions. In addition, the administrator role has account management (including new, modified, quota, delete and other operations), unit management and system information query functions. Users can conveniently store relevant data and process results by using the platform. 3. The MapReduce programming model of the core component of scientific computing platform is studied. On the basis of the current perfect template matching algorithm, we use MapReduce programming specification to complete template matching, and use KNN and chi-square minimization algorithm to test the data to verify the improved algorithm. The performance comparison and analysis are carried out in single machine and cluster environment respectively.
【學(xué)位授予單位】：山東大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2013
【分類號】：TP333
本文編號：2414149
[Abstract]:With the development of detectors and space technology, astronomical observation extends from visible light, radio wave band to electromagnetic wave band including infrared, ultraviolet, X ray and 緯 ray, forming full band astronomy. Now it has reached a new stage, that is, the period of full-band-large sample-huge information. Astronomy has become the leader in the field of science with huge amounts of data. Due to the large amount of astronomical data and the rapid growth of astronomical data, the amount of data generated by these survey projects can usually reach TB or even PB level. For example, the Sloan Digital Sky Survey (SDSS,) took 10 years to cover 8000 square degrees of sky, obtaining about 108 stars, galaxies and quasars about 40TB imaging and spectral data. With the launch of the LAMOST survey program, the spectral observations of 10 million galaxies, 1 million quasars and 10 million stars will produce ten times as much data as SDSS, which will pose a great challenge to the storage and processing of massive data. In order to meet the requirements of LAMOST, a scientific computing platform for astronomical data processing is constructed and a customizable cloud storage system is designed and implemented. The main work of this paper is as follows: 1. A set of scientific computing platform based on Hadoop open source framework and suitable for astronomical data processing is built on 24 servers of LAMOST data processing center, which includes NumPy,SciPy,PyFITS and other commonly used toolkits. Use Python and Shell to complete automatic deployment packages to add and delete physical nodes and set load balancing quickly. 2. Based on Hadoop core component HDFS, a multi-user cloud storage system is designed and implemented, which provides users with new folder, file upload, download file / folder, delete file / folder, recycle bin, etc. Notepad and personal information management functions. In addition, the administrator role has account management (including new, modified, quota, delete and other operations), unit management and system information query functions. Users can conveniently store relevant data and process results by using the platform. 3. The MapReduce programming model of the core component of scientific computing platform is studied. On the basis of the current perfect template matching algorithm, we use MapReduce programming specification to complete template matching, and use KNN and chi-square minimization algorithm to test the data to verify the improved algorithm. The performance comparison and analysis are carried out in single machine and cluster environment respectively.
【學(xué)位授予單位】：山東大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2013
【分類號】：TP333
【參考文獻(xiàn)】
相關(guān)期刊論文 前7條
1 劉杰;潘景昌;韋鵬;劉猛;羅阿理;;基于光譜相似度的恒星大氣參數(shù)自動測量方法[J];光譜學(xué)與光譜分析;2012年12期
2 姜斌;衣振萍;馬紹漢;;一種基于PCA和系統(tǒng)成團(tuán)法的聚類軟件設(shè)計[J];計算機科學(xué);2008年04期
3 李成華;張新訪;金海;向文;;MapReduce:新型的分布式并行計算編程模型[J];計算機工程與科學(xué);2011年03期
4 傅穎勛;羅圣美;舒繼武;;安全云存儲系統(tǒng)與關(guān)鍵技術(shù)綜述[J];計算機研究與發(fā)展;2013年01期
5 張彥霞,趙永恒,崔辰州;天文學(xué)中的數(shù)據(jù)挖掘和知識發(fā)現(xiàn)[J];天文學(xué)進(jìn)展;2002年04期
6 侯建;帥仁俊;侯文;;基于云計算的海量數(shù)據(jù)存儲模型[J];通信技術(shù);2011年05期
7 褚耀泉;;LAMOST科學(xué)觀測計劃[J];中國科學(xué)技術(shù)大學(xué)學(xué)報;2007年06期
相關(guān)碩士學(xué)位論文 前3條
1 鄒彩輝;基于Hadoop平臺的自適應(yīng)局部超平面K近鄰算法的研究[D];華南理工大學(xué);2011年
2 董偉祥;LAMOST恒星大氣參數(shù)提取系統(tǒng)[D];山東大學(xué);2010年
3 劉杰;基于模板匹配的恒星大氣物理參數(shù)自動測量的研究[D];山東大學(xué);2012年
，本文編號：2414149
本文鏈接：http://sikaile.net/kejilunwen/jisuanjikexuelunwen/2414149.html
最近更新
教材專著
