澶辨晥閾炬帴澶勭悊 |
銆奌adoop鎶€鏈唴騫曪細娣卞叆瑙f瀽YARN鏋舵瀯璁捐涓庡疄鐜板師鐞嗐€婸DF 涓嬭澆
杞澆鑷細https://download.csdn.net/download/qq_20064763/10767644
涓嬭澆鍦板潃錛?/strong>
鐗堟潈褰掑嚭鐗堢ぞ鍜屽師浣滆€呮墍鏈夛紝閾炬帴宸插垹闄わ紝璇瘋喘涔版鐗?/b>
鐢?shù)瀛愮増浠呬緵棰勮锛屾敮鎸佹鐗堝Q屽枩嬈㈢殑璇瘋喘涔版鐗堜功綾嶏細銆奌adoop鎶€鏈唴騫曪細娣卞叆瑙f瀽YARN鏋舵瀯璁捐涓庡疄鐜板師鐞嗐€?/a>
鍥句功綆€浠嬶細
鏈功鏄?ldquo;Hadoop鎶€鏈唴騫?rdquo;緋誨垪鐨勭3鏈功錛屽墠闈袱鏈垎鍒Common銆丠DFS鍜孧apReduce榪涜浜嗘繁鍏ュ垎鏋愬拰璁茶В錛岃耽寰椾簡鏋佸ソ鐨勫彛紕戯紝Hadoop棰嗗煙鍑犱箮浜烘墜涓€鍐岋紝鏈功鍒欏YARN灞曞紑浜嗘繁鍏ョ殑鎺㈣錛屾槸棣栭儴鍏充簬YARN鐨勪笓钁椼€備粛鐒剁敱璧勬繁Hadoop鎶€鏈笓瀹惰懀瑗挎垚鎵х瑪錛屾牴鎹渶鏂扮殑Hadoop2.0鐗堟湰鎾板啓錛屾潈濞佺ぞ鍖篊hinaHadoop榧庡姏鎺ㄨ崘銆?/span> 鏈功浠庡簲鐢ㄨ搴︾郴緇熻瑙d簡YARN鐨勫熀鏈簱鍜岀粍浠剁敤娉曘€佸簲鐢ㄧ▼搴忚璁℃柟娉曘€乊ARN涓婃祦琛岀殑鍚勭璁$畻妗嗘灦錛圡apReduce銆乀ez銆丼torm銆丼park錛夛紝浠ュ強澶氫釜綾籝ARN鐨勫紑婧愯祫婧愮鐞嗙郴緇燂紙Corona鍜孧esos錛夛紱浠庢簮浠g爜瑙掑害娣卞叆鍒嗘瀽YARN鐨勮璁$悊蹇典笌鍩烘湰鏋舵瀯銆佸悇涓粍浠剁殑瀹炵幇鍘熺悊錛屼互鍙婂悇縐嶈綆楁鏋剁殑瀹炵幇緇嗚妭銆?/span> 鍏ㄤ功鍏卞洓閮ㄥ垎13绔狅細絎竴閮ㄥ垎錛堢1~2绔狅級涓昏浠嬬粛浜嗗浣曡幏鍙栥€侀槄璇誨拰璋冭瘯Hadoop鐨勬簮浠g爜錛屼互鍙奩ARN鐨勮璁℃€濇兂銆佸熀鏈灦鏋勫拰宸ヤ綔嫻佺▼錛涚浜岄儴鍒嗭紙絎?~7绔狅級緇撳悎婧愪唬鐮佽緇嗗墫鏋愬拰璁茶В浜哬ARN鐨勭涓夋柟寮€婧愬簱銆佸簳灞傞€氫俊搴撱€佹湇鍔″簱銆佷簨浠跺簱鐨勫熀鏈嬌鐢ㄥ拰瀹炵幇緇嗚妭錛岃緇嗚瑙d簡YARN鐨勫簲鐢ㄧ▼搴忚璁℃柟娉曪紝娣卞叆璁茶В鍜屽垎鏋愪簡ResourceManager銆佽祫婧愯皟搴﹀櫒銆丯odeManager絳夌粍浠剁殑瀹炵幇緇嗚妭錛涚涓夌瘒錛堢8~10绔狅級鍒欏紱葷嚎璁$畻妗嗘灦MapReduce銆丏AG璁$畻妗嗘灦Tez銆佸疄鏃惰綆楁鏋禨torm鍜屽唴瀛樿綆楁鏋禨park榪涜浜嗚緇嗙殑璁茶В錛涚鍥涢儴鍒嗭紙絎?1~13绔狅級棣栧厛瀵笷acebookCorona鍜孉pacheMesos榪涜浜嗘繁鍏ヨ瑙o紝鐒跺悗瀵筜ARN鐨勫彂灞曡秼鍔胯繘琛屼簡灞曟湜銆傞檮褰曢儴鍒嗘敹褰曚簡YARN瀹夎鎸囧崡銆乊ARN閰嶇疆鍙傛暟浠ュ強HadoopShell鍛戒護絳夐潪甯告湁鐢ㄧ殑璧勬枡銆?/span>
鐩稿叧鎴浘錛?br />
![]()
鍥句功鐩綍錛?/strong>
鍓嶃€€璦€ 絎竴閮ㄥ垎銆€鍑嗗綃?/span> 絎?绔犮€€鐜鍑嗗 2 1.1銆€鍑嗗瀛︿範鐜 2 1.1.1銆€鍩虹杞歡涓嬭澆 2 1.1.2銆€濡備綍鍑嗗Linux鐜 3 1.2銆€鑾峰彇Hadoop婧愪唬鐮?5 1.3銆€鎼緩Hadoop婧愪唬鐮侀槄璇葷幆澧?5 1.3.1銆€鍒涘緩Hadoop宸ョ▼ 5 1.3.2銆€Hadoop婧愪唬鐮侀槄璇繪妧宸?8 1.4銆€Hadoop婧愪唬鐮佺粍緇囩粨鏋?10 1.5銆€Hadoop鍒濅綋楠?12 1.5.1銆€鎼緩Hadoop鐜 12 1.5.2銆€Hadoop Shell浠嬬粛 15 1.6銆€緙栬瘧鍙婅皟璇旽adoop婧愪唬鐮?16 1.6.1銆€緙栬瘧Hadoop婧愪唬鐮?17 1.6.2銆€璋冭瘯Hadoop婧愪唬鐮?18 1.7銆€灝忕粨 20 絎?绔犮€€YARN璁捐鐞嗗康涓庡熀鏈灦鏋?21 2.1銆€YARN浜х敓鑳屾櫙 21 2.1.1銆€MRv1鐨勫眬闄愭€?21 2.1.2銆€杞婚噺綰у脊鎬ц綆楀鉤鍙?22 2.2銆€Hadoop鍩虹鐭ヨ瘑 23 2.2.1銆€鏈瑙i噴 23 2.2.2銆€Hadoop鐗堟湰鍙樿縼 25 2.3銆€YARN鍩烘湰璁捐鎬濇兂 29 2.3.1銆€鍩烘湰妗嗘灦瀵規(guī)瘮 29 2.3.2銆€緙栫▼妯″瀷瀵規(guī)瘮 30 2.4銆€YARN 鍩烘湰鏋舵瀯 31 2.4.1銆€YARN鍩烘湰緇勬垚緇撴瀯 32 2.4.2銆€YARN閫氫俊鍗忚 34 2.5銆€YARN宸ヤ綔嫻佺▼ 35 2.6銆€澶氳搴︾悊瑙ARN 36 2.6.1銆€騫惰緙栫▼ 36 2.6.2銆€璧勬簮綆$悊緋葷粺 36 2.6.3銆€浜戣綆?37 2.7銆€鏈功娑夊強鍐呭 38 2.8銆€灝忕粨 38 絎簩閮ㄥ垎銆€YARN鏍稿績璁捐綃?/span> 絎?绔犮€€YARN鍩虹搴?40 3.1銆€姒傝堪 40 3.2銆€絎笁鏂瑰紑婧愬簱 41 3.2.1銆€Protocol Buffers 41 3.2.2銆€Apache Avro 43 3.3銆€搴曞眰閫氫俊搴?46 3.3.1銆€RPC閫氫俊妯″瀷 46 3.3.2銆€Hadoop RPC鐨勭壒鐐規(guī)榪?48 3.3.3銆€RPC鎬諱綋鏋舵瀯 48 3.3.4銆€Hadoop RPC浣跨敤鏂規(guī)硶 49 3.3.5銆€Hadoop RPC綾昏瑙?51 3.3.6銆€Hadoop RPC鍙傛暟璋冧紭 57 3.3.7銆€YARN RPC瀹炵幇 57 3.3.8銆€YARN RPC搴旂敤瀹炰緥 61 3.4銆€鏈嶅姟搴撲笌浜嬩歡搴?65 3.4.1銆€鏈嶅姟搴?66 3.4.2銆€浜嬩歡搴?66 3.4.3銆€YARN鏈嶅姟搴撳拰浜嬩歡搴撶殑浣跨敤鏂規(guī)硶 68 3.4.4銆€浜嬩歡椹卞姩甯︽潵鐨勫彉鍖?70 3.5銆€鐘舵€佹満搴?72 3.5.1銆€YARN鐘舵€佽漿鎹㈡柟寮?72 3.5.2銆€鐘舵€佹満綾?73 3.5.3銆€鐘舵€佹満鐨勪嬌鐢ㄦ柟娉?73 3.5.4銆€鐘舵€佹満鍙鍖?76 3.6銆€婧愪唬鐮侀槄璇誨紩瀵?76 3.7銆€灝忕粨 77 3.8銆€闂璁ㄨ 77 絎?绔犮€€YARN搴旂敤紼嬪簭璁捐鏂規(guī)硶 78 4.1銆€姒傝堪 78 4.2銆€瀹㈡埛绔璁?79 4.2.1銆€瀹㈡埛绔紪鍐欐祦紼?80 4.2.2銆€瀹㈡埛绔紪紼嬪簱 84 4.3銆€ApplicationMaster璁捐 84 4.3.1銆€ApplicationMaster緙栧啓嫻佺▼ 84 4.3.2銆€ApplicationMaster緙栫▼搴?92 4.4銆€YARN 搴旂敤紼嬪簭瀹炰緥 95 4.4.1銆€DistributedShell 95 4.4.2銆€Unmanaged AM 99 4.5銆€婧愪唬鐮侀槄璇誨紩瀵?100 4.6銆€灝忕粨 100 4.7銆€闂璁ㄨ 100 絎?绔犮€€ResourceManager鍓栨瀽 102 5.1銆€姒傝堪 102 5.1.1銆€ResourceManager鍩烘湰鑱岃兘 102 5.1.2銆€ResourceManager鍐呴儴鏋舵瀯 103 5.1.3銆€ResourceManager浜嬩歡涓庝簨浠跺鐞嗗櫒 106 5.2銆€鐢ㄦ埛浜や簰妯″潡 108 5.2.1銆€ClientRMService 108 5.2.2銆€AdminService 109 5.3銆€ApplicationMaster綆$悊 109 5.4銆€NodeManager綆$悊 112 5.5銆€Application綆$悊 113 5.6銆€鐘舵€佹満綆$悊 114 5.6.1銆€RMApp鐘舵€佹満 115 5.6.2銆€RMAppAttempt鐘舵€佹満 119 5.6.3銆€RMContainer鐘舵€佹満 123 5.6.4銆€RMNode鐘舵€佹満 127 5.7銆€鍑犱釜甯歌琛屼負鍒嗘瀽 129 5.7.1銆€鍚姩ApplicationMaster 129 5.7.2銆€鐢寵涓庡垎閰岰ontainer 132 5.7.3銆€鏉€姝籄pplication 134 5.7.4銆€Container瓚呮椂 135 5.7.5銆€ApplicationMaster瓚呮椂 138 5.7.6銆€NodeManager瓚呮椂 138 5.8銆€瀹夊叏綆$悊 139 5.8.1銆€鏈浠嬬粛 139 5.8.2銆€Hadoop璁よ瘉鏈哄埗 139 5.8.3銆€Hadoop鎺堟潈鏈哄埗 142 5.9銆€瀹歸敊鏈哄埗 144 5.9.1銆€Hadoop HA鍩烘湰妗嗘灦 145 5.9.2銆€YARN HA瀹炵幇 148 5.10銆€婧愪唬鐮侀槄璇誨紩瀵?149 5.11銆€灝忕粨 151 5.12銆€闂璁ㄨ 152 絎?绔犮€€璧勬簮璋冨害鍣?153 6.1銆€璧勬簮璋冨害鍣ㄨ儗鏅?153 6.2銆€HOD璋冨害鍣?154 6.2.1銆€Torque璧勬簮綆$悊鍣?154 6.2.2銆€HOD浣滀笟璋冨害 155 6.3銆€YARN璧勬簮璋冨害鍣ㄧ殑鍩烘湰鏋舵瀯 157 6.3.1銆€鍩烘湰鏋舵瀯 157 6.3.2銆€璧勬簮琛ㄧず妯″瀷 160 6.3.3銆€璧勬簮璋冨害妯″瀷 161 6.3.4銆€璧勬簮鎶㈠崰妯″瀷 164 6.4銆€YARN灞傜駭闃熷垪綆$悊鏈哄埗 169 6.4.1銆€灞傜駭闃熷垪綆$悊鏈哄埗 169 6.4.2銆€闃熷垪鍛藉悕瑙勫垯 171 6.5銆€Capacity Scheduler 172 6.5.1銆€Capacity Scheduler鐨勫姛鑳?172 6.5.2銆€Capacity Scheduler瀹炵幇 176 6.6銆€Fair Scheduler 179 6.6.1銆€Fair Scheduler鍔熻兘浠嬬粛 180 6.6.2銆€Fair Scheduler瀹炵幇 182 6.6.3銆€Fair Scheduler涓嶤apacity Scheduler瀵規(guī)瘮 183 6.7銆€鍏朵粬璧勬簮璋冨害鍣ㄤ粙緇?184 6.8銆€婧愪唬鐮侀槄璇誨紩瀵?185 6.9銆€灝忕粨 186 6.10銆€闂璁ㄨ 187 絎?绔犮€€NodeManager鍓栨瀽 188 7.1銆€姒傝堪 188 7.1.1銆€NodeManager鍩烘湰鑱岃兘 188 7.1.2銆€NodeManager鍐呴儴鏋舵瀯 190 7.1.3銆€NodeManager浜嬩歡涓庝簨浠跺鐞嗗櫒 193 7.2銆€鑺傜偣鍋ュ悍鐘跺喌媯€嫻?194 7.2.1銆€鑷畾涔塖hell鑴氭湰 194 7.2.2銆€媯€嫻嬬鐩樻崯鍧忔暟鐩?196 7.3銆€鍒嗗竷寮忕紦瀛樻満鍒?196 7.3.1銆€璧勬簮鍙鎬т笌鍒嗙被 198 7.3.2銆€鍒嗗竷寮忕紦瀛樺疄鐜?200 7.4銆€鐩綍緇撴瀯綆$悊 203 7.4.1銆€鏁版嵁鐩綍綆$悊 203 7.4.2銆€鏃ュ織鐩綍綆$悊 203 7.5銆€鐘舵€佹満綆$悊 206 7.5.1銆€Application鐘舵€佹満 207 7.5.2銆€Container鐘舵€佹満 210 7.5.3銆€LocalizedResource鐘舵€佹満 213 7.6銆€Container鐢熷懡鍛ㄦ湡鍓栨瀽 214 7.6.1銆€Container璧勬簮鏈湴鍖?214 7.6.2銆€Container榪愯 218 7.6.3銆€Container璧勬簮娓呯悊 222 7.7銆€璧勬簮闅旂 224 7.7.1銆€Cgroups浠嬬粛 224 7.7.2銆€鍐呭瓨璧勬簮闅旂 228 7.7.3銆€CPU璧勬簮闅旂 230 7.8銆€婧愪唬鐮侀槄璇誨紩瀵?234 7.9銆€灝忕粨 235 7.10銆€闂璁ㄨ 236 絎笁閮ㄥ垎銆€璁$畻妗嗘灦綃?/span> 絎?绔犮€€紱葷嚎璁$畻妗嗘灦MapReduce 238 8.1銆€姒傝堪 238 8.1.1銆€鍩烘湰鏋勬垚 238 8.1.2銆€浜嬩歡涓庝簨浠跺鐞嗗櫒 240 8.2銆€MapReduce瀹㈡埛绔?241 8.2.1銆€ApplicationClientProtocol鍗忚 242 8.2.2銆€MRClientProtocol鍗忚 243 8.3銆€MRAppMaster宸ヤ綔嫻佺▼ 243 8.4銆€MR浣滀笟鐢熷懡鍛ㄦ湡鍙婄浉鍏崇姸鎬佹満 246 8.4.1銆€MR浣滀笟鐢熷懡鍛ㄦ湡 246 8.4.2銆€Job鐘舵€佹満 249 8.4.3銆€Task鐘舵€佹満 253 8.4.4銆€TaskAttempt鐘舵€佹満 255 8.5銆€璧勬簮鐢寵涓庡啀鍒嗛厤 259 8.5.1銆€璧勬簮鐢寵 259 8.5.2銆€璧勬簮鍐嶅垎閰?262 8.6銆€Container鍚姩涓庨噴鏀?263 8.7銆€鎺ㄦ祴鎵ц鏈哄埗 264 8.7.1銆€綆楁硶浠嬬粛 265 8.7.2銆€鎺ㄦ祴鎵ц鐩稿叧綾?266 8.8銆€浣滀笟鎭㈠ 267 8.9銆€鏁版嵁澶勭悊寮曟搸 269 8.10銆€鍘嗗彶浣滀笟綆$悊鍣?271 8.11銆€MRv1涓嶮Rv2瀵規(guī)瘮 273 8.11.1銆€MRv1 On YARN 273 8.11.2銆€MRv1涓嶮Rv2鏋舵瀯姣旇緝 274 8.11.3銆€MRv1涓嶮Rv2緙栫▼鎺ュ彛鍏煎鎬?274 8.12銆€婧愪唬鐮侀槄璇誨紩瀵?275 8.13銆€灝忕粨 277 8.14銆€闂璁ㄨ 277 絎?绔犮€€DAG璁$畻妗嗘灦Tez 278 9.1銆€鑳屾櫙 278 9.2銆€Tez鏁版嵁澶勭悊寮曟搸 281 9.2.1銆€Tez緙栫▼妯″瀷 281 9.2.2銆€Tez鏁版嵁澶勭悊寮曟搸 282 9.3銆€DAG Master瀹炵幇 284 9.3.1銆€DAG緙栫▼妯″瀷 284 9.3.2銆€MR鍒癉AG杞崲 286 9.3.3銆€DAGAppMaster 288 9.4銆€浼樺寲鏈哄埗 291 9.4.1銆€褰撳墠YARN妗嗘灦瀛樺湪鐨勯棶棰?291 9.4.2銆€Tez寮曞叆鐨勪紭鍖栨妧鏈?292 9.5銆€Tez搴旂敤鍦烘櫙 292 9.6銆€涓庡叾浠栫郴緇熸瘮杈?294 9.7銆€灝忕粨 295 絎?0绔犮€€瀹炴椂/鍐呭瓨璁$畻妗嗘灦Storm/Spark 296 10.1銆€Hadoop MapReduce鐨勭煭鏉?296 10.2銆€瀹炴椂璁$畻妗嗘灦Storm 296 10.2.1銆€Storm緙栫▼妯″瀷 297 10.2.2銆€Storm鍩烘湰鏋舵瀯 302 10.2.3銆€Storm On YARN 304 10.3銆€鍐呭瓨璁$畻妗嗘灦Spark 307 10.3.1銆€Spark緙栫▼妯″瀷 308 10.3.2銆€Spark鍩烘湰鏋舵瀯 312 10.3.3銆€Spark On YARN 316 10.3.4銆€Spark/Storm On YARN姣旇緝 317 10.4銆€灝忕粨 317 絎洓閮ㄥ垎銆€楂樼駭綃?/span> 絎?1绔犮€€Facebook Corona鍓栨瀽 320 11.1銆€姒傝堪 320 11.1.1銆€Corona鐨勫熀鏈灦鏋?320 11.1.2銆€Corona鐨凴PC鍗忚涓庡簭鍒楀寲妗嗘灦 322 11.2銆€Corona璁捐鐗圭偣 323 11.2.1銆€鎺ㄥ紡緗戠粶閫氫俊妯″瀷 323 11.2.2銆€鍩轟簬Hadoop 0.20鐗堟湰 324 11.2.3銆€浣跨敤Thrift 324 11.2.4銆€娣卞害闆嗘垚Fair Scheduler 324 11.3銆€宸ヤ綔嫻佺▼浠嬬粛 324 11.3.1銆€浣滀笟鎻愪氦 325 11.3.2銆€璧勬簮鐢寵涓庝換鍔″惎鍔?326 11.4銆€涓昏妯″潡浠嬬粛 327 11.4.1銆€ClusterManager 327 11.4.2銆€CoronaJobTracker 330 11.4.3銆€CoronaTaskTracker 333 11.5銆€灝忕粨 335 絎?2绔犮€€Apache Mesos鍓栨瀽 336 12.1銆€姒傝堪 336 12.2銆€搴曞眰緗戠粶閫氫俊搴?337 12.2.1銆€libprocess鍩烘湰鏋舵瀯 338 12.2.2銆€涓€涓畝鍗曠ず渚?338 12.3銆€Mesos鏈嶅姟 340 12.3.1銆€SchedulerProcess 341 12.3.2銆€Mesos Master 342 12.3.3銆€Mesos Slave 343 12.3.4銆€ExecutorProcess 343 12.4銆€Mesos宸ヤ綔嫻佺▼ 344 12.4.1銆€妗嗘灦娉ㄥ唽榪囩▼ 344 12.4.2銆€Framework Executor娉ㄥ唽榪囩▼ 345 12.4.3銆€璧勬簮鍒嗛厤鍒頒換鍔¤繍琛岃繃紼?345 12.4.4銆€浠誨姟鍚姩榪囩▼ 347 12.4.5銆€浠誨姟鐘舵€佹洿鏂拌繃紼?347 12.5銆€Mesos璧勬簮鍒嗛厤絳栫暐 348 12.5.1銆€Mesos璧勬簮鍒嗛厤妗嗘灦 349 12.5.2銆€Mesos璧勬簮鍒嗛厤綆楁硶 349 12.6銆€Mesos瀹歸敊鏈哄埗 350 12.6.1銆€Mesos Master瀹歸敊 350 12.6.2銆€Mesos Slave瀹歸敊 351 12.7銆€Mesos搴旂敤瀹炰緥 352 12.7.1銆€Hadoop On Mesos 352 12.7.2銆€Storm On Mesos 353 12.8銆€Mesos涓嶻ARN瀵規(guī)瘮 354 12.9銆€灝忕粨 355 絎?3绔犮€€YARN鎬葷粨涓庡彂灞曡秼鍔?356 13.1銆€璧勬簮綆$悊緋葷粺璁捐鍔ㄦ満 356 13.2銆€璧勬簮綆$悊緋葷粺鏋舵瀯婕斿寲 357 13.2.1銆€闆嗕腑寮忔灦鏋?357 13.2.2銆€鍙屽眰璋冨害鏋舵瀯 358 13.2.3銆€鍏變韓鐘舵€佹灦鏋?358 13.3銆€YARN鍙戝睍瓚嬪娍 359 13.3.1銆€YARN鑷韓鐨勫畬鍠?359 13.3.2銆€浠ARN涓烘牳蹇冪殑鐢熸€佺郴緇?361 13.3.3銆€YARN鍛ㄨ竟宸ュ叿鐨勫畬鍠?363 13.4銆€灝忕粨 363 闄勫綍A銆€YARN瀹夎鎸囧崡 364 闄勫綍B銆€YARN閰嶇疆鍙傛暟浠嬬粛 367 闄勫綍C銆€Hadoop Shell鍛戒護浠嬬粛 371 闄勫綍D銆€鍙傝€冭祫鏂?374 |