| 澶辨晥閾炬帴澶勭悊 |
|
Flink鍩虹璁蹭箟 PDF 涓嬭澆
鏈珯鏁寸悊涓嬭澆錛?/strong>
閾炬帴錛?a target="_blank">https://pan.baidu.com/s/1SOkuJAZ1DfQuFQ5SnO3qKg
鎻愬彇鐮侊細q2pz
鐩稿叧鎴浘錛?/strong>
涓昏鍐呭錛?/strong>
絎竴绔?Flink綆€浠嬨€愪簡瑙ｃ€?/div>
1.1.Flink鐨勫紩鍏?/div>
榪欏嚑騫村ぇ鏁版嵁鐨勯閫熷彂灞曪紝鍑虹幇浜嗗緢澶氱儹闂ㄧ殑寮€婧愮ぞ鍖猴紝鍏朵腑钁楀悕鐨勬湁 Hadoop銆丼torm錛屼互鍙婂悗鏉ョ殑 Spark錛屼粬浠兘鏈夌潃鍚勮嚜涓撴敞鐨勫簲鐢ㄥ満鏅€係park 鎺€寮€浜嗗唴瀛樿綆楃殑鍏堟渤錛屼篃浠ュ唴瀛樹負璧屾敞錛岃耽寰椾簡鍐呭瓨璁＄畻鐨勯閫熷彂灞曘€係park 鐨勭伀鐑垨澶氭垨灝戠殑鎺╃洊浜嗗叾浠栧垎甯冨紡璁＄畻鐨勭郴緇熻韓褰便€傚氨鍍?Flink錛屼篃灝卞湪榪欎釜鏃跺€欓粯榛樼殑鍙戝睍鐫€銆?/div>
鍦ㄥ浗澶栦竴浜涚ぞ鍖猴紝鏈夊緢澶氫漢灝嗗ぇ鏁版嵁鐨勮綆楀紩鎿庡垎鎴愪簡 4 浠ｏ紝褰撶劧錛屼篃鏈夊緢澶氫漢涓嶄細璁ゅ悓銆傛垜浠厛濮戜笖榪欎箞璁や負鍜岃璁恒€?/div>
棣栧厛絎竴浠ｇ殑璁＄畻寮曟搸錛屾棤鐤戝氨鏄?Hadoop 鎵胯澆鐨?MapReduce銆傝繖閲屽ぇ瀹跺簲璇ラ兘涓嶄細瀵?MapReduce 闄岀敓錛屽畠?yōu)畣璁〗帡鍒嗕皋Z袱涓樁孌碉紝鍒嗗埆涓?Map 鍜?Reduce銆傚浜庝笂灞傚簲鐢ㄦ潵璇達紝灝變笉寰椾笉鎯蟲柟璁炬硶鍘繪媶鍒嗙畻娉曪紝鐢氳嚦浜庝笉寰椾笉鍦ㄤ笂灞傚簲鐢ㄥ疄鐜板涓?Job 鐨勪覆鑱旓紝浠ュ畬鎴愪竴涓畬鏁寸殑綆楁硶錛屼緥濡傝凱浠ｈ綆椼€?/div>
鐢變簬榪欐牱鐨勫紛绔紝鍌敓浜嗘敮鎸?DAG 妗嗘灦鐨勪駭鐢熴€傚洜姝わ紝鏀寔 DAG 鐨勬鏋惰鍒掑垎涓虹浜屼唬璁＄畻寮曟搸銆傚 Tez 浠ュ強鏇翠笂灞傜殑 Oozie銆傝繖閲屾垜浠笉鍘葷粏絀跺悇縐?DAG 瀹炵幇涔嬮棿鐨勫尯鍒紝涓嶈繃瀵逛簬褰撴椂鐨?Tez 鍜?Oozie 鏉ヨ錛屽ぇ澶氳繕鏄壒澶勭悊鐨勪換鍔°€?/div>
鎺ヤ笅鏉ュ氨鏄互 Spark 涓轟唬琛ㄧ殑絎笁浠ｇ殑璁＄畻寮曟搸銆傜涓変唬璁＄畻寮曟搸鐨勭壒鐐逛富瑕佹槸 Job 鍐呴儴鐨?DAG 鏀寔錛堜笉璺ㄨ秺 Job錛夛紝浠ュ強寮鴻皟鐨勫疄鏃惰綆椼€傚湪榪欓噷錛屽緢澶氫漢涔熶細璁や負絎笁浠ｈ綆楀紩鎿庝篃鑳藉寰堝ソ鐨勮繍琛屾壒澶勭悊鐨?Job銆?/div>
闅忕潃絎笁浠ｈ綆楀紩鎿庣殑鍑虹幇錛屼績榪涗簡涓婂眰搴旂敤蹇€熷彂灞曪紝渚嬪鍚勭榪唬璁＄畻鐨勬€ц兘浠ュ強瀵規(guī)祦璁＄畻鍜?SQL 絳夌殑鏀寔銆侳link 鐨勮癁鐢熷氨琚綊鍦ㄤ簡絎洓浠ｃ€傝繖搴旇涓昏琛ㄧ幇鍦?Flink 瀵規(guī)祦璁＄畻鐨勬敮鎸侊紝浠ュ強鏇翠竴姝ョ殑瀹炴椂鎬т笂闈€傚綋鐒?Flink 涔熷彲浠ユ敮鎸?Batch 鐨勪換鍔★紝浠ュ強 DAG 鐨勮繍綆椼€?/div>
澶ф暟鎹殑4浠ｈ綆楀紩鎿?nbsp;
絎?浠ｏ細 Hadoop MapReduc鎵瑰鐞?Mapper銆丷educer 2.
絎?浠ｏ細 DAG妗嗘灦錛圤ozie 銆乀ez錛夛紝Tez + MapReduce 鎵瑰鐞?1涓猅ez = MR(1) + MR(2) + ... + MR(n) 鐩告瘮MR鏁堢巼鏈夋墍鎻愬崌
絎?浠ｏ細 Spark 鎵瑰鐞嗐€佹祦澶勭悊銆丼QL楂樺眰API鏀寔 鑷甫DAG 鍐呭瓨榪唬璁＄畻銆佹€ц兘杈冧箣鍓嶅ぇ騫呮彁
絎?浠ｏ細 Flink 鎵瑰鐞嗐€佹祦澶勭悊銆丼QL楂樺眰API鏀寔 鑷甫DAG 嫻佸紡璁＄畻鎬ц兘鏇撮珮銆佸彲闈犳€ф洿楂?/div>
棣栧厛錛屾垜浠彲浠ラ€氳繃涓嬮潰鐨勬€ц兘嫻嬭瘯鍒濇浜嗚В涓や釜妗嗘灦鐨勬€ц兘鍖哄埆錛屽畠浠兘鍙互鍩轟簬鍐呭瓨璁＄畻妗嗘灦榪涜瀹炴椂璁＄畻錛屾墍浠ラ兘鎷ユ湁闈炲父濂界殑璁＄畻鎬ц兘銆傜粡榪囨祴璇曪紝Flink璁＄畻鎬ц兘涓婄暐濂姐€?/div>
嫻嬭瘯鐜錛?nbsp;
1.CPU錛?000涓紱
2.鍐呭瓨錛氬崟鏈?28GB錛?nbsp;
3.鐗堟湰錛欻adoop 2.3.0錛孲park 1.4錛孎link 0.9
4.鏁版嵁錛?00MB錛?GB錛?TB錛?nbsp;
5.綆楁硶錛欿-means錛氫互絀洪棿涓璌涓偣涓轟腑蹇冭繘琛岃仛綾伙紝瀵規(guī)渶闈犺繎瀹冧滑鐨勫璞″綊綾匯€傞€氳繃榪唬鐨勬柟娉曪紝閫愭鏇存柊鍚勮仛綾諱腑蹇冪殑鍊鹼紝鐩磋嚦寰楀埌鏈€濂界殑鑱氱被緇撴灉銆?nbsp;
6.榪唬錛欿=10錛?緇勬暟鎹?nbsp;
Spark鍜孎link鍏ㄩ儴閮借繍琛屽湪Hadoop YARN涓婏紝鎬ц兘涓篎link > Spark > Hadoop(MR)錛岃凱浠ｆ鏁拌秺澶氳秺鏄庢樉錛屾€ц兘涓婏紝Flink浼樹簬Spark鍜孒adoop鏈€涓昏鐨勫師鍥犳槸Flink鏀寔澧為噺榪唬錛屽叿鏈夊榪唬鑷姩浼樺寲鐨勫姛鑳?/div>
Flink鍜宻park鐨勫樊寮?/div>
SparkStreaming Flink
瀹氫箟 寮規(guī)€х殑鍒嗗竷寮忔暟鎹泦錛屽茍闈炵湡姝ｇ殑瀹炴椂璁＄畻 鐪熸鐨勬祦璁＄畻錛屽氨鍍弒torm涓€鏍鳳紱
浣唂link鍚屾椂鏀寔鏈夐檺鐨勬暟鎹祦璁＄畻錛堟壒澶勭悊錛?/div>
楂樺閿?span style="white-space:pre"> 鍩轟簬RDD鍜?checkpoint,姣旇緝娌夐噸 checkpoint錛堝揩鐓э級,闈炲父杞婚噺綰?/div>
鍐呭瓨綆＄悊 JVM鐩稿叧鎿嶄綔鏆撮湶緇欑敤鎴?span style="white-space:pre"> Flink鍦↗VM涓疄鐜扮殑鏄嚜宸辯殑鍐呭瓨綆＄悊
寤舵椂 涓瓑100ms 浣?0ms
1.2.浠€涔堟槸Flink
Flink璧鋒簮浜嶴tratosphere欏圭洰錛孲tratosphere鏄湪2010~2014騫寸敱3鎵€鍦板鏌忔灄鐨勫ぇ瀛﹀拰嬈ф床鐨勪竴浜涘叾浠栫殑澶у鍏卞悓榪涜鐨勭爺絀墮」鐩紝2014騫?鏈圫tratosphere鐨勪唬鐮佽澶嶅埗騫舵崘璧犵粰浜咥pache杞歡鍩洪噾浼氾紝鍙傚姞榪欎釜瀛靛寲欏圭洰鐨勫垵濮嬫垚鍛樻槸Stratosphere緋葷粺鐨勬牳蹇冨紑鍙戜漢鍛橈紝2014騫?2鏈堬紝Flink涓€璺冩垚涓篈pache杞歡鍩洪噾浼氱殑欏剁駭欏圭洰銆?/div>
鍦ㄥ痙璇腑錛孎link涓€璇嶈〃紺哄揩閫熷拰鐏靛閥錛岄」鐩噰鐢ㄤ竴鍙澗榧犵殑褰╄壊鍥炬浣滀負logo錛岃繖涓嶄粎鏄洜涓烘澗榧犲叿鏈夊揩閫熷拰鐏靛閥鐨勭壒鐐癸紝榪樺洜涓烘煆鏋楃殑鏉鵑紶鏈変竴縐嶈糠浜虹殑綰㈡鑹詫紝鑰孎link鐨勬澗榧爈ogo鎷ユ湁鍙埍鐨勫熬宸達紝灝懼反鐨勯鑹蹭笌Apache杞歡鍩洪噾浼氱殑logo棰滆壊鐩稿懠搴旓紝涔熷氨鏄錛岃繖鏄竴鍙狝pache椋庢牸鐨勬澗榧犮€?/div>
鍥?Flink Logo
Flink涓婚〉鍦ㄥ叾欏墮儴灞曠ず浜嗚欏圭洰鐨勭悊蹇碉細“Apache Flink鏄負鍒嗗竷寮忋€侀珮鎬ц兘銆侀殢鏃跺彲鐢ㄤ互鍙婂噯紜殑嫻佸鐞嗗簲鐢ㄧ▼搴忔墦閫犵殑寮€婧愭祦澶勭悊妗嗘灦”銆?/div>
Apache Flink鏄竴涓鏋跺拰鍒嗗竷寮忓鐞嗗紩鎿庯紝鐢ㄤ簬瀵規(guī)棤鐣屽拰鏈夌晫鏁版嵁嫻佽繘琛屾湁鐘舵€佽綆椼€侳link琚璁″湪鎵€鏈夊父瑙佺殑闆嗙兢鐜涓繍琛岋紝浠ュ唴瀛樻墽琛岄€熷害鍜屼換鎰忚妯℃潵鎵ц璁＄畻銆?/div>
1.3.Flink嫻佸鐞嗙壒鎬?/div>
1)鏀寔楂樺悶鍚愩€佷綆寤惰繜銆侀珮鎬ц兘鐨勬祦澶勭悊
2)鏀寔甯︽湁浜嬩歡鏃墮棿鐨勭獥鍙ｏ紙Window錛夋搷浣?/div>
3)鏀寔鏈夌姸鎬佽綆楃殑Exactly-once璇箟
4)鏀寔楂樺害鐏墊椿鐨勭獥鍙ｏ紙Window錛夋搷浣滐紝鏀寔鍩轟簬time銆乧ount銆乻ession錛屼互鍙奷ata-driven鐨勭獥鍙ｆ搷浣?/div>
5)鏀寔鍏鋒湁鑳屽帇鍔熻兘鐨勬寔緇祦妯″瀷
6)鏀寔鍩轟簬杞婚噺綰у垎甯冨紡蹇収錛圫napshot錛夊疄鐜扮殑瀹歸敊
7)涓€涓繍琛屾椂鍚屾椂鏀寔Batch on Streaming澶勭悊鍜孲treaming澶勭悊
8)Flink鍦↗VM鍐呴儴瀹炵幇浜嗚嚜宸辯殑鍐呭瓨綆＄悊
9)鏀寔榪唬璁＄畻
10)鏀寔紼嬪簭鑷姩浼樺寲錛氶伩鍏嶇壒瀹氭儏鍐典笅Shuffle銆佹帓搴忕瓑鏄傝吹鎿嶄綔錛屼腑闂寸粨鏋滄湁蹇呰榪涜緙撳瓨
|
| Java1234瀹樻柟緹?5錛?/td> | 
|
| Java1234瀹樻柟緹?5錛?/td> | 838462530 |
鑻忓叕緗戝畨澶?32061202001004鍙?/p>
