| 澶辨晥閾炬帴澶勭悊 |
|
Kafka KSQL瀹炴垬 PDF 涓嬭澆
鏈珯鏁寸悊涓嬭澆錛?/strong>
閾炬帴錛?a target="_blank">https://pan.baidu.com/s/1jh6wv3C16keHYGPNfFzWvg
鎻愬彇鐮侊細(xì)pf0m
鐩稿叧鎴浘錛?/strong>
涓昏鍐呭錛?/strong>
1.鑳屾櫙
kafka鏃╂湡浣滀負(fù)涓€涓棩蹇楁秷鎭郴緇燂紝寰堝彈榪愮淮嬈㈣繋鐨勶紝閰嶅悎ELK鐜╄搗鏉ュ緢happy錛屽湪kafka鎱㈡參鐨勮漿鍚戞祦寮忓鉤鍙扮殑榪囩▼涓紝寮€鍙戜篃鎱㈡參浠嬪叆浜?jiǎn)锛屼竴浜涗笟鍔＄郴緇熶篃寮€濮嬪拰kafka瀵規(guī)帴璧鋒潵浜?jiǎn)锛屼篃杩樻槸寰堝彈澶у娆q庣殑錛岀敱浜庝笟鍔￠渶瑕侊紝涓€閮ㄥ垎灝忕櫧涔熷氨鍏嶄笉浜?jiǎn)鎺ヨЕkafka浜?jiǎn)锛寴q欎簺灝忕櫧鎬繪槸浼?xì)瀹夊涓嶄綇濂藉蹇?jī)錛岃綺劇‘鐨勬煡鐪媖afka涓殑鏌愪竴鏉℃暟鎹紝浣滀負(fù)鏈嶅姟鎻愪緵鏂癸紝鎴戜篃寰堟柟鍟婏紝璇ユ€庝箞鎬鹼紵涓氬姟鏂逛笉鏁㈠緱緗晩錛屽彧鑳藉啓consumer鍘繪秷璐癸紝鐒跺悗浜鴻倝鏌ヨ銆?/div>
2.闇€姹?/div>
鏈変粈涔堟柟娉曡兘鐩存帴鏌ヨkafka涓凡鏈夌殑鏁版嵁鍛紵閭ｆ椂鍊檖resto灝辨槧鍏ョ溂甯樹簡(jiǎn)錛屽垵姝ユ帰绱㈠悗鍙戠幇presto紜疄寮哄ぇ錛屽拰鎴戜滑鍦ㄧ敤鐨刬mpala鏈夌殑涓€鎷鹼紝鏀寔鐨勬暟鎹簮涔熸洿澶氾紝浠€涔坮edis銆乵ongo銆乲afka閮藉彲浠ョ敤sql鏉ユ煡璇紝鐪熸槸鏁戞槦鍟婏紝榪欐牱閭ｇ兢灝忕櫧灝卞彲浠ョ洿鎺ヤ嬌鐢╬resto鏉ユ煡璇㈤噷闈㈢殑鏁版嵁浜?jiǎn)銆備笉榪噋resto鍦ㄤ笉寮€鍙戞彃浠剁殑鎯呭喌涓嬶紝瀵筴afka鐨勬暟鎹湁鏍煎紡瑕佹眰錛屾敮鎸乯son銆乤vro銆傚叧浜巔resto鐨勮皟鐮旇presto瀹炴垬銆備絾鏄垜鍙槸鎯崇敤sql鏌ヨkafka錛岃€宲resto鍔熻兘榪囦簬寮哄ぇ錛屽繀鐒舵暣涓鏋跺氨鏄懼緱姣旇緝鍘氶噸浜?jiǎn)锛屽姛鑳藉鍢涖€傛湁浠€涔堣交閲忕駭鐨勫伐鍏峰憿錛?/div>
3.浠嬬粛
鏌愪竴澶╋紝kafka鐨勪翰鍎垮瓙KSQL灝辮癁鐢熶簡(jiǎn)錛孠SQL鏄竴涓敤浜嶢pache kafka鐨勬祦寮廠QL寮曟搸錛孠SQL闄嶄綆浜?jiǎn)杩涘叆娴佸鐞嗙殑闂ㄦ锛屾彁渚涗簡(jiǎn)涓€涓畝鍗曠殑銆佸畬鍏ㄤ氦浜掑紡鐨凷QL鎺ュ彛錛岀敤浜庡鐞咾afka鐨勬暟鎹紝鍙互璁╂垜浠湪嫻佹暟鎹笂鎸佺畫鎵ц SQL 鏌ヨ錛孠SQL鏀寔騫挎硾鐨勫己澶х殑嫻佸鐞嗘搷浣滐紝鍖呮嫭鑱氬悎銆佽繛鎺ャ€佺獥鍙ｃ€佷細(xì)璇濈瓑絳夈€?/div>
KSQL鍦ㄥ唴閮ㄤ嬌鐢↘afka鐨凷treams API錛屽茍涓斿畠浠叡浜笌Kafka嫻佸鐞嗙浉鍚岀殑鏍稿績(jī)鎶借薄錛孠SQL鏈変袱涓牳蹇?jī)鎶借薄锛屽畠浠搴斾簬鍒癒afka Streams涓殑涓や釜鏍稿績(jī)鎶借薄錛岃浣犲彲浠ュ鐞唊afka鐨則opic鏁版嵁銆傚叧浜庤繖涓や釜鏍稿績(jī)鎶借薄涓嬬珷鑺傝В璇匯€?/div>
4.鏋舵瀯
4.1閮ㄧ講鏋舵瀯
鐢變竴涓狵SQL鏈嶅姟鍣ㄨ繘紼嬫墽琛屾煡璇€備竴緇凨SQL榪涚▼鍙互浣滀負(fù)闆嗙兢榪愯銆傚彲浠ラ€氳繃鍚姩鏇村鐨凨SQL瀹炰緥鏉ュ姩鎬佹坊鍔犳洿澶氱殑澶勭悊鑳藉姏銆傝繖浜汯SQL瀹炰緥鏄閿欑殑錛屽鏋滀竴涓疄渚嬪け璐ヤ簡(jiǎn)錛屽叾浠栫殑灝變細(xì)鎺ョ瀹冪殑宸ヤ綔銆傛煡璇㈡槸浣跨敤浜や簰寮忕殑KSQL鍛戒護(hù)琛屽鎴風(fēng)鍚姩鐨勶紝璇ュ鎴風(fēng)閫氳繃REST API鍚戦泦緹ゅ彂閫佸懡浠ゃ€傚懡浠よ鍏佽媯€(gè)鏌ュ彲鐢ㄧ殑stream鍜宼able錛屽彂鍑烘柊鐨勬煡璇紝媯€(gè)鏌ョ姸鎬佸茍緇堟姝ｅ湪榪愯鐨勬煡璇€侹SQL鍐呴儴鏄嬌鐢↘afka鐨剆tream API鏋勫緩鐨勶紝瀹冪戶鎵夸簡(jiǎn)瀹冪殑寮規(guī)€у彲浼哥緝鎬с€佸厛榪涚殑鐘舵€佺鐞嗗拰瀹歸敊鍔熻兘錛屽茍鏀寔Kafka鏈€榪戝紩鍏ョ殑涓€嬈℃€у鐞嗚涔夈€侹SQL鏈嶅姟鍣ㄥ皢姝ゅ祵鍏ュ埌涓€涓垎甯冨紡SQL寮曟搸涓?鍖呮嫭涓€浜涚敤浜庢煡璇㈡€ц兘鐨勮嚜鍔ㄥ瓧鑺備唬鐮佺敓鎴?鍜屼竴涓敤浜庢煡璇㈠拰鎺у埗鐨凴EST API銆?/div>
4.2澶勭悊鏋舵瀯
5.鎶借薄姒傚康
KSQL綆€鍖栦簡(jiǎn)嫻佸簲鐢ㄧ▼搴忥紝瀹冮泦鎴愪簡(jiǎn)stream鍜宼able鐨勬蹇碉紝鍏佽浣跨敤琛ㄧず鐜板湪鍙戠敓鐨勪簨浠剁殑stream鏉ヨ繛鎺ヨ〃紺哄綋鍓嶇姸鎬佺殑table銆?Apache Kafka涓殑涓€涓猼opic鍙互琛ㄧず涓篕SQL涓殑STREAM鎴朤ABLE錛屽叿浣撳彇鍐充簬topic澶勭悊鐨勯鏈熻涔夈€備笅闈㈢湅鐪嬩袱涓牳蹇?jī)鐨勮В璇汇€?/div>
stream錛氭祦鏄棤闄愬埗鐨勭粨鏋勫寲鏁版嵁搴忓垪錛宻tream涓殑fact鏄笉鍙彉鐨勶紝榪欐剰鍛崇潃鍙互灝嗘柊fact鎻掑叆鍒皊tream涓紝浣嗘槸鐜版湁fact姘歌繙涓嶄細(xì)琚洿鏂版垨鍒犻櫎銆?stream鍙互浠嶬afka topic鍒涘緩錛屾垨鑰呬粠鐜版湁鐨剆tream鍜宼able涓淳鐢熴€?/div>
table錛氫竴涓猼able鏄竴涓猻tream鎴栧彟涓€涓猼able鐨勮鍥撅紝瀹冧唬琛ㄤ簡(jiǎn)涓€涓笉鏂彉鍖栫殑fact鐨勯泦鍚堬紝瀹冪浉褰撲簬浼犵粺鐨勬暟鎹簱琛紝浣嗛€氳繃嫻佸寲絳夋祦璇箟鏉ヤ赴瀵屻€傝〃涓殑浜嬪疄鏄彲鍙樼殑錛岃繖鎰忓懗鐫€鍙互灝嗘柊鐨勪簨瀹炴彃鍏ュ埌琛ㄤ腑錛岀幇鏈夌殑浜嬪疄鍙互琚洿鏂版垨鍒犻櫎銆傚彲浠ヤ粠Kafka涓婚涓垱寤鴻〃錛屼篃鍙互浠庣幇鏈夌殑嫻佸拰琛ㄤ腑媧劇敓琛ㄣ€?/div>
6.閮ㄧ講
ksql鏀寔kafka0.11涔嬪悗鐨勭増鏈紝鍦╟onfluent鐨刅3鍜孷4鐗堟湰涓粯璁ゅ茍娌℃湁鍔犲叆ksql server紼嬪簭錛屽綋鐒禫3鍜孷4鏄敮鎸乲sql鐨勶紝鍦╒5鐗堟湰涓凡緇忛粯璁ゅ姞鍏sql浜?jiǎn)锛屼皋Z簡(jiǎn)鏂逛究婕旂ず錛屾垜浠嬌鐢╟onfluent kafka V5鐗堟湰婕旂ず錛寊k鍜宬afka涔熸槸鍗曞疄渚嬪惎鍔ㄣ€?/div>
6.1涓嬭澆
wget https://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz
tar zxvf confluent-oss-5.0.0-2.11.tar.gz -C /opt/programs/confluent_5.0.0
6.2鍚姩zk
cd /opt/programs/confluent_5.0.0
bin/zookeeper-server-start -daemon etc/kafka/zookeeper.properties
6.3鍚姩kafka
cd /opt/programs/confluent_5.0.0
bin/kafka-server-start -daemon etc/kafka/server.properties
6.4鍒涘緩topic鍜宒ata
confluent鑷甫浜?jiǎn)涓€涓猭sql-datagen宸ュ叿錛屽彲浠ュ垱寤哄拰浜х敓鐩稿叧鐨則opic鍜屾暟鎹紝ksql-datagen鍙互鎸囧畾鐨勫弬鏁板涓嬶細(xì)
[bootstrap-server=<kafka bootstrap server(s)> (defaults to localhost:9092)]
[quickstart=<quickstart preset> (case-insensitive; one of 'orders', 'users', or 'pageviews')]
schema=<avro schema file>
[schemaRegistryUrl=<url for Confluent Schema Registry> (defaults to http://localhost:8081)]
format=<message format> (case-insensitive; one of 'avro', 'json', or 'delimited')
topic=<kafka topic name>
key=<name of key column>
[iterations=<number of rows> (defaults to 1,000,000)]
[maxInterval=<Max time in ms between rows> (defaults to 500)]
[propertiesFile=<file specifying Kafka client properties>]
鍒涘緩pageviews錛屾暟鎹牸寮忎負(fù)delimited
cd /opt/programs/confluent_5.0.0/bin
./ksql-datagen quickstart=pageviews format=delimited topic=pageviews maxInterval=500
ps錛氫互涓婂懡浠や細(xì)婧愭簮涓嶆柇鍦╯tdin涓婅緭鍑烘暟鎹紝灝辨槸宸ュ叿鑷繁浜х敓鐨勬暟鎹紝濡備笅鏍蜂緥
8001 --> ([ 1539063767860 | 'User_6' | 'Page_77' ]) ts:1539063767860
8011 --> ([ 1539063767981 | 'User_9' | 'Page_75' ]) ts:1539063767981
8021 --> ([ 1539063768086 | 'User_5' | 'Page_16' ]) ts:1539063768086
涓嶈繃浣跨敤consumer娑堣垂鍑烘潵鐨勬暟鎹槸濡備笅鏍峰紡
1539066430530,User_5,Page_29
1539066430915,User_6,Page_74
1539066431192,User_4,Page_28
1539066431621,User_6,Page_38
1539066431772,User_7,Page_29
1539066432122,User_8,Page_34
鍒涘緩users錛屾暟鎹牸寮忎負(fù)json
cd /opt/programs/confluent_5.0.0/bin
./ksql-datagen quickstart=users format=json topic=users maxInterval=100
ps錛氫互涓婂懡浠や細(xì)婧愭簮涓嶆柇鍦╯tdin涓婅緭鍑烘暟鎹紝灝辨槸宸ュ叿鑷繁浜х敓鐨勬暟鎹紝濡備笅鏍蜂緥
User_5 --> ([ 1517896551436 | 'User_5' | 'Region_5' | 'MALE' ]) ts:1539063787413
User_7 --> ([ 1513998830510 | 'User_7' | 'Region_4' | 'MALE' ]) ts:1539063787430
User_6 --> ([ 1514865642822 | 'User_6' | 'Region_2' | 'MALE' ]) ts:1539063787481
涓嶈繃浣跨敤consumer娑堣垂鍑烘潵鐨勬暟鎹槸濡備笅鏍峰紡
{"registertime":1507118206666,"userid":"User_6","regionid":"Region_7","gender":"OTHER"}
{"registertime":1506192314325,"userid":"User_1","regionid":"Region_1","gender":"MALE"}
{"registertime":1489277749526,"userid":"User_6","regionid":"Region_4","gender":"FEMALE"}
{"registertime":1497188917765,"userid":"User_9","regionid":"Region_3","gender":"OTHER"}
{"registertime":1493121964253,"userid":"User_4","regionid":"Region_3","gender":"MALE"}
{"registertime":1515609444511,"userid":"User_5","regionid":"Region_9","gender":"FEMALE"}
|
| Java1234瀹樻柟緹?5錛?/td> | 
|
| Java1234瀹樻柟緹?5錛?/td> | 838462530 |
鑻忓叕緗戝畨澶?32061202001004鍙?/p>
