Blink First Time
2019-05-27 本文已影响2人
it_zzy
Blink First Time
背景
blink开源，目前在https://github.com/apache/flink/tree/blink
使用步骤
1.git clone
➜ code_github git clone https://github.com/apache/flink.git
2.切到blink 分支
flink git:(master) git checkout blink
Checking out files: 100% (11549/11549), done.
Branch 'blink' set up to track remote branch 'blink' from 'origin'.
Switched to a new branch 'blink'
3.修改pom，注释掉
<module>flink-tests</module>
<module>flink-yarn-tests</module>
<module>flink-fs-tests</module>
<module>flink-end-to-end-tests</module>
- build
mvn clean install -Dmaven.test.skip -Dcheckstyle.skip -Dlicense.skip=true -Drat.ignoreErrors=true
部分日志:
[INFO] Building tar: /Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1.tar.gz
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (create-build-target-link) @ flink-dist_2.11 ---
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (integration-tests) @ flink-dist_2.11 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ flink-dist_2.11 ---
[INFO] Installing /Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-dist_2.11-1.5.1.jar to /Users/zzy/Documents/admaster/maven_repo/com/alibaba/blink/flink-dist_2.11/1.5.1/flink-dist_2.11-1.5.1.jar
[INFO] Installing /Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/pom.xml to /Users/zzy/Documents/admaster/maven_repo/com/alibaba/blink/flink-dist_2.11/1.5.1/flink-dist_2.11-1.5.1.pom
[INFO] Installing /Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1.tar.gz to /Users/zzy/Documents/admaster/maven_repo/com/alibaba/blink/flink-dist_2.11/1.5.1/flink-dist_2.11-1.5.1.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] force-shading ...................................... SUCCESS [ 3.936 s]
[INFO] flink 1.5.1 ........................................ SUCCESS [ 7.404 s]
[INFO] flink-annotations .................................. SUCCESS [ 1.128 s]
[INFO] flink-shaded-hadoop ................................ SUCCESS [ 0.045 s]
[INFO] flink-shaded-hadoop2 ............................... SUCCESS [ 10.061 s]
[INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [ 15.434 s]
[INFO] flink-shaded-yarn-tests ............................ SUCCESS [ 12.405 s]
[INFO] flink-shaded-curator ............................... SUCCESS [ 0.648 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [ 0.076 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [ 0.553 s]
[INFO] flink-metrics ...................................... SUCCESS [ 0.066 s]
[INFO] flink-metrics-core ................................. SUCCESS [ 3.425 s]
[INFO] flink-core ......................................... SUCCESS [ 8.611 s]
[INFO] flink-java ......................................... SUCCESS [ 3.451 s]
[INFO] flink-queryable-state .............................. SUCCESS [ 0.061 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [ 0.517 s]
[INFO] flink-filesystems .................................. SUCCESS [ 0.049 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [ 1.105 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [ 1.191 s]
[INFO] flink-runtime ...................................... SUCCESS [ 29.281 s]
[INFO] flink-optimizer .................................... SUCCESS [ 0.794 s]
[INFO] flink-clients ...................................... SUCCESS [ 0.692 s]
[INFO] flink-streaming-java ............................... SUCCESS [ 3.745 s]
[INFO] flink-scala ........................................ SUCCESS [ 18.012 s]
[INFO] flink-examples ..................................... SUCCESS [ 0.207 s]
[INFO] flink-examples-batch ............................... SUCCESS [ 11.974 s]
[INFO] flink-test-utils ................................... SUCCESS [ 2.087 s]
[INFO] flink-state-backends ............................... SUCCESS [ 0.035 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [ 0.651 s]
[INFO] flink-libraries .................................... SUCCESS [ 0.028 s]
[INFO] flink-cep .......................................... SUCCESS [ 0.624 s]
[INFO] flink-java8 ........................................ SUCCESS [ 11.433 s]
[INFO] flink-mapr-fs ...................................... SUCCESS [ 1.065 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 17.399 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [ 17.899 s]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 15.572 s]
[INFO] flink-runtime-web .................................. SUCCESS [06:31 min]
[INFO] flink-streaming-scala .............................. SUCCESS [01:01 min]
[INFO] flink-table-common ................................. SUCCESS [ 2.550 s]
[INFO] flink-python ....................................... SUCCESS [ 0.834 s]
[INFO] flink-service ...................................... SUCCESS [ 0.470 s]
[INFO] flink-table ........................................ SUCCESS [01:37 min]
[INFO] flink-connectors ................................... SUCCESS [ 0.351 s]
[INFO] flink-orc .......................................... SUCCESS [ 0.748 s]
[INFO] flink-jdbc ......................................... SUCCESS [ 0.362 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [ 5.048 s]
[INFO] flink-hbase ........................................ SUCCESS [ 4.883 s]
[INFO] flink-hcatalog ..................................... SUCCESS [ 3.769 s]
[INFO] flink-formats ...................................... SUCCESS [ 0.047 s]
[INFO] flink-avro ......................................... SUCCESS [ 3.043 s]
[INFO] flink-json ......................................... SUCCESS [ 0.350 s]
[INFO] flink-metrics-jmx .................................. SUCCESS [ 0.248 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [ 1.811 s]
[INFO] flink-connector-kafka-0.8 .......................... SUCCESS [ 4.041 s]
[INFO] flink-connector-kafka-0.9 .......................... SUCCESS [ 3.068 s]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [ 7.390 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [ 3.355 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [ 1.187 s]
[INFO] flink-connector-elasticsearch ...................... SUCCESS [ 9.907 s]
[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 13.047 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 21.743 s]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [ 0.897 s]
[INFO] flink-connector-twitter ............................ SUCCESS [ 3.209 s]
[INFO] flink-connector-nifi ............................... SUCCESS [ 0.543 s]
[INFO] flink-connector-cassandra .......................... SUCCESS [ 3.388 s]
[INFO] flink-connector-filesystem ......................... SUCCESS [ 0.660 s]
[INFO] flink-connector-hive ............................... SUCCESS [ 6.394 s]
[INFO] flink-examples-streaming ........................... SUCCESS [ 14.000 s]
[INFO] flink-examples-table ............................... SUCCESS [ 12.201 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [ 0.282 s]
[INFO] flink-gelly ........................................ SUCCESS [ 1.347 s]
[INFO] flink-gelly-scala .................................. SUCCESS [ 11.416 s]
[INFO] flink-gelly-examples ............................... SUCCESS [ 5.726 s]
[INFO] flink-sql-parser ................................... SUCCESS [ 2.988 s]
[INFO] flink-sql-client ................................... SUCCESS [ 1.518 s]
[INFO] flink-ml ........................................... SUCCESS [ 23.872 s]
[INFO] flink-cep-scala .................................... SUCCESS [ 5.095 s]
[INFO] flink-streaming-python ............................. SUCCESS [ 4.220 s]
[INFO] flink-scala-shell .................................. SUCCESS [ 6.045 s]
[INFO] flink-quickstart ................................... SUCCESS [ 0.726 s]
[INFO] flink-quickstart-java .............................. SUCCESS [ 4.403 s]
[INFO] flink-quickstart-scala ............................. SUCCESS [ 0.142 s]
[INFO] flink-contrib ...................................... SUCCESS [ 0.030 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [ 0.366 s]
[INFO] flink-container .................................... SUCCESS [ 0.120 s]
[INFO] flink-mesos ........................................ SUCCESS [ 9.726 s]
[INFO] flink-metrics-ganglia .............................. SUCCESS [ 0.328 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [ 0.439 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [ 0.281 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [ 0.130 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [ 0.223 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [ 0.148 s]
[INFO] flink-yarn ......................................... SUCCESS [ 5.032 s]
[INFO] flink-kubernetes ................................... SUCCESS [ 1.812 s]
[INFO] flink-docs ......................................... SUCCESS [ 0.611 s]
[INFO] flink-yarn-shuffle ................................. SUCCESS [ 3.146 s]
[INFO] flink-dist 1.5.1 ................................... SUCCESS [ 40.690 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:52 min
[INFO] Finished at: 2019-05-27T14:31:01+08:00
[INFO] ------------------------------------------------------------------------
5.进入build-target/bin 目录，执行./start-cluster.sh
flink git:(blink) ✗ ll build-target
lrwxr-xr-x 1 zzy stafll build-target/bin/uild-target -> /Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1-bin/flink-1.5.1
total 248
-rwxr-xr-x 1 zzy staff 24481 May 27 11:47 config.sh
-rwxr-xr-x 1 zzy staff 2224 May 27 08:29 flink
-rwxr-xr-x 1 zzy staff 2823 May 27 11:47 flink-console.sh
-rwxr-xr-x 1 zzy staff 6521 May 27 11:47 flink-daemon.sh
-rwxr-xr-x 1 zzy staff 1271 May 27 08:29 flink.bat
-rwxr-xr-x 1 zzy staff 1482 May 27 11:47 historyserver.sh
-rwxr-xr-x 1 zzy staff 2652 May 27 11:47 jobmanager.sh
-rwxr-xr-x 1 zzy staff 1660 May 27 11:47 kubernetes-session.sh
-rwxr-xr-x 1 zzy staff 1802 May 27 08:29 mesos-appmaster-job.sh
-rwxr-xr-x 1 zzy staff 1971 May 27 11:47 mesos-appmaster.sh
-rwxr-xr-x 1 zzy staff 2013 May 27 11:47 mesos-taskmanager.sh
-rwxr-xr-x 1 zzy staff 1182 May 27 08:29 pyflink-stream.sh
-rwxr-xr-x 1 zzy staff 1164 May 27 11:47 pyflink.bat
-rwxr-xr-x 1 zzy staff 1107 May 27 11:47 pyflink.sh
-rwxr-xr-x 1 zzy staff 4215 May 27 11:47 sql-client.sh
-rwxr-xr-x 1 zzy staff 3364 May 27 08:29 start-cluster.bat
-rwxr-xr-x 1 zzy staff 1836 May 27 08:29 start-cluster.sh
-rwxr-xr-x 1 zzy staff 3162 May 27 11:47 start-scala-shell.sh
-rwxr-xr-x 1 zzy staff 1854 May 27 08:29 start-zookeeper-quorum.sh
-rwxr-xr-x 1 zzy staff 1616 May 27 08:29 stop-cluster.sh
-rwxr-xr-x 1 zzy staff 1845 May 27 08:29 stop-zookeeper-quorum.sh
-rwxr-xr-x 1 zzy staff 3762 May 27 11:47 taskmanager.sh
-rwxr-xr-x 1 zzy staff 1638 May 27 11:47 yarn-session.sh
-rwxr-xr-x 1 zzy staff 2281 May 27 08:29 zookeeper.sh
执行后
➜ flink git:(blink) ✗./build-target/bin/start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host zzy.
/Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1-bin/flink-1.5.1/bin/flink-daemon.sh: line 142: /tmp/flink-zzy-standalonesession.pid: Permission denied
log4j:WARN No appenders could be found for logger (org.apache.flink.configuration.GlobalConfiguration).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Starting taskexecutor daemon on host zzy.
/Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1-bin/flink-1.5.1/bin/flink-daemon.sh: line 142: /tmp/flink-zzy-taskexecutor.pid: Permission denied
访问8081：
image.png
word count实战
1.进入flink-examples/flink-examples-streaming
mvn clean install -Dmaven.test.skip
2.将1产生的jar包上传
image.png
点击show plan
image.png
3.提交任务，执行
点击submit后
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
再来一个例子上传flink-examples-streaming_2.11-1.5.1-TopSpeedWindowing.jar
image.png
点击expand all
image.png
点击collapse all
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
image.png
Flink-sql
下面测试下Flink-sql，首先启动sql-client
➜ flink git:(blink) ✗ ./build-target/bin/sql-client.sh embedded
No default environment specified.
Searching for '/Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1-bin/flink-1.5.1/conf/sql-client-defaults.yaml'...found.
Reading default environment from: file:/Users/zzy/Documents/zuoyebang/code_github/flink/flink-dist/target/flink-1.5.1-bin/flink-1.5.1/conf/sql-client-defaults.yaml
No session environment specified.
Validating current environment...done.
▒▓██▓██▒
▓████▒▒█▓▒▓███▓▒
▓███▓░░ ▒▒▒▓██▒ ▒
░██▒ ▒▒▓▓█▓▓▒░ ▒████
██▒ ░▒▓███▒ ▒█▒█▒
░▓█ ███ ▓░▒██
▓█ ▒▒▒▒▒▓██▓░▒░▓▓█
█░ █ ▒▒░ ███▓▓█ ▒█▒▒▒
████░ ▒▓█▓ ██▒▒▒ ▓███▒
░▒█▓▓██ ▓█▒ ▓█▒▓██▓ ░█░
▓░▒▓████▒ ██ ▒█ █▓░▒█▒░▒█▒
███▓░██▓ ▓█ █ █▓ ▒▓█▓▓█▒
░██▓ ░█░ █ █▒ ▒█████▓▒ ██▓░▒
███░ ░ █░ ▓ ░█ █████▒░░ ░█░▓ ▓░
██▓█ ▒▒▓▒ ▓███████▓░ ▒█▒ ▒▓ ▓██▓
▒██▓ ▓█ █▓█ ░▒█████▓▓▒░ ██▒▒ █ ▒ ▓█▒
▓█▓ ▓█ ██▓ ░▓▓▓▓▓▓▓▒ ▒██▓ ░█▒
▓█ █ ▓███▓▒░ ░▓▓▓███▓ ░▒░ ▓█
██▓ ██▒ ░▒▓▓███▓▓▓▓▓██████▓▒ ▓███ █
▓███▒ ███ ░▓▓▒░░ ░▓████▓░ ░▒▓▒ █▓
█▓▒▒▓▓██ ░▒▒░░░▒▒▒▒▓██▓░ █▓
██ ▓░▒█ ▓▓▓▓▒░░ ▒█▓ ▒▓▓██▓ ▓▒ ▒▒▓
▓█▓ ▓▒█ █▓░ ░▒▓▓██▒ ░▓█▒ ▒▒▒░▒▒▓█████▒
██░ ▓█▒█▒ ▒▓▓▒ ▓█ █░ ░░░░ ░█▒
▓█ ▒█▓ ░ █░ ▒█ █▓
█▓ ██ █░ ▓▓ ▒█▓▓▓▒█░
█▓ ░▓██░ ▓▒ ▓█▓▒░░░▒▓█░ ▒█
██ ▓█▓░ ▒ ░▒█▒██▒ ▓▓
▓█▒ ▒█▓▒░ ▒▒ █▒█▓▒▒░░▒██
░██▒ ▒▓▓▒ ▓██▓▒█▒ ░▓▓▓▓▒█▓
░▓██▒ ▓░ ▒█▓█ ░░▒▒▒
▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░▓▓ ▓░▒█░
______ _ _ _ _____ ____ _ _____ _ _ _ BETA
| ____| (_) | | / ____|/ __ \| | / ____| (_) | |
| |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
| __| | | | '_ \| |/ / \___ \| | | | | | | | | |/ _ \ '_ \| __|
| | | | | | | | < ____) | |__| | |____ | |____| | | __/ | | | |_
|_| |_|_|_| |_|_|\_\ |_____/ \___\_\______| \_____|_|_|\___|_| |_|\__|
Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to exit.
Flink SQL>
定义输入的表
Flink SQL> create table csv_source ( a varchar ) with ( type = 'csv', path = 'file:///Users/zzy/Documents/zuoyebang/code_github/input.csv' );
[INFO] Table has been created.
定义输出的表
Flink SQL> create table csv_sink ( a varchar, c bigint ) with ( type = 'csv', updatemode = 'upsert', path = 'file:///Users/zzy/Documents/zuoyebang/code_github/output.csv' );
[INFO] Table has been created.
Flink SQL> show tables;
csv_source
csv_sink
Flink SQL> insert into csv_sink select a, count(*) from csv_source group by a;
[INFO] Submitting SQL update statement to the cluster...
[INFO] Table update statement has been successfully submitted to the cluster:
Cluster ID: StandaloneClusterId
Job ID: ca4cbb86996d641c358fee6a6f975e24
Web interface: http://localhost:8081
去flink webui上看到有一个default:sql...的任务
image.png
image.png
参考：
