mac安装apache flink
2018-11-28 本文已影响98人
nextliving
本文使用mac平台专属工具homebrew安装apache flink,并简单启动一个本地集群(local cluster)。
安装flink
mac上可以直接使用homebrew的命令$ brew install apache-flink安装flink:
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> New Formulae
curl-openssl gitmoji kubespy qalculate-gtk
fx healpix maven@3.5
==> Updated Formulae
openssl ✔ libphonenumber
sqlite ✔ libpq
abcm2ps libpqxx
abyss libqalculate
......
......
.....
ibhttpserver yq
libimagequant z3
liblcf zimg
libomp
==> Renamed Formulae
gutenberg -> zola mat -> mat2
==> Deleted Formulae
maven@3.0 maven@3.1
==> Downloading https://www.apache.org/dyn/closer.lua?path=flink/flink-1.6.2/fli
==> Downloading from http://mirror.bit.edu.cn/apache/flink/flink-1.6.2/flink-1.6
######################################################################## 100.0%
🍺 /usr/local/Cellar/apache-flink/1.6.2: 111 files, 324MB, built in 4 minutes 46 seconds
安装完成以后使用$ flink --version命令查看版本，终端输出：
Version: 1.6.2, Commit ID: 3456ad0
flink项目结构
flink 1.6.2项目文件夹目录结构如下：
.
├── INSTALL_RECEIPT.json
├── LICENSE
├── NOTICE
├── README.txt
├── bin
│ └── flink
└── libexec
├── bin
│ ├── config.sh
│ ├── flink
│ ├── flink-console.sh
│ ├── flink-daemon.sh
│ ├── historyserver.sh
│ ├── jobmanager.sh
│ ├── mesos-appmaster-job.sh
│ ├── mesos-appmaster.sh
│ ├── mesos-taskmanager.sh
│ ├── pyflink-stream.sh
│ ├── pyflink.sh
│ ├── sql-client.sh
│ ├── standalone-job.sh
│ ├── start-cluster.sh
│ ├── start-scala-shell.sh
│ ├── start-zookeeper-quorum.sh
│ ├── stop-cluster.sh
│ ├── stop-zookeeper-quorum.sh
│ ├── taskmanager.sh
│ ├── yarn-session.sh
│ └── zookeeper.sh
├── conf
│ ├── flink-conf.yaml
│ ├── log4j-cli.properties
│ ├── log4j-console.properties
│ ├── log4j-yarn-session.properties
│ ├── log4j.properties
│ ├── logback-console.xml
│ ├── logback-yarn.xml
│ ├── logback.xml
│ ├── masters
│ ├── slaves
│ ├── sql-client-defaults.yaml
│ └── zoo.cfg
├── examples
│ ├── batch
│ │ ├── ConnectedComponents.jar
│ │ ├── DistCp.jar
│ │ ├── EnumTriangles.jar
│ │ ├── KMeans.jar
│ │ ├── PageRank.jar
│ │ ├── TransitiveClosure.jar
│ │ ├── WebLogAnalysis.jar
│ │ └── WordCount.jar
│ ├── gelly
│ │ └── flink-gelly-examples_2.11-1.6.2.jar
│ ├── python
│ │ ├── batch
│ │ │ ├── TPCHQuery10.py
│ │ │ ├── TPCHQuery3.py
│ │ │ ├── TriangleEnumeration.py
│ │ │ ├── WebLogAnalysis.py
│ │ │ ├── WordCount.py
│ │ │ └── __init__.py
│ │ └── streaming
│ │ ├── fibonacci.py
│ │ └── word_count.py
│ └── streaming
│ ├── IncrementalLearning.jar
│ ├── Iteration.jar
│ ├── Kafka010Example.jar
│ ├── SessionWindowing.jar
│ ├── SocketWindowWordCount.jar
│ ├── StateMachineExample.jar
│ ├── TopSpeedWindowing.jar
│ ├── Twitter.jar
│ ├── WindowJoin.jar
│ └── WordCount.jar
├── lib
│ ├── flink-dist_2.11-1.6.2.jar
│ ├── flink-python_2.11-1.6.2.jar
│ ├── flink-shaded-hadoop2-uber-1.6.2.jar
│ ├── log4j-1.2.17.jar
│ └── slf4j-log4j12-1.7.7.jar
├── libexec
│ ├── config.sh
│ ├── flink
│ ├── flink-console.sh
│ ├── flink-daemon.sh
│ ├── historyserver.sh
│ ├── jobmanager.sh
│ ├── mesos-appmaster-job.sh
│ ├── mesos-appmaster.sh
│ ├── mesos-taskmanager.sh
│ ├── pyflink-stream.sh
│ ├── pyflink.sh
│ ├── sql-client.sh
│ ├── standalone-job.sh
│ ├── start-cluster.sh
│ ├── start-scala-shell.sh
│ ├── start-zookeeper-quorum.sh
│ ├── stop-cluster.sh
│ ├── stop-zookeeper-quorum.sh
│ ├── taskmanager.sh
│ ├── yarn-session.sh
│ └── zookeeper.sh
├── log
│ └── flink-chenxin-client-MacBookPro.log
└── opt
├── flink-cep-scala_2.11-1.6.2.jar
├── flink-cep_2.11-1.6.2.jar
├── flink-gelly-scala_2.11-1.6.2.jar
├── flink-gelly_2.11-1.6.2.jar
├── flink-metrics-datadog-1.6.2.jar
├── flink-metrics-dropwizard-1.6.2.jar
├── flink-metrics-ganglia-1.6.2.jar
├── flink-metrics-graphite-1.6.2.jar
├── flink-metrics-prometheus-1.6.2.jar
├── flink-metrics-slf4j-1.6.2.jar
├── flink-metrics-statsd-1.6.2.jar
├── flink-ml_2.11-1.6.2.jar
├── flink-queryable-state-runtime_2.11-1.6.2.jar
├── flink-s3-fs-hadoop-1.6.2.jar
├── flink-s3-fs-presto-1.6.2.jar
├── flink-sql-client-1.6.2.jar
├── flink-streaming-python_2.11-1.6.2.jar
├── flink-swift-fs-hadoop-1.6.2.jar
└── flink-table_2.11-1.6.2.jar
15 directories, 111 files
启动本地集群
执行命令$ cd /usr/local/cellar/apache-flink/1.6.2/libexec进入libexec文件夹，执行命令$ ./bin/start-cluster.sh启动本地集群(local cluster),终端输出
Starting cluster.
Starting standalonesession daemon on host MacBookPro.
Starting taskexecutor daemon on host MacBookPro.
这些信息太粗略了，去/usr/local/cellar/apache-flink/1.6.2/libexec/log下查看flink-chenxin-standalonesession-0-MacBookPro.log，日志如下：
2018-11-28 21:33:41,089 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-11-28 21:33:41,091 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.6.2, Rev:3456ad0, Date:17.10.2018 @ 18:46:46 GMT)
2018-11-28 21:33:41,091 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: chenxin
2018-11-28 21:33:41,384 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-11-28 21:33:41,451 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: chenxin
2018-11-28 21:33:41,451 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.92-b14
2018-11-28 21:33:41,451 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes
2018-11-28 21:33:41,451 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.7.5
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/usr/local/Cellar/apache-flink/1.6.2/libexec/log/flink-chenxin-standalonesession-0-MacBookPro.log
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/usr/local/Cellar/apache-flink/1.6.2/libexec/conf/log4j.properties
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/usr/local/Cellar/apache-flink/1.6.2/libexec/conf/logback.xml
2018-11-28 21:33:41,453 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:
2018-11-28 21:33:41,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2018-11-28 21:33:41,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /usr/local/Cellar/apache-flink/1.6.2/libexec/conf
2018-11-28 21:33:41,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode
2018-11-28 21:33:41,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-11-28 21:33:41,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /usr/local/Cellar/apache-flink/1.6.2/libexec/lib/flink-python_2.11-1.6.2.jar:/usr/local/Cellar/apache-flink/1.6.2/libexec/lib/flink-shaded-hadoop2-uber-1.6.2.jar:/usr/local/Cellar/apache-flink/1.6.2/libexec/lib/log4j-1.2.17.jar:/usr/local/Cellar/apache-flink/1.6.2/libexec/lib/slf4j-log4j12-1.7.7.jar:/usr/local/Cellar/apache-flink/1.6.2/libexec/lib/flink-dist_2.11-1.6.2.jar:::
2018-11-28 21:33:41,454 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2018-11-28 21:33:41,455 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-11-28 21:33:41,466 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2018-11-28 21:33:41,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2018-11-28 21:33:41,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2018-11-28 21:33:41,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2018-11-28 21:33:41,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-11-28 21:33:41,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2018-11-28 21:33:41,467 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2018-11-28 21:33:41,482 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.
2018-11-28 21:33:41,482 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2018-11-28 21:33:41,524 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.
2018-11-28 21:33:41,558 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to chenxin (auth:SIMPLE)
2018-11-28 21:33:41,586 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2018-11-28 21:33:41,609 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at localhost:6123
2018-11-28 21:33:42,114 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2018-11-28 21:33:42,171 INFO akka.remote.Remoting - Starting remoting
2018-11-28 21:33:42,354 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@localhost:6123]
2018-11-28 21:33:42,363 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@localhost:6123
2018-11-28 21:33:42,380 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-11-28 21:33:42,388 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /var/folders/pz/1sq2sn2922d0bzrpycsht1d40000gn/T/blobStore-a34dc063-20a2-4f91-9853-d8852f0e6229
2018-11-28 21:33:42,392 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:58832 - max concurrent requests: 50 - max backlog: 1000
2018-11-28 21:33:42,411 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2018-11-28 21:33:42,416 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /var/folders/pz/1sq2sn2922d0bzrpycsht1d40000gn/T/executionGraphStore-d014dd84-3370-482c-bf5d-a46269ce75a8, expiration time 3600000, maximum cache size 52428800 bytes.
2018-11-28 21:33:42,446 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /var/folders/pz/1sq2sn2922d0bzrpycsht1d40000gn/T/blobStore-2a864ac9-2728-4b96-9398-721b756c399c
2018-11-28 21:33:42,454 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-11-28 21:33:42,457 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /var/folders/pz/1sq2sn2922d0bzrpycsht1d40000gn/T/flink-web-572c3581-6395-4a0f-b0b6-673945ce72c8/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2018-11-28 21:33:42,458 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /var/folders/pz/1sq2sn2922d0bzrpycsht1d40000gn/T/flink-web-572c3581-6395-4a0f-b0b6-673945ce72c8/flink-web-upload for file uploads.
2018-11-28 21:33:42,463 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
2018-11-28 21:33:42,702 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /usr/local/Cellar/apache-flink/1.6.2/libexec/log/flink-chenxin-standalonesession-0-MacBookPro.log
2018-11-28 21:33:42,702 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /usr/local/Cellar/apache-flink/1.6.2/libexec/log/flink-chenxin-standalonesession-0-MacBookPro.out
2018-11-28 21:33:42,850 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at localhost:8081
2018-11-28 21:33:42,850 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://localhost:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
2018-11-28 21:33:42,850 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://localhost:8081.
2018-11-28 21:33:42,927 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-11-28 21:33:42,951 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-11-28 21:33:42,971 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
2018-11-28 21:33:42,972 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
2018-11-28 21:33:42,990 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000
2018-11-28 21:33:42,992 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
2018-11-28 21:33:43,445 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering TaskManager with ResourceID a1fc84e9fc2f53dad7998bb834eb15cd (akka.tcp://flink@macbookpro:58833/user/taskmanager_0) at ResourceManager
查看任务页面
如果在上一步正常启动，那么可以在浏览器中访问http://localhost:8081/，会展示如下页面：
1.png
点击左侧的
Running Jobs会展示如下页面：
2.png
点击左侧的
Task Managers会展示如下页面：
3.png
点击左侧的
Job Manager会展示如下页面：
4.png
点击左侧的
Submit new Job会展示如下页面：
5.png
可以在这里提交job,一般一个job就是一个jar包。
