Spark开发--Local模式

2019-12-02 本文已影响0人无剑_君

一、Local模式

Local 模式是最简单的一种Spark运行方式，它采用单节点多线程（cpu)方式运行，local模式是一种OOTB（开箱即用）的方式，只需要在spark-env.sh导出JAVA_HOME,无需其他任何配置即可使用，因而常用于开发和学习。
本地单机模式下，所有的Spark进程均运行于同一个JVM中，并行处理则通过多线程来实现。在默认情况下，单机模式启动与本地系统的CPU核心数目相同的线程。如果要设置并行的级别，则以local[N]的格式来指定一个master变量，N表示要使用的线程数目。
方式：

./spark-shell --master local[n]

n代表线程数

二、前置条件

1、Java8安装

下载地址：http://openjdk.java.net/
https://adoptopenjdk.net/releases.html

下载

root@master:~# apt install openjdk-8-jdk -y
# 验证
root@master:~# java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
# JDK路径
root@master:~# whereis java
java: /usr/bin/java /usr/share/java /usr/share/man/man1/java.1.gz
# 搜索java系统命令的位置
root@master:~# which java
/usr/bin/java

# 下载安装
root@master:~# wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz

2、Scala scala 2.13.1

1) 下载

下载地址：https://www.scala-lang.org/download/

查看版本

信息查看

下载

# 下载
root@slave2:~# root@master:~# wget https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz

# 解压
root@master:~# tar -zxvf scala-2.12.10.tgz -C /usr/local

注：如果要开发请使用2.12.10版本。

2) 配置环境变量

root@master:~# vi /etc/profile
#末尾添加
# 环境变量
export SCALA_HOME=/usr/local/scala-2.12.10
export PATH=$PATH:$SCALA_HOME/bin
# 立即生效
root@master:~# source /etc/profile

# 验证
root@master:~# scala -version
Scala code runner version 2.12.10-- Copyright 2002-2019, LAMP/EPFL and Lightbend, Inc.

三、下载安装Spark

下载安装
下载地址：http://spark.apache.org/downloads.html
下载地址

# 下载
root@master:~# wget https://www-us.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz

# 解压
root@master:~# tar -zxvf spark-2.4.4-bin-hadoop2.7.tgz -C /usr/local

配置环境变量

# 配置环境变量
export SCALA_HOME=/usr/local/scala-2.12.10
export HADOOP_HOME=/usr/local/hadoop-2.9.2
export SPARK_HOME=/usr/local/spark-2.4.4-bin-hadoop2.7
export PATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin

# 环境变量立即生效
root@master:~# source /etc/profile

配置Spark

# 复制配置文件
root@master:~# cp /usr/local/spark-2.4.4-bin-hadoop2.7/conf/spark-env.sh.template /usr/local/spark-2.4.4-bin-hadoop2.7/conf/spark-env.sh
# 配置服务器IP
root@master:~# vi /usr/local/spark-2.4.4-bin-hadoop2.7/conf/spark-env.sh

#  添加内容
SPARK_LOCAL_IP=192.168.247.131

四、测试运行

指令示例
1）单线程

root@master:~# spark-shell --master local
19/12/01 18:03:58 WARN Utils: Your hostname, master resolves to a loopback address: 127.0.1.1; using 192.168.247.131 instead (on interface ens33)
19/12/01 18:03:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/12/01 18:03:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.247.131:4040
Spark context available as 'sc' (master = local, app id = local-1575194657764).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_222)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

# 进程查看
root@master:~# jps
128720 SparkSubmit
129448 Jps

Web查看：http://192.168.247.131:4040/

Web查看

2）多线程

# 代表会有4个线程（每个线程一个core）来并发执行应用程序。
root@master:~# spark-shell --master local[4] 
19/12/01 18:06:31 WARN Utils: Your hostname, master resolves to a loopback address: 127.0.1.1; using 192.168.247.131 instead (on interface ens33)
19/12/01 18:06:31 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/12/01 18:06:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.247.131:4040
Spark context available as 'sc' (master = local[4], app id = local-1575194812191).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_222)
Type in expressions to have them evaluated.
Type :help for more information.

scala>