hadoop1.0 安装
2019-06-11 本文已影响0人
Kcrise
- 集群规划及环境
VMware Workstation 15
CentOS 7.6
192.168.10.21 (master)
192.168.10.22 (slave1)
192.168.10.23 (slave2)
- 虚拟机安装、网络配置
略
- 安装jdk 1.8
略
- hadoop 1.2.1 安装
master主机上传hadoop安装包
[root@master]/root$l
总用量 37208
-rw-r--r--. 1 root root 38096663 6月 3 20:50 hadoop-1.2.1-bin.tar.gz
-rw-------. 1 root root 1257 6月 4 22:39 anaconda-ks.cfg
解压缩包
[root@master]/root$tar -gxvf hadoop-1.2.1-bin.tar.gz -C /usr/local/src
5.修改配置文件
(1) masters
[root@master]/root$cd /usr/local/src/hadoop-1.2.1/conf/
[root@master]/usr/local/src/hadoop-1.2.1/conf$vi master
localhost修改为
master
(2)slaves
[root@master]/usr/local/src/hadoop-1.2.1/conf$vi slaves
localhost修改为
slave1
slave2
(3)core-site.xml
[root@master]/usr/local/src/hadoop-1.2.1/conf$vi core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/src/hadoop-1.2.1/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.10.21:9000</value>
</property>
</configuration>
(4)mapred-site.xml
[root@master]/usr/local/src/hadoop-1.2.1/conf$vi mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://192.168.10.21:9001</value>
</property>
</configuration>
(5)hdfs-site.xml
[root@master]/usr/local/src/hadoop-1.2.1/conf$vi hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
(6)hadoop-env.sh
[root@master]/usr/local/src/hadoop-1.2.1/conf$vi hadoop-env.sh
行尾追加
export JAVA_HOME=/opt/jdk
6.初始化hdfs
[root@master]/root$cd /usr/local/src/hadoop-1.2.1/bin
[root@master]/usr/local/src/hadoop-1.2.1/bin$./hadoop namenode -format
19/06/10 10:31:43 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.10.21
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.8.0_162
************************************************************/
19/06/10 10:31:43 INFO util.GSet: Computing capacity for map BlocksMap
19/06/10 10:31:43 INFO util.GSet: VM type = 64-bit
19/06/10 10:31:43 INFO util.GSet: 2.0% max memory = 932184064
19/06/10 10:31:43 INFO util.GSet: capacity = 2^21 = 2097152 entries
19/06/10 10:31:43 INFO util.GSet: recommended=2097152, actual=2097152
19/06/10 10:31:43 INFO namenode.FSNamesystem: fsOwner=root
19/06/10 10:31:43 INFO namenode.FSNamesystem: supergroup=supergroup
19/06/10 10:31:43 INFO namenode.FSNamesystem: isPermissionEnabled=true
19/06/10 10:31:43 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
19/06/10 10:31:43 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
19/06/10 10:31:43 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
19/06/10 10:31:43 INFO namenode.NameNode: Caching file names occuring more than 10 times
19/06/10 10:31:43 INFO common.Storage: Image file /usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds.
19/06/10 10:31:43 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/edits
19/06/10 10:31:43 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/edits
19/06/10 10:31:43 INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2.1/tmp/dfs/name has been successfully formatted.
19/06/10 10:31:43 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.10.21
************************************************************/
7.启动hadoop
[root@master]/root$cd /usr/local/src/hadoop-1.2.1/bin
[root@master]/usr/local/src/hadoop-1.2.1/bin$./start-all.sh
starting namenode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-namenode-master.out
slave1: starting datanode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-slave1.out
slave2: starting datanode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-slave2.out
master: starting secondarynamenode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-master.out
starting jobtracker, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-jobtracker-master.out
slave1: starting tasktracker, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-slave1.out
slave2: starting tasktracker, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-slave2.out
8.配置创建软连接
[root@master]/usr/local/src$l
总用量 4
drwxr-xr-x. 16 root root 4096 6月 10 10:36 hadoop-1.2.1
lrwxrwxrwx. 1 root root 12 6月 10 10:47 hadoop -> hadoop-1.2.1
[root@master]/usr/local/src$ln -s hadoop-1.2.1 hadoop
8.配置PATH
[root@master]/usr/local/src$vi /etc/profile
新增
#######################hadoop##########################
export HADOOP_INSTALL=/usr/local/src/hadoop
export PATH=$PATH:HADOOP_INSTALL/bin
9.同步/etc/profile文件到slave1和slave2
[root@master]/root$put /etc/profile /etc
profile 100% 2770 685.2KB/s 00:00
profile 100% 2770 1.4MB/s 00:00
[root@master]/root$
9.修改windos主机的HOSTS文件
C:\Windows\System32\drivers\etc\HOSTS
192.168.10.21 master
192.168.10.22 slave1
192.168.10.23 slave2
10.测试
(1)浏览器访问namenode节点
http://192.168.10.21:50070 ok
(2)hadoop 命令行模式
[root@master]/root$hadoop fs -ls /
Found 1 items
drwxr-xr-x - root supergroup 0 2019-06-10 10:36 /usr
上传文件
[root@master]/root$hadoop fs -put data.txt /
[root@master]/root$hadoop fs -ls /
Found 2 items
-rw-r--r-- 3 root supergroup 1074963960 2019-06-11 15:21 /data.txt
drwxr-xr-x - root supergroup 0 2019-06-10 16:43 /usr
[root@master]/root$
11.监控界面
NameNode
http://192.168.10.21:50070
SecondaryNameNode
http://master:50090/status.jsp
DataNode
http://slave1:50075/
http://slave2:50075/
JobTracker
http://master:50030/jobtracker.jsp
TaskTracker
http://slave1:50060/tasktracker.jsp
http://slave2:50060/tasktracker.jsp
