Hadoop搭建和使用

2018-08-01 本文已影响0人 Claire_cc

1.搭建Hadoop

安装环境：虚拟机（VMware）Ubuntu 16.04.3
Step 1：安装JDK
（1）从Oracle官网
http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk8-downloads-2133151-zhs.html下载JDK包
（2）解压：tar -zxvf jdk-8u117-linux-x64.tar.gz
（3）修改全局配置文件vim /etc/profile并激活：source /etc/profile

图1 检查是否安装成功：java –version

图2
Step2：安装ssh
Apt-get install openssh-server
报错Unable to fetch some archives, maybe run apt-get update or try with --fix-missing，则输入：apt-get upgrade
报错Unable to correct problems, you have held broken packages.输入
apt-get autoremove openssh-client openssh-server
sudo apt-get install openssh-client openssh-server
安装成功后，输入ssh localhost测试

图3
Step3：安装Hadoop
(1)下载解压和Step2类似不再重复
(2)创建hadoop用户和组，并授予执行权限
sudo addgroup hadoop
sudo usermod -a -G hadoop juchang
sudo gedit etc/sudoers
在root ALL=(ALL:ALL) ALL下添加hadoop ALL=(ALL:ALL) ALL.
sudo chmod -R 755 /opt/hadoop
sudo chown -R xxx:hadoop /opt/hadoop
(3)和Step2一样，修改/etc/profile并激活

图4 测试hadoop version

图5

2.运行WordCount实例（非分布式）

hadoop默认是非分布式模式且自带wordcount的demo
cd /opt/hadoop
mkdir input
cp README.txt input
bin/hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.7.6-sources.jar org.apache.hadoop.examples.WordCount input output

运行结果：