12-HDFS详解二
1.HDFS写流程
写的操作对于我们客户端来说是透明的
drwxr-xr-x
hadoop rwx
supergroup r-x
other r-x
权限:
[root@hadoop000 ~]# su - hadoop -c "hdfs dfs -put /tmp/test.log /"
[root@hadoop000 ~]# su - hadoop -c "hdfs dfs -ls /"
Found 6 items
-rw-r--r-- 1 hadoop supergroup 18 2018-05-19 20:47 /rz.log
-rw-r--r-- 2 hadoop supergroup 18 2018-05-19 21:20 /rz.log1
-rw-r--r-- 1 hadoop supergroup 18 2018-05-20 20:04 /rz.log123
-rw-r--r-- 1 hadoop supergroup 0 2018-05-20 20:10 /test.log
drwx------ - hadoop supergroup 0 2018-05-16 22:49 /tmp
drwxr-xr-x - hadoop supergroup 0 2018-05-16 22:49 /user
[hadoop@hadoop000 ~]$ hdfs dfs -mkdir /root
[hadoop@hadoop000 ~]$ hdfs dfs -chown -R root:root /root
[hadoop@hadoop000 ~]$ hdfs dfs -ls /
Found 7 items
drwxr-xr-x - root root 0 2018-05-20 20:12 /root
-rw-r--r-- 1 hadoop supergroup 18 2018-05-19 20:47 /rz.log
-rw-r--r-- 2 hadoop supergroup 18 2018-05-19 21:20 /rz.log1
-rw-r--r-- 1 hadoop supergroup 18 2018-05-20 20:04 /rz.log123
-rw-r--r-- 1 hadoop supergroup 0 2018-05-20 20:10 /test.log
drwx------ - hadoop supergroup 0 2018-05-16 22:49 /tmp
drwxr-xr-x - hadoop supergroup 0 2018-05-16 22:49 /user
2.pid
pid默认存储在/tmp目录
pid内容为进程号
ps -ef|grep hadoop 出现PID A,B,C 可能误杀B,C
[hadoop@hadoop000 sbin]$ cat hadoop-daemon.sh |grep pid
# HADOOP_PID_DIR The pid files are stored. /tmp by default.
pid=$HADOOP_PID_DIR/hadoop-$HADOOP_IDENT_STRING-$command.pid
if [ -f $pid ]; then
if kill -0 `cat $pid` > /dev/null 2>&1; then
echo $command running as process `cat $pid`. Stop it first.
echo $! > $pid
if [ -f $pid ]; then
TARGET_PID=`cat $pid`
rm -f $pid
删除pid，进程会挂吗？能够提供服务吗？
进程不挂，服务正常
生产上:
pid 绝对不能放到/tmp?
一个月/tmp 会自动删除pid
mkdir /data/hadoop/tmp
chmod -R 777 /data/hadoop/tmp
vi hadoop-env.sh
export HADOOP_PID_DIR=/data/hadoop/tmp
3.hadoop命令[hadoop@hadoop000 hadoop-2.8.1]$ hadoop fsUsage: hadoop fs [generic options][-put [-f] [-p] [-l] [-d]...][-get [-f] [-p] [-ignoreCrc] [-crc]...][-copyFromLocal [-f] [-p] [-l] [-d]...][-copyToLocal [-f] [-p] [-ignoreCrc] [-crc]...]
[hadoop@hadoop000 hadoop-2.8.1]$ hadoop checknative
18/05/20 20:54:13 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
18/05/20 20:54:13 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /opt/software/hadoop-2.8.1/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:10301
bzip2: false
openssl: true /usr/lib64/libcrypto.so
4.hdfs命令[hadoop@hadoop000 ~]$ hdfs dfs Usage: hadoop fs [generic options][hadoop@hadoop000 hadoop]$ hdfs dfsadminUsage: hdfs dfsadminNote: Administrative commands can only be run as the HDFS superuser. [-report [-live] [-dead] [-decommissioning]] [-safemode]
[hadoop@hadoop000 hadoop]$ hdfs fsck
5.垃圾箱
[hadoop@hadoop000 hadoop]$ hdfs dfs -rm /rz.log123
Deleted /rz.log123
[hadoop@hadoop000 hadoop]$ hdfs dfs -rm /rz.log1
18/05/20 21:25:05 INFO fs.TrashPolicyDefault:
Moved: 'hdfs://192.168.137.251:9000/rz.log1' to trash
at: hdfs://192.168.137.251:9000/user/hadoop/.Trash/Current/rz.log1
[hadoop@hadoop000 hadoop]$
[hadoop@hadoop000 hadoop]$ hdfs dfs -rm /rz.log1
18/05/20 21:25:05 INFO fs.TrashPolicyDefault: Moved: 'hdfs://192.168.137.251:9000/rz.log1' to trash at: hdfs://192.168.137.251:9000/user/hadoop/.Trash/Current/rz.log1
[hadoop@hadoop000 hadoop]$ hdfs dfs -ls /
Found 5 items
drwxr-xr-x - root root 0 2018-05-20 20:13 /root
-rw-r--r-- 1 hadoop supergroup 18 2018-05-19 20:47 /rz.log
-rw-r--r-- 1 hadoop supergroup 0 2018-05-20 20:10 /test.log
drwx------ - hadoop supergroup 0 2018-05-16 22:49 /tmp
drwxr-xr-x - hadoop supergroup 0 2018-05-16 22:49 /user
[hadoop@hadoop000 hadoop]$ hdfs dfs -mv /user/hadoop/.Trash/Current/rz.log1 /rz.log1
[hadoop@hadoop000 hadoop]$ hdfs dfs -ls /
Found 6 items
drwxr-xr-x - root root 0 2018-05-20 20:13 /root
-rw-r--r-- 1 hadoop supergroup 18 2018-05-19 20:47 /rz.log
-rw-r--r-- 2 hadoop supergroup 18 2018-05-19 21:20 /rz.log1
-rw-r--r-- 1 hadoop supergroup 0 2018-05-20 20:10 /test.log
drwx------ - hadoop supergroup 0 2018-05-16 22:49 /tmp
drwxr-xr-x - hadoop supergroup 0 2018-05-16 22:49 /user
[hadoop@hadoop000 hadoop]$
[hadoop@hadoop000 hadoop]$ hdfs dfs -rm -skipTrash /rz.log1
Deleted /rz.log1
