rsync远程同步|xsync集群分发(hadoop完全分布式准
前置环境准备：
centos7克隆ip|机器名|映射关系|别名配置（hadoop完全分布式准备一）
scp命令copy文件和配置(hadoop完全分布式准备二)
rsync远程同步工具
优点
rsync主要用于备份和镜像。具有速度快、避免复制相同内容和支持符号链接的优点。
rsync和scp区别:用 rsync做文件的复制要比scp的速度快, rsync只对差异文件做更新。scp是把所有文件都复制过去。
基本语法
rsync -rvl $pdir/$fname $user(@hadoop$host: $pdir/$fname
命令 选项参数 要拷贝的文件路径/名称 目的用户@主机:目的路径名称
选项参数说明
选项 功能
-r 递归
-v 显示复制过程
-l Copy符号链接
案例
将hadoop101机器上的/opt/software 同步到hadoop102的服务器上的对应目录下。
[shaozhiqi@hadoop101 ~]$ rsync -rvl /opt/software shaozhiqi@hadoop102:/opt/
shaozhiqi@hadoop102's password:
\sending incremental file list
software/hadoop-3.1.2.tar.gz
software/jdk-8u211-linux-x64.tar.gz
sent 527488768 bytes received 51 bytes 14066368.51 bytes/sec
total size is 527424191 speedup is 1.00
[shaozhiqi@hadoop101 ~]$
查看102机器目录同步成功
[shaozhiqi@hadoop102 software]$ ll
total 515068
-rw-rw-r--. 1 shaozhiqi shaozhiqi 332433589 Jun 23 04:55 hadoop-3.1.2.tar.gz
-rw-rw-r--. 1 shaozhiqi shaozhiqi 194990602 Jun 23 04:56 jdk-8u211-linux-x64.tar.gz
[shaozhiqi@hadoop102 software]$
xsync集群分发脚本
(1)需求:循环复制文件到所有节点的相同目录下
(2)需求分析:
在 home/shaozhiqi/bin这个目录下存放的脚本, shaozhiqi用户可以在系统任何地方直接执行。
可用案例：
如果我们修改了其中一台hadoop的某个xml的，需要在其他机器上都同步过去就可以这样做
需求实现
在home/shao目录下创建bin目录,并在bin目录下 xsync创建文件,文件
脚本实现
#!/bin/bash
#1获取输入参数个数,如果没有参数,直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi
#2获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname
#3获取上级目录到绝对路径
pdir=`cd -P $(dirname $pl); pwd`
echo pdir=$pdir
#4获取当前用户名称
user=`whoami`
#5循环
for (( host =103; host<105;host++)); do
echo -------hadoop$host -----------------------
rsync -rvl $pdir/$fname $user@hadoop$host:$pdir
done
basename shell 语法
[shaozhiqi@hadoop102 opt]$ basename module/hadoop-3.1.2/etc/hadoop/core-site.xml
core-site.xml
[shaozhiqi@hadoop102 opt]$ basename core-site.xml
core-site.xml
dirname shell语法
[shaozhiqi@hadoop102 opt]$ dirname module/hadoop-3.1.2/etc/hadoop/core-site.xml
module/hadoop-3.1.2/etc/hadoop
[shaozhiqi@hadoop102 opt]$
whoami
[shaozhiqi@hadoop102 opt]$ whoami
shaozhiqi
[shaozhiqi@hadoop102 opt]$
赋权限
[shaozhiqi@hadoop102 bin]$ chmod 777 testxsync
[shaozhiqi@hadoop102 bin]$ ll
total 4
-rwxrwxrwx. 1 shaozhiqi shaozhiqi 520 Jun 23 05:48 testxsync
执行前查看103
[shaozhiqi@hadoop103 ~]$ ll
total 0
执行前查看104
[shaozhiqi@hadoop104 ~]$ ll
total 0
执行脚本
[shaozhiqi@hadoop102 bin]$ testxsync bin/
fname=bin
dirname: missing operand
Try 'dirname --help' for more information.
pdir=/home/shaozhiqi
-------hadoop103 -----------------------
shaozhiqi@hadoop103's password:
sending incremental file list
bin/
bin/.testxsync.swo
bin/.testxsync.swp
bin/testxsync
sent 25281 bytes received 73 bytes 2668.84 bytes/sec
total size is 25052 speedup is 0.99
-------hadoop104 -----------------------
shaozhiqi@hadoop104's password:
Permission denied, please try again.
shaozhiqi@hadoop104's password:
sending incremental file list
bin/
bin/.testxsync.swo
bin/.testxsync.swp
bin/testxsync
sent 25281 bytes received 73 bytes 2414.67 bytes/sec
total size is 25052 speedup is 0.99
[shaozhiqi@hadoop102 bin]$
执行后查看103、104脚本执行成功
[shaozhiqi@hadoop103 ~]$ ll
total 0
drwxrwxr-x. 2 shaozhiqi shaozhiqi 64 Jun 23 06:13 bin
[shaozhiqi@hadoop103 ~]$
[shaozhiqi@hadoop104 ~]$ ll
total 0
drwxrwxr-x. 2 shaozhiqi shaozhiqi 64 Jun 23 06:13 bin
[shaozhiqi@hadoop104 ~]$
说明
我们[shaozhiqi@hadoop103 ~]$ mkdir 的文件都会在 /home/shaozhiqi/bin
在 home/shaozhiqi/bin这个目录下存放的脚本, shaozhiqi用户可以在系统任何地方直接执行。
如果有意外，导致无法识别，那就把它放在、usr/local/bin下
