Cloudera Hadoop ACL实战
2017-12-06 本文已影响0人
shaopi0211
现状
随着业务复杂度提升，需要将hadoop资源开放给其他组使用时，简单的HDFS权限已无法满足当前业务权限管理需求。
环境
- 操作系统系统:Centos7
- JDK:1.7
- CDH 版本:5.8.4
操作步骤
- CM界面中修改hdfs参数:
dfs.namenode.acls.enabled为enable。 - 重启HDFS服务。
ACL实战
为用户添加acl权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls -d /user/hbase
drwx------ - hbase supergroup 0 2017-08-03 10:47 /user/hbase
[dengsc@nfjd-hadoop-test01 ~]$ id
uid=2190(dengsc) gid=100(users) groups=100(users),1002(data),20007(gb)
# 无权限访问此目录
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/hbase/
ls: Permission denied: user=dengsc, access=READ_EXECUTE, inode="/user/hbase":hbase:supergroup:drwx------
# 使用hdfs超级用户授权
[dengsc@nfjd-hadoop-test01 ~]$ sudo su - hdfs
Last login: Fri Oct 13 15:03:57 CST 2017 on pts/0
-bash-4.2$ id
uid=596(hdfs) gid=598(hdfs) groups=598(hdfs),600(hadoop)
-bash-4.2$ hdfs dfs -setfacl -m user:dengsc:rwx /user/hbase
-bash-4.2$ echo $?
0
# 发现权限为最后多了一个 + 号
-bash-4.2$ hdfs dfs -ls -d /user/hbase
drwxrwx---+ - hbase supergroup 0 2017-08-03 10:47 /user/hbase
# 切换至dengsc用户
-bash-4.2$ logout
[dengsc@nfjd-hadoop-test01 ~]$ id
uid=2190(dengsc) gid=100(users) groups=100(users),1002(data),20007(gb)
# 已有目录访问权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/hbase/
Found 6 items
drwx------ - hbase supergroup 0 2017-09-14 08:00 /user/hbase/.Trash
drwx------ - hbase supergroup 0 2017-08-05 21:35 /user/hbase/.staging
drwxr-xr-x - hbase supergroup 0 2017-07-27 11:32 /user/hbase/123
drwxr-xr-x - hbase supergroup 0 2017-07-27 10:44 /user/hbase/1234
drwxr-xr-x - hbase supergroup 0 2017-08-03 10:47 /user/hbase/ip_coordinate_geo_infos
drwxr-xr-x - hbase supergroup 0 2017-08-03 10:45 /user/hbase/mytest
# 获取当前目录的ACL权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -getfacl /user/hbase
# file: /user/hbase
# owner: hbase
# group: supergroup
user::rwx
user:dengsc:rwx
group::---
mask::rwx
other::---
为用户组添加acl权限
# 查看目录权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls -d /user/log/dengsc/
drwx------ - hdfs hdfs 0 2017-10-16 20:56 /user/log/dengsc
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/log/dengsc/
ls: Permission denied: user=dengsc, access=READ_EXECUTE, inode="/user/log/dengsc":hdfs:hdfs:drwx------
# 使用hdfs用户授权
[dengsc@nfjd-hadoop-test01 ~]$ sudo su - hdfs
Last login: Mon Oct 16 20:56:49 CST 2017 on pts/0
-bash-4.2$ hdfs dfs -setfacl -m group:data:rwx /user/log/dengsc
# 获取acl权限信息
-bash-4.2$ hdfs dfs -getfacl /user/log/dengsc
# file: /user/log/dengsc
# owner: hdfs
# group: hdfs
user::rwx
group::---
group:data:rwx
mask::rwx
other::---
# 在NameNode 所有主机添加data用户组,并将dengsc用户添加至组中
[dengsc@nfjd-hadoop02-node178 ~]$ sudo groupadd data && sudo usermod -a -G data dengsc
[dengsc@nfjd-hadoop02-node178 ~]$ grep dengsc /etc/group
data:x:20005:dengsc,zhengjj
# 在hadoop client节点访问前面的目录，已有访问权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/log/dengsc
Found 1 items
-rw-r--r-- 2 dengsc hdfs 0 2017-10-16 21:02 /user/log/dengsc/a.txt
限制用户组访问目录
# 查看目录权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -ls -d /user/dengsc
drwxr-xrwx+ - dengsc supergroup 0 2017-09-21 10:45 /user/dengsc
# 切换至xujun用户
[dengsc@nfjd-hadoop-test01 ~]$ sudo su - xujun
Last login: Mon Oct 16 10:14:03 CST 2017 from 172.16.101.204 on pts/0
# 能访问/user/dengsc目录
[xujun@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/dengsc/
Found 12 items
drwx------ - dengsc supergroup 0 2017-09-27 08:00 /user/dengsc/.Trash
drwxr-xr-x - dengsc supergroup 0 2017-06-06 13:44 /user/dengsc/.sparkStaging
drwx------ - dengsc supergroup 0 2017-09-02 19:21 /user/dengsc/.staging
drwxr-xr-x - dengsc supergroup 0 2017-04-11 09:35 /user/dengsc/QuasiMonteCarlo_1491874546037_2098352235
drwxr-xr-x - dengsc supergroup 0 2017-04-11 09:40 /user/dengsc/QuasiMonteCarlo_1491874794721_615029672
drwxr-xr-x - dengsc supergroup 0 2017-04-11 16:01 /user/dengsc/QuasiMonteCarlo_1491897628774_542221176
drwxr-xr-x - dengsc supergroup 0 2017-09-20 20:40 /user/dengsc/cold
drwxr-xr-x - dengsc supergroup 0 2017-05-15 20:56 /user/dengsc/input
drwxr-xrwx - dengsc supergroup 0 2017-07-27 13:38 /user/dengsc/python
---------- 3 dengsc supergroup 0 2017-06-26 14:46 /user/dengsc/test.x
-rw-r--r-- 3 dengsc supergroup 21 2017-09-14 15:37 /user/dengsc/test1.txt
drwxr-xr-t - dengsc supergroup 0 2017-09-21 15:45 /user/dengsc/warm
[xujun@nfjd-hadoop-test01 ~]$ logout
# 设置ACL权限
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -setfacl -m group:data:--- /user/dengsc/
[dengsc@nfjd-hadoop-test01 ~]$ hdfs dfs -getfacl /user/dengsc/
# file: /user/dengsc
# owner: dengsc
# group: supergroup
user::rwx
user:dsc:---
group::r-x
group:data:---
mask::r-x
other::rwx
# NameNode主机添加xujun至data组
[dengsc@nfjd-hadoop02-node178 ~]$ sudo usermod -a -G data xujun
[dengsc@nfjd-hadoop02-node178 ~]$ grep data /etc/group
data:x:20005:dengsc,zhengjj,xujun
# xujun用户已无法访问/user/dengsc目录
[xujun@nfjd-hadoop-test01 ~]$ hdfs dfs -ls /user/dengsc/
ls: Permission denied: user=xujun, access=READ_EXECUTE, inode="/user/dengsc":dengsc:supergroup:drwxr-xrwx
清除ACL权限
[dengsc@nfjd-hadoop02-node178 ~]$ hdfs dfs -getfacl /user/dengsc
# file: /user/dengsc
# owner: dengsc
# group: supergroup
user::rwx
user:dsc:---
group::r-x
group:data:---
mask::r-x
other::rwx
# 清除group:data的ACL权限
[dengsc@nfjd-hadoop02-node178 ~]$ hdfs dfs -setfacl -x group:data /user/dengsc
[dengsc@nfjd-hadoop02-node178 ~]$ hdfs dfs -getfacl /user/dengsc
# file: /user/dengsc
# owner: dengsc
# group: supergroup
user::rwx
user:dsc:---
group::r-x
mask::r-x
other::rwx
[dengsc@nfjd-hadoop02-node178 ~]$ sudo su - xujun
Last login: Tue Aug 1 13:36:20 CST 2017 from 172.16.101.204 on pts/0
# xujun用户重获/user/dengsc目录权限
[xujun@nfjd-hadoop02-node178 ~]$ hdfs dfs -ls /user/dengsc
Found 12 items
drwx------ - dengsc supergroup 0 2017-09-27 08:00 /user/dengsc/.Trash
drwxr-xr-x - dengsc supergroup 0 2017-06-06 13:44 /user/dengsc/.sparkStaging
drwx------ - dengsc supergroup 0 2017-09-02 19:21 /user/dengsc/.staging
drwxr-xr-x - dengsc supergroup 0 2017-04-11 09:35 /user/dengsc/QuasiMonteCarlo_1491874546037_2098352235
drwxr-xr-x - dengsc supergroup 0 2017-04-11 09:40 /user/dengsc/QuasiMonteCarlo_1491874794721_615029672
drwxr-xr-x - dengsc supergroup 0 2017-04-11 16:01 /user/dengsc/QuasiMonteCarlo_1491897628774_542221176
drwxr-xr-x - dengsc supergroup 0 2017-09-20 20:40 /user/dengsc/cold
drwxr-xr-x - dengsc supergroup 0 2017-05-15 20:56 /user/dengsc/input
drwxr-xrwx - dengsc supergroup 0 2017-07-27 13:38 /user/dengsc/python
---------- 3 dengsc supergroup 0 2017-06-26 14:46 /user/dengsc/test.x
-rw-r--r-- 3 dengsc supergroup 21 2017-09-14 15:37 /user/dengsc/test1.txt
drwxr-xr-t - dengsc supergroup 0 2017-09-21 15:45 /user/dengsc/warm
总结
添加ACL限制
hdfs dfs -setfacl -m group:partner:--- /user/log
查看acl限制
hdfs dfs -getfacl /user/log
删除acl限制
hdfs dfs -setfacl -x group:partner /user/log
将用户添加至partner组
sudo usermod -a -G partner ronghui
将用户从partner组移除
sudo gpasswd -d ronghui partner
添加组
sudo groupadd partner
添加用户
sudo useradd secret -g users -d /home/secret/ -s /bin/bash;
删除用户
sudo userdel secret
删除组
sudo groupdel partner
关于default ACL继承 BUG
apache社区补丁在3.0版本解决
https://issues.apache.org/jira/browse/HDFS-6962
Cloudera 社区在cdh5.8.4解决
https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_58.html#fixed_issues584
需要修改参数dfs.namenode.posix.acl.inheritance.enabled值为true
cloudera hadoop 只需要在NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml配置中添加参数dfs.namenode.posix.acl.inheritance.enabled=true,并重启两个NameNode服务。
default ACL继承实战
# user hdfs
# 创建父目录
$ hdfs dfs -mkdir /user/dengsc/acl
$ hdfs dfs -chgrp hdfs /user/dengsc/acl
$ hdfs dfs -chmod 770 /user/dengsc/acl
# 设置父目录defualt ACL权限
$ hdfs dfs -setfacl -m default:user:dengsc:rwx /user/dengsc/acl
$ hdfs dfs -mkdir /user/dengsc/acl/ccc
# 子目录继承父目录ACL权限
$ hdfs dfs -getfacl /user/dengsc/acl/ccc
# file: /user/dengsc/acl/ccc
# owner: hdfs
# group: hdfs
user::rwx
user:dengsc:rwx
group::rwx
mask::rwx
other::---
default:user::rwx
default:user:dengsc:rwx
default:group::rwx
default:mask::rwx
default:other::---
# 设置父目录ACL权限
$ hdfs dfs -setfacl -m user:dengsc:rwx /user/dengsc/acl
# user dengsc
$ logout
# 检查父目录ACL权限是否生效
$ hdfs dfs -mkdir /user/dengsc/acl/ddd
# 检查default ACL权限是否继承
$ hdfs dfs -mkdir /user/dengsc/acl/ddd/ppp
$ hdfs dfs -getfacl /user/dengsc/acl/ddd/ppp
# file: /user/dengsc/acl/ddd/ppp
# owner: dengsc
# group: hdfs
user::rwx
user:dengsc:rwx
group::rwx
mask::rwx
other::---
default:user::rwx
default:user:dengsc:rwx
default:group::rwx
default:mask::rwx
default:other::---
# 检查默认umask是否存在问题
$ hdfs dfs -mkdir /user/dengsc/ppp
$ hdfs dfs -ls -d /user/dengsc/ppp
drwxr-xr-x - dengsc supergroup 0 2017-12-05 19:14 /user/dengsc/ppp
