下载github仓库文件

从github仓库下载最新文件并解压至 hadoop-kerberos-master
ifilonenko/hadoop-kerberos: Docker compose: single-node, pseudo-distributed, kerberized, hadoop cluster (github.com)

下载hadoop

https://mirror-hk.koddos.net/apache/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz

下载后的压缩文件复制到hadoop-kerberos-master目录下

image.png

修改dockerfile

仓库中的版本是2.7，通过上面的连接下载的为hadoop-2.10.1，所以需要修改dockerfile中的版本号
改为以下内容

ADD hadoop-2.10.1.tar.gz /
RUN ln -s hadoop-2.10.1 hadoop

修改docker-compose.yml

原始配置文件中只暴露hdfs的9000端口，通过修改配置暴露出hdfs的50470（webui）端口和kerberos的88端口，经过修改后的docker-compose.yml文件如下

version: "2"
services:
  kerberos:
    ports:
      - 88:88/udp
    container_name: kerberos.example
    hostname: kerberos.example.com
    entrypoint: /start-kdc.sh
    build:
      context: .
      args:
        - http_proxy
        - https_proxy
    volumes:
      - server-keytab:/var/keytabs

  nn:
    ports:
      - 9000:9000
      - 50470:50470
    container_name: nn.example
    hostname: nn.example.com
    user: hdfs
    entrypoint: /start-namenode.sh
    build:
      context: .
      args:
        - http_proxy
        - https_proxy
    volumes:
      - server-keytab:/var/keytabs
      - /hadoop

  dn1:
    container_name: dn1.example
    hostname: dn1.example.com
    user: hdfs
    entrypoint: /start-datanode.sh
    build:
      context: .
      args:
        - http_proxy
        - https_proxy
    volumes:
      - server-keytab:/var/keytabs

  data-populator:
    container_name: data-populator.example
    hostname: data-populator.example.com
    user: hdfs
    entrypoint: /populate-data.sh
    build:
      context: .
      args:
        - http_proxy
        - https_proxy
    volumes:
      - server-keytab:/var/keytabs

networks:
  default:
    external:
      name: com

volumes:
  server-keytab:

初次安装

执行以下命令直接创建镜像并启动容器

docker-compose up -d --force-recreate --build

服务启动错误

如果遇到nn.example服务启动后自动退出的情况，需要全局查找 krb5.conf文件并注释 #renew_lifetime = 7d
在linux系统下先使用docker-compose down 命令卸载相关容器，通过find命令查找所有 krb5.conf文件，逐一修改

其他命令

1 环境清理

docker-compose down
docker volume rm hadoop-kerberos-master_server-keytab
##docker-compose up -d --force-recreate --build

2 重新启动hadoop

docker-compose up -d --force-recreate --build

3 在容器中登录hdfs

docker exec -it nn.example /bin/bash
kinit -kt /var/keytabs/hdfs.keytab hdfs/nn.example.com
hdfs dfs -ls /

4 复制keytab文件

docker cp  nn.example:/var/keytabs/hdfs.keytab ./hdfs.keytab

5 复制conf文件

docker cp  kerberos.example:/etc/krb5.conf ./krb5.conf

通过java连接kerberos hdfs

        String path;
        String user;
        String url;
        String keyTab; 
        System.setProperty("hadoop.home.dir", "E:\\TOOLS\\hadoop\\hadoop");
        //System.setProperty("sun.security.krb5.debug", "true");
        System.setProperty("java.security.krb5.conf", "E:\\data\\hdfs\\krb5.conf");

        path = "/user/ifilonenko";
        user = "hdfs/nn.example.com";
        url = "hdfs://192.168.172.149:9000/"; 
        keyTab = "E:\\data\\hdfs\\hdfs.keytab";

        org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
        conf.set("fs.defaultFS", url);
        conf.set("hadoop.security.authentication", "kerberos");

        try {
            UserGroupInformation.setConfiguration(conf);
            UserGroupInformation.loginUserFromKeytab(user, keyTab);
            FileSystem fs = FileSystem.get(conf);
            FileStatus files[] = fs.listStatus(new Path(path));
            log.info("11111111111111111111111111");
            for (FileStatus file : files) {
                log.info(file.getPath().toString());
            }
            log.info("2222222222222222222222222222");
        } catch (Exception e) {
            e.printStackTrace();

        }