一篇漫画理解HDFS读写工作原理

一篇漫画让你了解Hadoop HDFS！

默认的3副本存放规则：

1. 如果写入Client是HDFS集群中的DN，则1st副本存放在本Client所在主机；
2. 如果写入Client不是HDFS集群中DN，则1st副本随机存放在集群中某个DN;
3. 2nd副本存放于和1st副本不同机架的某DN节点上；
4. 3rd副本存放于和2nd副本相同机架的另外一个DN上。

/**
 * The class is responsible for choosing the desired number of targets
 * for placing block replicas.
 * The replica placement strategy is that if the writer is on a datanode,
 * the 1st replica is placed on the local machine, 
 * otherwise a random datanode. The 2nd replica is placed on a datanode
 * that is on a different rack. The 3rd replica is placed on a datanode
 * which is on a different node of the rack as the second replica.
 */
@InterfaceAudience.Private
public class BlockPlacementPolicyDefault extends BlockPlacementPolicy {

  private static final String enableDebugLogging =
    "For more information, please enable DEBUG log level on "
    + BlockPlacementPolicy.class.getName();

  private static final ThreadLocal<StringBuilder> debugLoggingBuilder
      = new ThreadLocal<StringBuilder>() {
        @Override
        protected StringBuilder initialValue() {
          return new StringBuilder();
        }
      };

关于block副本数需要注意的地方：

1. 设置的副本数不能超过集群中DataNode的数量
2. 每个DataNode只能存放某block的一个副本
3. 每个机架最多只能存放某个block的2个副本