Performance Parameter define / change

Parameters are defined in HDFS-site.xml.

Cloudera manager has friendly GUI for end-user to change the para, without going with xml file modification manually.

Start Cloudera manager:

    On terminal, run:  $ sudo /home/cloudera/cloudera-manager --express --force

    Then, on firefox: access : quickstart.cloudera:7180/cmf/services/8/config

4 main parameters impact performances:

    DFS Block size  -- dfs.blocksize : default 64M.  Impact directly the name node mamory usage and mumber of map tasks.

    HDFS Replication -- dfs.replication : default 3. Reducing replication has a trade off with regards to robustness. It mitigates the failure and is achieved from perspectives below:

        periodicaly heartbeat from data node to name node.

        file's checksum stored in name node, to verify the re-read from other healthy nodes.

    Number of handlers on each data node -- dfs.datanode.handler.count

    Maximum number of blocks per file -- dfs.namenode.fs-limits.max-blocks-per-file

lesson 7 - slides