最新要闻

广告

手机

iphone11大小尺寸是多少？苹果iPhone11和iPhone13的区别是什么？

iphone11大小尺寸是多少？苹果iPhone11和iPhone13的区别是什么？

 警方通报辅警执法直播中被撞飞：犯罪嫌疑人已投案

警方通报辅警执法直播中被撞飞：犯罪嫌疑人已投案

家电

  1. 首页
  2. 移动通信
  3. 内容

【环球时快讯】HDFS Balancer负载均衡器

来源：博客园


(资料图)

目录
  • 1、背景
  • 2、什么是平衡
    • 2.1 每个DataNode的利用率计算
    • 2.2 集群的利用率
    • 2.3 平衡
  • 3、hdfs balancer语法
  • 4、运行一个简单的balance案例
    • 4.1 设置平衡数据传输带宽
    • 4.2 执行banalce
  • 5、参考文档

1、背景

当我们的hadoop集群运行了一段时间之后，各个DataNode上的数据分布并不一定是均匀分布的。比如说：我们向现有集群中添加了一个新的DataNode。

2、什么是平衡

此处是我自己的一个简单的理解

所谓的平衡指的是 每个DataNode的利用率与 集群的利用率之间相差不超过给定的阈值百分比。此处的平衡指的是各个DataNode之间的平衡，同一个DataNode之间的各个磁盘是不会平衡的。

2.1 每个DataNode的利用率计算

DataNode的利用率=dfs已用的空间 / 分配给dfs的空间。

注意：分配给dfs的空间不是磁盘的总空间。

2.2 集群的利用率

集群的利用率=各datanode dfs已使用的空间 / 各datanode总空间

2.3 平衡

假设平衡的阈值是 5%，集群的利用率是 37.5，那么每个节点的利用率在32.5%到42.5%之间都认为是均衡的。也就是说，极端情况下，DataNode的利用率最大相差10%。

3、hdfs balancer语法

[hadoopdeploy@hadoop01 ~]$ hdfs balancer --helpUsage: hdfs balancer[-policy ]the balancing policy: datanode or blockpool[-threshold ]Percentage of disk capacity[-exclude [-f  | ]]Excludes the specified datanodes.[-include [-f  | ]]Includes only the specified datanodes.[-source [-f  | ]]Pick only the specified datanodes as source nodes.[-blockpools ]The balancer will only run on blockpools included in this list.[-idleiterations ]Number of consecutive idle iterations (-1 for Infinite) before exit.[-runDuringUpgrade]Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.[-asService]Run as a long running service.Generic options supported are:-conf         specify an application configuration file-D                define a value for a given property-fs  specify default filesystem URL to use, overrides "fs.defaultFS" property from configurations.-jt   specify a ResourceManager-files                 specify a comma-separated list of files to be copied to the map reduce cluster-libjars                specify a comma-separated list of jar files to be included in the classpath-archives           specify a comma-separated list of archives to be unarchived on the compute machinesThe general command line syntax is:command [genericOptions] [commandOptions]
参数描述
threshold磁盘容量的百分比。默认值为10%，表示上下浮动10%。
policy平衡策略。 datanode（默认）：当每一个DataNode是平衡的时候，集群就是平衡的。blockpool：当每一个DataNode中的blockpool是平衡的，集群就是平衡的。
exclude不参与平衡的DataNode节点
include参与平衡的DataNode节点
source仅选取指定的数据节点作为源节点
blockpoolsBalancer仅在指定的blockpools中运行
idleiterations退出前的连续空闲迭代次数（-1表示无限）
-runDuringUpgrade是否在正在进行的HDFS升级过程中运行平衡器。通常不需要这样做，因为这不会影响过度使用的计算机上的已用空间。
-asService作为长期运行的服务运行

4、运行一个简单的balance案例

4.1 设置平衡数据传输带宽

[hadoopdeploy@hadoop01 ~]$ hdfs dfsadmin  -setBalancerBandwidth 10485760Balancer bandwidth is set to 10485760[hadoopdeploy@hadoop01 ~]$

当我们的集群负载需要调低这个值，当我们的集群负载较低时，可以适当调高这个值。

4.2 执行banalce

[hadoopdeploy@hadoop01 ~]$ hdfs balancer -policy datanode -threshold 52023-03-26 14:10:09,785 INFO balancer.Balancer: Using a threshold of 5.02023-03-26 14:10:09,786 INFO balancer.Balancer: namenodes  = [hdfs://hadoop01:8020]2023-03-26 14:10:09,786 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]2023-03-26 14:10:09,786 INFO balancer.Balancer: included nodes = []2023-03-26 14:10:09,786 INFO balancer.Balancer: excluded nodes = []2023-03-26 14:10:09,786 INFO balancer.Balancer: source nodes = []Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved  NameNode2023-03-26 14:10:09,787 INFO balancer.NameNodeConnector: getBlocks calls for hdfs://hadoop01:8020 will be rate-limited to 20 per second2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.namenode.get-blocks.max-qps = 20 (default=20)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 100 (default=100)2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.bandwidthPerSec = 104857600 (default=104857600)2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.141:98662023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.140:98662023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.142:98662023-03-26 14:10:10,402 INFO balancer.Balancer: 0 over-utilized: []2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 underutilized: []2023-3-26 14:10:10                0                  0 B                 0 B                0 B                  0  hdfs://hadoop01:8020The cluster is balanced. Exiting...2023-3-26 14:10:10       Balancing took 810.0 milliseconds[hadoopdeploy@hadoop01 ~]$

5、参考文档

1、https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer2、https://help.aliyun.com/document_detail/449686.html

关键词：