Apache Phoenix（十八）Phoenix结合Hbase
2020-03-03 本文已影响0人
我知他风雨兼程途径日暮不赏
HBASE Help描述
Set a quota for a user, table, or namespace.
Syntax : set_quota TYPE => <type>, <args>
TYPE => THROTTLE
User can either set quota on read, write or on both the requests together(i.e., read+write)
The read, write, or read+write(default throttle type) request limit can be expressed using
the form 100req/sec, 100req/min and the read, write, read+write(default throttle type) limit
can be expressed using the form 100k/sec, 100M/min with (B, K, M, G, T, P) as valid size unit
and (sec, min, hour, day) as valid time unit.
Currently the throttle limit is per machine - a limit of 100req/min
means that each machine can execute 100req/min.
For example:
hbase> set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => '10req/sec'
hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => READ, USER => 'u1', LIMIT => '10req/sec'
hbase> set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => '10M/sec'
hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, USER => 'u1', LIMIT => '10M/sec'
hbase> set_quota TYPE => THROTTLE, USER => 'u1', TABLE => 't2', LIMIT => '5K/min'
hbase> set_quota TYPE => THROTTLE, USER => 'u1', NAMESPACE => 'ns2', LIMIT => NONE
hbase> set_quota TYPE => THROTTLE, NAMESPACE => 'ns1', LIMIT => '10req/sec'
hbase> set_quota TYPE => THROTTLE, TABLE => 't1', LIMIT => '10M/sec'
hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, TABLE => 't1', LIMIT => '10M/sec'
hbase> set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => NONE
hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, USER => 'u1', LIMIT => NONE
hbase> set_quota USER => 'u1', GLOBAL_BYPASS => true
TYPE => SPACE
Users can either set a quota on a table or a namespace. The quota is a limit on the target is size on the FileSystem and some action to take when the target exceeds that limit. The limit
is in bytes and can expressed using standard metric suffixes (B, K, M, G, T, P), defaulting
to bytes if not provided. Different quotas can be applied to one table at the table and namespace
level; table-level quotas take priority over namespace-level quotas.
There are a limited number of policies to take when a quota is violation, listed in order of
least strict to most strict.
NO_INSERTS - No new data is allowed to be ingested (e.g. Put, Increment, Append).
NO_WRITES - Same as NO_INSERTS but Deletes are also disallowed.
NO_WRITES_COMPACTIONS - Same as NO_WRITES but compactions are also disallowed.
DISABLE - The table(s) are disabled.
For example:
hbase> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => '1G', POLICY => NO_INSERTS
hbase> set_quota TYPE => SPACE, TABLE => 't2', LIMIT => '50G', POLICY => DISABLE
hbase> set_quota TYPE => SPACE, TABLE => 't3', LIMIT => '2T', POLICY => NO_WRITES_COMPACTIONS
hbase> set_quota TYPE => SPACE, NAMESPACE => 'ns1', LIMIT => '50T', POLICY => NO_WRITES
Space quotas can also be removed via this command. To remove a space quota, provide NONE
for the limit.
For example:
hbase> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => NONE
hbase> set_quota TYPE => SPACE, NAMESPACE => 'ns1', LIMIT => NONE
实验
实验目的：限制一个命名空间无限膨胀，假定一个用户一个命名空间，达到类似云盘限制配额功能。
准备
phoenix创建命名空间：
create schema quota;
创建phoenix测试表：
CREATE TABLE quota.example (
my_pk bigint not null,
m.first_name varchar(50),
m.last_name varchar(50)
CONSTRAINT pk PRIMARY KEY (my_pk))
hbase-site.xml修改
设置quota限额开启，并设置限额更新时间（毫秒为单位），重启hbase。
<property>
<name>hbase.quota.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.quota.refresh.period</name>
<value>1000</value>
</property>
新增限额策略
这里表示：整个命名空间为quota的全部表大小限制成1B，如果超过1B无法数据写入。
hbase(main):010:0> set_quota TYPE => SPACE, NAMESPACE => 'QUOTA' LIMIT => '1B', POLICY => NO_WRITES
测试配额是否生效
写入数据：
UPSERT INTO QUOTA.EXAMPLE values(92331,'hah','HAHA');
抛出对应的异常：
Error: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NO_WRITES Puts are disallowed due to a space quota.
at org.apache.hadoop.hbase.quotas.policies.NoWritesViolationPolicyEnforcement.check(NoWritesViolationPolicyEnforcement.java:46)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1012)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
: 1 time, servers with issues: master,16020,1583226486320 (state=,code=0)
org.apache.phoenix.execute.CommitException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NO_WRITES Puts are disallowed due to a space quota.
at org.apache.hadoop.hbase.quotas.policies.NoWritesViolationPolicyEnforcement.check(NoWritesViolationPolicyEnforcement.java:46)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1012)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
: 1 time, servers with issues: master,16020,1583226486320
at org.apache.phoenix.execute.MutationState.send(MutationState.java:1058)
at org.apache.phoenix.execute.MutationState.send(MutationState.java:1344)
at org.apache.phoenix.execute.MutationState.commit(MutationState.java:1167)
at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:670)
at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:666)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:666)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:411)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:813)
at sqlline.SqlLine.begin(SqlLine.java:686)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:291)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: NO_WRITES Puts are disallowed due to a space quota.
at org.apache.hadoop.hbase.quotas.policies.NoWritesViolationPolicyEnforcement.check(NoWritesViolationPolicyEnforcement.java:46)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1012)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
: 1 time, servers with issues: master,16020,1583226486320
at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1225)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:455)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438)
at org.apache.phoenix.execute.MutationState.send(MutationState.java:993)
... 18 more
20/03/03 17:50:23 WARN client.AsyncRequestFutureImpl: id=1, table=QUOTA:EXAMPLE, attempt=1/16, failed=1ops, last exception=org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_WRITES Puts are disallowed due to a space quota.
at org.apache.hadoop.hbase.quotas.policies.NoWritesViolationPolicyEnforcement.check(NoWritesViolationPolicyEnforcement.java:46)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1012)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
on master,16020,1583226486320, tracking started Tue Mar 03 17:50:23 CST 2020; not retrying 1 - final failure
