Sharding-JDBC 配置分析
2020-05-27 本文已影响0人
晴天哥_王志
Sharding-JDBC系列
- Sharding-JDBC 核心组件介绍
- Sharding-JDBC 配置分析
- Sharding-JDBC 执行整体流程
- Sharding-JDBC 分库配置解析过程
- Sharding-JDBC 分表配置解析过程
- Sharding-JDBC 分库分表配置解析过程
数据分片配置解析流程
ShardingRule创建流程
public class ShardingRule implements BaseRule {
private final ShardingRuleConfiguration ruleConfiguration;
private final ShardingDataSourceNames shardingDataSourceNames;
private final Collection<TableRule> tableRules;
private final Collection<BindingTableRule> bindingTableRules;
private final Collection<String> broadcastTables;
private final ShardingStrategy defaultDatabaseShardingStrategy;
private final ShardingStrategy defaultTableShardingStrategy;
private final ShardingKeyGenerator defaultShardingKeyGenerator;
private final Collection<MasterSlaveRule> masterSlaveRules;
private final EncryptRule encryptRule;
public ShardingRule(final ShardingRuleConfiguration shardingRuleConfig, final Collection<String> dataSourceNames) {
Preconditions.checkArgument(null != shardingRuleConfig, "ShardingRuleConfig cannot be null.");
Preconditions.checkArgument(null != dataSourceNames && !dataSourceNames.isEmpty(), "Data sources cannot be empty.");
this.ruleConfiguration = shardingRuleConfig;
shardingDataSourceNames = new ShardingDataSourceNames(shardingRuleConfig, dataSourceNames);
// 1、创建tableRules
tableRules = createTableRules(shardingRuleConfig);
broadcastTables = shardingRuleConfig.getBroadcastTables();
// 2、创建bindingTableRules
bindingTableRules = createBindingTableRules(shardingRuleConfig.getBindingTableGroups());
defaultDatabaseShardingStrategy = createDefaultShardingStrategy(shardingRuleConfig.getDefaultDatabaseShardingStrategyConfig());
defaultTableShardingStrategy = createDefaultShardingStrategy(shardingRuleConfig.getDefaultTableShardingStrategyConfig());
defaultShardingKeyGenerator = createDefaultKeyGenerator(shardingRuleConfig.getDefaultKeyGeneratorConfig());
// 3、创建masterSlaveRules
masterSlaveRules = createMasterSlaveRules(shardingRuleConfig.getMasterSlaveRuleConfigs());
encryptRule = createEncryptRule(shardingRuleConfig.getEncryptRuleConfig());
}
}
- 负责解析ShardingRuleConfiguration生成ShardingRule。
- createTableRules负责创建创建tableRules。
- createBindingTableRules负责创建bindingTableRules。
- createMasterSlaveRules负责创建masterSlaveRules。
RuleConfiguration
@Getter
@Setter
public final class ShardingRuleConfiguration implements RuleConfiguration {
// 表规则配置
private Collection<TableRuleConfiguration> tableRuleConfigs = new LinkedList<>();
// 相同表分片规则的组，如果表分片规则相同，则可以放在一个组里。
private Collection<String> bindingTableGroups = new LinkedList<>();
// 广播的表
private Collection<String> broadcastTables = new LinkedList<>();
// 默认数据源名称
private String defaultDataSourceName;
// 默认数据库的分片算法配置
private ShardingStrategyConfiguration defaultDatabaseShardingStrategyConfig;
// 默认表的分片算法配置
private ShardingStrategyConfiguration defaultTableShardingStrategyConfig;
// 自动生成键的配置
private KeyGeneratorConfiguration defaultKeyGeneratorConfig;
// 主备配置信息
private Collection<MasterSlaveRuleConfiguration> masterSlaveRuleConfigs = new LinkedList<>();
// 加密配置规则
private EncryptRuleConfiguration encryptRuleConfig;
}
@Getter
@Setter
public final class TableRuleConfiguration {
// 逻辑表名
private final String logicTable;
// 实际物理的表，按照dataBase+table的维度
private final String actualDataNodes;
// database的分片策略
private ShardingStrategyConfiguration databaseShardingStrategyConfig;
// table的分片的策略
private ShardingStrategyConfiguration tableShardingStrategyConfig;
// 自动生成键的配置
private KeyGeneratorConfiguration keyGeneratorConfig;
public TableRuleConfiguration(final String logicTable) {
this(logicTable, null);
}
public TableRuleConfiguration(final String logicTable, final String actualDataNodes) {
Preconditions.checkArgument(!Strings.isNullOrEmpty(logicTable), "LogicTable is required.");
this.logicTable = logicTable;
this.actualDataNodes = actualDataNodes;
}
}
@Getter
public class MasterSlaveRuleConfiguration implements RuleConfiguration {
// 名称
private final String name;
// 主数据源
private final String masterDataSourceName;
// 从数据源
private final List<String> slaveDataSourceNames;
// 负载均衡算法
private final LoadBalanceStrategyConfiguration loadBalanceStrategyConfiguration;
}
- 各类Rule的创建依据的是RuleConfiguration，目前主要由ShardingRuleConfiguration、TableRuleConfiguration、MasterSlaveRuleConfiguration。
- ShardingRuleConfiguration是整体的规则配置。
- TableRuleConfiguration是表的规则配置，核心变量是logicTable和actualDataNodes。
- MasterSlaveRuleConfiguration是主从的配置。
TableRule创建流程
public class ShardingRule implements BaseRule {
private Collection<TableRule> createTableRules(final ShardingRuleConfiguration shardingRuleConfig) {
Collection<TableRuleConfiguration> tableRuleConfigurations = shardingRuleConfig.getTableRuleConfigs();
Collection<TableRule> result = new ArrayList<>(tableRuleConfigurations.size());
// 遍历tableRuleConfigurations生成TableRule
for (TableRuleConfiguration each : tableRuleConfigurations) {
// 针对每个TableRuleConfiguration来生成TableRule
// 指定shardingDataSourceNames为DataSource的名称
result.add(new TableRule(each, shardingDataSourceNames, getDefaultGenerateKeyColumn(shardingRuleConfig)));
}
return result;
}
}
@Getter
@ToString(exclude = {"dataNodeIndexMap", "actualTables", "actualDatasourceNames", "datasourceToTablesMap"})
public final class TableRule {
public TableRule(final TableRuleConfiguration tableRuleConfig, final ShardingDataSourceNames shardingDataSourceNames, final String defaultGenerateKeyColumn) {
// 逻辑表名
logicTable = tableRuleConfig.getLogicTable().toLowerCase();
// 通过splitAndEvaluate针对实际的dataNodes会根据配置进行实际解析
List<String> dataNodes = new InlineExpressionParser(tableRuleConfig.getActualDataNodes()).splitAndEvaluate();
dataNodeIndexMap = new HashMap<>(dataNodes.size(), 1);
// actualDataNodes会建立dataSource和logic表的关系
actualDataNodes = isEmptyDataNodes(dataNodes)
? generateDataNodes(tableRuleConfig.getLogicTable(), shardingDataSourceNames.getDataSourceNames()) : generateDataNodes(dataNodes, shardingDataSourceNames.getDataSourceNames());
actualTables = getActualTables();
databaseShardingStrategy = null == tableRuleConfig.getDatabaseShardingStrategyConfig() ? null : ShardingStrategyFactory.newInstance(tableRuleConfig.getDatabaseShardingStrategyConfig());
tableShardingStrategy = null == tableRuleConfig.getTableShardingStrategyConfig() ? null : ShardingStrategyFactory.newInstance(tableRuleConfig.getTableShardingStrategyConfig());
generateKeyColumn = getGenerateKeyColumn(tableRuleConfig.getKeyGeneratorConfig(), defaultGenerateKeyColumn);
shardingKeyGenerator = containsKeyGeneratorConfiguration(tableRuleConfig)
? new ShardingKeyGeneratorServiceLoader().newService(tableRuleConfig.getKeyGeneratorConfig().getType(), tableRuleConfig.getKeyGeneratorConfig().getProperties()) : null;
checkRule(dataNodes);
}
private List<DataNode> generateDataNodes(final String logicTable, final Collection<String> dataSourceNames) {
List<DataNode> result = new LinkedList<>();
int index = 0;
for (String each : dataSourceNames) {
DataNode dataNode = new DataNode(each, logicTable);
result.add(dataNode);
dataNodeIndexMap.put(dataNode, index);
actualDatasourceNames.add(each);
addActualTable(dataNode.getDataSourceName(), dataNode.getTableName());
index++;
}
return result;
}
}
public final class DataNode {
private static final String DELIMITER = ".";
private final String dataSourceName;
private final String tableName;
}
- TableRule的是依据TableRuleConfiguration来进行创建。
- TableRule的logicTable直接获取TableRuleConfiguration的逻辑表名。
- TableRule的actualDataNodes会解析TableRuleConfiguration的ActualDataNodes生成最细粒度的数据节点。
BindingTableRule创建流程
public class ShardingRule implements BaseRule {
private Collection<BindingTableRule> createBindingTableRules(final Collection<String> bindingTableGroups) {
Collection<BindingTableRule> result = new ArrayList<>(bindingTableGroups.size());
for (String each : bindingTableGroups) {
result.add(createBindingTableRule(each));
}
return result;
}
private BindingTableRule createBindingTableRule(final String bindingTableGroup) {
List<TableRule> tableRules = new LinkedList<>();
for (String each : Splitter.on(",").trimResults().splitToList(bindingTableGroup)) {
tableRules.add(getTableRule(each));
}
return new BindingTableRule(tableRules);
}
}
@RequiredArgsConstructor
@Getter
public final class BindingTableRule {
private final List<TableRule> tableRules;
}
- BindingTableRule的创建会使用逗号分割bindingTableGroups的字符串。
MasterSlaveRule创建流程
public class ShardingRule implements BaseRule {
private Collection<MasterSlaveRule> createMasterSlaveRules(final Collection<MasterSlaveRuleConfiguration> masterSlaveRuleConfigurations) {
Collection<MasterSlaveRule> result = new ArrayList<>(masterSlaveRuleConfigurations.size());
for (MasterSlaveRuleConfiguration each : masterSlaveRuleConfigurations) {
result.add(new MasterSlaveRule(each));
}
return result;
}
}
@Getter
public class MasterSlaveRule implements BaseRule {
private final String name;
private final String masterDataSourceName;
private final List<String> slaveDataSourceNames;
private final MasterSlaveLoadBalanceAlgorithm loadBalanceAlgorithm;
private final MasterSlaveRuleConfiguration ruleConfiguration;
public MasterSlaveRule(final MasterSlaveRuleConfiguration config) {
name = config.getName();
masterDataSourceName = config.getMasterDataSourceName();
slaveDataSourceNames = config.getSlaveDataSourceNames();
loadBalanceAlgorithm = createMasterSlaveLoadBalanceAlgorithm(config.getLoadBalanceStrategyConfiguration());
ruleConfiguration = config;
}
}
- MasterSlaveRule的参数全部来自MasterSlaveRuleConfiguration的解析。
- MasterSlaveRule的masterDataSourceName为主节点的配置。
- MasterSlaveRule的slaveDataSourceNames为从节点的配置。
分表的配置解析
ShardingRuleConfiguration
public final class ShardingTablesConfigurationPrecise implements ExampleConfiguration {
@Override
public DataSource getDataSource() throws SQLException {
// 1、创建ShardingRuleConfiguration对象
ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
// 2、绑定TableRuleConfiguration的对象
shardingRuleConfig.getTableRuleConfigs().add(getOrderTableRuleConfiguration());
// 3、绑定TableRuleConfiguration的对象
shardingRuleConfig.getTableRuleConfigs().add(getOrderItemTableRuleConfiguration());
// 4、绑定BindingTable
shardingRuleConfig.getBindingTableGroups().add("t_order, t_order_item");
// 5、绑定BroadcastTables
shardingRuleConfig.getBroadcastTables().add("t_address");
// 6、绑定默认分片策略
shardingRuleConfig.setDefaultTableShardingStrategyConfig(new StandardShardingStrategyConfiguration("order_id", new PreciseModuloShardingTableAlgorithm()));
return ShardingDataSourceFactory.createDataSource(createDataSourceMap(), shardingRuleConfig, new Properties());
}
private static TableRuleConfiguration getOrderTableRuleConfiguration() {
TableRuleConfiguration result = new TableRuleConfiguration("t_order", "demo_ds.t_order_${[0, 1]}");
result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_id", getProperties()));
return result;
}
private static TableRuleConfiguration getOrderItemTableRuleConfiguration() {
TableRuleConfiguration result = new TableRuleConfiguration("t_order_item", "demo_ds.t_order_item_${[0, 1]}");
result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_item_id", getProperties()));
return result;
}
private static Map<String, DataSource> createDataSourceMap() {
Map<String, DataSource> result = new HashMap<>();
result.put("demo_ds", DataSourceUtil.createDataSource("demo_ds"));
return result;
}
private static Properties getProperties() {
Properties result = new Properties();
result.setProperty("worker.id", "123");
return result;
}
}
- java硬编码的分表配置的逻辑代码。
ShardingRuleConfiguration
- 分表的配置展示。
{
"bindingTableGroups": ["t_order, t_order_item"],
"broadcastTables": ["t_address"],
"defaultTableShardingStrategyConfig": {
"preciseShardingAlgorithm": {},
"shardingColumn": "order_id"
},
"masterSlaveRuleConfigs": [],
"tableRuleConfigs": [{
"actualDataNodes": "demo_ds.t_order_${[0, 1]}",
"keyGeneratorConfig": {
"column": "order_id",
"properties": {
"worker.id": "123"
},
"type": "SNOWFLAKE"
},
"logicTable": "t_order"
}, {
"actualDataNodes": "demo_ds.t_order_item_${[0, 1]}",
"keyGeneratorConfig": {
"column": "order_item_id",
"properties": {
"worker.id": "123"
},
"type": "SNOWFLAKE"
},
"logicTable": "t_order_item"
}]
}
- 分表的配置的JSON呈现。
ShardingRule
ShardingRule
- 分表配置解析生成的ShardingRule，核心在于actualDataNodes的生成。
- demo_ds.t_order_${[0, 1]}最终会变成demo_ds.t_order_0 和 demo_ds.t_order_1。
分库分表配置解析
ShardingRuleConfiguration
public final class ShardingDatabasesAndTablesConfigurationPrecise implements ExampleConfiguration {
@Override
public DataSource getDataSource() throws SQLException {
ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
shardingRuleConfig.getTableRuleConfigs().add(getOrderTableRuleConfiguration());
shardingRuleConfig.getTableRuleConfigs().add(getOrderItemTableRuleConfiguration());
shardingRuleConfig.getBindingTableGroups().add("t_order, t_order_item");
shardingRuleConfig.getBroadcastTables().add("t_address");
shardingRuleConfig.setDefaultDatabaseShardingStrategyConfig(new InlineShardingStrategyConfiguration("user_id", "demo_ds_${user_id % 2}"));
shardingRuleConfig.setDefaultTableShardingStrategyConfig(new StandardShardingStrategyConfiguration("order_id", new PreciseModuloShardingTableAlgorithm()));
return ShardingDataSourceFactory.createDataSource(createDataSourceMap(), shardingRuleConfig, new Properties());
}
private static TableRuleConfiguration getOrderTableRuleConfiguration() {
TableRuleConfiguration result = new TableRuleConfiguration("t_order", "demo_ds_${0..1}.t_order_${[0, 1]}");
result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_id", getProperties()));
return result;
}
private static TableRuleConfiguration getOrderItemTableRuleConfiguration() {
TableRuleConfiguration result = new TableRuleConfiguration("t_order_item", "demo_ds_${0..1}.t_order_item_${[0, 1]}");
result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_item_id", getProperties()));
return result;
}
private static Map<String, DataSource> createDataSourceMap() {
Map<String, DataSource> result = new HashMap<>();
result.put("demo_ds_0", DataSourceUtil.createDataSource("demo_ds_0"));
result.put("demo_ds_1", DataSourceUtil.createDataSource("demo_ds_1"));
return result;
}
private static Properties getProperties() {
Properties result = new Properties();
result.setProperty("worker.id", "123");
return result;
}
}
- java硬编码的分库分表的逻辑代码。
ShardingRuleConfiguration
- 分库分表的配置展示。
{
"bindingTableGroups": ["t_order, t_order_item"],
"broadcastTables": ["t_address"],
"defaultDatabaseShardingStrategyConfig": {
"algorithmExpression": "demo_ds_${user_id % 2}",
"shardingColumn": "user_id"
},
"defaultTableShardingStrategyConfig": {
"preciseShardingAlgorithm": {},
"shardingColumn": "order_id"
},
"masterSlaveRuleConfigs": [],
"tableRuleConfigs": [{
"actualDataNodes": "demo_ds_${0..1}.t_order_${[0, 1]}",
"keyGeneratorConfig": {
"column": "order_id",
"properties": {
"worker.id": "123"
},
"type": "SNOWFLAKE"
},
"logicTable": "t_order"
}, {
"actualDataNodes": "demo_ds_${0..1}.t_order_item_${[0, 1]}",
"keyGeneratorConfig": {
"column": "order_item_id",
"properties": {
"worker.id": "123"
},
"type": "SNOWFLAKE"
},
"logicTable": "t_order_item"
}]
}
- 分库分表的数据格式JSON呈现。
- 分库分表的demo_ds和t_order_item表明两个库和两个表合计2*2=4个实际数据节点。
ShardingRule
ShardingRule
- 分库分表生成的ShardingRule，核心在于actualDataNodes的生成。
- 核心的actualDataNodes已经按照库表维度进行了拆分。
主从配置解析
MasterSlaveConfiguration
public final class MasterSlaveConfiguration implements ExampleConfiguration {
@Override
public DataSource getDataSource() throws SQLException {
MasterSlaveRuleConfiguration masterSlaveRuleConfig = new MasterSlaveRuleConfiguration("demo_ds_master_slave", "demo_ds_master", Arrays.asList("demo_ds_slave_0", "demo_ds_slave_1"));
return MasterSlaveDataSourceFactory.createDataSource(createDataSourceMap(), masterSlaveRuleConfig, new Properties());
}
private Map<String, DataSource> createDataSourceMap() {
Map<String, DataSource> result = new HashMap<>();
result.put("demo_ds_master", DataSourceUtil.createDataSource("demo_ds_master"));
result.put("demo_ds_slave_0", DataSourceUtil.createDataSource("demo_ds_slave_0"));
result.put("demo_ds_slave_1", DataSourceUtil.createDataSource("demo_ds_slave_1"));
return result;
}
}
- java硬编码的主从配置的逻辑代码。
MasterSlaveConfiguration
- 主从的配置展示。
{
"masterDataSourceName": "demo_ds_master",
"name": "demo_ds_master_slave",
"slaveDataSourceNames": ["demo_ds_slave_0", "demo_ds_slave_1"]
}
- 主从配置的数据格式JSON呈现。
MasterSlaveRule
MasterSlaveRule
- 主从生成的ShardingRule。
分片的主从配置
ShardingRuleConfiguration
public final class ShardingMasterSlaveConfigurationPrecise implements ExampleConfiguration {
@Override
public DataSource getDataSource() throws SQLException {
ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
shardingRuleConfig.getTableRuleConfigs().add(getOrderTableRuleConfiguration());
shardingRuleConfig.getTableRuleConfigs().add(getOrderItemTableRuleConfiguration());
shardingRuleConfig.getBindingTableGroups().add("t_order, t_order_item");
shardingRuleConfig.getBroadcastTables().add("t_address");
shardingRuleConfig.setDefaultDatabaseShardingStrategyConfig(new StandardShardingStrategyConfiguration("user_id", new PreciseModuloShardingDatabaseAlgorithm()));
shardingRuleConfig.setDefaultTableShardingStrategyConfig(new StandardShardingStrategyConfiguration("order_id", new PreciseModuloShardingTableAlgorithm()));
shardingRuleConfig.setMasterSlaveRuleConfigs(getMasterSlaveRuleConfigurations());
return ShardingDataSourceFactory.createDataSource(createDataSourceMap(), shardingRuleConfig, new Properties());
}
private static TableRuleConfiguration getOrderTableRuleConfiguration() {
TableRuleConfiguration result = new TableRuleConfiguration("t_order", "ds_${0..1}.t_order_${[0, 1]}");
result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_id", getProperties()));
return result;
}
private static TableRuleConfiguration getOrderItemTableRuleConfiguration() {
TableRuleConfiguration result = new TableRuleConfiguration("t_order_item", "ds_${0..1}.t_order_item_${[0, 1]}");
result.setKeyGeneratorConfig(new KeyGeneratorConfiguration("SNOWFLAKE", "order_item_id", getProperties()));
return result;
}
private static List<MasterSlaveRuleConfiguration> getMasterSlaveRuleConfigurations() {
MasterSlaveRuleConfiguration masterSlaveRuleConfig1 = new MasterSlaveRuleConfiguration("ds_0", "demo_ds_master_0", Arrays.asList("demo_ds_master_0_slave_0", "demo_ds_master_0_slave_1"));
MasterSlaveRuleConfiguration masterSlaveRuleConfig2 = new MasterSlaveRuleConfiguration("ds_1", "demo_ds_master_1", Arrays.asList("demo_ds_master_1_slave_0", "demo_ds_master_1_slave_1"));
return Lists.newArrayList(masterSlaveRuleConfig1, masterSlaveRuleConfig2);
}
private static Map<String, DataSource> createDataSourceMap() {
final Map<String, DataSource> result = new HashMap<>();
result.put("demo_ds_master_0", DataSourceUtil.createDataSource("demo_ds_master_0"));
result.put("demo_ds_master_0_slave_0", DataSourceUtil.createDataSource("demo_ds_master_0_slave_0"));
result.put("demo_ds_master_0_slave_1", DataSourceUtil.createDataSource("demo_ds_master_0_slave_1"));
result.put("demo_ds_master_1", DataSourceUtil.createDataSource("demo_ds_master_1"));
result.put("demo_ds_master_1_slave_0", DataSourceUtil.createDataSource("demo_ds_master_1_slave_0"));
result.put("demo_ds_master_1_slave_1", DataSourceUtil.createDataSource("demo_ds_master_1_slave_1"));
return result;
}
private static Properties getProperties() {
Properties result = new Properties();
result.setProperty("worker.id", "123");
return result;
}
}
- java硬编码的分片主从配置的逻辑代码。
ShardingRuleConfiguration
- 分片主从的配置展示。
{
"bindingTableGroups": ["t_order, t_order_item"],
"broadcastTables": ["t_address"],
"defaultDatabaseShardingStrategyConfig": {
"preciseShardingAlgorithm": {},
"shardingColumn": "user_id"
},
"defaultTableShardingStrategyConfig": {
"preciseShardingAlgorithm": {},
"shardingColumn": "order_id"
},
"masterSlaveRuleConfigs": [{
"masterDataSourceName": "demo_ds_master_0",
"name": "ds_0",
"slaveDataSourceNames": ["demo_ds_master_0_slave_0", "demo_ds_master_0_slave_1"]
}, {
"masterDataSourceName": "demo_ds_master_1",
"name": "ds_1",
"slaveDataSourceNames": ["demo_ds_master_1_slave_0", "demo_ds_master_1_slave_1"]
}],
"tableRuleConfigs": [{
"actualDataNodes": "ds_${0..1}.t_order_${[0, 1]}",
"keyGeneratorConfig": {
"column": "order_id",
"properties": {
"worker.id": "123"
},
"type": "SNOWFLAKE"
},
"logicTable": "t_order"
}, {
"actualDataNodes": "ds_${0..1}.t_order_item_${[0, 1]}",
"keyGeneratorConfig": {
"column": "order_item_id",
"properties": {
"worker.id": "123"
},
"type": "SNOWFLAKE"
},
"logicTable": "t_order_item"
}]
}
- 分片主从配置的数据格式JSON呈现。
ShardingRule
ShardingRule
- 分片主从生成的ShardingRule。
