Dubbo Admin 实现原理

2020-01-02 本文已影响0人晴天哥_王志

开篇

这篇文章主要用于讲解清楚Dubbo Admin的监控数据的来源，如何通过zookeeper作为注册中心来获取实际运行中的数据进行服务治理。
这篇文章主要侧重于数据方面的获取包括consumers, configurators, routers, providers，具体如何进行服务治理后面文章会具体进行分析。
文章是基于dubbo-2.6.0的版本进行分析。

注册节点

[zk] ls /dubbo/com.alibaba.dubbo.demo.DemoService
[consumers, configurators, routers, providers]

zookeeper服务树

Dubbo的服务树如上图所示，以服务如com.foo.BarService作为第一层节点Service层。
Service层包含Type层，包括consumers, configurators, routers, providers，监控数据主要是指Type层子节点的数据监控。

实现过程概述

1.依次按照/dubbo，/dubbo/service，/dubbo/service/type的顺序进行发现并最终订阅/dubbo/service/type的节点。
2.首次获取所有/dubbo/service/type的所有子节点并保存导数据结构registryCache当中。
3.后续/dubbo/service/type的节点变更通过zookeeper事件通知机制更新到registryCache当中。
4.registryCache保存了zookeeper上Dubbo服务节点上的所有信息，按照
ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>的数据结构进行保存，其中category包含providers,consumers,routers,configurators。

实现过程源码分析

dubbo-admin目录

dubbo-admin的目录结构如上图，RegistryServerSync作为同步数据的核心类。
RegistryServerSync的数据用于同步并监听zookeeper中关于Dubbo服务的数据。

RegistryServerSync的bean定义

<dubbo:application name="dubbo-admin"/>

<dubbo:registry address="${dubbo.registry.address}" check="false" file="false"/>

<dubbo:reference id="registryService" interface="com.alibaba.dubbo.registry.RegistryService" check="false"/>

<bean id="configService" class="com.alibaba.dubbo.governance.service.impl.ConfigServiceImpl"/>
<bean id="consumerService" class="com.alibaba.dubbo.governance.service.impl.ConsumerServiceImpl"/>
<bean id="overrideService" class="com.alibaba.dubbo.governance.service.impl.OverrideServiceImpl"/>
<bean id="ownerService" class="com.alibaba.dubbo.governance.service.impl.OwnerServiceImpl"/>
<bean id="providerService" class="com.alibaba.dubbo.governance.service.impl.ProviderServiceImpl"/>
<bean id="routeService" class="com.alibaba.dubbo.governance.service.impl.RouteServiceImpl"/>
<bean id="userService" class="com.alibaba.dubbo.governance.service.impl.UserServiceImpl">
    <property name="rootPassword" value="${dubbo.admin.root.password}"/>
    <property name="guestPassword" value="${dubbo.admin.guest.password}"/>
</bean>

<bean id="governanceCache" class="com.alibaba.dubbo.governance.sync.RegistryServerSync"/>

RegistryServerSync的定义在META-INF/spring/dubbo-admin.xml当中。

RegistryServerSync初始化订阅过程

public class RegistryServerSync implements InitializingBean, DisposableBean, NotifyListener {

    private static final Logger logger = LoggerFactory.getLogger(RegistryServerSync.class);

    // admin://192.168.1.5?category=providers,consumers,routers,configurators&check=false&classifier=*&enabled=*&group=*&interface=*&version=*
    private static final URL SUBSCRIBE = new URL(Constants.ADMIN_PROTOCOL, NetUtils.getLocalHost(), 0, "",
            Constants.INTERFACE_KEY, Constants.ANY_VALUE,
            Constants.GROUP_KEY, Constants.ANY_VALUE,
            Constants.VERSION_KEY, Constants.ANY_VALUE,
            Constants.CLASSIFIER_KEY, Constants.ANY_VALUE,
            Constants.CATEGORY_KEY, Constants.PROVIDERS_CATEGORY + ","
            + Constants.CONSUMERS_CATEGORY + ","
            + Constants.ROUTERS_CATEGORY + ","
            + Constants.CONFIGURATORS_CATEGORY,
            Constants.ENABLED_KEY, Constants.ANY_VALUE,
            Constants.CHECK_KEY, String.valueOf(false));

    private static final AtomicLong ID = new AtomicLong();

    /**
     * Make sure ID never changed when the same url notified many times
     */
    private final ConcurrentHashMap<String, Long> URL_IDS_MAPPER = new ConcurrentHashMap<String, Long>();

    // ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>
    private final ConcurrentMap<String, ConcurrentMap<String, Map<Long, URL>>> registryCache = new ConcurrentHashMap<String, ConcurrentMap<String, Map<Long, URL>>>();
    @Autowired
    private RegistryService registryService;

    public ConcurrentMap<String, ConcurrentMap<String, Map<Long, URL>>> getRegistryCache() {
        return registryCache;
    }

    public void afterPropertiesSet() throws Exception {
        logger.info("Init Dubbo Admin Sync Cache...");
        registryService.subscribe(SUBSCRIBE, this);
    }
}

category参数

RegistryServerSync在afterPropertiesSet()方法内部执行订阅操作。
SUBSCRIBE变量中category=providers,consumers,routers,configurators表示我们需要订阅service下的这4类节点，其中routers和configurators就是跟服务治理相关。
SUBSCRIBE的check=false&classifier=&enabled=&group=&interface=&version=*等变量都是通用符*，这里我们关注下interface=*的变量，后面执行实际订阅会根据该变量做判断。

RegistryServerSync执行订阅过程

参见ZookeeperRegistry#doSubscribe

    protected void doSubscribe(final URL url, final NotifyListener listener) {
        try {
            // admin订阅过程中，首次传入参数interface的值为*，所以走的这个分支
            if (Constants.ANY_VALUE.equals(url.getServiceInterface())) {
                String root = toRootPath();
                ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url);
                if (listeners == null) {
                    zkListeners.putIfAbsent(url, new ConcurrentHashMap<NotifyListener, ChildListener>());
                    listeners = zkListeners.get(url);
                }
                ChildListener zkListener = listeners.get(listener);
                if (zkListener == null) {
                    listeners.putIfAbsent(listener, new ChildListener() {
                        public void childChanged(String parentPath, List<String> currentChilds) {
                            for (String child : currentChilds) {
                                child = URL.decode(child);
                                if (!anyServices.contains(child)) {
                                    anyServices.add(child);
                                    subscribe(url.setPath(child).addParameters(Constants.INTERFACE_KEY, child,
                                            Constants.CHECK_KEY, String.valueOf(false)), listener);
                                }
                            }
                        }
                    });
                    zkListener = listeners.get(listener);
                }
                zkClient.create(root, false);
                // 获取 /dubbo目录的子节点，返回service层所有service接口
                // 如/dubbo/com.alibaba.dubbo.demo.DemoServiceEcho
                List<String> services = zkClient.addChildListener(root, zkListener);
                if (services != null && services.size() > 0) {
                    for (String service : services) {
                        service = URL.decode(service);
                        anyServices.add(service);
                        // 针对每个service进行subscribe操作
                        subscribe(url.setPath(service).addParameters(Constants.INTERFACE_KEY, service,
                                Constants.CHECK_KEY, String.valueOf(false)), listener);
                    }
                }
            } else {
                List<URL> urls = new ArrayList<URL>();
                // 获取service下的category，consumers, configurators, routers, providers
                // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/providers
                // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/consumers
                // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/routers
                // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/configurators
                for (String path : toCategoriesPath(url)) {
                    ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url);
                    if (listeners == null) {
                        zkListeners.putIfAbsent(url, new ConcurrentHashMap<NotifyListener, ChildListener>());
                        listeners = zkListeners.get(url);
                    }
                    ChildListener zkListener = listeners.get(listener);
                    if (zkListener == null) {
                        listeners.putIfAbsent(listener, new ChildListener() {
                            public void childChanged(String parentPath, List<String> currentChilds) {
                                ZookeeperRegistry.this.notify(url, listener, toUrlsWithEmpty(url, parentPath, currentChilds));
                            }
                        });
                        zkListener = listeners.get(listener);
                    }
                    zkClient.create(path, false);
                    // 针对每个service的监听以下节点
                    // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/providers
                    // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/consumers
                    // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/routers
                    // /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/configurators
                    List<String> children = zkClient.addChildListener(path, zkListener);
                    if (children != null) {
                        urls.addAll(toUrlsWithEmpty(url, path, children));
                    }
                }
                // urls为某个服务下所有的子节点内容
                // dubbo://172.17.32.8:20880/com.alibaba.dubbo.demo.DemoServiceEcho?anyhost=true&application=demo-provider&bean.name=com.alibaba.dubbo.demo.DemoServiceEcho&dubbo=2.0.2&generic=false&interface=com.alibaba.dubbo.demo.DemoServiceEcho&methods=sayHello&pid=79153&side=provider&timestamp=1577783208090
                // empty://192.168.1.5/com.alibaba.dubbo.demo.DemoServiceEcho?category=consumers&check=false&classifier=*&enabled=*&group=*&interface=com.alibaba.dubbo.demo.DemoServiceEcho&version=*
                // route://0.0.0.0/com.alibaba.dubbo.demo.DemoServiceEcho?category=routers&dynamic=false&enabled=true&force=true&name=com.alibaba.dubbo.demo.DemoServiceEcho blackwhitelist&priority=0&router=condition&rule=consumer.host+%3D+1.1.1.1+%3D%3E+false&runtime=false
                // override://192.168.1.5/com.alibaba.dubbo.demo.DemoServiceEcho?category=configurators&dynamic=false&enabled=true&weight=12
                // override://192.168.1.5/com.alibaba.dubbo.demo.DemoServiceEcho?category=configurators&dynamic=false&enabled=true&weight=12
                // override://1.1.1.1/com.alibaba.dubbo.demo.DemoServiceEcho?category=configurators&dynamic=false&enabled=true&weight=13
                notify(url, listener, urls);
            }
        } catch (Throwable e) {
            throw new RpcException("Failed to subscribe " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
        }
    }

订阅的过程是一个逐层发现并订阅的过程。
获取根节点/dubbo下的所有service节点集合，如/dubbo/com.alibaba.dubbo.demo.DemoServiceEcho。
针对每个service节点获取该节点的所有子节点(描述方便用type表示)，包括providers,consumers,routers,configurators。
针对每个type如/dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/providers进行监听。
首次启动会获取每个type节点下的所有子节点进行第一轮初始化过程，后续的变更都是通过type节点本身的监听回调进行实现。
首次启动会把每个service下所有的type下的所有的子节点合并成urls后进行notify动作。

RegistryServerSync回调分组过程

参见AbstractRegistry#notify

public abstract class AbstractRegistry implements Registry {

    protected void notify(URL url, NotifyListener listener, List<URL> urls) {
        if (url == null) {
            throw new IllegalArgumentException("notify url == null");
        }
        if (listener == null) {
            throw new IllegalArgumentException("notify listener == null");
        }
        if ((urls == null || urls.size() == 0)
                && !Constants.ANY_VALUE.equals(url.getServiceInterface())) {
            logger.warn("Ignore empty notify urls for subscribe url " + url);
            return;
        }
        if (logger.isInfoEnabled()) {
            logger.info("Notify urls for subscribe url " + url + ", urls: " + urls);
        }
        // 根据category进行分组，包括consumers, configurators, routers, providers
        Map<String, List<URL>> result = new HashMap<String, List<URL>>();
        for (URL u : urls) {
            if (UrlUtils.isMatch(url, u)) {
                String category = u.getParameter(Constants.CATEGORY_KEY, Constants.DEFAULT_CATEGORY);
                List<URL> categoryList = result.get(category);
                if (categoryList == null) {
                    categoryList = new ArrayList<URL>();
                    result.put(category, categoryList);
                }
                categoryList.add(u);
            }
        }
        if (result.size() == 0) {
            return;
        }
        Map<String, List<URL>> categoryNotified = notified.get(url);
        if (categoryNotified == null) {
            notified.putIfAbsent(url, new ConcurrentHashMap<String, List<URL>>());
            categoryNotified = notified.get(url);
        }
        for (Map.Entry<String, List<URL>> entry : result.entrySet()) {
            String category = entry.getKey();
            List<URL> categoryList = entry.getValue();
            categoryNotified.put(category, categoryList);
            saveProperties(url);
            // 按照分组的结果进行回调通知
            listener.notify(categoryList);
        }
    }
}

RegistryServerSync的回调过程中根据category进行分组，分组包括consumers, configurators, routers, providers。
依次针对分组后的结果进行回调通知，执行RegistryServerSync的notify动作。

RegistryServerSync保存回调结果

参见RegistryServerSync#notify

public class RegistryServerSync implements InitializingBean, DisposableBean, NotifyListener {
    /**
     * Make sure ID never changed when the same url notified many times
     */
    private final ConcurrentHashMap<String, Long> URL_IDS_MAPPER = new ConcurrentHashMap<String, Long>();

    // ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>
    // servicename=groupName/serviceName:versionNum
    private final ConcurrentMap<String, ConcurrentMap<String, Map<Long, URL>>> registryCache = new ConcurrentHashMap<String, ConcurrentMap<String, Map<Long, URL>>>();


    public void notify(List<URL> urls) {
        if (urls == null || urls.isEmpty()) {
            return;
        }
        // Map<category, Map<servicename, Map<Long, URL>>>
        final Map<String, Map<String, Map<Long, URL>>> categories = new HashMap<String, Map<String, Map<Long, URL>>>();
        String interfaceName = null;
        for (URL url : urls) {
            String category = url.getParameter(Constants.CATEGORY_KEY, Constants.PROVIDERS_CATEGORY);
            // 针对empty的情况，移出已经消失的服务
            if (Constants.EMPTY_PROTOCOL.equalsIgnoreCase(url.getProtocol())) { // NOTE: group and version in empty protocol is *
                ConcurrentMap<String, Map<Long, URL>> services = registryCache.get(category);
                if (services != null) {
                    String group = url.getParameter(Constants.GROUP_KEY);
                    String version = url.getParameter(Constants.VERSION_KEY);
                    // NOTE: group and version in empty protocol is *
                    if (!Constants.ANY_VALUE.equals(group) && !Constants.ANY_VALUE.equals(version)) {
                        services.remove(url.getServiceKey());
                    } else {
                        for (Map.Entry<String, Map<Long, URL>> serviceEntry : services.entrySet()) {
                            String service = serviceEntry.getKey();
                            if (Tool.getInterface(service).equals(url.getServiceInterface())
                                    && (Constants.ANY_VALUE.equals(group) || StringUtils.isEquals(group, Tool.getGroup(service)))
                                    && (Constants.ANY_VALUE.equals(version) || StringUtils.isEquals(version, Tool.getVersion(service)))) {
                                services.remove(service);
                            }
                        }
                    }
                }
            } else {
                // 添加服务到全局的registryCache变量当中
                if (StringUtils.isEmpty(interfaceName)) {
                    interfaceName = url.getServiceInterface();
                }
                // 用于保存局部变量的categories
                Map<String, Map<Long, URL>> services = categories.get(category);
                if (services == null) {
                    services = new HashMap<String, Map<Long, URL>>();
                    categories.put(category, services);
                }
                // service=groupName/interfaceName:versionNum
                String service = url.getServiceKey();
                Map<Long, URL> ids = services.get(service);
                if (ids == null) {
                    ids = new HashMap<Long, URL>();
                    services.put(service, ids);
                }

                // Make sure we use the same ID for the same URL
                if (URL_IDS_MAPPER.containsKey(url.toFullString())) {
                    ids.put(URL_IDS_MAPPER.get(url.toFullString()), url);
                } else {
                    long currentId = ID.incrementAndGet();
                    ids.put(currentId, url);
                    URL_IDS_MAPPER.putIfAbsent(url.toFullString(), currentId);
                }
            }
        }
        if (categories.size() == 0) {
            return;
        }

        // 本次category对应的数据不为空进行添加动作
        for (Map.Entry<String, Map<String, Map<Long, URL>>> categoryEntry : categories.entrySet()) {
            String category = categoryEntry.getKey();
            ConcurrentMap<String, Map<Long, URL>> services = registryCache.get(category);
            if (services == null) {
                services = new ConcurrentHashMap<String, Map<Long, URL>>();
                registryCache.put(category, services);
            } else {
                Set<String> keys = new HashSet<String>(services.keySet());
                // 移出已经不存在的数据，同一个接口但是不在本次的回调数据当中
                for (String key : keys) {
                    if (Tool.getInterface(key).equals(interfaceName) && !categoryEntry.getValue().entrySet().contains(key)) {
                        services.remove(key);
                    }
                }
            }
            // 用最新的数据进行覆盖
            services.putAll(categoryEntry.getValue());
        }
    }
}

RegistryServerSync的notify()方法的urls参数是/dubbo/service/type下的子节点。
Dubbo的服务树的数据最终通过ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>的数据结构进行保存，保存的对象为registryCache，servicename=groupName/serviceName:versionNum的格式保存。
registryCache保存的Zookeeper上的Dubbo的服务节点内容。所有的服务治理操作依据的数据都在registryCache当中，任何服务节点的变更都会导致registryCache的数据更新。
registryCache是整个服务治理的数据核心。

Dubbo Admin 实现原理

开篇

注册节点

实现过程概述

实现过程源码分析

RegistryServerSync的bean定义

RegistryServerSync初始化订阅过程

RegistryServerSync执行订阅过程

RegistryServerSync回调分组过程

RegistryServerSync保存回调结果

猜你喜欢

热点阅读