vivo 互联网技术

缓存监控治理在游戏业务的实践和探索

2025-03-19  本文已影响0人  vivo互联网技术

作者：来自 vivo 互联网服务器团队- Wang Zhi

通过对 Redis 和 Caffeine 的缓存监控快速发现和定位问题降低故障的影响面。

一、缓存监控的背景

二、远程缓存的监控介绍

2.1 监控的方案

2.1.1 监控目的

2.1.2 监控方案

2.1.3 监控大盘

【Redis Server系统监控指标】

说明：

http://doc.redisfans.com/server/info.html。

【Redis 业务维度前缀监控指标】

说明：

2.2 监控的实现

2.2.1 前缀 key 设计

Redis Key 的设计

public class RedisKeyConstants {

    public static final String    REDIS_GAMEGROUP_NEW_KEY              = "newgamegroup";
    public static final String    REDIS_GAMEGROUP_DETAIL_KEY          = "gamegroup:detail";
    public static final String    REDIS_KEY_IUNIT_STRATEGY_COUNT      = "activity:ihandler:strategy:count";
    public static final String    CONTENT_DISTRIBUTE_CURRENT          = "content:distribute:current";
    public static final String    RECOMMEND_NOTE                      = "recommend:note";
}

public class RedisUtils {

    public static final String    COMMON_REDIS_KEY_SPLIT    = ":";

    public static String buildRedisKey(String key, Object... params) {
        if (params == null || params.length == 0) {
            return key;
        }

        for (Object param : params) {
            key += COMMON_REDIS_KEY_SPLIT + param;
        }

        return key;
    }
}

说明：

2.2.2 监控实现

@Slf4j
@Aspect
@Order(0)
@Component
public class RedisMonitorAspect {
    private static final String PREFIX_CONFIG = "redis.monitor.prefix";
    private static final Set<String> PREFIX_SET = new HashSet<>();
    @Resource
    private MonitorComponent monitorComponent;
    static {
        // 更新前缀匹配的名单
        String prefixValue = VivoConfigManager.getString(PREFIX_CONFIG, "");
        refreshConf(prefixValue);
        // 增加配置变更的回调
        VivoConfigManager.addListener(new VivoConfigListener() {
            @Override
            public void eventReceived(PropertyItem propertyItem, ChangeEventType changeEventType) {
                if (StringUtils.equalsIgnoreCase(propertyItem.getName(), PREFIX_CONFIG)) {
                    refreshConf(propertyItem.getValue());
                }
            }
        });
    }
    /**
     * 更新前缀匹配的名单
     * @param prefixValue
     */
    private static void refreshConf(String prefixValue) {
        if (StringUtils.isNotEmpty(prefixValue)) {
            String[] prefixArr = StringUtils.split(prefixValue, ",");
            Arrays.stream(prefixArr).forEach(item -> PREFIX_SET.add(item));
        }
    }
    @Pointcut("execution(* com.vivo.joint.dal.common.redis.dao.RedisDao.set*(..))")
    public void point() {
    }
    @Around("point()")
    public Object around(ProceedingJoinPoint pjp) throws Throwable {
        //业务逻辑异常情况直接抛到业务层处理
        Object result = pjp.proceed();
        try {
            if (VivoConfigManager.getBoolean("joint.center.redis.monitor.switch", true)) {
                Object[] args = pjp.getArgs();
                if (null != args && args.length > 0) {
                    String redisKey = String.valueOf(args[0]);
                    if (VivoConfigManager.getBoolean("joint.center.redis.monitor.send.log.switch", true)) {
                        LOGGER.info("更新redis的缓存 {}", redisKey);
                    }
                    String monitorKey = null;
                    // 先指定前缀匹配
                    if (!PREFIX_SET.isEmpty()) {
                        for (String prefix : PREFIX_SET) {
                            if (StringUtils.startsWithIgnoreCase(redisKey, prefix)) {
                                monitorKey = prefix;
                                break;
                            }
                        }
                    }
                    if (StringUtils.isEmpty(monitorKey) && StringUtils.contains(redisKey, ":")) {
                        // 需要考虑前缀的格式，保证数据写入不能膨胀
                        monitorKey = StringUtils.substringBeforeLast(redisKey, ":");
                    }
                    monitorComponent.sendRedisMonitorData(monitorKey);
                }
            }
        } catch (Exception e) {
        }
        return result;
    }
}

printf("hello world!");

说明：

2.3 监控的案例

public static final String REDISKEY_USER_POPUP_PLAN = "popup:user:plan";

    public PopupWindowPlan findPlan(FindPlanParam param) {
        String openId = param.getOpenId();
        String imei = param.getImei();
        String gamePackage = param.getGamePackage();
        Integer planType = param.getPlanType();
        String appId = param.getAppId();

        // 1、获取缓存的数据
        PopupWindowPlan cachedPlan = getPlanFromCache(openId, imei, gamePackage, planType);
        if (cachedPlan != null) {
            monitorPopWinPlan(cachedPlan);

            return cachedPlan;
        }

        // 2、未命中换成后从持久化部分获取对应的 PopupWindowPlan 对象

        // 3、保存到Redis换成
      setPlanToCache(openId, imei, gamePackage, plan);

        return cachedPlan;
    }

    // 从缓存中获取数据的逻辑
    private PopupWindowPlan getPlanFromCache(String openId, String imei, String gamePackage, Integer planType) {

        String key = RedisUtils.buildRedisKey(RedisKeyConstants.REDISKEY_USER_POPUP_PLAN, openId, imei, gamePackage, planType);
        String cacheValue = redisDao.get(key);
        if (StringUtils.isEmpty(cacheValue)) {
            return null;
        }

        try {
            PopupWindowPlan plan = objectMapper.readValue(cacheValue, PopupWindowPlan.class);
            return plan;
        } catch (Exception e) {
        }

        return null;
    }

    // 保存数据到缓存当中
    private void setPlanToCache(String openId, String imei, String gamePackage, PopupWindowPlan plan, Integer planType) {

        String key = RedisUtils.buildRedisKey(RedisKeyConstants.REDISKEY_USER_POPUP_PLAN, openId, imei, gamePackage, planType);
        try {
            String serializedStr = objectMapper.writeValueAsString(plan);
            redisDao.set(key, serializedStr, VivoConfigManager.getInteger(ConfigConstants.POPUP_PLAN_CACHE_EXPIRE_TIME, 300));
        } catch (Exception e) {
        }
    }

说明：

三、本地缓存的监控介绍

3.1 监控的方案

3.1.1 监控目的

3.1.2 监控方案

3.1.3 监控大盘

【Caffeine 系统监控指标】

说明：

3.2 监控的实现

public final class Caffeine<K, V> {

  /**
   * caffeine的实例名称
   */
  String instanceName;

  /**
   * caffeine的实例维护的Map信息
   */
  static Map<String, Cache> cacheInstanceMap = new ConcurrentHashMap<>();

  @NonNull
  public <K1 extends K, V1 extends V> Cache<K1, V1> build() {
    requireWeightWithWeigher();
    requireNonLoadingCache();

    @SuppressWarnings("unchecked")
    Caffeine<K1, V1> self = (Caffeine<K1, V1>) this;
    Cache localCache =  isBounded() ? new BoundedLocalCache.BoundedLocalManualCache<>(self) : new UnboundedLocalCache.UnboundedLocalManualCache<>(self);

    if (null != localCache && StringUtils.isNotEmpty(localCache.getInstanceName())) {
      cacheInstanceMap.put(localCache.getInstanceName(), localCache);
    }

    return localCache;
  }
}

static Cache<String, List<String>> accountWhiteCache = Caffeine.newBuilder().applyName("accountWhiteCache")
            .expireAfterWrite(VivoConfigManager.getInteger("trade.account.white.list.cache.ttl", 10), TimeUnit.MINUTES)
            .recordStats().maximumSize(VivoConfigManager.getInteger("trade.account.white.list.cache.size", 100)).build();

左右滑动查看完整代码

说明：

public static StatsData getCacheStats(String instanceName) {

    Cache cache = Caffeine.getCacheByInstanceName(instanceName);

    CacheStats cacheStats = cache.stats();
    StatsData statsData = new StatsData();

    statsData.setInstanceName(instanceName);
    statsData.setTimeStamp(System.currentTimeMillis()/1000);
    statsData.setMemoryUsed(String.valueOf(cache.getMemoryUsed()));
    statsData.setEstimatedSize(String.valueOf(cache.estimatedSize()));
    statsData.setRequestCount(String.valueOf(cacheStats.requestCount()));
    statsData.setHitCount(String.valueOf(cacheStats.hitCount()));
    statsData.setHitRate(String.valueOf(cacheStats.hitRate()));
    statsData.setMissCount(String.valueOf(cacheStats.missCount()));
    statsData.setMissRate(String.valueOf(cacheStats.missRate()));
    statsData.setLoadCount(String.valueOf(cacheStats.loadCount()));
    statsData.setLoadSuccessCount(String.valueOf(cacheStats.loadSuccessCount()));
    statsData.setLoadFailureCount(String.valueOf(cacheStats.loadFailureCount()));
    statsData.setLoadFailureRate(String.valueOf(cacheStats.loadFailureRate()));

    Optional<Eviction> optionalEviction = cache.policy().eviction();
    optionalEviction.ifPresent(eviction -> statsData.setMaximumSize(String.valueOf(eviction.getMaximum())));

    Optional<Expiration> optionalExpiration = cache.policy().expireAfterWrite();
    optionalExpiration.ifPresent(expiration -> statsData.setExpireAfterWrite(String.valueOf(expiration.getExpiresAfter(TimeUnit.SECONDS))));

    optionalExpiration = cache.policy().expireAfterAccess();
    optionalExpiration.ifPresent(expiration -> statsData.setExpireAfterAccess(String.valueOf(expiration.getExpiresAfter(TimeUnit.SECONDS))));

    optionalExpiration = cache.policy().refreshAfterWrite();
    optionalExpiration.ifPresent(expiration -> statsData.setRefreshAfterWrite(String.valueOf(expiration.getExpiresAfter(TimeUnit.SECONDS))));

    return statsData;
}

左右滑动查看完整代码

说明：

public static void sendReportData() {

    try {
        if (!VivoConfigManager.getBoolean("memory.caffeine.data.report.switch", true)) {
            return;
        }

        // 1、获取所有的cache实例对象
        Method listCacheInstanceMethod = HANDLER_MANAGER_CLASS.getMethod("listCacheInstance", null);
        List<String> instanceNames = (List)listCacheInstanceMethod.invoke(null, null);
        if (CollectionUtils.isEmpty(instanceNames)) {
            return;
        }

        String appName = System.getProperty("app.name");
        String localIp = getLocalIp();
        String localPort = String.valueOf(NetPortUtils.getWorkPort());
        ReportData reportData = new ReportData();
        InstanceData instanceData = new InstanceData();
        instanceData.setAppName(appName);
        instanceData.setIp(localIp);
        instanceData.setPort(localPort);

        // 2、遍历cache实例对象获取缓存监控数据
        Method getCacheStatsMethod = HANDLER_MANAGER_CLASS.getMethod("getCacheStats", String.class);
        Map<String, StatsData> statsDataMap = new HashMap<>();
        instanceNames.stream().forEach(instanceName -> {

            try {
                StatsData statsData = (StatsData)getCacheStatsMethod.invoke(null, instanceName);

                statsDataMap.put(instanceName, statsData);
            } catch (Exception e) {

            }
        });

        // 3、构建上报对象
        reportData.setInstanceData(instanceData);
        reportData.setStatsDataMap(statsDataMap);

        // 4、发送Http的POST请求
        HttpPost httpPost = new HttpPost(getReportDataUrl());
        httpPost.setConfig(requestConfig);

        StringEntity stringEntity = new StringEntity(JSON.toJSONString(reportData));
        stringEntity.setContentType("application/json");
        httpPost.setEntity(stringEntity);

        HttpResponse response = httpClient.execute(httpPost);
        String result = EntityUtils.toString(response.getEntity(),"UTF-8");
        EntityUtils.consume(response.getEntity());

        logger.info("Caffeine 数据上报成功 URL {} 参数 {} 结果 {}", getReportDataUrl(), JSON.toJSONString(reportData), result);
    } catch (Throwable throwable) {
        logger.error("Caffeine 数据上报失败 URL {} ", getReportDataUrl(), throwable);
    }
}

左右滑动查看完整代码

说明：

3.3 监控的案例

说明：

四、结束语

上一篇 下一篇

猜你喜欢

热点阅读