【Ovirt 笔记】虚拟资源管理分析与整理

2017-05-26 本文已影响24人 58bc06151329

分析整理的版本为 Ovirt 3.4.5 版本。

主机

ResourceManager

engine 将 InitBackendServicesOnStartupBean 交由 EJB 容器进行管理，注解为 @Singleton 和 @Startup，在应用启动前，对 InitBackendServicesOnStartupBean 进行初始化，并且实例化 ResourceManager。

@Singleton
@Startup
@DependsOn({ "Backend"})
public class InitBackendServicesOnStartupBean implements InitBackendServicesOnStartup{
......

@Override
@PostConstruct
public void create() {
......
ResourceManager.getInstance().init();
......

ResourceManager 实例化，其中最重要的操作如下：

for (VDS curVds : allVdsList) {
    AddVds(curVds, true);
}
IrsBrokerCommand.init();

AddVds 根据数据库中的主机，初始化一个 vdsManager，而这个 vdsManager 用于管理所有的主机。vdsManager 主要建立了与 vdsm 的 xml-rpc 连接。

VdsManager vdsManager = VdsManager.buildVdsManager(vds);

private void InitVdsBroker() {
    log.infoFormat("Initialize vdsBroker ({0},{1})", _vds.getHostName(), _vds.getPort());

    // Get the values of the timeouts:
    int clientTimeOut = Config.<Integer> getValue(ConfigValues.vdsTimeout) * 1000;
    int connectionTimeOut = Config.<Integer>getValue(ConfigValues.vdsConnectionTimeout) * 1000;
    int clientRetries = Config.<Integer>getValue(ConfigValues.vdsRetries);

    Pair<VdsServerConnector, HttpClient> returnValue =
            XmlRpcUtils.getConnection(_vds.getHostName(),
                    _vds.getPort(),
                    clientTimeOut,
                    connectionTimeOut,
                    clientRetries,
                    VdsServerConnector.class,
                    Config.<Boolean> getValue(ConfigValues.EncryptHostCommunication));
    _vdsProxy = new VdsServerWrapper(returnValue.getFirst(), returnValue.getSecond());
}

IrsBrokerCommand.init() 初始化所有数据中心的 IrsProxyData，也就是 SPM。管理每个数据中心的资源，如存储域，加入到 _irsProxyData 的缓存中，IrsProxyData 的构造方法中启动了定时任务 _updatingTimer_Elapsed 用来管理对应数据中心的主存储域以及其它的存储域。

public static void init() {
    for (StoragePool sp : DbFacade.getInstance().getStoragePoolDao().getAll()) {
        if (!_irsProxyData.containsKey(sp.getId())) {
            _irsProxyData.put(sp.getId(), new IrsProxyData(sp.getId()));
        }
    }
}

public IrsProxyData(Guid storagePoolId) {
    _storagePoolId = storagePoolId;
    int storagePoolRefreshTime = Config.<Integer> getValue(ConfigValues.StoragePoolRefreshTimeInSeconds);
    storagePoolRefreshJobId = SchedulerUtilQuartzImpl.getInstance().scheduleAFixedDelayJob(this,
            "_updatingTimer_Elapsed", new Class[0], new Object[0], storagePoolRefreshTime,
            storagePoolRefreshTime, TimeUnit.SECONDS);
}

@OnTimerMethodAnnotation("_updatingTimer_Elapsed")
public void _updatingTimer_Elapsed() {
    try {
        synchronized (syncObj) {
            if (!_disposed) {
                StoragePool storagePool = DbFacade.getInstance().getStoragePoolDao()
                        .get(_storagePoolId);

                if (storagePool != null) {
                    // when there are no hosts in status up, it means that there shouldn't be domain monitoring
                    // so all the domains need to move to "unknown" status as otherwise their status won't change.
                    if (DbFacade.getInstance()
                            .getVdsDao()
                            .getAllForStoragePoolAndStatus(_storagePoolId, reportingVdsStatus)
                            .isEmpty()) {
                        StoragePoolDomainHelper.updateApplicablePoolDomainsStatuses(_storagePoolId,
                                StoragePoolDomainHelper.storageDomainMonitoredStatus,
                                StorageDomainStatus.Unknown, "no reporting hosts");
                    }

                    if (storagePool.getStatus() == StoragePoolStatus.Up
                            || storagePool.getStatus() == StoragePoolStatus.NonResponsive || storagePool
                                    .getStatus() == StoragePoolStatus.Contend) {
                        proceedStoragePoolStats(storagePool);
                    }
                }

            }
        }
    } catch (Exception ex) {
    }
}

负责了 _asyncRunningVms 缓存的管理。该缓存为实时状态刷新提供了数据源。

public boolean AddAsyncRunningVm(Guid vmId) {
    boolean returnValue = false;
    if (_asyncRunningVms.putIfAbsent(vmId, Boolean.TRUE) == null) {
        returnValue = true;
    }
    return returnValue;
}

public void RemoveAsyncRunningVm(Guid vmId) {
    _asyncRunningVms.remove(vmId);
    getEventListener().removeAsyncRunningCommand(vmId);
}

public void succededToRunVm(Guid vmId, Guid vdsId) {
    if (_asyncRunningVms.containsKey(vmId)) {
        getEventListener().runningSucceded(vmId);
    }
    RemoveAsyncRunningVm(vmId);
}
......

根据 gwt-rpc 传递的 command 简称，通过 Class.fromName（commmandName） 创建对应的 Command 的实例，执行对应的业务逻辑。

private static final String VDSCommandPrefix = "VDSCommand";
......
private static String GetCommandTypeName(VDSCommandType command) {
    String packageName = command.getPackageName();
    String commandName = String.format("%s.%s%s", packageName, command, VDSCommandPrefix);
    return commandName;
}
......
private <P extends VDSParametersBase> VDSCommandBase<P> CreateCommand(
        VDSCommandType commandType, P parameters) {
    try {
        @SuppressWarnings("unchecked")
        Class<VDSCommandBase<P>> type =
                (Class<VDSCommandBase<P>>) Class.forName(GetCommandTypeName(commandType));
        Constructor<VDSCommandBase<P>> constructor =
                ReflectionUtils.findConstructor(type, parameters.getClass());

        if (constructor != null) {
            return constructor.newInstance(new Object[] { parameters });
        }
    } catch (Exception e) {
        if (e.getCause() != null) {
            log.error("CreateCommand failed", e.getCause());
            throw new RuntimeException(e.getCause().getMessage(), e.getCause());
        }
        log.error("CreateCommand failed", e);
    }
    return null;
}
......

VDSBrokerFrontendImpl

Backend 中进行实例化。主要负责了 VDSCommad 的执行和异步 Command 的缓存。

_resourceManger = new VDSBrokerFrontendImpl();

public VDSReturnValue RunVdsCommand(VDSCommandType commandType, VDSParametersBase parameters) {
    return VdsHandler.handleVdsResult(getResourceManager().runVdsCommand(commandType, parameters));
}

public VDSReturnValue RunAsyncVdsCommand(VDSCommandType commandType, VdsAndVmIDVDSParametersBase parameters,
                                         IVdsAsyncCommand command) {
    VDSReturnValue result = RunVdsCommand(commandType, parameters);
    if (result.getSucceeded()) {
        // Add async command to cached commands
        IVdsAsyncCommand prevCommand = _asyncRunningCommands.put(parameters.getVmId(), command);
        if (prevCommand != null && !prevCommand.equals(command)) {
            prevCommand.reportCompleted();
        }
    } else {
        throw new VdcBLLException(result.getVdsError().getCode(), result.getExceptionString());
    }

    return result;
}

虚拟机

平台中，虚拟机的状态共 18 种，对应着虚拟机当前的状态，而状态是根据用户的操作以及任务的执行以及 onTimer 中定时任刷新底层 vdsm 的虚拟机状态而后做一些判断共同决定的。

虚拟机对应的操作为开机，关机，断电，休眠，迁移，重启，快照，还有一些涉及磁盘的操作，对应的业务有导入，导出以及创建磁盘等。

平台可以通过界面用户触发或者 API 调用的方式进行虚拟机操作。最终都是通过相应的 command 实现业务逻辑。

业务逻辑中：

进行参数、权限、资源的检测与组装。
通过 xml-rpc 调用 vdsm 接口，由其完成与 libvirt 的交互达到管理 qemu 进程，最终操作 KVM 完成虚拟机操作。

运行虚拟机

运行虚拟机，可以分为多种情况，一般是普通运行，还有用于安装系统的只运行一次，运行休眠的虚拟机，以及无状态模式运行也就是运行池中的虚拟机。

普通运行，通过 RunVmCommand 实现。
只运行一次，通过 RunVmOnceCommand 实现，该模式主要用于安装系统时使用，配置有启动项，cd、floppy 等，开机后配置不保存。
无状态模式启动，同样使用了 RunVmCommand 实现，通过传递参数决定是否以无状态模式启动。无状态模式启动，在开机前会创建快照，关机后会恢复该快照（或者下次开机前恢复），保持开机前的状态。

虚拟机的运行，会根据集群策略，自动选择一台合适的主机运行。

虽然是不同的启动开机方式，但是都是调用 CreateVmVDSCommand。

VMStatus vmStatus = (VMStatus) getBackend()
            .getResourceManager()
            .RunAsyncVdsCommand(VDSCommandType.CreateVm, initCreateVmParams(), this).getReturnValue();

普通运行、只运行一次和无状态运行，最终都调用了 CreateVDSCommand。

return new CreateVDSCommand<CreateVmVDSCommandParameters>(getParameters());

CreateVDSCommand 根据不同的参数，组装 xml，通过 xml-rpc 执行 vdsm 的 create 方法。

mVmReturn = getBroker().create(createInfo);

public OneVmReturnForXmlRpc create(Map createInfo) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.create(createInfo);
        OneVmReturnForXmlRpc wrapper = new OneVmReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }
}

主机 vdsm 执行 create 方法。/usr/share/vdsm/API.py

def create(self, vmParams):
        """
        Start up a virtual machine.

        :param vmParams: required and optional VM parameters.
        :type vmParams: dict
        """
        vmParams['vmId'] = self._UUID
        try:
            if vmParams.get('vmId') in self._cif.vmContainer:
                self.log.warning('vm %s already exists' % vmParams['vmId'])
                return errCode['exist']

            if 'hiberVolHandle' in vmParams:
                vmParams['restoreState'], paramFilespec = \
                    self._getHibernationPaths(vmParams.pop('hiberVolHandle'))
                try:   # restore saved vm parameters
                    # NOTE: pickled params override command-line params. this
                    # might cause problems if an upgrade took place since the
                    # parmas were stored.
                    fname = self._cif.prepareVolumePath(paramFilespec)
......

运行休眠的虚拟机，需要恢复内存，执行的是 ResumeVDSCommand，调用 ResumeBrokerVDSCommand ，根据实时的状态进行回调。

ResumeBrokerVDSCommand<VdsAndVmIDVDSParametersBase> command =
                new ResumeBrokerVDSCommand<VdsAndVmIDVDSParametersBase>(parameters);
        command.execute();

mVmReturn = getBroker().resume(mVmId.toString());
proceedProxyReturnValue();

public OneVmReturnForXmlRpc pause(String vmId) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.pause(vmId);
        OneVmReturnForXmlRpc wrapper = new OneVmReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }

}

主机 vdsm 执行 cont 方法。/usr/share/vdsm/API.py

def cont(self):
    v = self._cif.vmContainer.get(self._UUID)
    if not v:
        return errCode['noVM']
    return v.cont()

关机/断电虚拟机

关机执行 ShutdownVmCommand，如果不能关闭，再执行断电 StopVmCommand。

if (canShutdownVm()) {
    // shutting down desktop and waiting for it in a separate thread to
    // become 'down':
    log.infoFormat("Sending shutdown command for VM {0}.", getVmName());

    int secondsToWait = getParameters().getWaitBeforeShutdown() ? Config
            .<Integer> getValue(ConfigValues.VmGracefulShutdownTimeout) : 0;

    // sending a shutdown command to the VM:
    setActionReturnValue(Backend
            .getInstance()
            .getResourceManager()
            .RunVdsCommand(VDSCommandType.DestroyVm,
                    new DestroyVmVDSCommandParameters(getVdsId(), getVmId(), false, true, secondsToWait))
            .getReturnValue());
}
else {
    // cannot shutdown -> send a StopVm command instead ('destroy'):
    // don't log -> log will appear for the StopVmCommand we are about to run:
    setCommandShouldBeLogged(false);

    log.infoFormat("Cannot shutdown VM {0}, status is not up. Stopping instead.", getVmName());

    StopVmParameters stopVmParams = new StopVmParameters(getVmId(), StopVmTypeEnum.CANNOT_SHUTDOWN);
    // stopVmParams.ParametersCurrentUser = CurrentUser;
    stopVmParams.setSessionId(getParameters().getSessionId());
    Backend.getInstance().runInternalAction(VdcActionType.StopVm, stopVmParams);
}

关机和断电最终都调用了 DestroyVDSCommand，只是传递的参数不同，最终执行 vdsm 不同的方法。关机 shutdown，断电 destroy。

if (getParameters().getGracefully()) {
    status = getBroker().shutdown(getParameters().getVmId().toString(),
            String.valueOf(getParameters().getSecondsToWait()),
            Config.<String> getValue(ConfigValues.VmGracefulShutdownMessage));
} else {
    status = getBroker().destroy(getParameters().getVmId().toString(), getParameters().getDestroyInfo());
}

public StatusOnlyReturnForXmlRpc shutdown(String vmId, String timeout, String message) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.shutdown(vmId, timeout, message);
        StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }

}

public StatusOnlyReturnForXmlRpc destroy(String vmId, Map destroyInfo) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.destroy(vmId, destroyInfo);
        StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }

}

主机 vdsm 关闭执行 shutdown 断电执行 destroy 方法。/usr/share/vdsm/API.py

def shutdown(self, delay=None, message=None, reboot=False, timeout=None,
             force=False):
    """
    Shut a VM down politely.

    :param message: message to be shown to guest user before shutting down
                    his machine.
    :param delay: grace period (seconds) to let guest user close his
                  applications.
    :param reboot: True if reboot is desired, False for shutdown
    :param timeout: number of seconds to wait before trying next
                    shutdown/reboot method
    :param force: True if shutdown/reboot desired by any means necessary
                  (forceful reboot/shutdown if all graceful methods fail)
    """
    try:
        v = self._cif.vmContainer[self._UUID]
    except KeyError:
        return errCode['noVM']
    if not delay:
        delay = config.get('vars', 'user_shutdown_timeout')
    if not message:
        message = USER_SHUTDOWN_MESSAGE
    if not timeout:
        timeout = config.getint('vars', 'sys_shutdown_timeout')
    return v.shutdown(delay, message, reboot, timeout, force)

def destroy(self, hostID, deprecatedSCSIKey):
        return self._irs.destroyStoragePool(self._UUID, hostID)

休眠虚拟机

休眠执行 HibernateVmCommand，会创建磁盘，并且保存内存数据 CreateImageVDSCommand。再调用 HibernateVDSCommand。保存虚拟机状态 SavingState 更新数据库。

Backend
        .getInstance()
        .getResourceManager()
        .RunVdsCommand(
                VDSCommandType.CreateImage,
                new CreateImageVDSCommandParameters(
                        getVm().getStoragePoolId(),
                        getStorageDomainId(),
                        image1GroupId,
                        getVm().getTotalMemorySizeInBytes(),
                        getMemoryVolumeType(),
                        VolumeFormat.RAW,
                        hiberVol1,
                        ""));

Backend.getInstance().getResourceManager().RunVdsCommand(VDSCommandType.Hibernate, para);

private VDSReturnValue runHibernateBrokerVDSCommand() {
    HibernateBrokerVDSCommand<HibernateVDSCommandParameters> command =
            new HibernateBrokerVDSCommand<HibernateVDSCommandParameters>(getParameters());
    command.execute();
    return command.getVDSReturnValue();
}

private void changeVmStatusToSavingState() {
    TransactionSupport.executeInNewTransaction(
            new TransactionMethod<Object>() {
                @Override
                public Object runInTransaction() {
                    VmDynamic vmDynamic = DbFacade.getInstance().getVmDynamicDao().get(getParameters().getVmId());
                    vmDynamic.setStatus(VMStatus.SavingState);
                    _vdsManager.updateVmDynamic(vmDynamic);
                    return null;
                }
            });
}

status = getBroker().hibernate(getParameters().getVmId().toString(),
getParameters().getHibernationVolHandle(),
getParameters().getEncryptionInfo());

public StatusOnlyReturnForXmlRpc hibernate(String vmId, String hiberVolHandle, Map<String, Object> encryptionInfo) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.hibernate(vmId, hiberVolHandle, encryptionInfo);
        StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }

}

主机 vdsm 休眠执行 hibernate 方法。/usr/share/vdsm/API.py

def hibernate(self, hibernationVolHandle, encryptionInfo):
    """
    Hibernate a VM.

    :param hiberVolHandle: opaque string, indicating the location of
                           hibernation images.
    :param encryptionInfo: a map value indicates the vm and template encyption
                            password
    """
    params = {'vmId': self._UUID, 'mode': 'file',
              'hiberVolHandle': hibernationVolHandle,
              'vmMigrateDestPw': encryptionInfo['vmSuspendPw'],
              'tmMigrateDestPw': encryptionInfo['tmSuspendPw'],
              'secretUUID': encryptionInfo['secretUUID']}
    response = self.migrate(params)
    if not response['status']['code']:
        response['status']['message'] = 'Hibernation process starting'
    return response

创建快照（保存内存）

快照的创建实际对应于底层是创建一个 volume ，执行 CreateSnapshotCommand，调用 CreateSnapshotVDSCommand。

vdsReturnValue =
        Backend
                .getInstance()
                .getResourceManager()
                .RunVdsCommand(
                        VDSCommandType.CreateSnapshot,
                        new CreateSnapshotVDSCommandParameters(getStoragePoolId(),
                                getDestinationStorageDomainId(),
                                getImageGroupId(),
                                getImage().getImageId(),
                                getDiskImage().getSize(),
                                mNewCreatedDiskImage.getVolumeType(),
                                mNewCreatedDiskImage.getVolumeFormat(),
                                getDiskImage().getId(),
                                getDestinationImageId(),
                                ""));

uuidReturn = getIrsProxy().createVolume(getParameters().getStorageDomainId().toString(),
getParameters().getStoragePoolId().toString(),
getParameters().getImageGroupId().toString(),
(Long.valueOf(getParameters().getImageSizeInBytes())).toString(),
getParameters().getVolumeFormat().getValue(),
getParameters().getImageType().getValue(),
2,
getParameters().getNewImageID().toString(),
getParameters().getNewImageDescription(),
getParameters().getSourceImageGroupId().toString(),
getParameters().getImageId().toString());

public OneUuidReturnForXmlRpc createVolume(String sdUUID, String spUUID, String imgGUID, String size,
        int volFormat, int volType, int diskType, String volUUID, String descr, String srcImgGUID, String srcVolUUID) {
    Map<String, Object> xmlRpcReturnValue = irsServer.createVolume(sdUUID, spUUID, imgGUID, size, volFormat,
            volType, diskType, volUUID, descr, srcImgGUID, srcVolUUID);
    OneUuidReturnForXmlRpc wrapper = new OneUuidReturnForXmlRpc(xmlRpcReturnValue);
    return wrapper;
}

主机 vdsm 创建卷，执行 Volume 类的 create 方法。/usr/share/vdsm/API.py

class Volume(APIBase):
    ctorArgs = ['volumeID', 'storagepoolID', 'storagedomainID', 'imageID']

    class Types:
        UNKNOWN = storage.volume.UNKNOWN_VOL
        PREALLOCATED = storage.volume.PREALLOCATED_VOL
        SPARSE = storage.volume.SPARSE_VOL

    class Formats:
        UNKNOWN = storage.volume.UNKNOWN_FORMAT
        COW = storage.volume.COW_FORMAT
        RAW = storage.volume.RAW_FORMAT

    class Roles:
        SHARED = storage.volume.SHARED_VOL
        LEAF = storage.volume.LEAF_VOL

    BLANK_UUID = storage.volume.BLANK_UUID
......
    def create(self, size, volFormat, preallocate, diskType, desc,
                   srcImgUUID, srcVolUUID):
        return self._irs.createVolume(self._sdUUID, self._spUUID,
                                      self._imgUUID, size, volFormat,
                                      preallocate, diskType, self._UUID, desc,
                                      srcImgUUID, srcVolUUID)

如果保存内存，会执行异步任务 SnapshotVDSCommand。

protected void endVmCommand() {
......
liveSnapshotSucceeded = performLiveSnapshot(createdSnapshot);
......

protected boolean performLiveSnapshot(final Snapshot snapshot) {
    try {
        TransactionSupport.executeInScope(TransactionScopeOption.Suppress, new TransactionMethod<Void>() {
            @Override
            public Void runInTransaction() {
                runVdsCommand(VDSCommandType.Snapshot, buildLiveSnapshotParameters(snapshot));
                return null;
            }
        });
    } catch (VdcBLLException e) {
        handleVdsLiveSnapshotFailure(e);
        return false;
    }
    return true;
}

private StatusOnlyReturnForXmlRpc executeSnapshotVerb() {
    String vmId = getParameters().getVmId().toString();
    return getParameters().isMemoryVolumeExists()
            ? getBroker().snapshot(vmId,
                    createDisksMap(),
                    getParameters().getMemoryVolume(),
                    getParameters().getSnapshotInfo())
            : getBroker().snapshot(vmId, createDisksMap(), getParameters().getSnapshotInfo());
}

public StatusOnlyReturnForXmlRpc snapshot(String vmId,
        Map<String, String>[] disks,
        String memory,
        Map<String, Object> vmEncrypParams) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.snapshot(vmId, disks, memory, vmEncrypParams);
        StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }
}

主机 vdsm 执行 snapshot 方法。/usr/share/vdsm/API.py

 def snapshot(self, snapDrives, snapMemVolHandle=None, vmEncrypParams={}):
    v = self._cif.vmContainer.get(self._UUID)
    if not v:
        return errCode['noVM']
    memoryParams = {}
    if snapMemVolHandle:
        memoryParams['dst'], memoryParams['dstparams'] = \
            self._getHibernationPaths(snapMemVolHandle)
    return v.snapshot(snapDrives, memoryParams, vmEncrypParams)

迁移虚拟机

迁移在虚拟机开机的情况下，从一台主机，迁移到一台新的主机。MigrateVmCommand 选择迁往的目的主机（根据集群策略）。调用 MigrateVDSCommand。改变虚拟机的状态为 MigratingFrom，更新数据库。

setActionReturnValue(Backend
                    .getInstance()
                    .getResourceManager()
                    .RunAsyncVdsCommand(
                            VDSCommandType.Migrate,
                            createMigrateVDSCommandParameters(),
                            this)
                            .getReturnValue());

MigrateBrokerVDSCommand<?> command = new MigrateBrokerVDSCommand<>(getParameters());
        command.execute();

protected void executeVdsBrokerCommand() {
        status = getBroker().migrate(migrationInfo);
        proceedProxyReturnValue();
    }

public StatusOnlyReturnForXmlRpc migrate(Map<String, String> migrationInfo) {
    try {
        Map<String, Object> xmlRpcReturnValue = vdsServer.migrate(migrationInfo);
        StatusOnlyReturnForXmlRpc wrapper = new StatusOnlyReturnForXmlRpc(xmlRpcReturnValue);
        return wrapper;
    } catch (UndeclaredThrowableException ute) {
        throw new XmlRpcRunTimeException(ute);
    }

}

主机 vdsm 执行 migrate 方法。/usr/share/vdsm/API.py

def migrate(self, params):
    """
    Migrate a VM to a remote host.

    :param params: a dictionary containing:
        *dst* - remote host or hibernation image filename
        *dstparams* - hibernation image filename for vdsm parameters
        *mode* - ``remote``/``file``
        *method* - ``online``
        *downtime* - allowed down time during online migration
        *dstqemu* - remote host address dedicated for migration
    """
    params['vmId'] = self._UUID
    self.log.debug(params)
    try:
        v = self._cif.vmContainer[self._UUID]
    except KeyError:
        return errCode['noVM']

    vmParams = v.status()
    if vmParams['status'] in (vmstatus.WAIT_FOR_LAUNCH, vmstatus.DOWN):
        return errCode['noVM']
    if params.get('mode') == 'file':
        if 'dst' not in params:
            params['dst'], params['dstparams'] = \
                self._getHibernationPaths(params['hiberVolHandle'])
    else:
        params['mode'] = 'remote'
    return v.migrate(params)

迁移会消耗一定的时间，在这个过程中，engine 会定时查询 vdsm 的虚拟机状态，从而更新虚拟机的实时状态并进行相应的处理数据的更新。VdsManager 的 onTimer。

public void onTimer() {
......
_vdsUpdater = new VdsUpdateRunTimeInfo(VdsManager.this, _vds, monitoringStrategy) _vdsUpdater.refresh();

磁盘资源

场景	VDS 命令	vdsm 接口	VolumeType	存储域	VolumeFormat
创建模板	CopyImageVDSCommand	CopyImage	Sparse	与虚拟机一致	RAW→COW
/	/	/	与虚拟机一致	与虚拟机一致	COW
/	/	/	Preallocated	与虚拟机一致	RAW
模版 clone 创建 vm	CopyImageVDSCommand	CopyImage	Preallocated	与模板一致	RAW
模版 thin 创建 vm	CreateSnapshotVDSCommand	createVolume	Sparse	与模板一致	COW
创建 vm/增加磁盘	CreateImageVDSCommand	createVolume	Preallocated/Sparse	File	RAW
/	/	/	Sparse	Block	COW
/	/	/	Preallocated	Block	RAW
快照创建 vm	CopyImageVDSCommand	CopyImage	与快照一致	与快照一致	COW
/	/	/	Sparse	Block	RAW→COW
/	/	/	Preallocated	与快照一致	RAW
创建快照	CreateSnapshotVDSCommand	createVolume	Sparse	与虚拟机一致	COW
导入导出域中虚拟机	CopyImageVDSCommand	copyImage	与导出域一致	与导出域一致	与导出域一致
导入外部供应商 image	CopyImageVDSCommand	copyImage	Preallocated	Block	RAW
/	/	/	Sparse→Preallocated	Block	RAW
/	/	/	Sparse	File	RAW
/	/	/	Sparse	与外部一致	COW
/	/	/	Preallocated→Sparse	与外部一致	COW
导入模版	CopyImageVDSCommand	copyImage	Sparse	Block	RAW→COW
/	/	/	Sparse	File	RAW
/	/	/	Preallocated	与模板一致	与模板一致
导出虚拟机	CopyImageVDSCommand	copyImage	与虚拟机一致	与虚拟机一致	与虚拟机一致
导出模版	CopyImageVDSCommand	copyImage	与模板一致	与模板一致	与模板一致

创建模板

为了适应软加密，将 VolumeType.Sparse、VolumeFormat.RAW 并且无论存储域格式，最终将 RAW 转为 COW 格式。

使用了 AddVmTemplateCommand→CopyImageVDSCommand，最终执行 CopyImage。

VDSReturnValue vdsReturnValue = Backend
    .getInstance()
    .getResourceManager()
    .RunVdsCommand(
            VDSCommandType.CopyImage,
            new CopyImageVDSCommandParameters(storagePoolId, getParameters().getStorageDomainId(),
                    getParameters().getVmId(), imageGroupId, snapshotId, destinationImageGroupID,
                    getDestinationImageId(), StringUtils.defaultString(newImage.getDescription()), getParameters()
                            .getDestinationStorageDomainId(), CopyVolumeType.SharedVol, targetFormat,
                            newImage.getVolumeType(), getDiskImage().isWipeAfterDelete(), false));

模版 clone 创建 vm

使用了 CreateCloneOfTemplateCommand→CopyImageVDSCommand，最终执行 CopyImage。

vdsReturnValue = runVdsCommand(VDSCommandType.CopyImage,
                    copyImageParas);

模版 thin 创建 vm

使用了 CreateSnapshotCommand→CreateSnapshotVDSCommand，最终执行 createVolume。

vdsReturnValue =
            Backend
                    .getInstance()
                    .getResourceManager()
                    .RunVdsCommand(
                            VDSCommandType.CreateSnapshot,
                            new CreateSnapshotVDSCommandParameters(getStoragePoolId(),
                                    getDestinationStorageDomainId(),
                                    getImageGroupId(),
                                    getImage().getImageId(),
                                    getDiskImage().getSize(),
                                    mNewCreatedDiskImage.getVolumeType(),
                                    mNewCreatedDiskImage.getVolumeFormat(),
                                    getDiskImage().getId(),
                                    getDestinationImageId(),
                                    ""));

创建 vm/增加磁盘

使用了 AddVmFromScratchCommand→AddImageFromScratch→CreateImageVDSCommand，最终执行 createVolume。

tmpRetValue = Backend.getInstance().runInternalAction(
VdcActionType.AddImageFromScratch,
tempVar,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));

VDSReturnValue vdsReturnValue = runVdsCommand(
VDSCommandType.CreateImage,
new CreateImageVDSCommandParameters(getParameters().getStoragePoolId(), getParameters()
        .getStorageDomainId(), getImageGroupId(), getParameters().getDiskInfo().getSize(),
        getParameters().getDiskInfo().getVolumeType(), getParameters().getDiskInfo()
                .getVolumeFormat(), getDestinationImageId(), ""));

快照创建 vm

使用了 AddVmFromSnapshotCommand→CopyImageGroupCommand→CopyImageVDSCommand，最终执行 CopyImage。

protected VdcActionType getChildActionType() {
        return VdcActionType.CopyImageGroup;
    }

protected VdcReturnValueBase executeChildCopyingCommand(VdcActionParametersBase parameters) {
    VdcReturnValueBase result = Backend.getInstance().runInternalAction(
            getChildActionType(),
                    parameters,
                    ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));
    return result;
}

vdsReturnValue = runVdsCommand(
    VDSCommandType.CopyImage,
    copyImageParas);

创建快照

使用了 CreateSnapshotCommand→CreateSnapshotVDSCommand，最终执行 createVolume。

vdsReturnValue =
            Backend
                    .getInstance()
                    .getResourceManager()
                    .RunVdsCommand(
                            VDSCommandType.CreateSnapshot,
                            new CreateSnapshotVDSCommandParameters(getStoragePoolId(),
                                    getDestinationStorageDomainId(),
                                    getImageGroupId(),
                                    getImage().getImageId(),
                                    getDiskImage().getSize(),
                                    mNewCreatedDiskImage.getVolumeType(),
                                    mNewCreatedDiskImage.getVolumeFormat(),
                                    getDiskImage().getId(),
                                    getDestinationImageId(),
                                    ""));

vdsReturnValue = runVdsCommand(
                    VDSCommandType.CopyImage,
                    copyImageParas);

导入导出域中虚拟机

使用了 ImportVmCommand→CopyImageGroupCommand→CopyImageVDSCommand，最终执行 CopyImage。

vdcRetValue = Backend.getInstance().runInternalAction(
            VdcActionType.CopyImageGroup,
            buildMoveOrCopyImageGroupParametersForMemoryConfImage(
                    containerId, guids.get(0), guids.get(4), guids.get(5)),
                    ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));

vdsReturnValue = runVdsCommand(
                    VDSCommandType.CopyImage,
                    copyImageParas);

导入模版

使用了 ImportVmTemplateCommand→CopyImageGroupCommand→CopyImageVDSCommand，最终执行 CopyImage。

VdcReturnValueBase vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
p,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));

vdsReturnValue = runVdsCommand(
                    VDSCommandType.CopyImage,
                    copyImageParas);

导出虚拟机

使用了 ExportVmCommand→CopyImageGroupCommand→CopyImageVDSCommand，最终执行 CopyImage。

VdcReturnValueBase vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
buildMoveOrCopyImageGroupParametersForMemoryDumpImage(
        containerID, guids.get(0), guids.get(2), guids.get(3)),
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));

vdsReturnValue = runVdsCommand(
                    VDSCommandType.CopyImage,
                    copyImageParas);

导出模版

使用了 ExportVmTemplateCommand→CopyImageGroupCommand→CopyImageVDSCommand，最终执行 CopyImage。

VdcReturnValueBase vdcRetValue = Backend.getInstance().runInternalAction(
VdcActionType.CopyImageGroup,
p,
ExecutionHandler.createDefaultContexForTasks(getExecutionContext()));

vdsReturnValue = runVdsCommand(
                    VDSCommandType.CopyImage,
                    copyImageParas);

copyImage

uuidReturn = getIrsProxy().copyImage(getParameters().getStorageDomainId().toString(),
getParameters().getStoragePoolId().toString(),
getParameters().getVmId().toString(),
getParameters().getImageGroupId().toString(),
getParameters().getImageId().toString(),
getParameters().getdstImageGroupId().toString(),
getParameters().getDstImageId().toString(),
getParameters().getCopyImageInfo(), // add by hzy for EncryptInfo
getParameters().getImageDescription(),
getParameters().getDstStorageDomainId().toString(),
getParameters().getCopyVolumeType().getValue(),
getParameters().getVolumeFormat().getValue(),
getParameters().getPreallocate().getValue(),
String.valueOf(getParameters().getPostZero()).toLowerCase(),
String.valueOf(getParameters().getForce()).toLowerCase());

public OneUuidReturnForXmlRpc copyImage(String sdUUID, String spUUID, String vmGUID, String srcImgGUID,
        String srcVolUUID, String dstImgGUID, String dstVolUUID, Map<String, Object> encryption, String descr, String dstSdUUID, int volType,
        int volFormat, int preallocate, String postZero, String force) {
    Map<String, Object> xmlRpcReturnValue = irsServer.copyImage(sdUUID, spUUID, vmGUID, srcImgGUID, srcVolUUID,
            dstImgGUID, dstVolUUID, encryption, descr, dstSdUUID, volType, volFormat, preallocate, postZero, force);
    OneUuidReturnForXmlRpc wrapper = new OneUuidReturnForXmlRpc(xmlRpcReturnValue);
    return wrapper;
}

主机 vdsm，执行 Volume 类的 copy 方法。/usr/share/vdsm/API.py

 def copy(self, dstSdUUID, dstImgUUID, dstVolUUID, encryption, desc, volType,
             volFormat, preallocate, postZero, force):
        vmUUID = ''   # vmUUID is never used
        return self._irs.copyImage(self._sdUUID, self._spUUID, vmUUID,
                                   self._imgUUID, self._UUID, dstImgUUID,
                                   dstVolUUID,encryption, desc, dstSdUUID, volType,
                                   volFormat, preallocate, postZero, force)

createVolume

uuidReturn = getIrsProxy().createVolume(getParameters().getStorageDomainId().toString(),
getParameters().getStoragePoolId().toString(),
getParameters().getImageGroupId().toString(),
(Long.valueOf(getParameters().getImageSizeInBytes())).toString(),
getParameters().getVolumeFormat().getValue(),
getParameters().getImageType().getValue(),
2,
getParameters().getNewImageID().toString(),
getParameters().getNewImageDescription(),
getParameters().getSourceImageGroupId().toString(),
getParameters().getImageId().toString());

public OneUuidReturnForXmlRpc createVolume(String sdUUID, String spUUID, String imgGUID, String size,
        int volFormat, int volType, int diskType, String volUUID, String descr, String srcImgGUID, String srcVolUUID) {
    Map<String, Object> xmlRpcReturnValue = irsServer.createVolume(sdUUID, spUUID, imgGUID, size, volFormat,
            volType, diskType, volUUID, descr, srcImgGUID, srcVolUUID);
    OneUuidReturnForXmlRpc wrapper = new OneUuidReturnForXmlRpc(xmlRpcReturnValue);
    return wrapper;
}

主机 vdsm，执行 Volume 类的 create 方法。/usr/share/vdsm/API.py

class Volume(APIBase):
    ctorArgs = ['volumeID', 'storagepoolID', 'storagedomainID', 'imageID']

    class Types:
        UNKNOWN = storage.volume.UNKNOWN_VOL
        PREALLOCATED = storage.volume.PREALLOCATED_VOL
        SPARSE = storage.volume.SPARSE_VOL

    class Formats:
        UNKNOWN = storage.volume.UNKNOWN_FORMAT
        COW = storage.volume.COW_FORMAT
        RAW = storage.volume.RAW_FORMAT

    class Roles:
        SHARED = storage.volume.SHARED_VOL
        LEAF = storage.volume.LEAF_VOL

    BLANK_UUID = storage.volume.BLANK_UUID
......
    def create(self, size, volFormat, preallocate, diskType, desc,
                   srcImgUUID, srcVolUUID):
        return self._irs.createVolume(self._sdUUID, self._spUUID,
                                      self._imgUUID, size, volFormat,
                                      preallocate, diskType, self._UUID, desc,
                                      srcImgUUID, srcVolUUID)

SPM

SPM 是存储池管理器，用于管理所在数据中心的所有存储域，运行在数据中心的其中一台主机上，通过协调存储域中的元数据来控制对存储的访问。虚拟化平台会保证一直有一个 SPM 在正常运行，在 SPM 的主机出现问题时，会选择另外一台主机。

实现的核心是通过 IrsBrokerCommand抽象类，所有的 irsCommand 都继承该类。调用 vdsm 的接口时，通过 getIrsProxy() 调用。
getIrsProxy 是通过 IrsBrokerCommand 的一个 ConcurrentHashMap，_irsProxyData 的 map 变量，保存了 key 为 storagePool的 id，value 为 IrsProxyData 对象，IrsProxyData 保存了与 SPM 主机接口调用的代理 IIrsServer。

private static Map<Guid, IrsProxyData> _irsProxyData = new ConcurrentHashMap<Guid, IrsProxyData>();

一般 IrsBrokerCommand 调用，都是通过 IrsProxyData 找到 IIrsServer 接口 privatemIrsProxy。

private IIrsServer privatemIrsProxy;

private IIrsServer getmIrsProxy() {
    return privatemIrsProxy;
}

gethostFromVds 方法找到合适的主机（状态是 Up，根据优先级，同级的话随机），然后选为 SPM，privatemIrsProxy 为 null，重新选择 SPM。

public IIrsServer getIrsProxy() {
    if (getmIrsProxy() == null) {
        final StoragePool storagePool = DbFacade.getInstance().getStoragePoolDao().get(_storagePoolId);
        // don't try to start spm on uninitialized pool
        if (storagePool.getStatus() != StoragePoolStatus.Uninitialized) {
            String host =
                    TransactionSupport.executeInScope(TransactionScopeOption.Suppress,
                            new TransactionMethod<String>() {
                                @Override
                                public String runInTransaction() {
                                    return gethostFromVds();
                                }
                            });

            if (host != null) {
                // Get the values of the timeouts:
                int clientTimeOut = Config.<Integer> getValue(ConfigValues.vdsTimeout) * 1000;
                int connectionTimeOut = Config.<Integer>getValue(ConfigValues.vdsConnectionTimeout) * 1000;
                int clientRetries = Config.<Integer> getValue(ConfigValues.vdsRetries);

                Pair<IrsServerConnector, HttpClient> returnValue =
                        XmlRpcUtils.getConnection(host,
                                getmIrsPort(),
                                clientTimeOut,
                                connectionTimeOut,
                                clientRetries,
                                IrsServerConnector.class,
                                Config.<Boolean> getValue(ConfigValues.EncryptHostCommunication));
                privatemIrsProxy = new IrsServerWrapper(returnValue.getFirst(), returnValue.getSecond());
                runStoragePoolUpEvent(storagePool);
            }
        }
    }
    return getmIrsProxy();
}

一般场景下，一台主机挂掉，另一台主机出现 Connecting 状态，是因为调用接口时，出现网络异常 VDSNetworkException。调用 IrsBrokerCommand 的 failover 方法。执行失败后，可尝试恢复两次，恢复次数可设置。

private void failover() {
    if ((getParameters().getIgnoreFailoverLimit() || _failoverCounter < Config
            .<Integer> getValue(ConfigValues.SpmCommandFailOverRetries) - 1)
            && getCurrentIrsProxyData().getHasVdssForSpmSelection() && getCurrentIrsProxyData().failover()) {
        _failoverCounter++;
        executeCommand();
    } else {
        getVDSReturnValue().setSucceeded(false);
    }
}
......
public boolean failover() {
    Guid vdsId = mCurrentVdsId;
    nullifyInternalProxies();
    boolean performFailover = false;
    if (vdsId != null) {
        try {
            VDSReturnValue statusResult = ResourceManager.getInstance().runVdsCommand(VDSCommandType.SpmStatus,
                    new SpmStatusVDSCommandParameters(vdsId, _storagePoolId));
            if (statusResult != null
                    && statusResult.getSucceeded()
                    && (((SpmStatusResult) statusResult.getReturnValue()).getSpmStatus() == SpmStatus.SPM || ((SpmStatusResult) statusResult
                            .getReturnValue()).getSpmStatus() == SpmStatus.Contend)) {
                performFailover = ResourceManager
                        .getInstance()
                        .runVdsCommand(VDSCommandType.SpmStop,
                                new SpmStopVDSCommandParameters(vdsId, _storagePoolId)).getSucceeded();
            } else {
                performFailover = true;
            }
        } catch (Exception ex) {
            // try to failover to another host if failed to get spm
            // status or stop spm
            // (in case mCurrentVdsId has wrong id for some reason)
            log.errorFormat("Could not get spm status on host {0} for spmStop.", vdsId);
            performFailover = true;
        }
    }

    if (performFailover) {
        log.infoFormat("Irs placed on server {0} failed. Proceed Failover", vdsId);
        mTriedVdssList.add(vdsId);
        return true;
    } else {
        log.errorFormat("IRS failover failed - cant allocate vds server");
        return false;
    }
}

【Ovirt 笔记】虚拟资源管理分析与整理

主机

ResourceManager

VDSBrokerFrontendImpl

虚拟机

运行虚拟机

关机/断电虚拟机

休眠虚拟机

创建快照（保存内存）

迁移虚拟机

磁盘资源

创建模板

模版 clone 创建 vm

模版 thin 创建 vm

创建 vm/增加磁盘

快照创建 vm

创建快照

导入导出域中虚拟机

导入模版

导出虚拟机

导出模版

copyImage

createVolume

SPM

猜你喜欢

热点阅读