opendaylight源码分析之yang parser
2021-11-15 本文已影响0人
movee
1 基本流程
1.1 编译yang文件，生成parse tree
需要编译的yang文件
InputStream exampleYang = Main.class.getResourceAsStream("/example.yang");
StatementStreamSource exampleSource = new YangStatementSourceImpl("example", exampleYang);
InputStream testYang = Main.class.getResourceAsStream("/test.yang");
StatementStreamSource testSource = new YangStatementSourceImpl("test", testYang);
也可以使用YangStatementStreamSource表示需要编译的yang文件
RevisionSourceIdentifier srcId = RevisionSourceIdentifier.create("/example.yang", "2018-08-14");
YangTextFileSchemaSource textFileSrc = new YangTextFileSchemaSource(srcId, new File("/example.yang"));
StatementStreamSource exampleSource = YangStatementStreamSource.create(textFileSrc);
或
RevisionSourceIdentifier srcId = RevisionSourceIdentifier.create("/example.yang", "2018-08-14");
ResourceYangTextSchemaSource resourceFileSrc = new ResourceYangTextSchemaSource(srcId, new URL("/example.yang"));
StatementStreamSource exampleSource = YangStatementStreamSource.create(resourceFileSrc);
创建StatementStreamSource对象时会调用antlr编译yang文件，得到parse tree，代码如下：
public static StatementContext parseYangSource(final InputStream stream) throws IOException,
YangSyntaxErrorException {
final YangStatementLexer lexer = new YangStatementLexer(new ANTLRInputStream(stream));
final CommonTokenStream tokens = new CommonTokenStream(lexer);
final YangStatementParser parser = new YangStatementParser(tokens);
//disconnect from console error output
parser.removeErrorListeners();
// 添加error listener
final YangErrorListener errorListener = new YangErrorListener();
parser.addErrorListener(errorListener);
final StatementContext result = parser.statement();
errorListener.validate();
// Walk the resulting tree and replace each children with an immutable list, lowering memory requirements
// and making sure the resulting tree will not get accidentally modified. An alternative would be to use
// org.antlr.v4.runtime.Parser.TrimToSizeListener, but that does not make the tree immutable.
ParseTreeWalker.DEFAULT.walk(MAKE_IMMUTABLE_LISTENER, result);
return result;
}
StatementStreamSource封装了两个关键信息：语法分析树StatementContext和语法分析树遍历的监听器YangStatementParserListenerImpl
1.2 translate parse tree，得到SchemaContext
下面分析如何对parse tree进行translate，得到便于处理的SchemaContext，代码如下：
final CrossSourceStatementReactor.BuildAction reactor = YangInferencePipeline.RFC6020_REACTOR.newBuild();
// 添加需要分析的parse tree
reactor.addSources(exampleSource, testSource);
// 分析所有添加的parse tree，生成schema数据结构
SchemaContext context = reactor.buildEffective();
对parse tree的遍历分为6个阶段，也就是说会遍历parse tree 6次，每一次遍历时，处理yang文件的不同statement。所有的阶段执行完后，则处理完所有的statement。每个阶段处理的statement，由这个阶段对应的StatementSupportBundle定义。
reactor.buildEffective()的工作由BuildGlobalContext完成，不出意外，BuildGlobalContext包含了每个阶段支持分析的statement定义，以及每个yang文件对应的处理类SourceSpecificContext, 以及跟踪整个分析过程的一些状态
每一个phase的处理步骤：
private void executePhases() throws ReactorException {
for (final ModelProcessingPhase phase : PHASE_EXECUTION_ORDER) {
// 只做了一些检查
startPhase(phase);
// 分析statement
loadPhaseStatements();
// 当前实现，这一步没做什么动作
completePhaseActions();
// 只做了一些检查
endPhase(phase);
}
}
每一个phase会调用所有yang文件对应的SourceSpecificContext的loadStatements()函数
/**
*
* 这几个函数的可以写成一个函数，最终都会调用listener遍历parser树
*
* @throws SourceException
*/
void loadStatements() throws SourceException {
LOG.trace("Source {} loading statements for phase {}", source, inProgressPhase);
switch (inProgressPhase) {
case SOURCE_PRE_LINKAGE:
source.writePreLinkage(new StatementContextWriter(this, inProgressPhase), stmtDef());
break;
case SOURCE_LINKAGE:
source.writeLinkage(new StatementContextWriter(this, inProgressPhase), stmtDef(), preLinkagePrefixes(), getRootVersion());
break;
case STATEMENT_DEFINITION:
source.writeLinkageAndStatementDefinitions(new StatementContextWriter(this, inProgressPhase), stmtDef(), prefixes(), getRootVersion());
break;
case FULL_DECLARATION:
source.writeFull(new StatementContextWriter(this, inProgressPhase), stmtDef(), prefixes(), getRootVersion());
break;
default:
break;
}
}
StatementContextWriter中有一个重要的成员
// statement处理完成前，current代表父statement，处理结束时，代表当前statement，为它的孩子节点创建StatementContextBase
// 对于yang module的第一个statement，即root statement，处理结束前，current值为null
private StatementContextBase<?, ?, ?> current;
遍历完后，StatementContextBase就代表了yang文件定义的schema tree
YangStmtMapping中为每一个yang statement关键字定义了一个对应的StatementDefinition枚举类型，定义了statement关键字对应的QName值，对应的DeclaredStatement类对象。QName格式为urn:ietf:params:xml:ns:yang:yin:1:<关键字>, 比如关键字module对应的QName值为urn:ietf:params:xml:ns:yang:yin:1:module。
几个常用的关键字对应的StatementDefinition为：
public enum YangStmtMapping implements StatementDefinition {
CONTACT(ContactStatement.class, "contact", "text", true),
CONTAINER(ContainerStatement.class, "container", "name"),
KEY(KeyStatement.class, "key", "value"),
LEAF(LeafStatement.class, "leaf", "name"),
LEAF_LIST(LeafListStatement.class, "leaf-list", "name"),
LIST(ListStatement.class, "list", "name"),
MODULE(ModuleStatement.class, "module", "name"),
NAMESPACE(NamespaceStatement.class, "namespace", "uri"),
NOTIFICATION(NotificationStatement.class, "notification", "name"),
PREFIX(PrefixStatement.class, "prefix", "value"),
REVISION(RevisionStatement.class, "revision", "date"),
RPC(RpcStatement.class, "rpc", "name"),
TYPE(TypeStatement.class, "type", "name"),
YANG_VERSION(YangVersionStatement.class, "yang-version", "value"),
}
同时为每一个yang statement定义了一个对应的StatementSupport实例，在YangInferencePipeline中为每一个phase添加处理的statement时，同时添加了对应的StatementSupport，常用的statement对应的StatementSupport为：
ModuleStatementSupport
PrefixStatementImpl
YangVersionStatementImpl
RevisionStatementImpl
ContactStatementImpl
TypeStatementImpl
KeyStatementImpl
ContainerStatementImpl
ListStatementImpl
RpcStatementImpl
NotificationStatementImpl
LeafStatementImpl
LeafListStatementImpl
这些类在实例化时都关联了对应的StatementDefinition
不管哪个phase，不管哪个yang文件，最后都是调用监听器YangStatementParserListenerImpl遍历parse tree StatementContext
发现与退出parse tree节点时的处理：
@Override
public void enterStatement(final StatementContext ctx) {
// 识别出的statement的第一个token在yang文件中的位置
final StatementSourceReference ref = DeclarationInTextSource.atPosition(sourceName, ctx.getStart().getLine(),
ctx.getStart().getCharPositionInLine());
// 读取keyword 的文本值, 比如 module，rpc，container这些关键字
final String keywordTxt = Verify.verifyNotNull(ctx.getChild(KeywordContext.class, 0)).getText();
// LOG.info("sourceName: {}, keyword: {}", sourceName, keywordTxt);
// keyword对应 的 Qname，通过StatementSupport的StatementDefinition可以找到关键字与Qname的映射
final QName validStatementDefinition = getValidStatementDefinition(prefixes, stmtDef, keywordTxt);
// 自己父节点的孩子个数加1
final int childId = counters.peek().getAndIncrement();
// push 一个counter到栈中，代表自己，每个自己的孩子会给这个计数器加1
counters.push(new Counter());
if (stmtDef == null || validStatementDefinition == null || !toBeSkipped.isEmpty()) {
// LOG.info("keywordTxt: {}, validStatementDefinition: {}", keywordTxt, validStatementDefinition);
SourceException.throwIf(writer.getPhase() == ModelProcessingPhase.FULL_DECLARATION, ref,
"%s is not a YANG statement or use of extension.", keywordTxt);
toBeSkipped.add(keywordTxt);
// 本阶段不处理的statement，本节点以及它的孩子节点，都不遍历
return;
}
// 读取keyword对应的参数值
final ArgumentContext argumentCtx = ctx.getChild(ArgumentContext.class, 0);
final String argument = argumentCtx != null ? Utils.stringFromStringContext(argumentCtx, yangVersion, ref)
: null;
// LOG.info("enter sourceName: {}, keyword: {}， argument: {}", sourceName, keywordTxt, argument);
writer.startStatement(childId, validStatementDefinition, argument, ref);
}
@Override
public void exitStatement(final StatementContext ctx) {
final StatementSourceReference ref = DeclarationInTextSource.atPosition(
sourceName, ctx.getStart().getLine(), ctx.getStart().getCharPositionInLine());
final KeywordContext keyword = ctx.getChild(KeywordContext.class, 0);
final String statementName = keyword.getText();
if (stmtDef != null && getValidStatementDefinition(prefixes, stmtDef, statementName) != null
&& toBeSkipped.isEmpty()) {
writer.endStatement(ref);
}
// LOG.info("exit sourceName: {}, keywork: {}", sourceName, statementName);
// No-op if the statement is not on the list
toBeSkipped.remove(statementName);
counters.pop();
}
enterStatement()最后调用StatementContextWriter的startStatement()函数:
/**
* 为current StatementContextBase创建子 StatementContextBase，如果current是null，说明是root StatementContext，直接new一个
* 其实对应的statement为 module
*/
StatementContextBase<?, ?, ?> createDeclaredChild(final StatementContextBase<?, ?, ?> current, final int childId,
final QName name, final String argument, final StatementSourceReference ref) {
// 非 root statement，即 非 modulestatement
if (current != null) {
// Fast path: we are entering a statement which was emitted in previous phase
StatementContextBase<?, ?, ?> existing = current.lookupSubstatement(childId);
// 是根据上下文推导出来的，一直向下遍历
while (existing != null && StatementSource.CONTEXT == existing.getStatementSource()) {
existing = existing.lookupSubstatement(childId);
}
// 直接返回
if (existing != null) {
return existing;
}
}
// 在BuildGlobalContext中查所有的，使用 yangversion + qname 查 是否已处理过，其实是查缓冲
// StatementDefinitionContext 中存放的是 statement对应的StatementSupport
StatementDefinitionContext<?, ?, ?> def = currentContext.getStatementDefinition(getRootVersion(), name);
if (def == null) {
// 全局 使用 qname 查 是否已处理过，其实是查缓冲
def = currentContext.getModelDefinedStatementDefinition(name);
if (def == null) {
// 从本地查，key为Qname，所有本阶段以及之前阶段支持的所有statement definition
final StatementSupport<?, ?, ?> extension = qNameToStmtDefMap.get(name);
if (extension != null) {
// 缓存到全局 BuildGlobalContext 中
def = new StatementDefinitionContext<>(extension);
currentContext.putModelDefinedStatementDefinition(name, def);
}
}
} else if (current != null && StmtContextUtils.isUnrecognizedStatement(current)) {
/*
* This code wraps statements encountered inside an extension so
* they do not get confused with regular statements.
*/
def = Preconditions.checkNotNull(current.definition().getAsUnknownStatementDefinition(def),
"Unable to create unknown statement definition of yang statement %s in unknown statement %s", def,
current);
}
// LOG.info("Qname: {}, argument: {}, StatementDefinitionContext: {}", name, argument, def);
// 处理到这个地方，正在处理的statement一定是本阶段能处理的statement
InferenceException.throwIfNull(def, ref, "Statement %s does not have type mapping defined.", name);
if (def.hasArgument()) {
SourceException.throwIfNull(argument, ref, "Statement %s requires an argument", name);
} else {
SourceException.throwIf(argument != null, ref, "Statement %s does not take argument", name);
}
/*
* If the current statement definition has argument specific
* sub-definitions, get argument specific sub-definition based on given
* argument (e.g. type statement need to be specialized based on its
* argument).
*/
if (def.hasArgumentSpecificSubDefinitions()) {
def = def.getSubDefinitionSpecificForArgument(argument);
}
// 非 root statement，即 非 modulestatement
if (current != null) {
return current.createSubstatement(childId, def, ref, argument);
}
/*
* If root is null or root version is other than default,
* we need to create new root.
*
* 如果current为null，表示当前遍历到的statement是yang模块的root statement，即module
*
*/
if (root == null) {
root = new RootStatementContext<>(this, def, ref, argument);
// LOG.info("rootStatement: {}, rootArgument: {}", root.definition().getStatementName(), root.rawStatementArgument());
} else if (!RootStatementContext.DEFAULT_VERSION.equals(root.getRootVersion())
&& inProgressPhase == ModelProcessingPhase.SOURCE_LINKAGE) {
root = new RootStatementContext<>(this, def, ref, argument, root.getRootVersion(), root.getRootIdentifier());
} else {
// root不是null，校验当前module的名字与root代表的module的名字需要一致
final QName rootStatement = root.definition().getStatementName();
final String rootArgument = root.rawStatementArgument();
Preconditions.checkState(Objects.equals(def.getStatementName(), rootStatement)
&& Objects.equals(argument, rootArgument),
"Root statement was already defined as '%s %s'.", rootStatement, rootArgument);
}
return root;
}
public final <CA, CD extends DeclaredStatement<CA>, CE extends EffectiveStatement<CA, CD>> StatementContextBase<CA, CD, CE> createSubstatement(
final int offset, final StatementDefinitionContext<CA, CD, CE> def, final StatementSourceReference ref,
final String argument) {
final ModelProcessingPhase inProgressPhase = getRoot().getSourceContext().getInProgressPhase();
Preconditions.checkState(inProgressPhase != ModelProcessingPhase.EFFECTIVE_MODEL,
"Declared statement cannot be added in effective phase at: %s", getStatementSourceReference());
final Optional<StatementContextBase<?, ?, ?>> implicitStatement = definition.beforeSubStatementCreated(this,
offset, def, ref, argument);
if (implicitStatement.isPresent()) {
return implicitStatement.get().createSubstatement(offset, def, ref, argument);
}
final StatementContextBase<CA, CD, CE> ret = new SubstatementContext<>(this, def, ref, argument);
substatements = substatements.put(offset, ret);
def.onStatementAdded(ret);
return ret;
}
可见，处理完一个节点后，根节点module对应的StatementContextWriter的StatementContextBase的值为RootStatementContext，其他的节点对应的StatementContextWriter的StatementContextBase的值为SubstatementContext，RootStatementContext和SubstatementContext有一个叫做substatements的成员，记录了该节点对应的孩子节点所对应的SubstatementContext，RootStatementContext和SubstatementContext都存储了对应statement的StatementSupport实例。这样遍历完后，就为每个yang module建立了一个schema树，这棵树的root节点存储在SourceSpecificContext的RootStatementContext<?, ?, ?> root字段中
最后调用BuildGlobalContext的transformEffective()
private EffectiveSchemaContext transformEffective() throws ReactorException {
Preconditions.checkState(finishedPhase == ModelProcessingPhase.EFFECTIVE_MODEL);
final List<DeclaredStatement<?>> rootStatements = new ArrayList<>(sources.size());
final List<EffectiveStatement<?, ?>> rootEffectiveStatements = new ArrayList<>(sources.size());
try {
for (final SourceSpecificContext source : sources) {
final RootStatementContext<?, ?, ?> root = source.getRoot();
try {
rootStatements.add(root.buildDeclared());
rootEffectiveStatements.add(root.buildEffective());
} catch (final RuntimeException ex) {
throw propagateException(source, ex);
}
}
} finally {
RecursiveObjectLeaker.cleanup();
}
sealMutableStatements();
return new EffectiveSchemaContext(rootStatements, rootEffectiveStatements);
}
其中调用了RootStatementContext的buildDeclared()和buildEffective()
@Override
public D buildDeclared() {
Preconditions.checkArgument(completedPhase == ModelProcessingPhase.FULL_DECLARATION
|| completedPhase == ModelProcessingPhase.EFFECTIVE_MODEL);
if (declaredInstance == null) {
declaredInstance = definition().getFactory().createDeclared(this);
}
return declaredInstance;
}
@Override
public E buildEffective() {
if (effectiveInstance == null) {
effectiveInstance = definition().getFactory().createEffective(this);
}
return effectiveInstance;
}
最终调用ModuleStatementSupport的createDeclared()和createEffective()
@Override
public ModuleStatement createDeclared(final StmtContext<String, ModuleStatement, ?> ctx) {
return new ModuleStatementImpl(ctx);
}
@Override
public EffectiveStatement<String, ModuleStatement> createEffective(
final StmtContext<String, ModuleStatement, EffectiveStatement<String, ModuleStatement>> ctx) {
return new ModuleEffectiveStatementImpl(ctx);
}
EffectiveSchemaContext构造函数内容：
public EffectiveSchemaContext(final List<DeclaredStatement<?>> rootDeclaredStatements,
final List<EffectiveStatement<?, ?>> rootEffectiveStatements) {
this.rootDeclaredStatements = ImmutableList.copyOf(rootDeclaredStatements);
this.rootEffectiveStatements = ImmutableList.copyOf(rootEffectiveStatements);
final Set<Module> modulesInit = new HashSet<>();
for (EffectiveStatement<?, ?> rootEffectiveStatement : rootEffectiveStatements) {
if (rootEffectiveStatement instanceof ModuleEffectiveStatementImpl) {
Module module = (Module) rootEffectiveStatement;
modulesInit.add(module);
}
}
this.modules = ImmutableSet.copyOf(ModuleDependencySort.sort(modulesInit));
final SetMultimap<URI, Module> nsMap = Multimaps.newSetMultimap(new TreeMap<>(), MODULE_SET_SUPPLIER);
final SetMultimap<String, Module> nameMap = Multimaps.newSetMultimap(new TreeMap<>(), MODULE_SET_SUPPLIER);
final Set<ModuleIdentifier> modIdBuilder = new HashSet<>();
for (Module m : modulesInit) {
nameMap.put(m.getName(), m);
nsMap.put(m.getNamespace(), m);
modIdBuilder.add(ModuleIdentifierImpl.create(m.getName(), Optional.of(m.getNamespace()),
Optional.of(m.getRevision())));
resolveSubmoduleIdentifiers(m.getSubmodules(), modIdBuilder);
}
namespaceToModules = ImmutableSetMultimap.copyOf(nsMap);
nameToModules = ImmutableSetMultimap.copyOf(nameMap);
moduleIdentifiers = ImmutableSet.copyOf(modIdBuilder);
}
1.3 总结
可见最后生成的SchemaContext是EffectiveSchemaContext的实例，EffectiveSchemaContext中保存了所有yang module的Module对象列表，同时保存了module statement对应的DeclaredStatement和EffectiveStatement的列表，module statement对应的DeclaredStatement是ModuleStatementImpl的实例，Module和EffectiveStatement是ModuleEffectiveStatementImpl的实例，ModuleStatementImpl和ModuleEffectiveStatementImpl类都有一个RootStatementContext字段，RootStatementContext保存了它的所有孩子节点，孩子节点对应的类为SubstatementContext，它同样保存了它的所有孩子节点，从而形成一颗schema树
常用的DeclaredStatement
ContactStatementImpl
ContainerStatementImpl
KeyStatementImpl
LeafListStatementImpl
LeafStatementImpl
ListStatementImpl
ModuleStatementImpl
NotificationStatementImpl
OrderedByStatementImpl
PrefixStatementImpl
RevisionStatementImpl
RpcStatementImpl
TypeStatementImpl
ValueStatementImpl
常用的EffectiveStatement(SchemaNode)
ContactEffectiveStatementImpl
ContainerEffectiveStatementImpl
KeyEffectiveStatementImpl
LeafEffectiveStatementImpl
LeafListEffectiveStatementImpl
ListEffectiveStatementImpl
ModuleEffectiveStatementImpl
NotificationEffectiveStatementImpl
OrderedByEffectiveStatementImpl
PrefixEffectiveStatementImpl
RevisionEffectiveStatementImpl
RpcEffectiveStatementImpl
TypeDefEffectiveStatementImpl
ValueEffectiveStatementImpl
