Apache Storm 笔记storm大数据，机器学习，人工智能

JStorm运行依赖storm-core的任务：依赖冲突与解决

2017-11-30  本文已影响98人  峰巢

发表于个人博客，原文发表地址。
JStorm虽然用Java语言重新实现了storm，但是对于storm的external部分并未实现。现在在JStorm基础上构建topology时，需要使用storm的external部分中的storm-sql-core以及storm-sql-runtime，这样在构建topology的项目中需要同时依赖storm-core、jstorm-core，出现了冲突。

本地调试问题

Found multiple defaults.yaml resources

根据前面描述的情况，maven的pom.xml文件将包含如下的依赖jstorm-core、storm-core、storm-sql-core、storm-sql-runtime。

<dependency>
  <groupId>com.alibaba.jstorm</groupId>
  <artifactId>jstorm-core</artifactId>
  <version>2.2.1</version>
  <!--本地调试时注释一下scope -->
  <!--<scope>provided</scope>-->
</dependency>
<dependency>
  <groupId>org.apache.storm</groupId>
  <artifactId>storm-core</artifactId>
  <version>1.1.1</version>
  <!--<scope>provided</scope>-->
</dependency>
<dependency>
  <groupId>org.apache.storm</groupId>
  <artifactId>storm-sql-core</artifactId>
  <version>1.1.1</version>
</dependency>
<dependency>
  <groupId>org.apache.storm</groupId>
  <artifactId>storm-sql-runtime</artifactId>
  <version>1.1.1</version>
  <!--<scope>provided</scope>-->
</dependency>

此时按照JStorm本地调试的模式在ide中运行，

LocalCluster cluster = new LocalCluster();
cluster.submitTopology(TOPOLOGY_NAME, config, builder.createTopology());
Utils.sleep(20000);
cluster.killTopology(TOPOLOGY_NAME);
cluster.shutdown();

出现如下报错：

Exception in thread "main" java.lang.ExceptionInInitializerError
at backtype.storm.topology.BaseConfigurationDeclarer.(BaseConfigurationDeclarer.java:29)
at backtype.storm.topology.TopologyBuilder$ConfigGetter.(TopologyBuilder.java:433)
at backtype.storm.topology.TopologyBuilder$SpoutGetter.(TopologyBuilder.java:450)
at backtype.storm.topology.TopologyBuilder.setSpout(TopologyBuilder.java:289)
at com.ctg.itrdc.ruleengine.JsonTopology.main(JsonTopology.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.RuntimeException: Invalid configuration defaults.yaml:Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/E:/Program%20Files%20(x86)/apache-maven-3.3.1-bin/repos/repository/com/alibaba/jstorm/jstorm-core/2.2.1/jstorm-core-2.2.1.jar!/defaults.yaml, jar:file:/E:/Program%20Files%20(x86)/apache-maven-3.3.1-bin/repos/repository/org/apache/storm/storm-core/1.1.1/storm-core-1.1.1.jar!/defaults.yaml]
at com.alibaba.jstorm.utils.LoadConf.findAndReadYaml(LoadConf.java:77)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:355)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:453)
at backtype.storm.utils.Utils.(Utils.java:112)
... 10 more
Caused by: java.io.IOException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/E:/Program%20Files%20(x86)/apache-maven-3.3.1-bin/repos/repository/com/alibaba/jstorm/jstorm-core/2.2.1/jstorm-core-2.2.1.jar!/defaults.yaml, jar:file:/E:/Program%20Files%20(x86)/apache-maven-3.3.1-bin/repos/repository/org/apache/storm/storm-core/1.1.1/storm-core-1.1.1.jar!/defaults.yaml]
at com.alibaba.jstorm.utils.LoadConf.getConfigFileInputStream(LoadConf.java:101)
at com.alibaba.jstorm.utils.LoadConf.findAndReadYaml(LoadConf.java:55)
... 13 more

可以看出是因为同时依赖了jstorm-core、storm-core导致存在多个配置文件加载出错。

解决方法

解决的方法比较暴力，pom.xml文件不需要做修改，保持jstorm-core、storm-core等被注释了，找到冲突的本地maven仓库中的/repository/org/apache/storm/storm-core/1.1.1/storm-core-1.1.1.jar，将jar包下的defaults.yaml删除。

defaults.yaml删除
采用本地模式提交topology，注意有关提交topology的类，依赖自JStorm中的import backtype.storm.xxx、不要继承自import org.apache.storm.xxx。以上，主要是删除storm中有冲突的配置文件defaults.yaml。

集群模式问题

问题1:Invalid signature file digest for Manifest main attributes

修改pom.xml，将jstorm-core、storm-core设置为<scope>provided</scope>，修改本地提交代码为集群提交：
StormSubmitter. submitTopology(topo_name , config, builder.createTopology() );
打包后去集群上执行，提交失败，报错：
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

$ ./jstorm jar ~/topology/ruleengine-0.0.1-SNAPSHOT.jar com.ctg.itrdc.ruleengine.JsonTopology ruleengine2
/usr/java/jdk1.8.0_111/bin/java
Exception in thread "main" java.lang.ExceptionInInitializerError
    at backtype.storm.command.config_value.main(config_value.java:40)
Caused by: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
    at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:314)
    at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:268)
    at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
    at java.util.jar.JarVerifier.update(JarVerifier.java:228)
    at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
    at java.util.jar.JarFile.getInputStream(JarFile.java:450)
    at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:940)
    at sun.misc.Resource.cachedInputStream(Resource.java:77)
    at sun.misc.Resource.getByteBuffer(Resource.java:160)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at backtype.storm.utils.Utils.<clinit>(Utils.java:103)
    ... 1 more
Failed to get config java.library.path
None

ruleengine2
cannot concatenate 'str' and 'NoneType' objects
Syntax: [jstorm jar topology-jar-path class ...]

    Runs the main method of class with the specified arguments.
    The jstorm jars and configs in $JSTORM_CONF_DIR/storm.yaml are put on the classpath.
    The process is configured so that StormSubmitter
    (https://github.com/alibaba/jstorm/wiki/JStorm-Chinese-Documentation)
    will upload the jar at topology-jar-path when the topology is submitted.

解决方法1

谷歌后发现，Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes错误是maven打包时设置的问题，修改打包配置如下：

<build>
    <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>1.7.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.handlers</resource>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>META-INF/spring.schemas</resource>
                                </transformer>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.ctg.itrdc.ruleengine.JsonTopology</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
    </plugins>
</build>

主要是添加如下部分。

<filters>
    <filter>
        <artifact>*:*</artifact>
        <excludes>
            <exclude>META-INF/*.SF</exclude>
            <exclude>META-INF/*.DSA</exclude>
            <exclude>META-INF/*.RSA</exclude>
        </excludes>
    </filter>
</filters>

问题 2:storm-core类加载失败

按照在Git上的看到的回复，集群运行时，设置依赖的时候jstorm和storm的依赖应该都设置成provided。
解决了问题1后，成功提交topology，能够成功运行，在web-UI中查看日志也没有出错，但是topology运行的结果不对。

猜测是storm-sql-core以及storm-sql-runtime依赖了storm-core，但是因为jstorm-core、storm-core设置为<scope>provided</scope>，但是JStorm集群中只有jstorm-core是provided，而storm-core仍然是没有提供。

坑的是类加载失败错误是直接system.out.print，而不是会打印log，所以在Web-UI中是看不到具体错误的。
继续按照猜测走下去，在pom.xml中将storm-core的<scope>provided</scope>注释掉:

<!--<scope>provided</scope>-->

打包提交，成功加载了storm-core的类，且topology正常运行没有报错,问题解决。

上一篇 下一篇

猜你喜欢

热点阅读