BigData~09:Sqoop
2019-05-18 本文已影响0人
帅可儿妞
使用 Sqoop 把数据从 Hive 导出到 MySQL,中间遇到了一些坑,现总结如下:
一、报错总结
- 在 CDH 上添加了 sqoop 服务之后:
[root@db1 ~]# sqoop help Warning: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Error: Could not find or load main class org.apache.sqoop.Sqoop
- 后来,在另外一个机器上做的,这个问题就没有遇到,遇到在添加吧
- -bash: --username: command not found
[root@db2 sqoop]# sqoop export \ > --connect jdbc:mysql://localhost:3306/fact?useUnicode=true&characterEncoding=utf8 \ > --username txdevfactroot \ > --password 'admin' \ > --table wechart_user_increase_fact \ > --input-fields-terminated-by '\001' \ > --update-key id \ > --update-mode allowinsert \ > --export-dir /user/hive/warehouse/test.db/user_increase_wechart_fact/ [1] 28195 -bash: --username: command not found [root@db2 sqoop]# Warning: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 19/05/18 12:22:44 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.15.1 Export requires a --table or a --call argument. Try --help for usage instructions. [1]+ Exit 1 sqoop export --connect jdbc:mysql://172.25.48.2:3306/fact?useUnicode=true
- 凡是遇到--参数找不到的,把上一条命令使用双引号括起来即可
- ERROR tool.ExportTool: Error during export:
19/05/18 10:52:28 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:439) at org.apache.sqoop.manager.MySQLManager.upsertTable(MySQLManager.java:145) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:85) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:111) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
- 当然了,这个原因应该有好多,我的原因是:hive 的表中没有 id,MySQL 中有 id,这个 id 是自增的,想当然的认为没有问题,后来问题一直解决不了,就在 Hive 得到表中添加了一个使用 UUID 的 id 字段,当然了,MySQL 中需要把自增干掉;