01.Hive报错篇-DDL后查询数据失败

Hive执行DDL操作后的常见错误，错误分析及处理方式

1. 创建一张orc表，并且向该表插入数据，这时候再新增字段，插入新的数据，然后count这张表的时候就会报该异常！报错如下：

2021-03-03 11:13:01,989 INFO [IPC Server handler 24 on 33697] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1584525046250_5922717_m_000001_0: Error: java.lang.RuntimeException: Error creating a batch
    at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.createValue(VectorizedOrcInputFormat.java:111)
    at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.createValue(VectorizedOrcInputFormat.java:49)
    at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createValue(CombineHiveRecordReader.java:94)
    at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createValue(CombineHiveRecordReader.java:43)
    at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.createValue(HadoopShimsSecure.java:155)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.createValue(MapTask.java:180)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: No type found for column type entry 11
    at org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.addScratchColumnsToBatch(VectorizedRowBatchCtx.java:604)
    at org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.createVectorizedRowBatch(VectorizedRowBatchCtx.java:339)
    at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcInputFormat$VectorizedOrcRecordReader.createValue(VectorizedOrcInputFormat.java:109)

处理方案：

• 方案一： 在执行sql前执行set hive.vectorized.execution.enabled = false;
• 方案二：修复分区
  • 第一种情况：一层分区的情况；执行 MSCK REPAIR TABLE table_name;
  • 第二种情况：多层分区情况；执行 set hive.msck.path.validation=ignore; MSCK REPAIR TABLE table_name;

2. orc分区表，修改字段类型，查询历史分区数据会报错，报错如下：

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {2019-12-05, 757664, 390, 10448, 3, 102, 2017-08-05 00:03:54.0, , null, null, , , 0, null, , , null, , , null, 102, 2017-08-05 00:03:54.0, 2017-08-05 00:03:54.0, , 1, haier, 2017-08-04 22:44:07.0, null, 2019-12-05 18:34:59.0, null, 10448, 2020-06-02 15:31:40, 2020-06-02 15:31:40, 2020-06-02 15:42:08, 2020-11-02 18:24:01, null}
  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:100)
  at org.apache.hadoop.hive.ql.exec.MapOperator.process(Ma
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {2019-12-05, 757664, 390, 10448, 3, 102, 2017-08-05 00:03:54.0, , null, null, , , 0, null, , , null, , , null, 102, 2017-08-05 00:03:54.0, 2017-08-05 00:03:54.0, , 1, haier, 2017-08-04 22:44:07.0, null, 2019-12-05 18:34:59.0, null, 10448, 2020-06-02 15:31:40, 2020-06-02 15:31:40, 2020-06-02 15:42:08, 2020-11-02 18:24:01, null}
  at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:505)
  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
  ... 8 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.IntWritable
  at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
  at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getLong(PrimitiveObjectInspectorUtils.java:671)
  at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$LongConverter.convert(PrimitiveObjectInspectorConverter.java:180)pOperator.java:496)
  at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:406)
  at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:137)

原因为：单纯修改字段类型，不会涉及到历史分区元数据信息的修改；
处理方案：

• 添加动态分区，执行修改语句

-- 首先开启动态分区
SET hive.exec.dynamic.partition = true;  
-- 指定分区修改字段类型
ALTER TABLE tab_name partition(pdate) change oldCol newCol newType comment '';

• 添加级联参数，级联执行

ALTER TABLE tab_name change oldCol newCol newType comment '' CASCADE;