hive压缩
2022-08-18 本文已影响0人
后知不觉1
hive支持的文件格式
- hive版本 2.1.1
- hadoop版本 2.7.3
| 文件格式 | 理论压缩格式 | 当前集群支持情况 | 建表语句 |
|---|---|---|---|
| textfile | DEFLATE | 支持 | 无法通过建表属性加载压缩格式，自动压缩。需要全局开启压缩，输出压缩格式或者在session中设置#hive配置开启压缩\nset hive.exec.compress.intermediate=true; set hive.exec.compress.output=true; #指定引擎压缩\nset mapreduce.map.output.compress=true;set mapreduce.output.fileoutputformat.compress=true;\nset mapreduce.output.fileoutputformat.compress.type=BLOCK;\n#指定压缩化类\nsetmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.BZip2Codec;\nset mapreduce.output.fileoutputformat.compress.codec = org.apache.hadoop.io.compress.BZip2Codec;\n |
| gzip | 支持 | ||
| bzip2 | 支持 | ||
| orc | none（不压缩） | 支持 | CREATE TABLE compress_orc_none (name string, age string,roles string,gender string,work string,regine string) STORED AS orc tblproperties ("orc.compress"="none"); |
| ZLIB | 支持，默认 | CREATE TABLE compress_orc_zlib(name string, age string,roles string,gender string,work string,regine string) STORED AS orc tblproperties ("orc.compress"="ZLIB") | |
| SNAPPY | 支持 | CREATE TABLE compress_orc_snappy(name string, age string,roles string,gender string,work string,regine string) STORED AS orc tblproperties ("orc.compress"="SNAPPY") | |
| LZO | 否，需配置 | CREATE TABLE compress_orc_lzo(name string, age string,roles string,gender string,work string,regine string) STORED AS orc tblproperties ("orc.compress"="LZO") | |
| parquet | none（不压缩） | 支持，默认 | CREATE TABLE compress_parquet_snappy(name string, age string,roles string,gender string,work string,regine string) STORED AS PARQUET |
| SNAPPY | 支持 | CREATE TABLE compress_parquet_snappy(name string, age string,roles string,gender string,work string,regine string) STORED AS PARQUET tblproperties ("parquet.compression"="SNAPPY") | |
| GZIP | 支持 | CREATE TABLE compress_parquet_gzip(name string, age string,roles string,gender string,work string,regine string) STORED AS PARQUET tblproperties ("parquet.compression"="gzip") | |
| LZO | 否，需配置 | CREATE TABLE compress_parquet_snappy(name string, age string,roles string,gender string,work string,regine string) STORED AS PARQUET tblproperties ("parquet.compression"="lzo") |
