Hive日志分析实战

2018-03-14 本文已影响66人明明德撩码

对日志进行分析思路

日志内容格式

"27.38.5.159" "-" "31/Aug/2015:00:04:37 +0800" "GET /course/view.php?id=27 HTTP/1.1" "303" "440" - "http://www.ibeifeng.com/user.php?act=mycourse" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" "-" "learn.ibeifeng.com"

注意440 后面的- 没有被引号包围。创建

创建利用正则格式化数据的表

CREATE TABLE bf_log_src (
host STRING,
identity STRING,
user STRING,
time STRING,
request STRING,
status STRING,
size STRING,
referer STRING,
agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([^]) ([^]) ([^]) (-|\[^\]\]) ([^ "]|"[^"]") (-|[0-9]) (-|[0-9])(?: ([^ "]|".") ([^ "]|"."))?"
)
STORED AS TEXTFILE;