sparkcore进行wordcount词频统计
2019-05-20 本文已影响0人
大数据修行
准备一份数据/home/admin/data/helloworld.txt
hello world hello
hello world welcome
scala> val wc = sc.textFile("file:///home/admin/data/helloworld.txt")
scala> wc.flatMap(x=>x.split("\t")).map(x=>(x,1)).reduceByKey(_+_).collect()
res19: Array[(String, Int)] = Array((hello,3), (welcome,1), (world,2))
按照词频排序
scala> wc.flatMap(x=>x.split("\t")).map(x=>(x,1)).reduceByKey(_+_).sortBy(_._2,true).collect().foreach(println(_))
(welcome,1)
(world,2)
(hello,3)
