大牛分享爬取高德地图poi数据实战
高德地图搜索poi的api介绍地址
当前想法是爬取目标区域（作者所在小县城）的所有poi数据，存到数据库中作为原始数据，然后供其它系统调用，因为之前爬取过百度地图的poi数据，所以这次工作就驾轻就熟了。
1、首先注册一个高德地图的开发者账号，申请一个绑定Web服务的key，然后把刚注册的开发者账号认证一下： 申请账号、key就不赘述了，去高德地图开发平台很简单就能完成了，将账号认证是为了提高每日访问高德地图api接口的次数限制和并发请求。
2、根据上方api地址里面的介绍，总共分为4中搜索： 关键字搜索：通过用POI的关键字进行条件搜索，例如：肯德基、朝阳公园等；同时支持设置POI类型搜索，例如：银行 周边搜索：在用户传入经纬度坐标点附近，在设定的范围内，按照关键字或POI类型搜索； 多边形搜索：在多边形区域内进行搜索 ID查询：通过POI ID，查询某个POI详情，建议可同输入提示API配合使用
我的目标是某个区域的所有poi，所以选择的第三种：多边形搜索
3、多边形搜索最重要的参数就是polygon-》经纬度坐标对，我在百度地图坐标拾取系统拾取了我的目标区域的经纬度坐标对，如下图：
3步准备工作到这里就差不多结束了，在正式开始码代码之前先做个测试吧，用浏览器直接访问接口看看返回的数据（当然，高德的api接口有返回数据说明）
如上图，这里比较重要的一个属性是count，根据api的介绍count是搜索方案数目(最大值为1000)，所以说每次请求都会返回当前所搜所包含的poi个数，而大于1000的poi是没有办法获取到的。那么我如果想查询某个区域的全部数据，可以将这个区域再划分成更小的区域（显然是个递归操作）的集合，然后把这几个可以查到所有poi的区域的所有poi数据结合起来就是我最终需要的数据。可能口述不明朗，可以见下方草图：
如果需要Python方面的入门知识可以点击这个链接获取入门资料
好，可以开始撸代码了：
因为，整个调用API的过程都离不开经纬度，所以首先定义一个经纬度描述的类 `
//矩形块的经纬度标识， 左上角的经纬度 和右下角的经纬度classRectangleCoordinate{/**
* 矩形左上角经度
*/privatedoublex0;/**
* 矩形左上角纬度
*/privatedoubley0;/**
* 矩形右下角经度
*/privatedoublex1;/**
* 矩形右下角纬度
*/privatedoubley1;publicRectangleCoordinate(doublex0,doubley0,doublex1,doubley1){this.x0 = x0;this.y0 = y0;this.x1 = x1;this.y1 = y1; }/**
* [@return](https://my.oschina.net/u/556800) 获取矩形中心线的纬度
*/publicdoublegetAverageY(){return(y0 + y1) /2; }/**
* [@return](https://my.oschina.net/u/556800) 获取矩形中心线的经度
*/publicdoublegetAverageX(){return(x0 + x1) /2; }publicdoublegetX0(){returnx0; }publicvoidsetX0(doublex0){this.x0 = x0; }publicdoublegetY0(){returny0; }publicvoidsetY0(doubley0){this.y0 = y0; }publicdoublegetX1(){returnx1; }publicvoidsetX1(doublex1){this.x1 = x1; }publicdoublegetY1(){returny1; }publicvoidsetY1(doubley1){this.y1 = y1; } [@Override](https://my.oschina.net/u/1162528)publicStringtoString(){returnx0 +","+ y0 +"|"+ x1 +","+ y1; }}`
然后需要一个调用api，获取返回数据的方法，这个方法参数就是矩形块，当然还需要一个页数，即当前方法获取的是某个矩形区域的第X页的数据（每页上线25个poi，默认20个poi）
/** *@return获取矩形块的poi数据 */privateJSONObjectgetSearchResult(RectangleCoordinate coordinate,intpage){ RestTemplate restTemplate =newRestTemplate(); String url = getRequestGaodeUrl(coordinate,page); String result = restTemplate.getForObject(url, String.class);try{try{ Thread.sleep(50); }catch(InterruptedException e) { e.printStackTrace(); }returnJSONObject.parseObject(result); }catch(Exception e) { logger.error("an error occurred when getting response of gaode map data for coordinate:[{}]", coordinate.toString()); }returnnull; }
当然，上方已经说过，如果矩形块返回数据count=1000，就说明当前矩形块需要分割，我的想法比较简单，将矩形块按照上方草图，在水平中心和垂直分心分割，1个矩形块就分割成4个小矩形块了，方法如下：
/** *@return将矩形4等分成小矩形 然后返回4个 小矩形的经纬度集合 */privateListgetSplitRectangleList(RectangleCoordinate coordinate){ List splitRectangleList =newLinkedList<>(); splitRectangleList.add(newRectangleCoordinate(coordinate.getX0(), coordinate.getY0(), coordinate.getAverageX(), coordinate.getAverageY())); splitRectangleList.add(newRectangleCoordinate(coordinate.getAverageX(), coordinate.getY0(), coordinate.getX1(), coordinate.getAverageY())); splitRectangleList.add(newRectangleCoordinate(coordinate.getX0(), coordinate.getAverageY(), coordinate.getAverageX(), coordinate.getY1())); splitRectangleList.add(newRectangleCoordinate(coordinate.getAverageX(), coordinate.getAverageY(), coordinate.getX1(), coordinate.getY1()));returnsplitRectangleList; }
目前，可以获取到矩形区域经纬度对的集合了，也有获取api数据的方法了，然后就是遍历页数获取数据，自定义操作数据。 当某次分页请求返回的poi个数小于每页最大个数的时候就认为当前区域poi已经完全请求到了。
privatevoidstartAnaMainGaode(RectangleCoordinate coordinate)throwsAnalysisException{//当前爬取的数据的页数索引intpage_num =0;//当前爬取内容是否是最后一页booleanisLastPage =false; JSONObject searchResult; JSONArray datas =null; logger.info("ready to analysis coordinate:[{}]", coordinate.toString());while(!isLastPage) { logger.info("is going to get data for page_"+ page_num);try{ searchResult = getSearchResult(coordinate, page_num); datas = searchResult.getJSONArray("pois"); }catch(Exception e) { logger.error("an error occurred when getting response of gaode map data for coordinate:[{}]", coordinate.toString()); }if(datas !=null&& datas.size() <20) { isLastPage =true; logger.info("get result counts is [{}], now page index is [{}]", datas.size(), page_num); } saveIntoDbGaode(datas); page_num++; } }
privatevoidsaveIntoDbGaode(JSONArray result){ JSONObject resultItem;for(inti =0; i < result.size(); i++) { resultItem = result.getJSONObject(i);try{ results.add(getInsertUnitObject(resultItem)); }catch(Exception e) { logger.error("生成数据时异常，e: {}", e.getMessage()); e.printStackTrace(); } }if(results.size() > BATCHINSERTLIMIT || ISLAST) { logger.info("is ready to batch insert into unit, total count is {}", results.size());try{ dao.batchAddUnitGaode(results); }catch(Exception e) { logger.error("更新数据库异常，e: {}", e.getMessage()); } results =newJSONArray(); }}`
到此，基本方法都介绍过了，全部代码如下（因为都是简单方法和逻辑，不明白的留言交流）
//请求入口 publicvoidGaodePoiSearch(){//徐水区 final RectangleCoordinate searchAreaCoordinate = new RectangleCoordinate(115.521773, 39.106335, 115.801182, 38.943988); //保定市//final RectangleCoordinate searchAreaCoordinate = new RectangleCoordinate(114.332719,39.574064, 116.588688,38.179144);List validCoordinate = getValidCoordinate(searchAreaCoordinate); logger.info("get all valid coordinate,size is [{}]", validCoordinate.size());/**
* 获取到所有的小方块之后可以做一些处理， 比如存储到某个地方，以防发生异常，方便后面重新遍历，我这里暂未做处理
*/validCoordinate.forEach(coor -> {try{ startAnaMainGaode(coor); }catch(AnalysisException e) { e.printStackTrace(); } }); ISLAST =true; saveIntoDbGaode(newJSONArray());}/** * [@return](https://my.oschina.net/u/556800) 获取矩形块中 符合 调用api的 小矩形块的集合 * 因为高德地图某个矩形块只能获取前1000条，所以要将矩形块分割成可以获取到全部数据的矩形块 * 如果当前矩形块请求数据返回的count<1000 即为符合条件的，否则将矩形块4等分 然后递归 */privateListgetValidCoordinate(RectangleCoordinate coordinate){ List validCoordinate =newLinkedList<>(); JSONObject searchResult = getSearchResult(coordinate,0);if(searchResult.getIntValue("count") >=1000) { List splitRectangleList = getSplitRectangleList(coordinate); splitRectangleList.forEach(coor -> validCoordinate.addAll(getValidCoordinate(coor))); }else{ logger.info("add a valid coordinate [{}]", coordinate.toString()); validCoordinate.add(coordinate); }returnvalidCoordinate;}/** * [@return](https://my.oschina.net/u/556800) 将矩形4等分成小矩形 然后返回4个 小矩形的经纬度集合 */privateListgetSplitRectangleList(RectangleCoordinate coordinate){ List splitRectangleList =newLinkedList<>(); splitRectangleList.add(newRectangleCoordinate(coordinate.getX0(), coordinate.getY0(), coordinate.getAverageX(), coordinate.getAverageY())); splitRectangleList.add(newRectangleCoordinate(coordinate.getAverageX(), coordinate.getY0(), coordinate.getX1(), coordinate.getAverageY())); splitRectangleList.add(newRectangleCoordinate(coordinate.getX0(), coordinate.getAverageY(), coordinate.getAverageX(), coordinate.getY1())); splitRectangleList.add(newRectangleCoordinate(coordinate.getAverageX(), coordinate.getAverageY(), coordinate.getX1(), coordinate.getY1()));returnsplitRectangleList;}/** *@return获取矩形块的poi数据 */privateJSONObjectgetSearchResult(RectangleCoordinate coordinate,intpage){ RestTemplate restTemplate =newRestTemplate(); String url = getRequestGaodeUrl(coordinate,page); String result = restTemplate.getForObject(url, String.class);try{try{ Thread.sleep(50); }catch(InterruptedException e) { e.printStackTrace(); }returnJSONObject.parseObject(result); }catch(Exception e) { logger.error("an error occurred when getting response of gaode map data for coordinate:[{}]", coordinate.toString()); }returnnull;}privatevoidstartAnaMainGaode(RectangleCoordinate coordinate)throwsAnalysisException{//当前爬取的数据的页数索引intpage_num =0;//当前爬取内容是否是最后一页booleanisLastPage =false; JSONObject searchResult; JSONArray datas =null; logger.info("ready to analysis coordinate:[{}]", coordinate.toString());while(!isLastPage) { logger.info("is going to get data for page_"+ page_num);try{ searchResult = getSearchResult(coordinate, page_num); datas = searchResult.getJSONArray("pois"); }catch(Exception e) { logger.error("an error occurred when getting response of gaode map data for coordinate:[{}]", coordinate.toString()); }if(datas !=null&& datas.size() <20) { isLastPage =true; logger.info("get result counts is [{}], now page index is [{}]", datas.size(), page_num); } saveIntoDbGaode(datas); page_num++; }}privatevoidsaveIntoDbGaode(JSONArray result){ JSONObject resultItem;for(inti =0; i < result.size(); i++) { resultItem = result.getJSONObject(i);try{ results.add(getInsertUnitObject(resultItem)); }catch(Exception e) { logger.error("生成数据时异常，e: {}", e.getMessage()); e.printStackTrace(); } }if(results.size() > BATCHINSERTLIMIT || ISLAST) { logger.info("is ready to batch insert into unit, total count is {}", results.size());try{ dao.batchAddUnitGaode(results); }catch(Exception e) { logger.error("更新数据库异常，e: {}", e.getMessage()); } results =newJSONArray(); }}privateJSONObjectgetInsertUnitObject(JSONObject resultItem){ JSONObject unitDataObject =newJSONObject(); unitDataObject.put("uid", resultItem.getString("id")); unitDataObject.put("name", resultItem.getString("name")); unitDataObject.put("type", resultItem.getString("type")); unitDataObject.put("tag", resultItem.getString("type")); unitDataObject.put("address", resultItem.getString("address")); unitDataObject.put("province", resultItem.getString("pname")); unitDataObject.put("city", resultItem.getString("cityname")); unitDataObject.put("area", resultItem.getString("adname")); String tel = resultItem.getString("tel");if(tel !=null&& !"[]".equals(tel)) { unitDataObject.put("telephone", tel); }try{ JSONArray url = resultItem.getJSONArray("website");if(url !=null&& url.size() >0) { unitDataObject.put("detail_url", url.getString(0)); } }catch(Exception e) { unitDataObject.put("detail_url", resultItem.getString("website")); } JSONArray photos = resultItem.getJSONArray("photos");if(photos !=null&& photos.size() >0) { StringBuilder images =newStringBuilder();for(intj =0; j < photos.size(); j++) { images.append(j ==0?"":";").append(photos.getJSONObject(j).getString("url")); } unitDataObject.put("images", images.toString()); } String entr_location = resultItem.getString("location");if(StringUtils.isEmpty(entr_location)) { entr_location = resultItem.getString("entr_location"); }if(!StringUtils.isEmpty(entr_location)) { unitDataObject.put("lng", entr_location.split(",")[0]); unitDataObject.put("lat", entr_location.split(",")[1]); }returnunitDataObject;}privateStringgetRequestGaodeUrl(RectangleCoordinate coordinate,intpage){return"https://restapi.amap.com/v3/place/polygon?"+"key=xxxxxxxxxxxxxxxxxxxxxxx&polygon="+ coordinate.toString() +"&page="+ page +"&types=010000|"+"010100|010101|010102|010103|010104|010105|010107|010108|010109|010110|010111|010112|010200|010300|010400|"+"010401|010500|010600|010700|010800|010900|010901|011000|011100|020000|020100|020101|020102|020103|020104|"+"020105|020106|020200|020201|020202|020203|020300|020301|020400|020401|020402|020403|020404|020405|020406|"+"020407|020408|020600|020601|020602|020700|020701|020702|020703|020800|020900|020904|020905|021000|021001|"+"021002|021003|021004|021100|021200|021201|021202|021203|021300|021301|021400|021401|021500|021501|021600|"+"021601|021602|021700|021701|021702|021800|021802|021803|021804|021900|022000|022100|022200|022300|022301|"+"022400|022500|022501|022502|022600|022700|022800|022900|023000|023100|023200|023300|023301|023400|023500|"+"025000|025100|025200|025300|025400|025500|025600|025700|025800|025900|026000|026100|026200|026300|029900|"+"030000|030100|030200|030201|030202|030203|030204|030205|030206|030300|030301|030302|030303|030400|030401|"+"030500|030501|030502|030503|030504|030505|030506|030507|030508|030700|030701|030702|030800|030801|030802|"+"030803|030900|031000|031004|031005|031100|031101|031102|031103|031104|031200|031300|031301|031302|031303|"+"031400|031401|031500|031501|031600|031601|031700|031701|031702|031800|031801|031802|031900|031902|031903|"+"031904|032000|032100|032200|032300|032400|032401|032500|032600|032601|032602|032700|032800|032900|033000|"+"033100|033200|033300|033400|033401|033500|033600|035000|035100|035200|035300|035400|035500|035600|035700|"+"035800|035900|036000|036100|036200|036300|039900|040000|040100|040101|040200|040201|050000|050100|050101|"+"050102|050103|050104|050105|050106|050107|050108|050109|050110|050111|050112|050113|050114|050115|050116|"+"050117|050118|050119|050120|050121|050122|050123|050200|050201|050202|050203|050204|050205|050206|050207|"+"050208|050209|050210|050211|050212|050213|050214|050215|050216|050217|050300|050301|050302|050303|050304|"+"050305|050306|050307|050308|050309|050310|050311|050400|050500|050501|050502|050503|050504|050600|050700|"+"050800|050900|060000|060100|060101|060102|060103|060200|060201|060202|060300|060301|060302|060303|060304|"+"060305|060306|060307|060308|060400|060401|060402|060403|060404|060405|060406|060407|060408|060409|060411|"+"060413|060414|060415|060500|060501|060502|060600|060601|060602|060603|060604|060605|060606|060700|060701|"+"060702|060703|060704|060705|060706|060800|060900|060901|060902|060903|060904|060905|060906|060907|061000|"+"061001|061100|061101|061102|061103|061104|061200|061201|061202|061203|061204|061205|061206|061207|061208|"+"061209|061210|061211|061212|061213|061214|061300|061301|061302|061400|061401|070000|070100|070200|070201|"+"070202|070203|070300|070301|070302|070303|070304|070305|070306|070400|070401|070500|070501|070600|070601|"+"070603|070604|070605|070606|070607|070608|070609|070610|070700|070701|070702|070703|070704|070705|070706|"+"070800|070900|071000|071100|071200|071300|071400|071500|071600|071700|071800|071801|071900|071901|071902|"+"071903|072000|072001|080000|080100|080101|080102|080103|080104|080105|080106|080107|080108|080109|080110|"+"080111|080112|080113|080114|080115|080116|080117|080118|080119|080200|080201|080202|080300|080301|080302|"+"080303|080304|080305|080306|080307|080308|080400|080401|080402|080500|080501|080502|080503|080504|080505|"+"080600|080601|080602|080603|090000|090100|090101|090102|090200|090201|090202|090203|090204|090205|090206|"+"090207|090208|090209|090210|090211|090300|090400|090500|090600|090601|090602|090700|090701|090702|100000|"+"100100|100101|100102|100103|100104|100105|100200|100201|110000|110100|110101|110102|110103|110104|110105|"+"110106|110200|110201|110202|110203|110204|110205|110206|110207|110208|110209|120000|120100|120200|120201|"+"120202|120203|120300|120301|120302|120303|120304|130000|130100|130101|130102|130103|130104|130105|130106|"+"130107|130200|130201|130202|130300|130400|130401|130402|130403|130404|130405|130406|130407|130408|130409|"+"130500|130501|130502|130503|130504|130505|130506|130600|130601|130602|130603|130604|130605|130606|130700|"+"130701|130702|130703|140000|140100|140101|140102|140200|140201|140300|140400|140500|140600|140700|140800|"+"140900|141000|141100|141101|141102|141103|141104|141105|141200|141201|141202|141203|141204|141205|141206|"+"141207|141300|141400|141500|150000|150100|150101|150102|150104|150105|150106|150107|150200|150201|150202|"+"150203|150204|150205|150206|150207|150208|150209|150210|150300|150301|150302|150303|150304|150400|150500|"+"150501|150600|150700|150701|150702|150703|150800|150900|150903|150904|150905|150906|150907|150908|150909|"+"151000|151100|151200|151300|160000|160100|160101|160102|160103|160104|160105|160106|160107|160108|160109|"+"160110|160111|160112|160113|160114|160115|160117|160118|160119|160120|160121|160122|160123|160124|160125|"+"160126|160127|160128|160129|160130|160131|160132|160133|160134|160135|160136|160137|160138|160139|160140|"+"160141|160142|160143|160144|160145|160146|160147|160148|160149|160150|160151|160152|160200|160300|160301|"+"160302|160303|160304|160305|160306|160307|160308|160309|160310|160311|160312|160314|160315|160316|160317|"+"160318|160319|160320|160321|160322|160323|160324|160325|160326|160327|160328|160329|160330|160331|160332|"+"160333|160334|160335|160336|160337|160338|160339|160340|160341|160342|160343|160344|160345|160346|160347|"+"160348|160349|160400|160401|160402|160403|160404|160405|160406|160407|160408|160500|160501|160600|170000|"+"170100|170200|170201|170202|170203|170204|170205|170206|170207|170208|170209|170300|170400|170401|170402|"+"170403|170404|170405|170406|170407|170408|180000|180100|180101|180102|180103|180104|180200|180201|180202|"+"180203|180300|180301|180302|180400|180500|190000|190100|190101|190102|190103|190104|190105|190106|190107|"+"190108|190109|190200|190201|190202|190203|190204|190205|190300|190301|190302|190303|190304|190305|190306|"+"190307|190308|190309|190310|190311|190400|190401|190402|190403|190500|190600|190700|200000|200100|200200|"+"200300|200301|200302|200303|200304|200400|220000|220100|220101|220102|220103|220104|220105|220106|220107|"+"220200|220201|220202|220203|220204|220205|970000|990000|991000|991001|991400|991401|991500&extensions=all";}/**
* 矩形块的经纬度标识， 左上角的经纬度 和右下角的经纬度
*/classRectangleCoordinate{/**
* 矩形左上角经度
*/privatedoublex0;/**
* 矩形左上角纬度
*/privatedoubley0;/**
* 矩形右下角经度
*/privatedoublex1;/**
* 矩形右下角纬度
*/privatedoubley1;publicRectangleCoordinate(doublex0,doubley0,doublex1,doubley1){this.x0 = x0;this.y0 = y0;this.x1 = x1;this.y1 = y1; }/** *@return获取矩形中心线的纬度 */publicdoublegetAverageY(){return(y0 + y1) /2; }/** *@return获取矩形中心线的经度 */publicdoublegetAverageX(){return(x0 + x1) /2; }publicdoublegetX0(){returnx0; }publicvoidsetX0(doublex0){this.x0 = x0; }publicdoublegetY0(){returny0; }publicvoidsetY0(doubley0){this.y0 = y0; }publicdoublegetX1(){returnx1; }publicvoidsetX1(doublex1){this.x1 = x1; }publicdoublegetY1(){returny1; }publicvoidsetY1(doubley1){this.y1 = y1; }@OverridepublicStringtoString(){returnx0 +","+ y0 +"|"+ x1 +","+ y1; }}`
更新（2018-09-20）：
1、时间问题，当前50ms请求一次api接口，跑完小县城的数据（几万条）大概需要十分钟左右吧，把整个市区主要数据跑完断断续续的用了一天吧，最后跑了近27W数据
原文链接：https://my.oschina.net/u/1417838/blog/2054570
