Java爬虫中HttpClient请求头的设置

在做爬虫的时候发现如果不设置请求头的话，每次httpclient发起的请求都是响应移动端板式的，无法抓取响应的内容，后面查了一下才知道，需要重新设置请求头，让服务端误以为是某个真实浏览器发起的请求：

HttpClient httpClient = new DefaultHttpClient();    
    
//創建一個httpGet方法     
    
HttpGet httpGet = new HttpGet("xxxxx");    
    
//設置httpGet的头部參數信息     
    
httpGet.setHeader("Accept", "Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");    
    
httpGet.setHeader("Accept-Charset", "GB2312,utf-8;q=0.7,*;q=0.7");    
    
httpGet.setHeader("Accept-Encoding", "gzip, deflate");    
    
httpGet.setHeader("Accept-Language", "zh-cn,zh;q=0.5");    
    
httpGet.setHeader("Connection", "keep-alive");    
    
httpGet.setHeader("Cookie", "");    
    
httpGet.setHeader("Host", "");    
    
httpGet.setHeader("refer", "");    
    
httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2");   

最基本的爬虫就用了几句代码就完成了，附上爬虫代码，主要是用到HttpClient、webmagic、jsoup

public class WebMagicService {
    public static void main(String[] args)  throws Exception{
        HttpClient httpClient = HttpClients.createDefault();
        HttpGet httpGet = new HttpGet("http://bbs.e763.com/");
        httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2");
        HttpResponse response = httpClient.execute(httpGet);
        String contents = EntityUtils.toString(response.getEntity(),"gbk");//utf-8
        Document document = Jsoup.parse(contents);
        Elements elements = document.select("div#hza11 div.boxtxthot a");
     //   System.out.println(contents);
        for (Element element : elements) {
            System.out.println(element.text()+ " : " + element.attr("href"));
        }
    }
}


爬取内容如下：
三大运营商9月1日起取消漫游费 用户无需申请自动生效 : viewthread.php?tid=907629
2年前月桂湖弃婴案：被蚂蚁咬过的孩子， 精神发育迟滞 : viewthread.php?tid=907607
.....