一、多字段

字段实现精确

• 增加一个keyword字段

使用不同的analyzer

• 不同语言
• pinyin字段的搜索
• 还支持为搜索和索引指定不同的analyzer

二、Exact Values vs Full Text

Exact Values：包括数字/日期/具体一个字符串（例如“Apple Store”）

• ES 中的 keyword

全文本，非结构化的文本数据

• ES中的text

  
  image.png

Exact Values 不需要被分词

• ES为每一个字段创建一个倒排索引

• Exact Value在索引时，不需要做特殊的分词处理

  
  image.png
• Character Filters
• Tokenizer
• Token Filter

三、自定义分词

当es自带的分词器无法满足时，可以自定义分词器。通过自组合不同的组件实现

3.1 Character Filters

在Tokenizer之前对文本进行处理，例如增加删除及替换字符。可以配置多个 Character Filters。会影响Tokenizer和offset信息

一些自带的 Character Filters

• HTML script -- 去除html标签
• Mapping -- 字符串替换
• Pattern replace -- 正则匹配替换

3.2 Tokenizer

• 将原始的文本按照一定的规则，切分为词（term or token）
• ES内置的Tokenizers
  whitespace / standard / uax_url_email / pattern / keyword / path hierarchy
• 可以用java 开发组件实现自己的 Tokenizer

3.3 Token Filters

• 将Tokenizer 输出的单词（term），进行增加，修改，删除

• 自带的 Token Filters
  Lowercase / stop /synonym (添加 近义词)

  
  image.png

词的执行过程
1：分析器的大体执行过程
char filter - >tokenizer -> token filter
2：分词的时机
分词在索引时做，也就是数据写入时。目的是创建倒排索引提高搜索速度。写入的原始数据保存在_source中

image.png

PUT logs/_doc/1
{"level":"DEBUG"}

GET /logs/_mapping

POST _analyze
{
  "tokenizer":"keyword",
  "char_filter":["html_strip"],
  "text": "<b>hello world</b>"
}


POST _analyze
{
  "tokenizer":"path_hierarchy",
  "text":"/user/ymruan/a/b/c/d/e"
}

#使用char filter进行替换
POST _analyze
{
  "tokenizer": "standard",
  "char_filter": [
      {
        "type" : "mapping",
        "mappings" : [ "- => _"]
      }
    ],
  "text": "123-456, I-test! test-990 650-555-1234"
}

//char filter 替换表情符号
POST _analyze
{
  "tokenizer": "standard",
  "char_filter": [
      {
        "type" : "mapping",
        "mappings" : [ ":) => happy", ":( => sad"]
      }
    ],
    "text": ["I am felling :)", "Feeling :( today"]
}

// white space and snowball
GET _analyze
{
  "tokenizer": "whitespace",
  "filter": ["stop","snowball"],
  "text": ["The gilrs in China are playing this game!"]
}


// whitespace与stop
GET _analyze
{
  "tokenizer": "whitespace",
  "filter": ["stop","snowball"],
  "text": ["The rain in Spain falls mainly on the plain."]
}


//remove 加入lowercase后，The被当成 stopword删除
GET _analyze
{
  "tokenizer": "whitespace",
  "filter": ["lowercase","stop","snowball"],
  "text": ["The gilrs in China are playing this game!"]
}

//正则表达式
GET _analyze
{
  "tokenizer": "standard",
  "char_filter": [
      {
        "type" : "pattern_replace",
        "pattern" : "http://(.*)",
        "replacement" : "$1"
      }
    ],
    "text" : "http://www.elastic.co"
}

自定义分析器 示例

自定义分析器标准格式是：
PUT /my_index
{
    "settings": {
        "analysis": {
            "char_filter": { ... custom character filters ... },//字符过滤器
            "tokenizer": { ... custom tokenizers ... },//分词器
            "filter": { ... custom token filters ... }, //词单元过滤器
            "analyzer": { ... custom analyzers ... }
        }
    }
}
============================实例===========================
PUT /my_index
{
    "settings": {
        "analysis": {
            "char_filter": {
                "&_to_and": {
                    "type": "mapping",
                    "mappings": [ "&=> and "]
            }},
            "filter": {
                "my_stopwords": {
                    "type": "stop",
                    "stopwords": [ "the", "a" ]
            }},
            "analyzer": {
                "my_analyzer": {
                    "type": "custom",
                    "char_filter": [ "html_strip", "&_to_and" ],
                    "tokenizer": "standard",
                    "filter": [ "lowercase", "my_stopwords" ]
            }}
}}}


============================实例===========================
比如自定义好的analyzer名字是my_analyzer,在此索引下的某个新增字段应用此分析器
PUT /my_index/_mapping
{
   "properties":{
        "username":{
             "type":"text",
              "analyzer" : "my_analyzer"
         },
        "password" : {
          "type" : "text"
        }
    
  }
}
=================插入数据====================
PUT /my_index/_doc/1
{
  "username":"The quick & brown fox ",
   "password":"The quick & brown fox "


}
====username采用自定义分析器my_analyzer,password采用默认的standard分析器==
===验证
GET /index_v1/_analyze
{
  "field":"username",
  "text":"The quick & brown fox"
}

GET /index_v1/_analyze
{
  "field":"password",
  "text":"The quick & brown fox"
}
//官网权威指南是真的讲得好，虽然版本太老，Elasticsearch 2.x 版本，一些api已经不适用了，自定义分析器地址：https://www.elastic.co/guide/cn/elasticsearch/guide/cn/custom-analyzers.html