Transformer模型的学习总结
2018-12-13 本文已影响2087人
奔向算法的喵
Transformer来自Google团队17年的文章Attention is all you need。
该文章的目的：减少计算量并且提高并行效率，同时不减弱最终的实验效果。
创新点：Transformer只采用了attention机制。不像传统的encoder-decoder的模型需要结合RNN或者CNN来使用。创新之处在于使用了scaled Dot-product Attention和Multi-Head Attention。
我觉得将Transformer解释的最容易懂的的还是The illustrated transformer，看完这篇博客就瞬间懂了很多东西。然后哈佛大学也给出了详细的pytorch版本的代码，有jupyter notebook详细的解释，看完也会有别样的收获哈。
一、原理分析
1、模型的架构图如下：
Attentin is all you need文中给出的架构图如下，能够比较详细的看到了一个encoder和decoder的细节。在这里，可以看到一个encoder里面有2个子层，然后一个decoder中含有3个子层，后面会详细说到这个结构。
然后我们从全局的视野来看，encoder和decoder部分都包含了6个encoder和decoder。进入到第一个encoder的inputs结合embedding和positional embedding。通过了6个encoder之后，输出到了decoder部分的每一个decoder中。
2、Encoder部分
Transformer的两个创新的结构图如下，明白了两个部分，后面的东西就清楚多了。
上面的Q，K和V，被作为一种抽象的向量，主要目的是用来做计算和辅助attention。根据文章我们知道Attention的计算公式如下:
在这里说明一下，在tensorflow的代码里面，维度变成了(batch_size, max_len, vector_dimension)。其实就是增加了一个batch_size也就是输入的句子数，然后max_len就是控制的一句话的长度，最后是字向量的维度大小。
上面说的是从下图中x1经过self-attention到了z1的状态，通过了self-attetion的张量还需要进过残差网络和LaterNorm的处理，然后进入到全连接的前馈网络中，前馈网络也是同样的操作，进行的残差处理和正规化。最后输出的张量才真正的进入到了下一个encoder之中，然后这样的操作，经过了6次，然后最后就能进入到decoder的部分了。
可以从上图中看出，在向量进入self-attention层之前，是将词的embedding和位置的encoding做了一个相加的处理。因为模型里面没有用到RNN和CNN的东西，所以该论文采用了位置的编码来解决序列信息获取的问题。这里的positional encoding需要说明一下：
最后一个decoder输出的向量会经过Linear层和softmax层。Linear层的作用就是对decoder部分出来的向量做映射成一个logits向量，然后softmax层根据这个logits向量，将其转换为了概率值，最后找到概率最大值的位置。这样就完成了解码的输出了。
二、代码分析
代码分析来自于https://github.com/EternalFeather/Transformer-in-generating-dialogue，该文件里面的文件组成和作用如下：
| 文件名字 | 作用 |
|---|---|
| params.py | 定义了该模型里面的所用到的超参数，比如学习率、隐藏单元葛个数等。 |
| make_dic.py | 用来做数据预处理的，作用是生成源语言和目标语言的Vocabulary文件。 |
| data_load.py | 该文件包含所有关于加载数据以及批量化数据的函数。 |
| modules.py | 核心代码部分，包括了embedding和positional embedding、、以及multihead-attention、正则化等函数 |
| train.py | 训练模型的代码，定义了模型，损失函数以及训练和保存模型的过程 |
| eval.py | 模型训练完之后，评估模型的性能 |
1、明确该模型的一些超参数(可调)
# -*- coding: utf-8 -*-
class Params:
'''
Parameters of our model
'''
src_train = "data/src-train.txt"
tgt_train = "data/tgt-train.txt"
src_test = "data/src-val.txt"
tgt_test = "data/tgt-val.txt"
num_identical = 6
maxlen = 10
hidden_units = 512
num_heads = 8
logdir = 'logdir'
batch_size = 32
num_epochs = 250
dropout = 0.1
learning_rate = 0.0001
word_limit_size = 20
word_limit_lower = 3
checkpoint = 'checkpoint'
| 参数名 | 大小 | 对应到代码里面 | 意义 |
|---|---|---|---|
| batch_size | 32 | N | 批量的大小 |
| learning_rate | 0.001 | lr | 学习率的大小 |
| maxlen | 10 | T,T_q,T_k | 一句话最长多少个字 |
| word_limit_size | 20 | 出现次数小于20的话，那么认作UNK | |
| hidden_units | 512 | num_units,S | 隐藏单元的个数、维度大小 |
| num_identical | 6 | encoder和decoder部分叠加的个数 | |
| num_epochs | 250 | 训练的时候，总的epochs数目 | |
| num_heads | 8 | multi-head attention里面的头数 | |
| dropout_rate | 0.1 | dropout大小 |
2、数据的预处理
- make_dic.py
from __future__ import print_function
from params import Params as pm
import codecs
import os
from collections import Counter
def make_dic(path, fname):
'''
Constructs vocabulary as a dictionary
Args:
path: [String], Input file path
fname: [String], Output file name
Build vocabulary line by line to dictionary/ path
'''
text = codecs.open(path, 'r', 'utf-8').read() #codes.open()得到的是一个对象，然后read()之后就变成了字符串了
words = text.split()
wordCount = Counter(words)
if not os.path.exists('dictionary'):
os.mkdir('dictionary')
with codecs.open('dictionary/{}'.format(fname), 'w', 'utf-8') as f:
f.write("{}\t1000000000\n{}\t1000000000\n{}\t1000000000\n{}\t1000000000\n".format("<PAD>","<UNK>","<STR>","<EOS>"))
for word, count in wordCount.most_common(len(wordCount)):
f.write(u"{}\t{}\n".format(word, count))
if __name__ == '__main__':
make_dic(pm.src_train, "en.vocab.tsv")
make_dic(pm.tgt_train, "de.vocab.tsv")
print("MSG : Constructing Dictionary Finished!")
运行这个文件之后，我们得到了en.vocab和de.vocab两个文件。en.vocab部分结果如下:
<PAD> 1000000000
<UNK> 1000000000
<STR> 1000000000
<EOS> 1000000000
有 17300
的 15767
` 12757
- 10831
卦 8461
八 7865
麼 7771
沒 7324
嗎 6024
是 5940
......
ASCII 1
SAISONduSOLEIL 1
豌 1
迺 1
ThuDec2223 1
snis 1
Ya 1
2100 1
雇 1
主要作用就是统计了一下出现单词的次数，然后按照出现的次数进行了一个排序，用于后续的data_load模块的操作
- data_load.py
# -*- coding: utf-8 -*-
from __future__ import print_function
from params import Params as pm
import codecs
import sys
import numpy as np
import tensorflow as tf
def load_vocab(vocab): # 'en.vocab.tsv' 'de.vocab.tsv'
'''
Load word token from encoding dictionary
Args:
vocab: [String], vocabulary files
'''
vocab = [line.split()[0] for line in codecs.open('dictionary/{}'.format(vocab), 'r', 'utf-8').read().splitlines() if int(line.split()[1]) >= pm.word_limit_size]
word2idx_dic = {word: idx for idx, word in enumerate(vocab)}
idx2word_dic = {idx: word for idx, word in enumerate(vocab)}
return word2idx_dic, idx2word_dic
load_vocab这个函数的作用就是处理前面根据词频产生的文档，进过处理之后：
#en.vocab:
word2idx ={'<PAD>': 0, '<UNK>': 1, '<STR>': 2, '<EOS>': 3, '有': 4,
'的': 5, '`': 6, '-': 7, '卦': 8, '八': 9, ..., '爬': 1642, 'U': 1643}
idx2word={{0: '<PAD>', 1: '<UNK>', 2: '<STR>', 3: '<EOS>', 4: '有',
5: '的', 6: '`', 7: '-', 8: '卦', 9: '八', ..., 1642: '爬', 1643: 'U'}}
#de.vocab
word2idx={'<PAD>': 0, '<UNK>': 1, '<STR>': 2, '<EOS>': 3, '-': 4,
'的': 5, '`': 6, '不': 7, '人': 8, '好': 9, ..., '遺': 1586, '搜': 1587}
idx2word={0: '<PAD>', 1: '<UNK>', 2: '<STR>',3:'<EOS>', 4: '-',
5: '的', 6: '`', 7: '不', 8: '人', 9: '好', ..., 1586: '遺', 1587: '搜'}
接着看generate_dataset这个函数，它的作用就是产生数据集，也就是将句子表示成了np array的形式。传入的是source句子和target句子。这里的集外词的id给为1。函数的前半部分做的是一个index化，后半部分做的是padding处理，我们可以知道<pad>符号在Vocabulary里面的位置为0，所以当句子的长度小于10的时候，我们就在不足的位置给加填上0，保证每个句子的长度都是相同的。
最终返回的X，Y的维度都是(句子数，最大的句子的长度)
def generate_dataset(source_sents, target_sents):
'''
Parse source sentences and target sentences from corpus with some formats
Parse word token of each sentences
Args:
source_sents: [List], encoding sentences from src-train file
target_sents: [List], decoding sentences from tgt-train file
Padding for word token sentence list
'''
en2idx, idx2en = load_vocab('en.vocab.tsv')
de2idx, idx2de = load_vocab('de.vocab.tsv')
in_list, out_list, Sources, Targets = [], [], [], []
for source_sent, target_sent in zip(source_sents, target_sents):
# 1 means <UNK>
inpt = [en2idx.get(word, 1) for word in (source_sent + u" <EOS>").split()]
outpt = [de2idx.get(word, 1) for word in (target_sent + u" <EOS>").split()]
if max(len(inpt), len(outpt)) <= pm.maxlen:
# sentence token list
in_list.append(np.array(inpt))
out_list.append(np.array(outpt))
# sentence list
Sources.append(source_sent)
Targets.append(target_sent)
X = np.zeros([len(in_list), pm.maxlen], np.int32)
Y = np.zeros([len(out_list), pm.maxlen], np.int32)
for i, (x, y) in enumerate(zip(in_list, out_list)):
X[i] = np.lib.pad(x, (0, pm.maxlen - len(x)), 'constant', constant_values = (0, 0))
Y[i] = np.lib.pad(y, (0, pm.maxlen - len(y)), 'constant', constant_values = (0, 0))
return X, Y, Sources, Targets
load_data这个函数的作用就是给generate_dataset里面提供参数用的。
def load_data(l_data):
'''
Read train-data from input datasets
Args:
l_data: [String], the file name of datasets which used to generate tokens
'''
if l_data == 'train':
en_sents = [line for line in codecs.open(pm.src_train, 'r', 'utf-8').read().split('\n') if line]
de_sents = [line for line in codecs.open(pm.tgt_train, 'r', 'utf-8').read().split('\n') if line]
if len(en_sents) == len(de_sents):
inpt, outpt, Sources, Targets = generate_dataset(en_sents, de_sents)
else:
print("MSG : Source length is different from Target length.")
sys.exit(0)
return inpt, outpt
elif l_data == 'test':
en_sents = [line for line in codecs.open(pm.src_test, 'r', 'utf-8').read().split('\n') if line]
de_sents = [line for line in codecs.open(pm.tgt_test, 'r', 'utf-8').read().split('\n') if line]
if len(en_sents) == len(de_sents):
inpt, outpt, Sources, Targets = generate_dataset(en_sents, de_sents)
else:
print("MSG : Source length is different from Target length.")
sys.exit(0)
return inpt, Sources, Targets
else:
print("MSG : Error when load data.")
sys.exit(0)
利用到了前面的load_data这个函数，参数可以是train或者test。
这里面的inpt, outpt维度(句子的总数, maxlen)，那么batch_num就是我们总共的要训练的批量数。
输出x,y的维度(N, T)==(batch_size, maxlen)
def get_batch_data():
'''
A batch dataset generator
'''
inpt, outpt = load_data("train")
batch_num = len(inpt) // pm.batch_size
inpt = tf.convert_to_tensor(inpt, tf.int32)
outpt = tf.convert_to_tensor(outpt, tf.int32)
# parsing data into queue used for pipeline operations as a generator.
input_queues = tf.train.slice_input_producer([inpt, outpt])
# multi-thread processing using batch
x, y = tf.train.shuffle_batch(input_queues,
num_threads = 8,
batch_size = pm.batch_size,
capacity = pm.batch_size * 64,
min_after_dequeue = pm.batch_size * 32,
allow_smaller_final_batch = False)
return x, y, batch_num
3、核心部分-modules.py
# -*- coding: utf-8 -*-
from __future__ import print_function
import tensorflow as tf
import numpy as np
import math
def normalize(inputs, epsilon = 1e-8, scope = "ln", reuse = None):
'''
Implement layer normalization
Args:
inputs: [Tensor], A tensor with two or more dimensions, where the first one is "batch_size"
epsilon: [Float], A small number for preventing ZeroDivision Error
scope: [String], Optional scope for "variable_scope"
reuse: [Boolean], If to reuse the weights of a previous layer by the same name
Returns:
A tensor with the same shape and data type as "inputs"
'''
with tf.variable_scope(scope, reuse = reuse):
inputs_shape = inputs.get_shape() # a.get_shape().as_list() --->a维度是(2,3)，那么这个返回就是 [2, 3]
params_shape = inputs_shape[-1 :] # params_shape就是最后的一个维度了
# tf.nn.moments 计算返回的 mean 和 variance 作为 tf.nn.batch_normalization 参数调用。
mean, variance = tf.nn.moments(inputs, [-1], keep_dims = True)
beta = tf.Variable(tf.zeros(params_shape))
gamma = tf.Variable(tf.ones(params_shape))
normalized = (inputs - mean) / ((variance + epsilon) ** (.5))
outputs = gamma * normalized + beta
return outputs
不论是self-attention还是feed-forward neural network，都会做一个正规化的操作。这里参数epsilon就是为了防止正规化的时候，分母为0，然后scope为命名空间。看代码就知道，这个和batch normalization的公式是一样的。目的肯定是使得训练过程，反向传播的时候的震荡幅度更小。好多复杂的网络在输出之后都会加上normalization。
def positional_encoding(inputs,
vocab_size,
num_units,
zero_pad = True,
scale = True,
scope = "positional_embedding",
reuse = None):
'''
Positional_Encoding for a given tensor.
Args:
inputs: [Tensor], A tensor contains the ids to be search from the lookup table, shape = [batch_size, 1 + len(inpt)]
vocab_size: [Int], Vocabulary size
num_units: [Int], Hidden size of embedding
zero_pad: [Boolean], If True, all the values of the first row(id = 0) should be constant zero
scale: [Boolean], If True, the output will be multiplied by sqrt num_units(check details from paper)
scope: [String], Optional scope for 'variable_scope'
reuse: [Boolean], If to reuse the weights of a previous layer by the same name
Returns:
A 'Tensor' with one more rank than inputs's, with the dimensionality should be 'num_units'
'''
"""
inputs (batch_size, 1+len(inputs)) 那么N就是batch_size, 然后T就是maxlen，大小为10
num_units 就是隐层单元的个数，维度的大小
"""
N, T = inputs.get_shape().as_list()
with tf.variable_scope(scope, reuse = reuse):
position_ind = tf.tile(tf.expand_dims(tf.range(T), 0), [N, 1])
# First part of the PE function: sin and cos argument
position_enc = np.array([
[pos / np.power(10000, 2.*i/num_units) for i in range(num_units)]
for pos in range(T)])
# Second part, apply the cosine to even columns and sin to odds.
position_enc[:, 0::2] = np.sin(position_enc[:, 0::2]) # dim 2i
position_enc[:, 1::2] = np.cos(position_enc[:, 1::2]) # dim 2i+1
# Convert to a tensor
lookup_table = tf.convert_to_tensor(position_enc)
if zero_pad:
lookup_table = tf.concat((tf.zeros(shape=[1, num_units]),
lookup_table[1:, :]), 0)
outputs = tf.nn.embedding_lookup(lookup_table, position_ind)
if scale:
outputs = outputs * num_units**0.5
return tf.cast(outputs, tf.float32)
这一部分的代码就是架构图里面的positional embedding的实现了，由于没有采用序列模型，那么这里讲位置进行了嵌入，就捕捉到了位置上面的信息。
def embedding(inputs,
vocab_size,
num_units,
zero_pad = True,
scale = True,
scope = "embedding",
reuse = None):
'''
Embed a given tensor.
Args:
inputs: [Tensor], A tensor contains the ids to be search from the lookup table
vocab_size: [Int], Vocabulary size
num_units: [Int], Hidden size of embedding
zero_pad: [Boolean], If True, all the values of the first row(id = 0) should be constant zero
scale: [Boolean], If True, the output will be multiplied by sqrt num_units(check details from paper)
scope: [String], Optional scope for 'variable_scope'
reuse: [Boolean], If to reuse the weights of a previous layer by the same name
Returns:
A 'Tensor' with one more rank than inputs's, with the dimensionality should be 'num_units'
'''
"""
inputs传进来就(batch_size, 10)
lookup_table维度(vocab_size, 512)，进行了随机的初始化
"""
# shape = [vocabsize, 8]
with tf.variable_scope(scope, reuse = reuse):
lookup_table = tf.get_variable('lookup_table',
dtype = tf.float32,
shape = [vocab_size, num_units],
initializer = tf.contrib.layers.xavier_initializer())
if zero_pad:
''' tf.zeros 维度(1, 512)
lookup_table[1:, :]的目的是抛开了<PAD>这玩意儿，赋值为0，然后进行了合并
现在look _table维度还是(vocab_size, 512 )
'''
lookup_table = tf.concat((tf.zeros(shape = [1, num_units]), lookup_table[1:, :]), 0)
# outputs 维度就是 (batch_size, 10, 512) ==[N ,T, S]
outputs = tf.nn.embedding_lookup(lookup_table, inputs)
if scale:
# embedding 那一步
outputs = outputs * math.sqrt(num_units)
return outputs
输入维度(batch_size, maxlen)==[N, T]
输出维度(batch_size, maxlen, S)==[N, T, S]
这里面注意初始化lookup_table时把id=0的那一行（第一行）初始化为全0的结果。scale为True，paper中在embedding里面说明为什么这里需要做一个scale的操作。
下面是multi-head attention，为该代码的核心部分。注释里面写清楚了维度的一个变化情况。
最后输出维度[N, T_q, S]。
def multihead_attention(queries,
keys,
num_units = None,
num_heads = 8,
dropout_rate = 0,
is_training = True,
causality = False,
scope = "multihead_attention",
reuse = None):
'''
Implement multihead attention
Args:
queries: [Tensor], A 3-dimensions tensor with shape of [N, T_q, S_q]
keys: [Tensor], A 3-dimensions tensor with shape of [N, T_k, S_k]
num_units: [Int], Attention size
num_heads: [Int], Number of heads
dropout_rate: [Float], A ratio of dropout
is_training: [Boolean], If true, controller of mechanism for dropout
causality: [Boolean], If true, units that reference the future are masked
scope: [String], Optional scope for "variable_scope"
reuse: [Boolean], If to reuse the weights of a previous layer by the same name
Returns:
A 3-dimensions tensor with shape of [N, T_q, S]
'''
""" queries = self.enc (batch_size, 10 ,512)==[N, T_q, S] keys也是self.enc
num_units =512, num_heads =10
"""
with tf.variable_scope(scope, reuse = reuse):
if num_units is None:
# length of sentence
num_units = queries.get_shape().as_list()[-1]
""" Linear layers in Figure 2(right) 就是Q、K、V进入scaled Dot-product Attention前的Linear的操作
# 首先是进行了全连接的线性变换
shape = [N, T_q, S] (batch_size, 10 ,512)， S可以理解为512"""
Q = tf.layers.dense(queries, num_units, activation = tf.nn.relu)
# shape = [N, T_k, S]
K = tf.layers.dense(keys, num_units, activation = tf.nn.relu)
# shape = [N, T_k, S]
V = tf.layers.dense(keys, num_units, activation = tf.nn.relu)
'''
Q_、K_、V_就是权重WQ、WK、WV。
shape (batch_size*8， 10, 512/8=64)
'''
# Split and concat
# shape = [N*h, T_q, S/h]
Q_ = tf.concat(tf.split(Q, num_heads, axis = 2), axis = 0)
# shape = [N*h, T_k, S/h]
K_ = tf.concat(tf.split(K, num_heads, axis = 2), axis = 0)
# shape = [N*h, T_k, S/h]
V_ = tf.concat(tf.split(V, num_heads, axis = 2), axis = 0)
# [N, T_q, S] * [N*h, T_k, S/h] 这一步的张量乘法是怎么做的？
# shape = [N*h, T_q, T_k] Q
outputs = tf.matmul(Q_, tf.transpose(K_, [0, 2, 1]))
# Scale
outputs = outputs / (K_.get_shape().as_list()[-1] ** 0.5)
# Masking
# shape = [N, T_k]
# 这里的tf.reduce_sum进行了降维，由三维降低到了2维度，然后是取绝对值，转成0-1之间的值
'''[N, T_k, 512]------> [N, T_k] -----》[N*h, T_k] -----》[N*h, T_q, T_k] '''
key_masks = tf.sign(tf.abs(tf.reduce_sum(keys, axis = -1)))
# shape = [N*h, T_k]
key_masks = tf.tile(key_masks, [num_heads, 1])
# shape = [N*h, T_q, T_k] tf.expand_dims就是扩维度
key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(queries)[1], 1])
# If key_masks == 0 outputs = [1]*length(outputs)
paddings = tf.ones_like(outputs) * (-math.pow(2, 32) + 1)
# shape = [N*h, T_q, T_k]
outputs = tf.where(tf.equal(key_masks, 0), paddings, outputs)
if causality: #如果为true的话，那么就是将这个东西未来的units给屏蔽了
# reduce dims : shape = [T_q, T_k]
diag_vals = tf.ones_like(outputs[0, :, :])
# shape = [T_q, T_k]
# use triangular matrix to ignore the affect from future words
# like : [[1,0,0]
# [1,2,0]
# [1,2,3]]
tril = tf.contrib.linalg.LinearOperatorTriL(diag_vals).to_dense()
# shape = [N*h, T_q, T_k]
masks = tf.tile(tf.expand_dims(tril, 0), [tf.shape(outputs)[0], 1, 1])
paddings = tf.ones_like(masks) * (-math.pow(2, 32) + 1)
# shape = [N*h, T_q, T_k]
outputs = tf.where(tf.equal(masks, 0), paddings, outputs)
# Output Activation
outputs = tf.nn.softmax(outputs)
# Query Masking
# shape = [N, T_q]
query_masks = tf.sign(tf.abs(tf.reduce_sum(queries, axis = -1)))
# shape = [N*h, T_q]
query_masks = tf.tile(query_masks, [num_heads, 1])
# shape = [N*h, T_q, T_k]
query_masks = tf.tile(tf.expand_dims(query_masks, -1), [1, 1, tf.shape(keys)[1]])
outputs *= query_masks
# Dropouts
outputs = tf.layers.dropout(outputs, rate = dropout_rate, training = tf.convert_to_tensor(is_training))
# Weighted sum
# shape = [N*h, T_q, S/h]
outputs = tf.matmul(outputs, V_)
# Restore shape
# shape = [N, T_q, S]
outputs = tf.concat(tf.split(outputs, num_heads, axis = 0), axis = 2)
# Residual connection
outputs += queries
# Normalize
# shape = [N, T_q, S]
outputs = normalize(outputs)
return outputs
两层卷积之间加了relu非线性操作。之后是residual操作加上inputs残差，然后是normalize。最后输出的维度还是[N, T_q, S]。
def feedforward(inputs,
num_units = [2048, 512],
scope = "multihead_attention",
reuse = None):
'''
Position-wise feed forward neural network
Args:
inputs: [Tensor], A 3d tensor with shape [N, T, S]
num_units: [Int], A list of convolution parameters
scope: [String], Optional scope for "variable_scope"
reuse: [Boolean], If to reuse the weights of a previous layer by the same name
Return:
A tensor converted by feedforward layers from inputs
'''
with tf.variable_scope(scope, reuse = reuse):
# params = {"inputs": inputs, "filters": num_units[0], "kernel_size": 1, \
# "activation": tf.nn.relu, "use_bias": True}
# outputs = tf.layers.conv1d(inputs = inputs, filters = num_units[0], kernel_size = 1, activation = tf.nn.relu, use_bias = True)
# outputs = tf.layers.conv1d(**params)
params = {"inputs": inputs, "num_outputs": num_units[0], \
"activation_fn": tf.nn.relu}
outputs = tf.contrib.layers.fully_connected(**params)
# params = {"inputs": inputs, "filters": num_units[1], "kernel_size": 1, \
# "activation": None, "use_bias": True}
params = {"inputs": inputs, "num_outputs": num_units[1], \
"activation_fn": None}
# outputs = tf.layers.conv1d(inputs = inputs, filters = num_units[1], kernel_size = 1, activation = None, use_bias = True)
# outputs = tf.layers.conv1d(**params)
outputs = tf.contrib.layers.fully_connected(**params)
# residual connection
outputs += inputs
outputs = normalize(outputs)
return outputs
最后是进行了一个平滑的操作，就是one_hot中的0改成了一个很小的数，1改成了一个比较接近于1的数。
def label_smoothing(inputs, epsilon = 0.1):
'''
Implement label smoothing
Args:
inputs: [Tensor], A 3d tensor with shape of [N, T, V]
epsilon: [Float], Smoothing rate
Return:
A tensor after smoothing
'''
''' inputs的维度应该是(batch_size, sentense_length, vector dimension)
N就是batch_size, T就是句子的长度，V就是向量的维度大小
'''
K = inputs.get_shape().as_list()[-1]
return ((1 - epsilon) * inputs) + (epsilon / K)
4、训练
- train.py
这里面的self.decoder_input采取了一个操作就是将每个句子加了一个初始化为2的id，然后除去了最后的一个句子结束符。然后它的维度还是[N ,T]
# -*- coding: utf-8 -*-
from __future__ import print_function
import tensorflow as tf
from params import Params as pm
from data_loader import get_batch_data, load_vocab
from modules import *
from tqdm import tqdm
import os
class Graph():
# 直接就是一个init初始化一下
def __init__(self, is_training = True):
self.graph = tf.Graph()
with self.graph.as_default():
if is_training:
self.inpt, self.outpt, self.batch_num = get_batch_data()
else:
'''inpt(None, maxlen) outpt(None, maxlen) maxlen=10'''
self.inpt = tf.placeholder(tf.int32, shape = (None, pm.maxlen))
self.outpt = tf.placeholder(tf.int32, shape = (None, pm.maxlen))
# start with 2(<STR>) and without 3(<EOS>)
self.decoder_input = tf.concat((tf.ones_like(self.outpt[:, :1])*2, self.outpt[:, :-1]), -1)
# 直接就拿到了en和de的Vocabulary，en的大小是1644，de的大小是1588
en2idx, idx2en = load_vocab('en.vocab.tsv')
de2idx, idx2de = load_vocab('de.vocab.tsv')
# Encoder
with tf.variable_scope("encoder"):
''' self.inpt维度是(batch_size, maxlen)
self.enc 维度是(batch_size, maxlen, 512)
'''
self.enc = embedding(self.inpt,
vocab_size = len(en2idx),
num_units = pm.hidden_units,
scale = True,
scope = "enc_embed")
# Position Encoding(use range from 0 to len(inpt) to represent position dim of each words)
# tf.tile(tf.expand_dims(tf.range(tf.shape(self.inpt)[1]), 0), [tf.shape(self.inpt)[0], 1]),
self.enc += positional_encoding(self.inpt,
vocab_size = pm.maxlen,
num_units = pm.hidden_units,
zero_pad = False,
scale = False,
scope = "enc_pe")
# Dropout
self.enc = tf.layers.dropout(self.enc,
rate = pm.dropout,
training = tf.convert_to_tensor(is_training))
# Identical
for i in range(pm.num_identical):
with tf.variable_scope("num_identical_{}".format(i)):
# Multi-head Attention
self.enc = multihead_attention(queries = self.enc,
keys = self.enc,
num_units = pm.hidden_units,
num_heads = pm.num_heads,
dropout_rate = pm.dropout,
is_training = is_training,
causality = False)
self.enc = feedforward(self.enc, num_units = [4 * pm.hidden_units, pm.hidden_units])
下面就是decoder部分的代码。这里可以参考前面decoder的结构，里面多出了一个attention部分，该部分接受到了encoder输出的张量和decoder中self-attention里面输入的张量，然后再进行了vanilla attention。
最终decoder部分输出张量的维度是[N ,T, 512]
# Decoder
with tf.variable_scope("decoder"):
self.dec = embedding(self.decoder_input,
vocab_size = len(de2idx),
num_units = pm.hidden_units,
scale = True,
scope = "dec_embed")
# Position Encoding(use range from 0 to len(inpt) to represent position dim)
self.dec += positional_encoding(self.decoder_input,
vocab_size = pm.maxlen,
num_units = pm.hidden_units,
zero_pad = False,
scale = False,
scope = "dec_pe")
# Dropout
self.dec = tf.layers.dropout(self.dec,
rate = pm.dropout,
training = tf.convert_to_tensor(is_training))
# Identical
for i in range(pm.num_identical):
with tf.variable_scope("num_identical_{}".format(i)):
# Multi-head Attention(self-attention)
self.dec = multihead_attention(queries = self.dec,
keys = self.dec,
num_units = pm.hidden_units,
num_heads = pm.num_heads,
dropout_rate = pm.dropout,
is_training = is_training,
causality = True,
scope = "self_attention")
# Multi-head Attention(vanilla-attention)
self.dec = multihead_attention(queries=self.dec,
keys=self.enc,
num_units=pm.hidden_units,
num_heads=pm.num_heads,
dropout_rate=pm.dropout,
is_training=is_training,
causality=False,
scope="vanilla_attention")
self.dec = feedforward(self.dec, num_units = [4 * pm.hidden_units, pm.hidden_units])
现在已经走到了decoder部分输出了：
self.logits：进行了Linear变化，维度是[N, T, len(de2idx)]
self.preds：取了self.logits里面最后一个维度里面最大值的下标，维度是[n ,T]
self.istarget：将self.preds中所有id不为0的位置的值用1.0代替，维度是[n ,T]
self.acc: 对比self.preds, self.outpt，对应位置相等那么就是1.0，否则就是0。
# Linear
self.logits = tf.layers.dense(self.dec, len(de2idx))
self.preds = tf.to_int32(tf.arg_max(self.logits, dimension = -1))
self.istarget = tf.to_float(tf.not_equal(self.outpt, 0))
self.acc = tf.reduce_sum(tf.to_float(tf.equal(self.preds, self.outpt)) * self.istarget) / (tf.reduce_sum(self.istarget))
tf.summary.scalar('acc', self.acc)
is_training 为True的时候，也就是训练的时候，就需要进行下面的操作了。
loss的维度是[N, T]
if is_training:
# smooth inputs
self.y_smoothed = label_smoothing(tf.one_hot(self.outpt, depth = len(de2idx)))
# loss function
self.loss = tf.nn.softmax_cross_entropy_with_logits(logits = self.logits, labels = self.y_smoothed)
self.mean_loss = tf.reduce_sum(self.loss * self.istarget) / (tf.reduce_sum(self.istarget))
self.global_step = tf.Variable(0, name = 'global_step', trainable = False)
# optimizer
self.optimizer = tf.train.AdamOptimizer(learning_rate = pm.learning_rate, beta1 = 0.9, beta2 = 0.98, epsilon = 1e-8)
self.train_op = self.optimizer.minimize(self.mean_loss, global_step = self.global_step)
tf.summary.scalar('mean_loss', self.mean_loss)
self.merged = tf.summary.merge_all()
if __name__ == '__main__':
'''en2idx{'<PAD>':0, ...}, idx2en{0:'<PAD>'}都是字典形式 长度是1684'''
'''de2idx{'<PAD>':0, ...}, idx2de{0:'<PAD>'}都是字典形式 长度是1597'''
en2idx, idx2en = load_vocab('en.vocab.tsv')
de2idx, idx2de = load_vocab('de.vocab.tsv')
g = Graph("train")
print("MSG : Graph loaded!")
# save model and use this model to training
supvisor = tf.train.Supervisor(graph = g.graph,logdir = pm.logdir,save_model_secs = 0)
with supvisor.managed_session() as sess:
for epoch in range(1, pm.num_epochs + 1):
if supvisor.should_stop():
break
# process bar
for step in tqdm(range(g.batch_num), total = g.batch_num, ncols = 70, leave = False, unit = 'b'):
sess.run(g.train_op)
if not os.path.exists(pm.checkpoint):
os.mkdir(pm.checkpoint)
g_step = sess.run(g.global_step)
supvisor.saver.save(sess, pm.checkpoint + '/model_epoch_%02d_gs_%d' % (epoch, g_step))
print("MSG : Done!")
三、留下一些问题思考
1、Transformer的创新之处在于哪里？和传统的encoder-decoder模型的区别在哪里？它想实现的目标是什么？
2、为什么要采用self-attention以及multi-head attention？
3、如何理解transformer里面所采用的mask？
最后，有些理解不到位的部分，请指教！
参考资料:
1、Attention is all you need论文
2、attention is all you need模型笔记
3、The illustrated transformer
4、The Annotated Transformer
