2018-07-06-Tensorflow-实现LSTM-Lan
2018-07-11 本文已影响0人
社交达人叔本华
本文是对tensorflow官网中的这篇教程进行的说明和整理，源代码可以在官网中找到。
1 LSTM 理解
1.1 标准模型
LSTM 的模型不是本文的重点，因此只是简单的进行介绍。主要参考这篇博客进行说明。
image.png
借用文章中的图说明一下问题。我们在本文中涉及到的是多层的lstm的模型，不是简单的单层的模型，这一点需要注意否则代码中很难看懂。从图中我们可以看出该模型的几个主要特征：
１．模型的结构呈现序列化。简化起见，我们只看最下面一层。可以看出lstm的模型呈现序列化的结构，即最小的单位为一个单元(cell)，每一个cell都以前一个cell的隐状态以及当前文本作为输入，输出自己的隐状态和输出文本(或输出向量)。
２．深层的lstm知识在原有的单层的基础上进行了堆叠。这样做的目的是为了能够让模型可调节的粒度更小，达到降低损失，提高准确率的目的。堆叠的具体方式是将原有的每个单元的输出作为更深层的输入之一。
1.2 代码模型
2 基本框架
2.1程序流程
2.2 代码注释
1.reader.py
"""
reader
author: xuqh
18-6-21 下午3:01
description: read data from ptb
"""
import os
import sys
from collections import Counter
import tensorflow as tf
Py3 = sys.version_info[0] == 3
def _read_words(filename):
"""
把文件分成单词组成列表读到内存
:param filename: file to read from
:return:list of words
"""
with tf.gfile.GFile(filename) as f:
if Py3:
return f.read().replace('\n', '<eos>').split()
else:
return f.read().decode('utf-8').replace('\n', '<eos>').split()
def _build_vocab(filename):
"""
build a vocabulary dictionary
构建{单词：下标}的字典
:param filename:
:return: dict of vocabulary
"""
data = _read_words(filename)
couter = Counter(data)
count_pairs = sorted(couter.items(), key=lambda x: (-x[1], x[0]))
words, _ = list(zip(*count_pairs))
word_to_id = dict(zip(words, range(len(words))))
return word_to_id
def _doc_to_ids(filename, word_to_id):
"""
mapping all words in file to ids according to word_to_id dictionary
把文件读分成单词，单词转化为下标，存成单词下标列表
:param filename:
:param word_to_id:
:return: list of ids
"""
data = _read_words(filename)
return [word_to_id[d] for d in data]
def ptb_raw_data(datapath=None):
"""
read all ptb data into lists of ids
读取train,test,valid数据，转化成下标列表
:param datapath:
:return:
"""
train_path = os.path.join(datapath, "ptb.train.txt")
valid_path = os.path.join(datapath, "ptb.valid.txt")
test_path = os.path.join(datapath, "ptb.test.txt")
# build dictionary
word_to_id = _build_vocab(train_path)
# get data
train_data = _doc_to_ids(train_path, word_to_id)
valid_data = _doc_to_ids(valid_path, word_to_id)
test_data = _doc_to_ids(test_path, word_to_id)
vocab_size = len(word_to_id)
return train_data, valid_data, test_data, vocab_size
def produce_ptb(raw_data, batch_size, num_steps, name=None):
"""
produce ptb data batches
生成批数据
:param raw_data:one of th output from ptb_raw_data function
:param batch_size:
:param num_steps:int,number of the unrolls--how many units in rnn
:param name:variable scope name
:returns
A pair of Tensors, each shaped [batch_size, num_steps]. The second element
of the tuple is the same data time-shifted to the right by one.
"""
with tf.name_scope(name, "PTBProducer", [raw_data, batch_size, num_steps]):
raw_data = tf.convert_to_tensor(raw_data, dtype=tf.int32, name="raw_data")
batch_size_int = batch_size
batch_size = tf.constant(batch_size, name="batch_size")
data_len = tf.size(raw_data, name="data_length")
batch_len = tf.floor_div(data_len, batch_size, name="get_batch_len")
data = tf.reshape(raw_data[0:batch_size * batch_len], shape=(batch_size, batch_len), name="get_data")
epoch_size = (batch_len - 1) // num_steps # 一个epoch有epoch_size个bactch的数据
assertation = tf.assert_positive(epoch_size, message="epoch size==0,decrease batch size or num_steps ")
with tf.control_dependencies([assertation]):
# only executed after assertation was implementated
# 此处代码每次执行之前都要先执行assertation
epoch_size = tf.identity(epoch_size, name="get_epoch_size")
i = tf.train.range_input_producer(epoch_size, shuffle=False).dequeue()
x = tf.strided_slice(data, [0, i * num_steps], [batch_size, (i + 1) * num_steps])
x.set_shape([batch_size_int, num_steps])
y = tf.strided_slice(data, [0, i * num_steps + 1], [batch_size, (i + 1) * num_steps + 1])
y.set_shape([batch_size_int, num_steps])
# 返回为两个tensor,一个是输入数据，一个是输出标签（因为任务是预测下一个单词，所以标签只是输入向后平移一个单词）
return x, y
2.ptb_lm.py
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Example / benchmark for building a PTB LSTM model.
Trains the model described in:
(Zaremba, et. al.) Recurrent Neural Network Regularization
http://arxiv.org/abs/1409.2329
There are 3 supported model configurations:
===========================================
| config | epochs | train | valid | test
===========================================
| small | 13 | 37.99 | 121.39 | 115.91
| medium | 39 | 48.45 | 86.16 | 82.07
| large | 55 | 37.87 | 82.62 | 78.29
The exact results may vary depending on the random initialization.
The hyperparameters used in the model:
- init_scale - the initial scale of the weights －权重的范围设定
- learning_rate - the initial value of the learning rate －学习率
- max_grad_norm - the maximum permissible norm of the gradient － 最大的梯度值，用于防止梯度爆炸（超过此值除以５）
- num_layers - the number of LSTM layers － LSTM模型的层数
- num_steps - the number of unrolled steps of LSTM － lstm静态展开成num_steps个基本单元
- hidden_size - the number of LSTM units -状态向量的维度
- max_epoch - the number of epochs trained with the initial learning rate -超过此值调整学习率learning_rate，方便快速收敛，防止产生震荡
- max_max_epoch - the total number of epochs for training －训练语料库的总次数
- keep_prob - the probability of keeping weights in the dropout layer - dropout操作的保留概率
- lr_decay - the decay of the learning rate for each epoch after "max_epoch" －调整学习率时，学习率的衰减率
- batch_size - the batch size －batch的大小
- rnn_mode - the low level implementation of lstm cell: one of CUDNN,
BASIC, or BLOCK, representing cudnn_lstm, basic_lstm, and
lstm_block_cell classes. － rnn模式，有基本（basic）;block;和cudnn三种方式
The data required for this example is in the data/ dir of the
PTB dataset from Tomas Mikolov's webpage:
$ wget http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz
$ tar xvf simple-examples.tgz
To run:
$ python ptb_word_lm.py --data_path=simple-examples/data/
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import time
import numpy as np
import tensorflow as tf
from tensorflow.python import debug as tf_debug
from tensorflow.python.client import device_lib
import reader
import util
flags = tf.flags
logging = tf.logging
flags.DEFINE_string(
"model", "small",
"A type of model. Possible options are: small, medium, large.")
flags.DEFINE_string("data_path", None,
"Where the training/test data is stored.")
flags.DEFINE_string("save_path", None,
"Model output directory.")
flags.DEFINE_bool("use_fp16", False,
"Train using 16-bit floats instead of 32bit floats")
flags.DEFINE_integer("num_gpus", 1,
"If larger than 1, Grappler AutoParallel optimizer "
"will create multiple training replicas with each GPU "
"running one replica.")
flags.DEFINE_string("rnn_mode", None,
"The low level implementation of lstm cell: one of CUDNN, "
"BASIC, and BLOCK, representing cudnn_lstm, basic_lstm, "
"and lstm_block_cell classes.")
FLAGS = flags.FLAGS
BASIC = "basic"
CUDNN = "cudnn"
BLOCK = "block"
def data_type():
return tf.float16 if FLAGS.use_fp16 else tf.float32
class PTBInput(object):
"""输入的数据封装到这个类中，可以顺便封装一些信息哇，比如批大小啦，num_steps大小啦，epoch的大小啦等等"""
def __init__(self, config, data, name=None):
self.batch_size = batch_size = config.batch_size
self.num_steps = num_steps = config.num_steps
self.epoch_size = ((len(data) // batch_size) - 1) // num_steps
self.input_data, self.targets = reader.produce_ptb(
data, batch_size, num_steps, name=name)
class PTBModel(object):
"""主要模型"""
def __init__(self, is_training, config, input_):
"""
初始化函数
:param is_training: 用于标记是否是训练模型
:param config: 模型的一些配置信息
:param input_: 输入数据
"""
self._is_training = is_training
self._input = input_
self._rnn_params = None
self._cell = None
self.batch_size = input_.batch_size
self.num_steps = input_.num_steps
size = config.hidden_size
vocab_size = config.vocab_size
with tf.device("/cpu:0"):
# 定义word_embedding，放进模型中进行训练
embedding = tf.get_variable(
"embedding", [vocab_size, size], dtype=data_type())
inputs = tf.nn.embedding_lookup(embedding, input_.input_data)
if is_training and config.keep_prob < 1:
# 设置输入数据的dropout
inputs = tf.nn.dropout(inputs, config.keep_prob)
# 构建模型，获取最终状态
output, state = self._build_rnn_graph(inputs, config, is_training)
# 计算损失函数
softmax_w = tf.get_variable(
"softmax_w", [size, vocab_size], dtype=data_type())
softmax_b = tf.get_variable("softmax_b", [vocab_size], dtype=data_type())
logits = tf.nn.xw_plus_b(output, softmax_w, softmax_b)
# Reshape logits to be a 3-D tensor for sequence loss
logits = tf.reshape(logits, [self.batch_size, self.num_steps, vocab_size])
# Use the contrib sequence loss and average over the batches
loss = tf.contrib.seq2seq.sequence_loss(
logits,
input_.targets,
tf.ones([self.batch_size, self.num_steps], dtype=data_type()),
average_across_timesteps=False,
average_across_batch=True)
# Update the cost
self._cost = tf.reduce_sum(loss)
self._final_state = state
if not is_training:
return
# 反向传播，优化参数
self._lr = tf.Variable(0.0, trainable=False)
tvars = tf.trainable_variables()
grads, _ = tf.clip_by_global_norm(tf.gradients(self._cost, tvars),
config.max_grad_norm)
optimizer = tf.train.GradientDescentOptimizer(self._lr)
self._train_op = optimizer.apply_gradients(
zip(grads, tvars),
global_step=tf.train.get_or_create_global_step())
# 定义学习率更新的操作；也可以使用feed_dict的方式，但是这样做更加的规范（高大上？）。
self._new_lr = tf.placeholder(
tf.float32, shape=[], name="new_learning_rate")
self._lr_update = tf.assign(self._lr, self._new_lr)
def _build_rnn_graph(self, inputs, config, is_training):
if config.rnn_mode == CUDNN:
return self._build_rnn_graph_cudnn(inputs, config, is_training)
else:
return self._build_rnn_graph_lstm(inputs, config, is_training)
def _build_rnn_graph_cudnn(self, inputs, config, is_training):
"""Build the inference graph using CUDNN cell."""
inputs = tf.transpose(inputs, [1, 0, 2])
self._cell = tf.contrib.cudnn_rnn.CudnnLSTM(
num_layers=config.num_layers,
num_units=config.hidden_size,
input_size=config.hidden_size,
dropout=1 - config.keep_prob if is_training else 0)
params_size_t = self._cell.params_size()
self._rnn_params = tf.get_variable(
"lstm_params",
initializer=tf.random_uniform(
[params_size_t], -config.init_scale, config.init_scale),
validate_shape=False)
c = tf.zeros([config.num_layers, self.batch_size, config.hidden_size],
tf.float32)
h = tf.zeros([config.num_layers, self.batch_size, config.hidden_size],
tf.float32)
self._initial_state = (tf.contrib.rnn.LSTMStateTuple(h=h, c=c),)
outputs, h, c = self._cell(inputs, h, c, self._rnn_params, is_training)
outputs = tf.transpose(outputs, [1, 0, 2])
outputs = tf.reshape(outputs, [-1, config.hidden_size])
return outputs, (tf.contrib.rnn.LSTMStateTuple(h=h, c=c),)
def _get_lstm_cell(self, config, is_training):
if config.rnn_mode == BASIC:
return tf.contrib.rnn.BasicLSTMCell(
config.hidden_size, forget_bias=0.0, state_is_tuple=True,
reuse=not is_training)
if config.rnn_mode == BLOCK:
return tf.contrib.rnn.LSTMBlockCell(
config.hidden_size, forget_bias=0.0)
raise ValueError("rnn_mode %s not supported" % config.rnn_mode)
def _build_rnn_graph_lstm(self, inputs, config, is_training):
"""
构建lstm的unrolling版本的模型
:param inputs: 输入数据
:param config: 配置信息
:param is_training: 是否是在构建训练模型
:return: state：最终的输出状态，包括h和ｃ；output每个cell的输出，然后展平了。
"""
"""Build the inference graph using canonical LSTM cells."""
# Slightly better results can be obtained with forget gate biases
# initialized to 1 but the hyperparameters of the model would need to be
# different than reported in the paper.
def make_cell():
"""
获取一个lstm的cell
:return:
"""
cell = self._get_lstm_cell(config, is_training)
if is_training and config.keep_prob < 1:
cell = tf.contrib.rnn.DropoutWrapper(
cell, output_keep_prob=config.keep_prob)
return cell
# 纵向的堆叠，增加深度
cell = tf.contrib.rnn.MultiRNNCell(
[make_cell() for _ in range(config.num_layers)], state_is_tuple=True)
self._initial_state = cell.zero_state(config.batch_size, data_type())
state = self._initial_state
# Simplified version of tf.nn.static_rnn().
# This builds an unrolled LSTM for tutorial purposes only.
# In general, use tf.nn.static_rnn() or tf.nn.static_state_saving_rnn().
#
# The alternative version of the code below is:
#
# inputs = tf.unstack(inputs, num=self.num_steps, axis=1)
# outputs, state = tf.nn.static_rnn(cell, inputs,
# initial_state=self._initial_state)
# 横向的展开增加时间广度
outputs = []
with tf.variable_scope("RNN"):
for time_step in range(self.num_steps):
if time_step > 0: tf.get_variable_scope().reuse_variables()
(cell_output, state) = cell(inputs[:, time_step, :], state)
outputs.append(cell_output)
output = tf.reshape(tf.concat(outputs, 1), [-1, config.hidden_size])
return output, state
def assign_lr(self, session, lr_value):
session.run(self._lr_update, feed_dict={self._new_lr: lr_value})
def export_ops(self, name):
"""Exports ops to collections."""
self._name = name
ops = {util.with_prefix(self._name, "cost"): self._cost}
if self._is_training:
ops.update(lr=self._lr, new_lr=self._new_lr, lr_update=self._lr_update)
if self._rnn_params:
ops.update(rnn_params=self._rnn_params)
for name, op in ops.items():
tf.add_to_collection(name, op)
self._initial_state_name = util.with_prefix(self._name, "initial")
self._final_state_name = util.with_prefix(self._name, "final")
util.export_state_tuples(self._initial_state, self._initial_state_name)
util.export_state_tuples(self._final_state, self._final_state_name)
def import_ops(self):
"""Imports ops from collections."""
if self._is_training:
self._train_op = tf.get_collection_ref("train_op")[0]
self._lr = tf.get_collection_ref("lr")[0]
self._new_lr = tf.get_collection_ref("new_lr")[0]
self._lr_update = tf.get_collection_ref("lr_update")[0]
rnn_params = tf.get_collection_ref("rnn_params")
if self._cell and rnn_params:
params_saveable = tf.contrib.cudnn_rnn.RNNParamsSaveable(
self._cell,
self._cell.params_to_canonical,
self._cell.canonical_to_params,
rnn_params,
base_variable_scope="Model/RNN")
tf.add_to_collection(tf.GraphKeys.SAVEABLE_OBJECTS, params_saveable)
self._cost = tf.get_collection_ref(util.with_prefix(self._name, "cost"))[0]
num_replicas = FLAGS.num_gpus if self._name == "Train" else 1
self._initial_state = util.import_state_tuples(
self._initial_state, self._initial_state_name, num_replicas)
self._final_state = util.import_state_tuples(
self._final_state, self._final_state_name, num_replicas)
@property
def input(self):
return self._input
#############################################################################################
# 将之前在各种函数中定义的成员变量封装，相当于定义了getter函数。统一管理，相对安全，可扩展性增强。
#############################################################################################
@property
def initial_state(self):
return self._initial_state
@property
def cost(self):
return self._cost
@property
def final_state(self):
return self._final_state
@property
def lr(self):
return self._lr
@property
def train_op(self):
return self._train_op
@property
def initial_state_name(self):
return self._initial_state_name
@property
def final_state_name(self):
return self._final_state_name
class SmallConfig(object):
"""Small config."""
init_scale = 0.1
learning_rate = 1.0
max_grad_norm = 5
num_layers = 2
num_steps = 20
hidden_size = 200
max_epoch = 1
max_max_epoch = 1
keep_prob = 1.0
lr_decay = 0.5
batch_size = 20
vocab_size = 10000
rnn_mode = BLOCK
class MediumConfig(object):
"""Medium config."""
init_scale = 0.05
learning_rate = 1.0
max_grad_norm = 5
num_layers = 2
num_steps = 35
hidden_size = 650
max_epoch = 6
max_max_epoch = 39
keep_prob = 0.5
lr_decay = 0.8
batch_size = 20
vocab_size = 10000
rnn_mode = BLOCK
class LargeConfig(object):
"""Large config."""
init_scale = 0.04
learning_rate = 1.0
max_grad_norm = 10
num_layers = 2
num_steps = 35
hidden_size = 1500
max_epoch = 14
max_max_epoch = 55
keep_prob = 0.35
lr_decay = 1 / 1.15
batch_size = 20
vocab_size = 10000
rnn_mode = BLOCK
class TestConfig(object):
"""Tiny config, for testing."""
init_scale = 0.1
learning_rate = 1.0
max_grad_norm = 1
num_layers = 1
num_steps = 2
hidden_size = 2
max_epoch = 1
max_max_epoch = 1
keep_prob = 1.0
lr_decay = 0.5
batch_size = 20
vocab_size = 10000
rnn_mode = BLOCK
def run_epoch(session, model, eval_op=None, verbose=False):
"""Runs the model on the given data."""
start_time = time.time()
costs = 0.0
iters = 0
# 每个epoch在执行之前都需要对模型进行初始化操作
state = session.run(model.initial_state)
# print(state)
fetches = {
"cost": model.cost,
"final_state": model.final_state,
}
if eval_op is not None:
fetches["eval_op"] = eval_op
for step in range(model.input.epoch_size):
# 每一步都需要做：将前一步的final_state作为当前模型的initial_state输入；训练模型还需要更新参数。
feed_dict = {}
for i, (c, h) in enumerate(model.initial_state):
# 用最上面一层的final_state给所有层的initial_state进行初始化
feed_dict[c] = state[i].c
feed_dict[h] = state[i].h
vals = session.run(fetches, feed_dict)
cost = vals["cost"]
# print(cost)
state = vals["final_state"]
# print(state)
costs += cost
iters += model.input.num_steps
if verbose and step % (model.input.epoch_size // 10) == 10:
print("%.3f perplexity: %.3f speed: %.0f wps" %
(step * 1.0 / model.input.epoch_size, np.exp(costs / iters),
iters * model.input.batch_size * max(1, FLAGS.num_gpus) /
(time.time() - start_time)))
return np.exp(costs / iters)
def get_config():
"""Get model config."""
config = None
if FLAGS.model == "small":
config = SmallConfig()
elif FLAGS.model == "medium":
config = MediumConfig()
elif FLAGS.model == "large":
config = LargeConfig()
elif FLAGS.model == "test":
config = TestConfig()
else:
raise ValueError("Invalid model: %s", FLAGS.model)
if FLAGS.rnn_mode:
config.rnn_mode = FLAGS.rnn_mode
if FLAGS.num_gpus != 1 or tf.__version__ < "1.3.0":
config.rnn_mode = BASIC
return config
def main(_):
if not FLAGS.data_path:
raise ValueError("Must set --data_path to PTB data directory")
gpus = [
x.name for x in device_lib.list_local_devices() if x.device_type == "GPU"
]
if FLAGS.num_gpus > len(gpus):
raise ValueError(
"Your machine has only %d gpus "
"which is less than the requested --num_gpus=%d."
% (len(gpus), FLAGS.num_gpus))
raw_data = reader.ptb_raw_data(FLAGS.data_path)
train_data, valid_data, test_data, _ = raw_data
config = get_config()
eval_config = get_config()
eval_config.batch_size = 1
eval_config.num_steps = 1
with tf.Graph().as_default():
initializer = tf.random_uniform_initializer(-config.init_scale,
config.init_scale)
# 将输入和模型这两张图的友谊的小桥搭建起来
with tf.name_scope("Train"):
train_input = PTBInput(config=config, data=train_data, name="TrainInput")
with tf.variable_scope("Model", reuse=None, initializer=initializer):
m = PTBModel(is_training=True, config=config, input_=train_input)
tf.summary.scalar("Training Loss", m.cost)
tf.summary.scalar("Learning Rate", m.lr)
with tf.name_scope("Valid"):
valid_input = PTBInput(config=config, data=valid_data, name="ValidInput")
with tf.variable_scope("Model", reuse=True, initializer=initializer):
mvalid = PTBModel(is_training=False, config=config, input_=valid_input)
tf.summary.scalar("Validation Loss", mvalid.cost)
with tf.name_scope("Test"):
test_input = PTBInput(
config=eval_config, data=test_data, name="TestInput")
with tf.variable_scope("Model", reuse=True, initializer=initializer):
mtest = PTBModel(is_training=False, config=eval_config,
input_=test_input)
models = {"Train": m, "Valid": mvalid, "Test": mtest}
# for name, model in models.items():
# model.export_ops(name)
# metagraph = tf.train.export_meta_graph()
# if tf.__version__ < "1.1.0" and FLAGS.num_gpus > 1:
# raise ValueError("num_gpus > 1 is not supported for TensorFlow versions "
# "below 1.1.0")
# soft_placement = False
# if FLAGS.num_gpus > 1:
# soft_placement = True
# util.auto_parallel(metagraph, m)
# with tf.Graph().as_default():
# tf.train.import_meta_graph(metagraph)
# for model in models.values():
# model.import_ops()
# 正经的运行，训练，预测
# Supervisor可以理解为更加高大上的一个session管理器。能帮我们这些少脑子的人做一些必须做但是经常忘记或者做不好的事情。比如：变量使用之前必须初始化？比如时刻保存这模型，防止断电？
sv = tf.train.Supervisor(logdir=FLAGS.save_path)
# config_proto = tf.ConfigProto(allow_soft_placement=soft_placement)
with sv.managed_session() as session:
# with sv.managed_session(config=config_proto) as session:
for i in range(config.max_max_epoch):
lr_decay = config.lr_decay ** max(i + 1 - config.max_epoch, 0.0)
m.assign_lr(session, config.learning_rate * lr_decay)
print("Epoch: %d Learning rate: %.3f" % (i + 1, session.run(m.lr)))
train_perplexity = run_epoch(session, m, eval_op=m.train_op,
verbose=True)
session = tf_debug.LocalCLIDebugWrapperSession(session)
print("Epoch: %d Train Perplexity: %.3f" % (i + 1, train_perplexity))
valid_perplexity = run_epoch(session, mvalid)
print("Epoch: %d Valid Perplexity: %.3f" % (i + 1, valid_perplexity))
test_perplexity = run_epoch(session, mtest)
print("Test Perplexity: %.3f" % test_perplexity)
if FLAGS.save_path:
print("Saving model to %s." % FLAGS.save_path)
sv.saver.save(session, FLAGS.save_path, global_step=sv.global_step)
if __name__ == "__main__":
# 内部执行过程为：解析FLAGs->执行main()函数
tf.app.run()
2.３ 知识点总结
1. 队列的使用。这方面的使用方法已经在这篇博客中详细介绍，在此不再赘述。
２．Supervisor的使用。同上。
３．Flags的使用
①功能描述：flags的设置主要是方便用户读取命令行的参数
②基本实现：定义+解析。定义的方式为首先使用 flags = tf.flags定义一个flags 变量。然后使用flags.DEFINE_string预定义可以输入的参数。（显然，不只是可以定义string，其他类型的也有。）；解析是完全自动化的过程。
４．Variable_scope的理解
参考这个提问
遇到问题
epoch理解
一个epoch 在深度学习中指的是一个语料库完整的运行完一遍。之所以会在这里产生困扰，是因为RNN系列的模型都会进行展开，展开就会有time_steps，而time_steps很容易和epoch混淆。简要的来说：
１．batch 的概念是纵向的，是每个单元同时训练多少个样例。batch的大小是样例的个数。
２．epoch的概念是横向的，指的是语料库本身。epoch的大小是语料库包含多少个基本单元（通常是单词）。
３．num_steps的概念是,假如我们把RNN模型展开成１０个单元，那么num_steps=10.
