cifar10
cifar10_input.py
文件结构
读取本地CIFAR-10的二进制文件共有四个函数
-
read_cifar10(filename_queue)
:读取和解析图片 -
_generate_image_and_label_batch(image, label, min_queue_examples, batch_size)
:建立一个图片和标签的batch队列 -
distorted_inputs(data_dir, batch_size)
:为训练改变输入的形状 -
inputs(eval_data, data_dir, batch_size)
:为评估构建输入
具体如下
def read_cifar10(filename_queue):
"""Reads and parses examples from CIFAR10 data files.
Recommendation: if you want N-way read parallelism, call this function
N times. This will give you N independent Readers reading different
files & positions within those files, which will give better mixing of
examples.
Args:
filename_queue: A queue of strings with the filenames to read from.
Returns:
An object representing a single example, with the following fields:
height: number of rows in the result (32)
width: number of columns in the result (32)
depth: number of color channels in the result (3)
key: a scalar string Tensor describing the filename & record number
for this example.
label: an int32 Tensor with the label in the range 0..9.
uint8image: a [height, width, depth] uint8 Tensor with the image data
"""
return result
def _generate_image_and_label_batch(image, label, min_queue_examples,
batch_size):
"""Construct a queued batch of images and labels.
Args:
image: 3-D Tensor of [height, width, 3] of type.float32.
label: 1-D Tensor of type.int32
min_queue_examples: int32, minimum number of samples to retain
in the queue that provides of batches of examples.
batch_size: Number of images per batch.
Returns:
images: Images. 4D tensor of [batch_size, height, width, 3] size.
labels: Labels. 1D tensor of [batch_size] size.
"""
return images, tf.reshape(label_batch, [batch_size])
def distorted_inputs(data_dir, batch_size):
"""Construct distorted input for CIFAR training using the Reader ops.
Args:
data_dir: Path to the CIFAR-10 data directory.
batch_size: Number of images per batch.
Returns:
images: Images. 4D tensor of [batch_size, IMAGE_SIZE, IMAGE_SIZE, 3] size.
labels: Labels. 1D tensor of [batch_size] size.
"""
return _generate_image_and_label_batch(float_image, read_input.label,
min_queue_examples, batch_size)
def inputs(eval_data, data_dir, batch_size):
"""Construct input for CIFAR evaluation using the Reader ops.
Args:
eval_data: bool, indicating if one should use the train or eval data set.
data_dir: Path to the CIFAR-10 data directory.
batch_size: Number of images per batch.
Returns:
images: Images. 4D tensor of [batch_size, IMAGE_SIZE, IMAGE_SIZE, 3] size.
labels: Labels. 1D tensor of [batch_size] size.
"""
return _generate_image_and_label_batch(float_image, read_input.label,
min_queue_examples, batch_size)
函数详解
read_cifar10(filename_queue)
最终返回一个result
(CIFAR10Record的实例),其中包含了
height、width:、depth
key:描述filename 和 record number的Tensor
label: 32位整型的 Tensor
uint8image: 图片数据
class CIFAR10Record(object):
pass
result = CIFAR10Record()
result.height = 32
result.width = 32
result.depth = 3
这里定义了一个类CIFAR10Record,pass
不做任何事情,一般用做占位语句。该处的 pass 便是占据一个位置,因为如果定义一个空函数程序会报错,当你没有想好函数的内容是可以用 pass 填充,使程序可以正常运行。
后面定义一个CIFAR10Record的实例result,设置实例result的属性值。他的优点是, 它是非常灵活的。类是在没有任何成员的情况下定义的。任何函数都可以决定要添加哪些字段。不同的调用可以创建此类的对象, 并以不同的方式填充它们 (因此, 同时, 您可以拥有具有不同成员的同一个类的对象)。
这种灵活性也是一种弊端。这缺乏结构: 很难查看代码并决定类将具有哪些成员。获取这样的对象并循环访问成员也不那么简单。最后, 该类是一个没有封装的极端情况。
更好的做法请查看CIFAR10Record 函数的参考资料
tf.FixedLengthRecordReader
tf.FixedLengthRecordReader是读取固定长度字节数信息(针对bin文件使用FixedLengthRecordReader读取比较合适),下次调用时会接着上次读取的位置继续读取文件,而不会从头开始读取。
参考资料tensorflow读取数据-tfrecord格式Tensorflow中使用tfrecord方式读取数据
tf.TFRecordReader.read(queue, name=None)
Returns the next record (key, value pair) produced by a reader.
返回一个阅读器生成的下一个记录(键值对)。
Will dequeue a work unit from queue if necessary (e.g. when the
Reader needs to start reading from a new file since it has
finished with the previous file).
如果有必要,将从队列中对一个工作单元进行排序(例如,当读者需要从一个新文件开始阅读时,因为它已经完成了前面的文件)。
Args:
queue: A Queue or a mutable string Tensor representing a handle
to a Queue, with string work items.
文件名队列句柄
name: A name for the operation (optional).
Returns:
A tuple of Tensors (key, value).
key: A string scalar Tensor.
value: A string scalar Tensor.
返回键值对,其中值表示读取的文件
distorted_inputs(data_dir, batch_size)
TF Boys (TensorFlow Boys ) 养成记(二): TensorFlow 数据读取
_generate_image_and_label_batch(image, label, min_queue_examples, batch_size)
# Creates batches of 32 images and 32 labels.
image_batch, label_batch = tf.train.shuffle_batch(
[single_image, single_label],
batch_size=32,
num_threads=4,
capacity=50000,
min_after_dequeue=10000)
参考资料
这段代码写的是从[single_image, single_label]利用4个线程读取32行
Args:
-
tensor_list
: 入队的张量列表 -
batch_size
: 表示进行一次批处理的tensors数量. -
num_threads
: The number of threads enqueuingtensor_list
.
设置num_threads的值大于1,使用多个线程在tensor_list中读取文件,这样保证了同一时刻只在一个文件中进行读取操作(但是读取速度依然优于单线程),而不是之前的同时读取多个文件,这种方案的优点是:
- 避免了两个不同的线程从同一文件中读取用一个样本
- 避免了过多的磁盘操作
-
capacity
: 容量:一个整数,队列中的最大的元素数.
这个参数一定要比min_after_dequeue参数的值大,并且决定了我们可以进行预处理操作元素的最大值. -
min_after_dequeue
: Minimum number elements in the queue after a
dequeue(出列), used to ensure a level of mixing of elements.
当一次出列操作完成后,队列中元素的最小数量,往往用于定义元素的混合级别.
定义了随机取样的缓冲区大小,此参数越大表示更大级别的混合但是会导致启动更加缓慢,并且会占用更多的内存
Returns:
A list of tensors with the same number and types as tensor_list
.
默认返回一个和读取tensor_list数据和类型一个tensor列表.