https://juejin.im/post/598972735188256de4693951

https://cloud.tencent.com/developer/article/1096650

我们想要的是构建一个生成式模型，而非仅仅是“记忆”图像数据的模糊结构。除了像前面那样从已有图像中编码出潜在向量，我们还不知道如何创造这些向量，也就无法凭空生成任何图像。

给编码网络增加一个约束，迫使它所生成的潜在向量大体上服从于单位高斯分布。

误差项：生成误差，用以衡量网络重构图像精确度的均方误差；潜在误差，用以衡量潜在变量在单位高斯分布上的契合程度的KL散度。

generation_loss = mean(square(generated_image - real_image))
latent_loss=KL-Divergence(latent_variable, unit_gaussian)
loss = generation_loss + latent_loss

为了优化KL散度，我们要用到重新参数化的一个简单技巧：生成一个均值向量一个标准差向量，而非直接生成实值向量。

image.png

注：z_mean and z_stddev are two vectors generated by encoder network
latent_loss = 0.5* tf.reduce_sum(tf.square(z_mean)+tf.square(z_stddev) - tf.log(tf.square(z_stddev))-1,1)

在计算解码网络的误差时，我们只需从标准差中取样，再加上均值向量，就能得到我们的潜在向量：

samples = tf.random_normal([batchsize,n_z],0,1,dtype=tf.float32)
sampled_z = z_mean + (z_stddev * samples)

全部代码:

reset_graph()

from functools import partial

n_inputs = 28 * 28
n_hidden1 = 500
n_hidden2 = 500
n_hidden3 = 20  # codings
n_hidden4 = n_hidden2
n_hidden5 = n_hidden1
n_outputs = n_inputs
learning_rate = 0.001

with tf.contrib.framework.arg_scope(
        [fully_connected],
        activation_fn=tf.nn.relu,
        weights_initializer=tf.contrib.layers.variance_scaling_initializer()):
    X = tf.placeholder(tf.float32, [None, n_inputs])
    hidden1 = fully_connected(X, n_hidden1)
    hidden2 = fully_connected(hidden1,n_hidden2)
    hidden3_mean = fully_connected(hidden2,n_hidden3,activation_fn=None)
    hidden3_gamma = fully_connected(hidden2,n_hidden3,activation_fn=None)
    hidden3_sigma = tf.exp(0.5 * hidden3_gamma)
    noise = tf.random_normal(tf.shape(hidden3_sigma),dtype=tf.float32)

    hidden3 = hidden3_mean + hidden3_sigma * noise
    hidden4 = fully_connected(hidden3,n_hidden4)
    hidden5 = fully_connected(hidden4,n_hidden5)
    logits = fully_connected(hidden5,n_outputs,activation_fn=None)
    outputs = tf.sigmoid(logits)

reconstruction_loss = tf.reduce_mean(
    tf.nn.sigmoid_cross_entropy_with_logits(labels=X,logits=logits))

latent_loss = 0.5 * tf.reduce_sum(tf.exp(hidden3_gamma) + tf.square(hidden3_mean)
                                 - 1 - hidden3_gamma)

cost = reconstruction_loss + latent_loss
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(cost)

init = tf.global_variables_initializer()
saver = tf.train.Saver()
n_epochs = 50
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([cost, reconstruction_loss, latent_loss], feed_dict={X: X_batch})
        print("\r{}".format(epoch), "Train total cost:", loss_val, "\tReconstruction loss:", reconstruction_loss_val, "\tLatent loss:", latent_loss_val)
        saver.save(sess, "./my_model_variational_variant.ckpt")

EM距离

https://blog.csdn.net/zhangping1987/article/details/25368183

https://hk.saowen.com/a/e14e3319b2b52da88b7a0fc074b93c153ff17b67e33e9ab07ff298d64d573f7a

代码：

import numpy as np
import cv

#p、q是兩個矩陣，第一列表示權值，後面三列表示直方圖或數量
p=np.asarray([[0.4,100,40,22],
            [0.3,211,20,2],
            [0.2,32,190,150],
            [0.1,2,100,100]],np.float32)
q=np.array([[0.5,0,0,0],
            [0.3,50,100,80],
            [0.2,255,255,255]],np.float32)
pp=cv.fromarray(p)
qq=cv.fromarray(q)
emd=cv.CalcEMD2(pp,qq,cv.CV_DIST_L2)

L<1 代表 f 是一个 1-Lipsschitz 函数。K-Lipsschitz 函数定义为对于 K >0，||f(x1) – f(x2)||≤K||x1 – x2||。

WGAN

http://www.10tiao.com/html/511/201702/2651993903/4.html

WAE

https://www.ctolib.com/topics-131397.html