內容摘要：

  • 8.1 TF高層封裝簡介，
  • 8.2 TensorFlow-Slim、TFLrean簡介
  • 8.3 Keras介紹
  • 8.4 Estimator介紹

8.1 TF高層封裝簡介

雖然tf可以靈活的支持不同神經網路結構，但是代碼還是相對冗長，為了更便捷實現神經網路結構，不同組織個人為tf提供了多種高層封裝。目前流行的主要有四種：TensorFlow-Slim、TFLrean、Keras、Estimator。TensorFlow-Slim是 谷歌公司給出的較早的高級封裝，現在仍然被較多使用；TFLrean相比TensorFlow-Slim更加簡潔。最終要的是Keras和Estimator，後面進行更詳細的介紹。

8.2 TensorFlow-Slim、TFLearn簡介

TensorFlow-Slim:

tensorflow-slim作用是使模型更加簡潔，一句話就可以定義一層網路，但是這些封裝使用不廣泛。這裡就不實際運行，直接貼上代碼和部分注釋備用:

import tensorflow as tfimport tensorflow.contrib.slim as slimimport numpy as npfrom tensorflow.examples.tutorials.mnist import input_data# 通過TensorFlow-Slim來定義LeNet-5的網路結構。def lenet5(inputs): # 將輸入轉化為四維數組，第一維代表batch大小，另外三維代表圖片 inputs = tf.reshape(inputs, [-1, 28, 28, 1]) # 定義卷積層 net = slim.conv2d(inputs, 32, [5, 5], padding=SAME, scope=layer1-conv) # 定義最大池化層 net = slim.max_pool2d(net, 2, stride=2, scope=layer2-max-pool) net = slim.conv2d(net, 64, [5, 5], padding=SAME, scope=layer3-conv) net = slim.max_pool2d(net, 2, stride=2, scope=layer4-max-pool) # 定義好的函數將四維轉換為二維，方便全連接使用 net = slim.flatten(net, scope=flatten) # 定義全連接層，共500個隱藏點 net = slim.fully_connected(net, 500, scope=layer5) net = slim.fully_connected(net, 10, scope=output) return net# 訓練模型def train(mnist): # 定義輸入 x = tf.placeholder(tf.float32, [None, 784], name=x-input) y_ = tf.placeholder(tf.float32, [None, 10], name=y-input) # 使用tensorflow-slim定義網路結構 y = lenet5(x) # 定義損失函數和訓練方法 cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1)) loss = tf.reduce_mean(cross_entropy) train_op = tf.train.GradientDescentOptimizer(0.01).minimize(loss) # 訓練過程 with tf.Session() as sess: tf.global_variables_initializer().run() for i in range(3000): xs, ys = mnist.train.next_batch(100) _, loss_value = sess.run([train_op, loss], feed_dict={x: xs, y_: ys}) if i % 1000 == 0: print("After %d training step(s), loss on training batch is %g." % (i, loss_value))def main(argv=None): mnist = input_data.read_data_sets("../../datasets/MNIST_data", one_hot=True) train(mnist)

tensorflow-slim最大的特點是對一些標準的神經網路模型進行了封裝。有興趣的也可以參見tensorflow-slim代碼庫地址：

tensorflow/tensorflow?

github.com圖標

TFLearn:

TFLearn與TF-slim相比更加簡潔，除了對神經網路結構進行封裝外還對訓練過程進行了封裝。參考代碼如下，沒有實現，僅供參考：

注，要TF中要單獨安裝：

pip install tflearn

import tflearnfrom tflearn.layers.core import input_data, dropout, fully_connectedfrom tflearn.layers.conv import conv_2d, max_pool_2dfrom tflearn.layers.estimator import regressionimport tflearn.datasets.mnist as mnist# 讀入數據集trainX, trainY, testX, testY = mnist.load_data( data_dir="../../datasets/MNIST_data", one_hot=True)# 將圖像數據resize成卷積卷積神經網路輸入的格式。trainX = trainX.reshape([-1, 28, 28, 1])testX = testX.reshape([-1, 28, 28, 1])# 構建神經網路。net = input_data(shape=[None, 28, 28, 1], name=input)net = conv_2d(net, 32, 5, activation=relu)net = max_pool_2d(net, 2)net = conv_2d(net, 64, 5, activation=relu)net = max_pool_2d(net, 2)net = fully_connected(net, 500, activation=relu)net = fully_connected(net, 10, activation=softmax)# 定義學習任務。指定優化器為sgd，學習率為0.01，損失函數為交叉熵。net = regression(net, optimizer=sgd, learning_rate=0.01, loss=categorical_crossentropy)# 通過定義的網路結構訓練模型，並在指定的驗證數據上驗證模型的效果。model = tflearn.DNN(net, tensorboard_verbose=0)model.fit(trainX, trainY, n_epoch=10, validation_set=([testX, testY]), show_metric=True)

8.3 Keras介紹

目前使用最廣泛的深度學習工具之一，支持TFboys，MXNet，CNTK和Theano。目前已經是TFboys官方的高層封裝之一。下面介紹Keras的簡單使用。

Keras基本用法

Keras API訓練模型可以定義一個Sequential類：

1）add函數完成網路層添加，卷積層、池化層、RNN機構（LSTM）、全連接層等等；

2）compile函數，指定優化函數、損失函數、以及訓練中需要監控的指標；

3）fit函數進行模型訓練，只需給出訓練數據、batch大小和訓練輪數；

接下來給出使用的詳細過程及code：

首先安裝：

備註：

1.失敗的話就多試幾次；2.再不行就用：conda install keras3.再不行就anaconda中直接添加

import kerasfrom keras.datasets import mnistfrom keras.models import Sequentialfrom keras.layers import Dense, Flatten, Conv2D, MaxPooling2Dfrom keras import backend as K# ————————1.0 數據準備————————num_classes = 10 # 類別img_rows, img_cols = 28, 28 # 圖片長和寬# 通過Keras封裝好的API載入MNIST數據。其中trainX就是一個60000 * 28 * 28的數組，trainY是每一張圖片對應的數字。# load_data函數中封裝了數據集的獲取位置，所以可以直接獲取；（複製地址到網頁中可直接下載備用）(trainX, trainY), (testX, testY) = mnist.load_data()# 根據對圖像編碼的格式要求來設置輸入層的格式if K.image_data_format() == channels_first: trainX = trainX.reshape(trainX.shape[0], 1, img_rows, img_cols) testX = testX.reshape(testX.shape[0], 1, img_rows, img_cols) # 因為MNIST中的圖片是黑色的，所以第一維的取值為1 input_shape = (1, img_rows, img_cols)else: trainX = trainX.reshape(trainX.shape[0], img_rows, img_cols, 1) testX = testX.reshape(testX.shape[0], img_rows, img_cols, 1) input_shape = (img_rows, img_cols, 1)# 將圖像轉化為0-1之間的實數，完成歸一化trainX = trainX.astype(float32)testX = testX.astype(float32)trainX /= 255.0testX /= 255.0# 將標準答案轉化為需要的格式（one-hot編碼），Y對應實際數字trainY = keras.utils.to_categorical(trainY, num_classes)testY = keras.utils.to_categorical(testY, num_classes)# ————————2.0 通過Keras的API定義卷積神經網路————————# 使用Keras API定義模型。model = Sequential()model.add(Conv2D(32, kernel_size=(5, 5), activation=relu, input_shape=input_shape))model.add(MaxPooling2D(pool_size=(2, 2)))model.add(Conv2D(64, (5, 5), activation=relu))model.add(MaxPooling2D(pool_size=(2, 2)))# 將卷積層的輸出拉直後作為下面全臉階層的輸入model.add(Flatten())# 全連接層，500個節點model.add(Dense(500, activation=relu))# 全連接層，得到最後輸出model.add(Dense(num_classes, activation=softmax))# 定義損失函數、優化函數和評測方法。model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.SGD(), metrics=[accuracy])# ————————3.0 通過Keras的API訓練模型並計算在測試數據上的準確率————————model.fit(trainX, trainY, batch_size=128, epochs=10, validation_data=(testX, testY))# 在測試數據上計算準確率。score = model.evaluate(testX, testY)print(Test loss:, score[0])print(Test accuracy:, score[1])

問題：

==========艱難解決的分割線（非正文部分）================

解決過程記錄：

win10下嘗試：

有帖子說是版本問題，僅僅是可能，但還是得試一下。然後，必須python2.7下的tensorflow才支持，所以我就試著安裝了python2.7。不過，試了好幾次一直配不上。查詢之後才發現win系統下的python2.7不支持tensorflow。

linux下：

無奈，就是自己搞了個ubuntu，linux系統上安裝了anaconda2並配置了python2.7下的tensorflow，然後問題又來了。主要是我使用的IDE是pycharm，pycharm上使用interpreter一直無法添加相應的tensorflow環境，what a fu....。嗯嗯，接著就是當天去上課。

win下繼續嘗試：

第二天我直接在windows上再試了一下，然後，tensorflow環境出了問題。於是乎，我重新添加了一個interpreter，中間小問題略過好幾條。然後重新import keras_cnn模塊，新問題變成了如下模樣：

嗯嗯， 我查詢了好多地方，ImportError: No module named scipy.lib，就是模塊導入失敗，缺少scipy.lib。於是我就安裝了scipy，並無卵用，有點崩潰。也許win10下真的不行吧，重新在linux下試試，不過這裡的原文件最好複製到一個新的位置，以免linux和win同時操作產生的一些已經遇到的問題。

linux繼續嘗試：

目標：如何添加上linux下python2.7的tensorflow的interpreter。

問題：又來了，pycharm看來最好不要win和linux下同時操作一個文件夾，這裡pycharm界面不顯示代碼和terminal等界面，所以重裝pychar。這時我嘗試過配置interpreter，所以重新安裝後，界面可以直接顯示，然後配置成功！

接下來就是進行keras等相關包的配置，這裡比較簡單。直接在interpreter裡面綠色加號裡面查找並添加就可以。【後悔了，太慢了，不如terminal中直接用pip進行安裝；不過或許是】

耐心等待，看來虛擬機還是慢！

然後運行，結果又有問題了：

想哭了。。。。繼續找問題：

注釋出了問題，直接貼下列代碼就好了：

# -*- coding: utf-8 -*-

重新嘗試——————————》》》成功！！

備註：正常運行部分見正文分割線後內容。

問題總結：

現象：提示如下，程序無報錯，環境為win10，anaconda3下python3.5環境中的tensorflow.

原因及解決辦法：作者使用python2.7下的tensorflow編寫程序，python3.5下相關部分不兼容（缺少包），所以 建議在linux下安裝python2.7並配置tensorflow（win下python2.7不支持tensorflow）。

備註：

1. 細節問題看上述記錄或者私信、留言，看到會回復。2. 以後沒沒有提示的話都是在linux中，python2.7下的tf中實現的。

===============成功的分割線（正文繼續）============

下面是下載數據集的部分：

緊接著運行，因為是虛擬機，給的資源有限，所以比較慢，10個Epoch還是hold不住啊：

最終結果：

LeNet-5模型實現:

下面展示如何在原生態的Keras在MNIST數據集上實現LeNet-5模型：

from keras.preprocessing import sequencefrom keras.models import Sequentialfrom keras.layers import Dense, Embeddingfrom keras.layers import LSTMfrom keras.datasets import imdb# ————————1.0 數據預處理——————————max_features = 20000 # 最多使用的單詞數量maxlen = 80 # 循環神經網路的階段長度batch_size = 32 # batch的大小# 載入數據並將單詞轉化為ID，max_features給出了最多使用的單詞數。# 評率較低的數據直接替換為統一的ID.# 通過Keras封裝的API會生成25000條訓練數據和25000測試數據。每一條一句話，每段話都有好壞標籤。(trainX, trainY), (testX, testY) = imdb.load_data(num_words=max_features)print(len(trainX), train sequences)print(len(testX), test sequences)# 在自然語言中，每一段話的長度是不一樣的，但循環神經網路的循環長度是固定的，所以這裡需要先將所有段落統一成固定長度。# 不夠的長度補0，超出的部分直接忽略超過的部分trainX = sequence.pad_sequences(trainX, maxlen=maxlen)testX = sequence.pad_sequences(testX, maxlen=maxlen)print(trainX shape:, trainX.shape)print(testX shape:, testX.shape)# ————————2.0 定義模型————————model = Sequential()# 構建embeding層。128代表其向量維度# 輸入層的單詞通過embeding層產生單詞向量並輸入到LSTM層，最後輸出結果model.add(Embedding(max_features, 128))model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))model.add(Dense(1, activation=sigmoid))model.compile(loss=binary_crossentropy, optimizer=adam, metrics=[accuracy])# ————————3.0 訓練評測模型————————model.fit(trainX, trainY, batch_size=batch_size, epochs=10, validation_data=(testX, testY))score = model.evaluate(testX, testY, batch_size=batch_size)print(Test loss:, score[0])print(Test accuracy:, score[1])

測試：

>>> import cx2_keras_rnnDownloading data from https://s3.amazonaws.com/text-datasets/imdb.npz 16384/17464789 [..............................] - ETA: 0s 24576/17464789 [..............................] - ETA: 6:23 40960/17464789 [..............................] - ETA: 7:40# ···················省略內容的分割線····················25000/25000 [==============================] - 254s 10ms/step - loss: 0.4591 - acc: 0.7814 - val_loss: 0.3813 - val_acc: 0.8331Epoch 2/10 32/25000 [..............................] - ETA: 4:56 - loss: 0.4126 - acc: 0.8125# ····················太久了就不繼續了的分割線·················

Keras高級介紹

帶返回值的結構

keras可以實現帶有返回值的網路結構，這裡展示瞭如何使用這種方式定義模型，即帶返回值得結構：

# -*- coding: utf-8 -*-import keras# from tflearn.layers.core import fully_connectedfrom keras.datasets import mnistfrom keras.layers import Input, Densefrom keras.models import Model# ————————1.0 數據預處理————————num_classes = 10img_rows, img_cols = 28, 28# 通過Keras封裝好的API載入MNIST數據。(trainX, trainY), (testX, testY) = mnist.load_data()trainX = trainX.reshape(trainX.shape[0], img_rows * img_cols)testX = testX.reshape(testX.shape[0], img_rows * img_cols)# 將圖像像素轉化為0到1之間的實數。trainX = trainX.astype(float32)testX = testX.astype(float32)trainX /= 255.0testX /= 255.0# 將標準答案轉化為需要的格式（one-hot編碼）。trainY = keras.utils.to_categorical(trainY, num_classes)testY = keras.utils.to_categorical(testY, num_classes)# ————————2.0 通過返回值的方式定義模型————————# 定義輸入，這裡指定的維度不用該考慮batch的大小inputs = Input(shape=(784,))# 定義一層全連接層，有500個隱藏節點，使用ReLU激活函數# 這一層的輸入為inputx = Dense(500, activation=relu)(inputs)# 定義輸出層，裡面沒有封裝sotfmax，所以要單獨添加predictions = Dense(10, activation=softmax)(x)# 通過Model類創建模型，和sequential類不同的是這裡需要指定輸入和輸出model = Model(inputs=inputs, outputs=predictions)model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.SGD(), metrics=[accuracy])# ————————3.0 訓練模型————————model.fit(trainX, trainY, batch_size=32, epochs=1, validation_data=(testX, testY))

結果：

>>> import cx31_keras_inception/home/sun/pycharm-community-2018.1.1/helpers/pydev/_pydev_bundle/pydev_import_hook.py:20: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. module = self._system_import(name, *args, **kwargs)Using TensorFlow backend.Train on 60000 samples, validate on 10000 samplesEpoch 1/12018-04-18 19:42:21.218010: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 32/60000 [..............................] - ETA: 34:20 - loss: 2.3628 - acc: 0.0625 224/60000 [..............................] - ETA: 5:07 - loss: 2.2318 - acc: 0.2009 480/60000 [..............................] - ETA: 2:29 - loss: 2.1934 - acc: 0.2167 736/60000 [..............................] - ETA: 1:41 - loss: 2.1499 - acc: 0.2595 928/60000 [..............................] - ETA: 1:23 - loss: 2.1167 - acc: 0.2985 1184/60000 [..............................] - ETA: 1:08 - loss: 2.0776 - acc: 0.3387 1472/60000 [..............................] - ETA: 56s - loss: 2.0316 - acc: 0.3899 1760/60000 [..............................] - ETA: 48s - loss: 1.9885 - acc: 0.4295# ····················省略內容的分割線············································58336/60000 [============================>.] - ETA: 0s - loss: 0.6077 - acc: 0.850858560/60000 [============================>.] - ETA: 0s - loss: 0.6067 - acc: 0.850958784/60000 [============================>.] - ETA: 0s - loss: 0.6059 - acc: 0.851058848/60000 [============================>.] - ETA: 0s - loss: 0.6056 - acc: 0.851258944/60000 [============================>.] - ETA: 0s - loss: 0.6051 - acc: 0.851359040/60000 [============================>.] - ETA: 0s - loss: 0.6048 - acc: 0.851459328/60000 [============================>.] - ETA: 0s - loss: 0.6034 - acc: 0.851759616/60000 [============================>.] - ETA: 0s - loss: 0.6018 - acc: 0.852059904/60000 [============================>.] - ETA: 0s - loss: 0.6004 - acc: 0.852360000/60000 [==============================] - 14s 231us/step- loss: 0.6002 - acc: 0.8523 - val_loss: 0.3413 - val_acc: 0.9085

下面給出keras實現Inception結構：

# -*- coding: utf-8 -*-from keras.layers import Conv2D, MaxPooling2D, Inputimport keras# ————————定義輸入圖像尺寸————input_img = Input(shape=(256, 256, 3))# ————————定義第一個分支————————tower_1 = Conv2D(64, (1, 1), padding=same, activation=relu)(input_img)tower_1 = Conv2D(64, (3, 3), padding=same, activation=relu)(tower_1)# ————————定義第二個分支————————# 與模型順序不同這裡使用的是input_img，而不是第一次的輸出tower_2 = Conv2D(64, (1, 1), padding=same, activation=relu)(input_img)tower_2 = Conv2D(64, (5, 5), padding=same, activation=relu)(tower_2)# ————————定義第三個分支————————tower_3 = MaxPooling2D(64, (3, 3), strides=(1, 1), padding=same)(input_img)tower_3 = Conv2D(64, (1, 1), padding=same, activation=relu)(tower_3)# ————————將三個分支通過concatenate的方式拼接在一起————————output = keras.layers.concatenate([tower_1, tower_2, tower_3], axis=1)

多輸入多輸出結構

輸入層1中含有784個節點，代表MNIST圖片中的784個像素。輸入層2含有10個節點，代表該圖片所對應的數字。輸出層在預測時僅僅依賴維度為1的隱藏層，因此預測的準確度比較低；而輸出層2的輸入中直接包含了正確答案，因此預測準確度很高。（疑問：包含答案進行預測？）

# -*- coding: utf-8 -*-import keras# from tflearn.layers.core import fully_connectedfrom keras.datasets import mnistfrom keras.layers import Input, Densefrom keras.models import Model# ————————1.0 數據預處理————————num_classes = 10img_rows, img_cols = 28, 28# 通過Keras封裝好的API載入MNIST數據。(trainX, trainY), (testX, testY) = mnist.load_data()trainX = trainX.reshape(trainX.shape[0], img_rows * img_cols)testX = testX.reshape(testX.shape[0], img_rows * img_cols)trainX = trainX.astype(float32)testX = testX.astype(float32)trainX /= 255.0testX /= 255.0trainY = keras.utils.to_categorical(trainY, num_classes)testY = keras.utils.to_categorical(testY, num_classes)# ————————2.0 定義模型————————# 定義兩個輸入。input1 = Input(shape=(784,), name="input1")input2 = Input(shape=(10,), name="input2")# 定義第一個輸出。# 定義一個只有一個隱藏節點的全連接網路x = Dense(1, activation=relu)(input1)output1 = Dense(10, activation=softmax, name = "output1")(x)# 定義第二個輸出。# 將一個隱藏節點的輸出和正確答案拼接在一起，作為第二個輸入層的輸入y = keras.layers.concatenate([x, input2])output2 = Dense(10, activation=softmax, name="output2")(y)# 定義多輸入多輸出模型model = Model(inputs=[input1, input2], outputs=[output1, output2])# 定義損失函數、優化函數和評測方法。若多個輸出的損失函數相同，可以指定一個損失函數；# 比如：loss = {output1:binary_crossentropy,output2:binary_crossentropy}model.compile(loss=keras.losses.categorical_crossentropy, optimizer=keras.optimizers.SGD(), loss_weights=[1, 0.1], metrics=[accuracy])# ————————3.0 訓練模型————————# 注意有兩個輸入和兩個輸出model.fit([trainX, trainY], [trainY, trainY], batch_size=128, epochs=20, validation_data=([testX, testY], [testY, testY]))

結果：

>>> import cx32_keras_mul_in_out/home/sun/pycharm-community-2018.1.1/helpers/pydev/_pydev_bundle/pydev_import_hook.py:20: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. module = self._system_import(name, *args, **kwargs)Using TensorFlow backend.Train on 60000 samples, validate on 10000 samplesEpoch 1/202018-04-19 02:38:25.667134: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 128/60000 [..............................] - ETA: 25:56 - loss: 2.5522 - output1_loss: 2.3070 - output2_loss: 2.4516 - output1_acc: 0.1094 - output2_acc: 0.0000e+00 256/60000 [..............................] - ETA: 13:51 - loss: 2.5507 - output1_loss: 2.3044 - output2_loss: 2.4636 - output1_acc: 0.1016 - output2_acc: 0.0000e+00/home/sun/anaconda2/envs/tensorflow/lib/python2.7/site-packages/keras/callbacks.py:120: UserWarning: Method on_batch_end() is slow compared to the batch update (0.175879). Check your callbacks. % delta_t_median) 384/60000 [..............................] - ETA: 9:59 - loss: 2.5516 - output1_loss: 2.3055 - output2_loss: 2.4606 - output1_acc: 0.0990 - output2_acc: 0.0000e+00 1664/60000 [..............................] - ETA: 2:17 - loss: 2.5524 - output1_loss: 2.3061 - output2_loss: 2.4636 - output1_acc: 0.0907 - output2_acc: 0.0000e+00 3072/60000 [>.............................] - ETA: 1:13 - loss: 2.5511 - output1_loss: 2.3054 - output2_loss: 2.4566 - output1_acc: 0.1068 - output2_acc: 0.0000e+00 4864/60000 [=>............................] - ETA: 45s - loss: 2.5500 - output1_loss: 2.3050 - output2_loss: 2.4497 - output1_acc: 0.1135 - output2_acc: 0.0000e+00 6016/60000 [==>...........................] - ETA: 36s - loss: 2.5497 - output1_loss: 2.3047 - output2_loss: 2.4506 - output1_acc: 0.1139 - output2_acc: 0.0000e+00 7168/60000 [==>...........................] - ETA: 30s - loss: 2.5495 - output1_loss: 2.3044 - output2_loss: 2.4508 - output1_acc: 0.1138 - output2_acc: 0.0000e+00# ··········································································53120/60000 [=========================>....] - ETA: 0s - loss: 1.8449 - output1_loss: 1.7006 - output2_loss: 1.4433 - output1_acc: 0.2917 - output2_acc: 0.854454272/60000 [==========================>...] - ETA: 0s - loss: 1.8444 - output1_loss: 1.7001 - output2_loss: 1.4428 - output1_acc: 0.2920 - output2_acc: 0.854355424/60000 [==========================>...] - ETA: 0s - loss: 1.8440 - output1_loss: 1.6997 - output2_loss: 1.4425 - output1_acc: 0.2920 - output2_acc: 0.854556192/60000 [===========================>..] - ETA: 0s - loss: 1.8435 - output1_loss: 1.6993 - output2_loss: 1.4421 - output1_acc: 0.2922 - output2_acc: 0.854957216/60000 [===========================>..] - ETA: 0s - loss: 1.8442 - output1_loss: 1.7000 - output2_loss: 1.4422 - output1_acc: 0.2919 - output2_acc: 0.854858112/60000 [============================>.] - ETA: 0s - loss: 1.8440 - output1_loss: 1.6998 - output2_loss: 1.4418 - output1_acc: 0.2925 - output2_acc: 0.855159648/60000 [============================>.] - ETA: 0s - loss: 1.8441 - output1_loss: 1.6999 - output2_loss: 1.4415 - output1_acc: 0.2927 - output2_acc: 0.854960000/60000 [==============================] - 3s 44us/step - loss: 1.8438 - output1_loss: 1.6996 - output2_loss: 1.4414 - output1_acc: 0.2926 - output2_acc: 0.8550 - val_loss: 1.8498 - val_output1_loss: 1.7070 - val_output2_loss: 1.4284 - val_output1_acc: 0.2950 - val_output2_acc: 0.8525

Keras和原生API結合使用

問題：1. 原生keras api對處理數據流的支持不是太好，基本是一次性將數據載入進內存；2. 原生keras api無法支持分散式訓練。

解決：將keras和原生態的TensorFlow API進行結合

下面給出在MNIST數據集上的應用：

# -*- coding: utf-8 -*-import tensorflow as tffrom tensorflow.examples.tutorials.mnist import input_datamnist_data = input_data.read_data_sets(D:/workstation/tensorflow/MNIST_data, one_hot=True)# 通過TensorFlow中的placeholder定義輸入。x = tf.placeholder(tf.float32, shape=(None, 784))y_ = tf.placeholder(tf.float32, shape=(None, 10))net = tf.keras.layers.Dense(500, activation=relu)(x)y = tf.keras.layers.Dense(10, activation=softmax)(net)# 定義預測的正確率作為指標acc_value = tf.reduce_mean( tf.keras.metrics.categorical_accuracy(y_, y))# 定義損失函數和優化和優化方法。【混用了keras和原生TF API】loss = tf.reduce_mean(tf.keras.losses.categorical_crossentropy(y_, y))train_step = tf.train.GradientDescentOptimizer(0.5).minimize(loss)# 使用原生態tf的方式進行訓練可以實現分散式效果with tf.Session() as sess: tf.global_variables_initializer().run() for i in range(3000): xs, ys = mnist_data.train.next_batch(100) _, loss_value = sess.run([train_step, loss], feed_dict={x: xs, y_: ys}) if i % 1000 == 0: print("After %d training step(s), loss on training batch is " "%g." % (i, loss_value)) print acc_value.eval(feed_dict={x: mnist_data.test.images, y_: mnist_data.test.labels})

結論：

>>> import cx33_keras_tf_APISuccessfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.Extracting D:/workstation/tensorflow/MNIST_data/train-images-idx3-ubyte.gzSuccessfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.Extracting D:/workstation/tensorflow/MNIST_data/train-labels-idx1-ubyte.gzSuccessfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.Extracting D:/workstation/tensorflow/MNIST_data/t10k-images-idx3-ubyte.gzSuccessfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.Extracting D:/workstation/tensorflow/MNIST_data/t10k-labels-idx1-ubyte.gzAfter 0 training step(s), loss on training batch is 2.26019.After 1000 training step(s), loss on training batch is 0.0932196.After 2000 training step(s), loss on training batch is 0.040669.0.9788

步數較少，適當增加效果可能會更好。代碼注釋較少，前面基本上介紹過了，只有部分混用部分需要注意一下。

Estimator介紹

官方提供的高層API，更好地整合了原生TensorFlow提供的功能。

# -*- coding: utf-8 -*-import numpy as npimport tensorflow as tffrom tensorflow.examples.tutorials.mnist import input_data# 輸出相應的tf日誌tf.logging.set_verbosity(tf.logging.INFO)mnist = input_data.read_data_sets("/mnt/hgfs/linux_work/tensorflow/MNIST_data", one_hot=False)# 定義模型的輸入。feature_columns = [tf.feature_column.numeric_column("image", shape=[784])]# 通過DNNClassifier定義模型。# feature_columns指定數據；model_dir則保存訓練中loss的變化以及一些其他指標保存到model_dir目錄estimator = tf.estimator.DNNClassifier(feature_columns=feature_columns, hidden_units=[500], n_classes=10, optimizer=tf.train.AdamOptimizer(), model_dir="log")# ————————2.0 訓練模型————————# 定義輸入和輸出數據。epochs指定訓練輪數；shrffle指定是否打亂；train_input_fn = tf.estimator.inputs.numpy_input_fn( x={"image": mnist.train.images}, y=mnist.train.labels.astype(np.int32), num_epochs=None, batch_size=128, shuffle=True)estimator.train(input_fn=train_input_fn, steps=10000)# ————————3.0 測試模型————————test_input_fn = tf.estimator.inputs.numpy_input_fn( x={"image": mnist.test.images}, y=mnist.test.labels.astype(np.int32), num_epochs=1, batch_size=128, shuffle=False)test_results = estimator.evaluate(input_fn=test_input_fn)accuracy_score = test_results["accuracy"]print("
Test accuracy: %g %%" % (accuracy_score*100))print test_results

結果：

>>> import cx41_estimator_DNN/home/sun/pycharm-community-2018.1.1/helpers/pydev/_pydev_bundle/pydev_import_hook.py:20: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. module = self._system_import(name, *args, **kwargs)Extracting D:/workstation/tensorflow/MNIST_data/train-images-idx3-ubyte.gzExtracting D:/workstation/tensorflow/MNIST_data/train-labels-idx1-ubyte.gzExtracting D:/workstation/tensorflow/MNIST_data/t10k-images-idx3-ubyte.gzExtracting D:/workstation/tensorflow/MNIST_data/t10k-labels-idx1-ubyte.gzINFO:tensorflow:Using default config.INFO:tensorflow:Using config: {_save_checkpoints_secs: 600, _session_config: None, _keep_checkpoint_max: 5, _task_type: worker, _global_id_in_cluster: 0, _is_chief: True, _cluster_spec: <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f81cea78910>, _evaluation_master: , _save_checkpoints_steps: None, _keep_checkpoint_every_n_hours: 10000, _service: None, _num_ps_replicas: 0, _tf_random_seed: None, _master: , _num_worker_replicas: 1, _task_id: 0, _log_step_count_steps: 100, _model_dir: log, _save_summary_steps: 100}INFO:tensorflow:Calling model_fn.INFO:tensorflow:Done calling model_fn.INFO:tensorflow:Create CheckpointSaverHook.INFO:tensorflow:Graph was finalized.2018-04-20 00:49:07.886575: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMAINFO:tensorflow:Restoring parameters from log/model.ckpt-1INFO:tensorflow:Running local_init_op.INFO:tensorflow:Done running local_init_op.INFO:tensorflow:Saving checkpoints for 2 into log/model.ckpt.INFO:tensorflow:loss = 263.1465, step = 2INFO:tensorflow:global_step/sec: 112.234# ··············硬體是硬傷的分割線·························

estmator可以更深層次的封裝神經網路的定義和訓練過程，用戶僅需要關注模型的輸入和模型的結構，其它的工作都可以通過estimator自動完成。但是estimator功能有限，為了更方便使用，estimator也可以實現自定義模型。

Estimator自定義模型

為了更方便選擇模型結構，模型使用的損失函數和每一層使用的激活函數都可以進行自定義。下面是自定義方式使用卷積神經網路解決MNIST問題：

# -*- coding: utf-8 -*-import numpy as npimport tensorflow as tffrom tensorflow.examples.tutorials.mnist import input_datatf.logging.set_verbosity(tf.logging.INFO)# 定義模型結構，可以是自定義或者高層封裝# istraining判定是否是訓練；最後返回前向傳播結果def lenet(x, is_training): x = tf.reshape(x, shape=[-1, 28, 28, 1]) conv1 = tf.layers.conv2d(x, 32, 5, activation=tf.nn.relu) conv1 = tf.layers.max_pooling2d(conv1, 2, 2) conv2 = tf.layers.conv2d(conv1, 64, 3, activation=tf.nn.relu) conv2 = tf.layers.max_pooling2d(conv2, 2, 2) fc1 = tf.contrib.layers.flatten(conv2) fc1 = tf.layers.dense(fc1, 1024) fc1 = tf.layers.dropout(fc1, rate=0.4, training=is_training) return tf.layers.dense(fc1, 10)# 自定義Estimator模型# features：輸入函數中提供的輸入層張量，即numpy_input_fn中x參數；# labels：指正確標籤，即numpy_input_fn中y參數；# mode：有三種train、evaluate、predict# params:可以是任何所需的參數；def model_fn(features, labels, mode, params): # 定義神經網路的結構並通過輸入獲取前向傳播結果 predict = lenet( features["image"], mode == tf.estimator.ModeKeys.TRAIN) # 如果是預測模式，則需要將結果返回即可 if mode == tf.estimator.ModeKeys.PREDICT: return tf.estimator.EstimatorSpec( mode=mode, predictions={"result": tf.argmax(predict, 1)}) # 定義損失函數 loss = tf.reduce_mean( tf.nn.sparse_softmax_cross_entropy_with_logits( logits=predict, labels=labels)) # 定義優化函數 optimizer = tf.train.GradientDescentOptimizer( learning_rate=params["learning_rate"]) # 定義訓練過程 train_op = optimizer.minimize( loss=loss, global_step=tf.train.get_global_step()) # 定義評測標準，運行evalute的時候會計算這裡 eval_metric_ops = { "accuracy": tf.metrics.accuracy( tf.argmax(predict, 1), labels) } # 返回模型訓練過程所需喲啊使用的損失函數、訓練過程和評測方法 return tf.estimator.EstimatorSpec( mode=mode, loss=loss, train_op=train_op, eval_metric_ops=eval_metric_ops)# 一下內容和estimator基本用法裡面的基本一致mnist = input_data.read_data_sets("/mnt/hgfs/linux_work/tensorflow/MNIST_data", one_hot=False)model_params = {"learning_rate": 0.01}estimator = tf.estimator.Estimator(model_fn=model_fn, params=model_params)train_input_fn = tf.estimator.inputs.numpy_input_fn( x={"image": mnist.train.images}, y=mnist.train.labels.astype(np.int32), num_epochs=None, batch_size=128, shuffle=True)estimator.train(input_fn=train_input_fn, steps=30000)# 在據上模型test_input_fn = tf.estimator.inputs.numpy_input_fn( x={"image": mnist.test.images}, y=mnist.test.labels.astype(np.int32), num_epochs=1, batch_size=128, shuffle=False)test_results = estimator.evaluate(input_fn=test_input_fn)accuracy_score = test_results["accuracy"]print("
Test accuracy: %g %%" % (accuracy_score*100))# 使用訓練好的模型在新數據上預測結果predict_input_fn = tf.estimator.inputs.numpy_input_fn( x={"image": mnist.test.images[:10]}, num_epochs=1, shuffle=False)predictions = estimator.predict(input_fn=predict_input_fn)for i, p in enumerate(predictions): print("Prediction %s: %s" % (i + 1, p["result"]))

結果：

>>> import cx43_estimator_selfdefineExtracting /mnt/hgfs/linux_work/tensorflow/MNIST_data/train-images-idx3-ubyte.gzExtracting /mnt/hgfs/linux_work/tensorflow/MNIST_data/train-labels-idx1-ubyte.gzExtracting /mnt/hgfs/linux_work/tensorflow/MNIST_data/t10k-images-idx3-ubyte.gzExtracting /mnt/hgfs/linux_work/tensorflow/MNIST_data/t10k-labels-idx1-ubyte.gzINFO:tensorflow:Using default config.WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp2wfNL0INFO:tensorflow:Using config: {_save_checkpoints_secs: 600, _session_config: None, _keep_checkpoint_max: 5, _task_type: worker, _global_id_in_cluster: 0, _is_chief: True, _cluster_spec: <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f437967f110>, _evaluation_master: , _save_checkpoints_steps: None, _keep_checkpoint_every_n_hours: 10000, _service: None, _num_ps_replicas: 0, _tf_random_seed: None, _master: , _num_worker_replicas: 1, _task_id: 0, _log_step_count_steps: 100, _model_dir: /tmp/tmp2wfNL0, _save_summary_steps: 100}INFO:tensorflow:Calling model_fn.INFO:tensorflow:Done calling model_fn.INFO:tensorflow:Create CheckpointSaverHook.INFO:tensorflow:Graph was finalized.2018-04-20 03:23:39.704554: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMAINFO:tensorflow:Running local_init_op.INFO:tensorflow:Done running local_init_op.INFO:tensorflow:Saving checkpoints for 1 into /tmp/tmp2wfNL0/model.ckpt.INFO:tensorflow:loss = 2.2842863, step = 1INFO:tensorflow:global_step/sec: 7.33635INFO:tensorflow:loss = 1.5192015, step = 101 (13.631 sec)# ·····················太慢了難以進行的分割線············

使用數據集（Dataset）作為Estmator輸入

這裡使用iris數據集，包含4個特徵分辨3中類型的植物。鳶尾花數據集算是數據集中的元老級人物，還有點小故事，有興趣的可以去搜一下。數據集中一共包含150個樣本，120條訓練數據，30條測試數據，且儲存文件為csv格式。下面是下載鏈接：

download.tensorflow.org download.tensorflow.org

# -*- coding: utf-8 -*-import tensorflow as tftf.logging.set_verbosity(tf.logging.INFO)# batch的自定義輸入函數需要每一次被調用時得到一個batch數據# 返回的batch數據中包含特徵batch和標籤batchdef my_input_fn(file_path, perform_shuffle=False, repeat_count=1): def decode_csv(line): # 將一行的數據解析出來 parsed_line = tf.decode_csv(line, [[0.], [0.], [0.], [0.], [0]]) # 返回的是一個字典，這裡需要和feature_columns的定義匹配 return {"x": parsed_line[:-1]}, parsed_line[-1:] # 使用數據集處理輸入的數據。 dataset = (tf.contrib.data.TextLineDataset(file_path) .skip(1) .map(decode_csv)) # 通過定義的數據集得到一個batch的輸入數據，即自定義的輸入過程的返回結果 if perform_shuffle: dataset = dataset.shuffle(buffer_size=256) dataset = dataset.repeat(repeat_count) dataset = dataset.batch(32) iterator = dataset.make_one_shot_iterator() batch_features, batch_labels = iterator.get_next() # 如果是為預測過程提供數據，那麼batch_labels可以直接使用None return batch_features, batch_labels# 定義Estimatorfeature_columns = [tf.feature_column.numeric_column("x", shape=[4])]classifier = tf.estimator.DNNClassifier( feature_columns=feature_columns, hidden_units=[10, 10], n_classes=3)# 使用lambda表達式將訓練相關的信息傳入自定義輸入數據處理函數並生成estimator需要的輸入函數classifier.train( input_fn=lambda: my_input_fn("/mnt/hgfs/linux_work/tensorflow/iris_training.csv", True, 100))# 進行評估test_results = classifier.evaluate( input_fn=lambda: my_input_fn("/mnt/hgfs/linux_work/tensorflow/iris_test.csv", False, 1))print("
Test accuracy: %g %%" % (test_results["accuracy"]*100))

結果：

>>> import cx42_estimator_dataset/home/sun/pycharm-community-2018.1.1/helpers/pydev/_pydev_bundle/pydev_import_hook.py:20: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. module = self._system_import(name, *args, **kwargs)INFO:tensorflow:Using default config.WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp01NtRBINFO:tensorflow:Using config: {_save_checkpoints_secs: 600, _session_config: None, _keep_checkpoint_max: 5, _task_type: worker, _global_id_in_cluster: 0, _is_chief: True, _cluster_spec: <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f6184247f50>, _evaluation_master: , _save_checkpoints_steps: None, _keep_checkpoint_every_n_hours: 10000, _service: None, _num_ps_replicas: 0, _tf_random_seed: None, _master: , _num_worker_replicas: 1, _task_id: 0, _log_step_count_steps: 100, _model_dir: /tmp/tmp01NtRB, _save_summary_steps: 100}WARNING:tensorflow:From /mnt/hgfs/linux_work/tensorflow/cx42_estimator_dataset.py:17: __init__ (from tensorflow.contrib.data.python.ops.readers) is deprecated and will be removed in a future version.Instructions for updating:Use `tf.data.TextLineDataset`.Traceback (most recent call last): File "<input>", line 1, in <module> File "/home/sun/pycharm-community-2018.1.1/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 20, in do_import module = self._system_import(name, *args, **kwargs) File "/mnt/hgfs/linux_work/tensorflow/cx42_estimator_dataset.py", line 40, in <module> input_fn=lambda: my_input_fn("/mnt/hgfs/linux_work/tensorflow/iris_training.csv", True, 100))# ···············省略內容的分割線························· File "/home/sun/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1585, in tf_map_func ret, [t.get_shape() for t in nest.flatten(ret)])AttributeError: list object has no attribute get_shape

備註：代碼運行遇到了問題，顯示模塊中缺少相關方法，這裡我查了一些資料但是沒有找到解答。

推薦閱讀：

相關文章