HomeDeep LearningDive Into TensorFlow, Part VI: Beyond Deep Learning
Deep Learning Specialization on Coursera

This is the sixth article in the series “Dive Into TensorFlow“, here is an index of all the articles in the series that have been published to date:

Part I: Getting Started with TensorFlow
Part II: Basic Concepts
Part III: Ubuntu16.04+GTX 1080+CUDA8.0+cuDNN5.0+TensorFlow
Part IV: Hello MNIST
Part V: Deep MNIST
Part VI: Beyond Deep Learning (this article)

In addition to deep learning, TensorFlow provides a high-level machine leaning api: TF Learn, which makes it easy to configure, train and evaluate machine learning models. TF Learn is a simplified interface for TensorFlow, to get people started on predictive analytics and data mining. The API covers a variety of needs: from linear models to Deep Learning applications like text and image understanding.

Why TensorFlow Learn?

To smooth the transition from the scikit-learn world of one-liner machine learning into the more open world of building different shapes of ML models. You can start by using fit/predict and slide into TensorFlow APIs as you are getting comfortable.

To provide a set of reference models that will be easy to integrate with existing code.

Learning from Small Data

Iris flower data set is a small data, but it’s perhaps the best known data set in the machine learning history:

The Iris flower data set or Fisher’s Iris data set is a multivariate data set introduced by Ronald Fisher in his 1936 paper The use of multiple measurements in taxonomic problems as an example of linear discriminant analysis. It is sometimes called Anderson’s Iris data set because Edgar Anderson collected the data to quantify the morphologic variation of Iris flowers of three related species. Two of the three species were collected in the GaspĂ© Peninsula “all from the same pasture, and picked on the same day and measured at the same time by the same person with the same apparatus”.

The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimetres. Based on the combination of these four features, Fisher developed a linear discriminant model to distinguish the species from each other.

Here is a picture to describe the iris data from “PCA example with Iris Data-set“:

Based on the scikit-learn, we can load the iris data quickly and train a linear classifier by TF Learn:

Python 2.7.6 (default, Jun  3 2014, 07:43:23) 
Type "copyright", "credits" or "license" for more information.
 
IPython 3.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
 
In [1]: from __future__ import absolute_import
 
In [2]: from __future__ import division
 
In [3]: from __future__ import print_function
 
In [4]: import tensorflow as tf
 
In [5]: import numpy as np
 
In [6]: from sklearn import datasets, metrics
 
In [7]: iris = datasets.load_iris()
 
In [8]: dir(iris)
Out[8]: 
['DESCR',
  ......
 'clear',
 'copy',
 'data',
 'feature_names',
 'fromkeys',
 'get',
 'has_key',
 'items',
 'iteritems',
 'iterkeys',
 'itervalues',
 'keys',
 'pop',
 'popitem',
 'setdefault',
 'target',
 'target_names',
 'update',
 'values',
 'viewitems',
 'viewkeys',
 'viewvalues']
 
In [9]: print(iris.DESCR)
Iris Plants Database
 
Notes
-----
Data Set Characteristics:
    :Number of Instances: 150 (50 in each of three classes)
    :Number of Attributes: 4 numeric, predictive attributes and the class
    :Attribute Information:
        - sepal length in cm
        - sepal width in cm
        - petal length in cm
        - petal width in cm
        - class:
                - Iris-Setosa
                - Iris-Versicolour
                - Iris-Virginica
    :Summary Statistics:
    ============== ==== ==== ======= ===== ====================
                    Min  Max   Mean    SD   Class Correlation
    ============== ==== ==== ======= ===== ====================
    sepal length:   4.3  7.9   5.84   0.83    0.7826
    sepal width:    2.0  4.4   3.05   0.43   -0.4194
    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
    petal width:    0.1  2.5   1.20  0.76     0.9565  (high!)
    ============== ==== ==== ======= ===== ====================
    :Missing Attribute Values: None
    :Class Distribution: 33.3% for each of 3 classes.
    :Creator: R.A. Fisher
    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
    :Date: July, 1988
 
This is a copy of UCI ML iris datasets.
http://archive.ics.uci.edu/ml/datasets/Iris
 
The famous Iris database, first used by Sir R.A Fisher
 
This is perhaps the best known database to be found in the
pattern recognition literature.  Fisher paper is a classic in the field and
is referenced frequently to this day.  (See Duda & Hart, for example.)  The
data set contains 3 classes of 50 instances each, where each class refers to a
type of iris plant.  One class is linearly separable from the other 2; the
latter are NOT linearly separable from each other.
 
References
----------
   - Fisher,R.A. "The use of multiple measurements in taxonomic problems"
     Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
     Mathematical Statistics" (John Wiley, NY, 1950).
   - Duda,R.O., & Hart,P.E. (1973) Pattern Classification and Scene Analysis.
     (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
   - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
     Structure and Classification Rule for Recognition in Partially Exposed
     Environments".  IEEE Transactions on Pattern Analysis and Machine
     Intelligence, Vol. PAMI-2, No. 1, 67-71.
   - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
     on Information Theory, May 1972, 431-433.
   - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al AUTOCLASS II
     conceptual clustering system finds 3 classes in the data.
   - Many, many more ...
 
 
In [10]: print(iris.data)
[[ 5.1  3.5  1.4  0.2]
 [ 4.9  3.   1.4  0.2]
 [ 4.7  3.2  1.3  0.2]
 [ 4.6  3.1  1.5  0.2]
 [ 5.   3.6  1.4  0.2]
 ......
 [ 6.7  3.   5.2  2.3]
 [ 6.3  2.5  5.   1.9]
 [ 6.5  3.   5.2  2. ]
 [ 6.2  3.4  5.4  2.3]
 [ 5.9  3.   5.1  1.8]]
 
In [11]: print(len(iris.data))
150
 
In [12]: print(iris.target)
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2]
 
In [13]: print(len(iris.target))
150
 
In [14]: print(iris.feature_names)
['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
 
In [15]: print(iris.feature_names)
['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
 
# Split iris data in train and test data
# A random permutation, to split the data randomly
In [16]: iris_X = iris.data
 
In [17]: iris_y = iris.target
 
In [18]: np.unique(iris_y)
Out[18]: array([0, 1, 2])
 
In [19]: np.random.seed(0)
 
In [20]: indices = np.random.permutation(len(iris_X))
 
In [21]: print(indices)
[114  62  33 107   7 100  40  86  76  71 134  51  73  54  63  37  78  90
  45  16 121  66  24   8 126  22  44  97  93  26 137  84  27 127 132  59
  18  83  61  92 112   2 141  43  10  60 116 144 119 108  69 135  56  80
 123 133 106 146  50 147  85  30 101  94  64  89  91 125  48  13 111  95
  20  15  52   3 149  98   6  68 109  96  12 102 120 104 128  46  11 110
 124  41 148   1 113 139  42   4 129  17  38   5  53 143 105   0  34  28
  55  75  35  23  74  31 118  57 131  65  32 138  14 122  19  29 130  49
 136  99  82  79 115 145  72  77  25  81 140 142  39  58  88  70  87  36
  21   9 103  67 117  47]
 
In [22]: iris_X_train = iris_X[indices[:-30]]
 
In [23]: iris_y_train = iris_y[indices[:-30]]
 
In [24]: print(iris_X_train)
[[ 5.8  2.8  5.1  2.4]
 [ 6.   2.2  4.   1. ]
 [ 5.5  4.2  1.4  0.2]
 [ 7.3  2.9  6.3  1.8]
 [ 5.   3.4  1.5  0.2]
 ......
 [ 4.9  2.4  3.3  1. ]
 [ 7.9  3.8  6.4  2. ]
 [ 6.7  3.1  4.4  1.4]
 [ 5.2  4.1  1.5  0.1]
 [ 6.   3.   4.8  1.8]]
 
In [25]: print(iris_y_train)
[2 1 0 2 0 2 0 1 1 1 2 1 1 1 1 0 1 1 0 0 2 1 0 0 2 0 0 1 1 0 2 1 0 2 2 1 0
 1 1 1 2 0 2 0 0 1 2 2 2 2 1 2 1 1 2 2 2 2 1 2 1 0 2 1 1 1 1 2 0 0 2 1 0 0
 1 0 2 1 0 1 2 1 0 2 2 2 2 0 0 2 2 0 2 0 2 2 0 0 2 0 0 0 1 2 2 0 0 0 1 1 0
 0 1 0 2 1 2 1 0 2]
 
In [26]: print(len(iris_X_train), len(iris_y_train))
120 120
 
In [27]: iris_X_test = iris_X[indices[-30:]]
 
In [28]: iris_y_test = iris_y[indices[-30:]]
 
In [29]: print(iris_X_test)
[[ 5.8  4.   1.2  0.2]
 [ 7.7  2.8  6.7  2. ]
 [ 5.1  3.8  1.5  0.3]
 [ 4.7  3.2  1.6  0.2]
 ......
 [ 6.3  2.3  4.4  1.3]
 [ 5.5  3.5  1.3  0.2]
 [ 5.1  3.7  1.5  0.4]
 [ 4.9  3.1  1.5  0.1]
 [ 6.3  2.9  5.6  1.8]
 [ 5.8  2.7  4.1  1. ]
 [ 7.7  3.8  6.7  2.2]
 [ 4.6  3.2  1.4  0.2]]
 
In [30]: print(iris_y_test)
[0 2 0 0 2 0 2 1 1 1 2 2 1 1 0 1 2 2 0 1 1 1 1 0 0 0 2 1 2 0]
 
In [31]: print(len(iris_X_test), len(iris_y_test))
30 30
 
In [32]: feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(iris_X_train)
 
In [33]: linear_classifier = tf.contrib.learn.LinearClassifier(n_classes=3, feature_columns=feature_columns)
 
In [34]: linear_classifier.fit(iris_X_train, iris_y_train, steps=2000)
Out[34]: Estimator(params={'enable_centered_bias': True, 'weight_column_name': None, 'optimizer': <tensorflow.python.training.ftrl.FtrlOptimizer object at 0x1196e0510>, 'feature_columns': [_RealValuedColumn(column_name='', dimension=4, default_value=None, dtype=tf.float64, normalizer=None)], 'n_classes': 3, 'joint_weights': False, 'gradient_clip_norm': None, 'num_ps_replicas': 0})
 
In [35]: iris_predictions = list(linear_classifier.predict(iris_X_test, as_iterable=True))
 
In [36]: score = metrics.accuracy_score(iris_y_test, iris_predictions)
 
In [37]: print("Accuracy: %f" % score)
Accuracy: 0.933333

Construct a Deep Neural Network Classifier

We can continue use DNNClassifier in TF Learn to learning a deep neural network classifier:

# Build 3 layer DNN with 10, 20, 10 units respectively.
In [38]: dnn_classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
   ....: hidden_units=[10, 20, 10], n_classes=3, model_dir="./tmp/iris_dnn_model")
 
In [39]: dnn_classifier.fit(x=iris_X_train, y=iris_y_train, steps=2000)
Out[39]: Estimator(params={'enable_centered_bias': True, 'activation_fn': <function relu at 0x114b4c5f0>, 'weight_column_name': None, 'hidden_units': [10, 20, 10], 'feature_columns': [_RealValuedColumn(column_name='', dimension=4, default_value=None, dtype=tf.float64, normalizer=None)], 'n_classes': 3, 'optimizer': 'Adagrad', 'dropout': None, 'gradient_clip_norm': None, 'num_ps_replicas': 0})
 
In [40]: accuracy_score = dnn_classifier.evaluate(x=iris_X_test, y=iris_y_test)["accuracy"]
 
In [41]: print("Accuracy: {0:f}".format(accuracy_score))
Accuracy: 0.966667

It seems that dnn classifier got a better accuracy result for iris test data. Now we can use the classifier model to estimate any new data:

In [43]: new_samples = np.array(
   ....: [[6.4, 3.2, 4.5, 1.5], [5.8, 3.1, 5.0, 1.7]], dtype=float)
 
In [44]: y = list(dnn_classifier.predict(new_samples, as_iterable=True))
 
In [45]: print('Predictions:{}'.format(str(y)))
Predictions:[array([1]), array([2])]
 
In [46]: y = list(linear_classifier.predict(new_samples, as_iterable=True))
 
In [48]: print('Predictions:{}'.format(str(y)))
Predictions:[1, 2]

Reference:
TF Learn
tf.contrib.learn Quickstart
Iris flower data set
UCI Iris Data Set
sklearn.datasets.load_iris
PCA example with Iris Data-set
Supervised learning: predicting an output variable from high-dimensional observations

Posted by TextMiner

Deep Learning Specialization on Coursera

Comments

Dive Into TensorFlow, Part VI: Beyond Deep Learning — No Comments

Leave a Reply

Your email address will not be published. Required fields are marked *