Dive Into TensorFlow, Part VI: Beyond Deep Learning
This is the sixth article in the series “Dive Into TensorFlow“, here is an index of all the articles in the series that have been published to date:
Part I: Getting Started with TensorFlow
Part II: Basic Concepts
Part III: Ubuntu16.04+GTX 1080+CUDA8.0+cuDNN5.0+TensorFlow
Part IV: Hello MNIST
Part V: Deep MNIST
Part VI: Beyond Deep Learning (this article)
In addition to deep learning, TensorFlow provides a high-level machine leaning api: TF Learn, which makes it easy to configure, train and evaluate machine learning models. TF Learn is a simplified interface for TensorFlow, to get people started on predictive analytics and data mining. The API covers a variety of needs: from linear models to Deep Learning applications like text and image understanding.
Why TensorFlow Learn?
To smooth the transition from the scikit-learn world of one-liner machine learning into the more open world of building different shapes of ML models. You can start by using fit/predict and slide into TensorFlow APIs as you are getting comfortable.
To provide a set of reference models that will be easy to integrate with existing code.
Learning from Small Data
Iris flower data set is a small data, but it’s perhaps the best known data set in the machine learning history:
The Iris flower data set or Fisher’s Iris data set is a multivariate data set introduced by Ronald Fisher in his 1936 paper The use of multiple measurements in taxonomic problems as an example of linear discriminant analysis. It is sometimes called Anderson’s Iris data set because Edgar Anderson collected the data to quantify the morphologic variation of Iris flowers of three related species. Two of the three species were collected in the Gaspé Peninsula “all from the same pasture, and picked on the same day and measured at the same time by the same person with the same apparatus”.
The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimetres. Based on the combination of these four features, Fisher developed a linear discriminant model to distinguish the species from each other.
Here is a picture to describe the iris data from “PCA example with Iris Data-set“:
Based on the scikit-learn, we can load the iris data quickly and train a linear classifier by TF Learn:
Python 2.7.6 (default, Jun 3 2014, 07:43:23) Type "copyright", "credits" or "license" for more information. IPython 3.1.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: from __future__ import absolute_import In [2]: from __future__ import division In [3]: from __future__ import print_function In [4]: import tensorflow as tf In [5]: import numpy as np In [6]: from sklearn import datasets, metrics In [7]: iris = datasets.load_iris() In [8]: dir(iris) Out[8]: ['DESCR', ...... 'clear', 'copy', 'data', 'feature_names', 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'target', 'target_names', 'update', 'values', 'viewitems', 'viewkeys', 'viewvalues'] In [9]: print(iris.DESCR) Iris Plants Database Notes ----- Data Set Characteristics: :Number of Instances: 150 (50 in each of three classes) :Number of Attributes: 4 numeric, predictive attributes and the class :Attribute Information: - sepal length in cm - sepal width in cm - petal length in cm - petal width in cm - class: - Iris-Setosa - Iris-Versicolour - Iris-Virginica :Summary Statistics: ============== ==== ==== ======= ===== ==================== Min Max Mean SD Class Correlation ============== ==== ==== ======= ===== ==================== sepal length: 4.3 7.9 5.84 0.83 0.7826 sepal width: 2.0 4.4 3.05 0.43 -0.4194 petal length: 1.0 6.9 3.76 1.76 0.9490 (high!) petal width: 0.1 2.5 1.20 0.76 0.9565 (high!) ============== ==== ==== ======= ===== ==================== :Missing Attribute Values: None :Class Distribution: 33.3% for each of 3 classes. :Creator: R.A. Fisher :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov) :Date: July, 1988 This is a copy of UCI ML iris datasets. http://archive.ics.uci.edu/ml/datasets/Iris The famous Iris database, first used by Sir R.A Fisher This is perhaps the best known database to be found in the pattern recognition literature. Fisher paper is a classic in the field and is referenced frequently to this day. (See Duda & Hart, for example.) The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are NOT linearly separable from each other. References ---------- - Fisher,R.A. "The use of multiple measurements in taxonomic problems" Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to Mathematical Statistics" (John Wiley, NY, 1950). - Duda,R.O., & Hart,P.E. (1973) Pattern Classification and Scene Analysis. (Q327.D83) John Wiley & Sons. ISBN 0-471-22361-1. See page 218. - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System Structure and Classification Rule for Recognition in Partially Exposed Environments". IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-2, No. 1, 67-71. - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule". IEEE Transactions on Information Theory, May 1972, 431-433. - See also: 1988 MLC Proceedings, 54-64. Cheeseman et al AUTOCLASS II conceptual clustering system finds 3 classes in the data. - Many, many more ... In [10]: print(iris.data) [[ 5.1 3.5 1.4 0.2] [ 4.9 3. 1.4 0.2] [ 4.7 3.2 1.3 0.2] [ 4.6 3.1 1.5 0.2] [ 5. 3.6 1.4 0.2] ...... [ 6.7 3. 5.2 2.3] [ 6.3 2.5 5. 1.9] [ 6.5 3. 5.2 2. ] [ 6.2 3.4 5.4 2.3] [ 5.9 3. 5.1 1.8]] In [11]: print(len(iris.data)) 150 In [12]: print(iris.target) [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2] In [13]: print(len(iris.target)) 150 In [14]: print(iris.feature_names) ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)'] In [15]: print(iris.feature_names) ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)'] # Split iris data in train and test data # A random permutation, to split the data randomly In [16]: iris_X = iris.data In [17]: iris_y = iris.target In [18]: np.unique(iris_y) Out[18]: array([0, 1, 2]) In [19]: np.random.seed(0) In [20]: indices = np.random.permutation(len(iris_X)) In [21]: print(indices) [114 62 33 107 7 100 40 86 76 71 134 51 73 54 63 37 78 90 45 16 121 66 24 8 126 22 44 97 93 26 137 84 27 127 132 59 18 83 61 92 112 2 141 43 10 60 116 144 119 108 69 135 56 80 123 133 106 146 50 147 85 30 101 94 64 89 91 125 48 13 111 95 20 15 52 3 149 98 6 68 109 96 12 102 120 104 128 46 11 110 124 41 148 1 113 139 42 4 129 17 38 5 53 143 105 0 34 28 55 75 35 23 74 31 118 57 131 65 32 138 14 122 19 29 130 49 136 99 82 79 115 145 72 77 25 81 140 142 39 58 88 70 87 36 21 9 103 67 117 47] In [22]: iris_X_train = iris_X[indices[:-30]] In [23]: iris_y_train = iris_y[indices[:-30]] In [24]: print(iris_X_train) [[ 5.8 2.8 5.1 2.4] [ 6. 2.2 4. 1. ] [ 5.5 4.2 1.4 0.2] [ 7.3 2.9 6.3 1.8] [ 5. 3.4 1.5 0.2] ...... [ 4.9 2.4 3.3 1. ] [ 7.9 3.8 6.4 2. ] [ 6.7 3.1 4.4 1.4] [ 5.2 4.1 1.5 0.1] [ 6. 3. 4.8 1.8]] In [25]: print(iris_y_train) [2 1 0 2 0 2 0 1 1 1 2 1 1 1 1 0 1 1 0 0 2 1 0 0 2 0 0 1 1 0 2 1 0 2 2 1 0 1 1 1 2 0 2 0 0 1 2 2 2 2 1 2 1 1 2 2 2 2 1 2 1 0 2 1 1 1 1 2 0 0 2 1 0 0 1 0 2 1 0 1 2 1 0 2 2 2 2 0 0 2 2 0 2 0 2 2 0 0 2 0 0 0 1 2 2 0 0 0 1 1 0 0 1 0 2 1 2 1 0 2] In [26]: print(len(iris_X_train), len(iris_y_train)) 120 120 In [27]: iris_X_test = iris_X[indices[-30:]] In [28]: iris_y_test = iris_y[indices[-30:]] In [29]: print(iris_X_test) [[ 5.8 4. 1.2 0.2] [ 7.7 2.8 6.7 2. ] [ 5.1 3.8 1.5 0.3] [ 4.7 3.2 1.6 0.2] ...... [ 6.3 2.3 4.4 1.3] [ 5.5 3.5 1.3 0.2] [ 5.1 3.7 1.5 0.4] [ 4.9 3.1 1.5 0.1] [ 6.3 2.9 5.6 1.8] [ 5.8 2.7 4.1 1. ] [ 7.7 3.8 6.7 2.2] [ 4.6 3.2 1.4 0.2]] In [30]: print(iris_y_test) [0 2 0 0 2 0 2 1 1 1 2 2 1 1 0 1 2 2 0 1 1 1 1 0 0 0 2 1 2 0] In [31]: print(len(iris_X_test), len(iris_y_test)) 30 30 In [32]: feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(iris_X_train) In [33]: linear_classifier = tf.contrib.learn.LinearClassifier(n_classes=3, feature_columns=feature_columns) In [34]: linear_classifier.fit(iris_X_train, iris_y_train, steps=2000) Out[34]: Estimator(params={'enable_centered_bias': True, 'weight_column_name': None, 'optimizer': <tensorflow.python.training.ftrl.FtrlOptimizer object at 0x1196e0510>, 'feature_columns': [_RealValuedColumn(column_name='', dimension=4, default_value=None, dtype=tf.float64, normalizer=None)], 'n_classes': 3, 'joint_weights': False, 'gradient_clip_norm': None, 'num_ps_replicas': 0}) In [35]: iris_predictions = list(linear_classifier.predict(iris_X_test, as_iterable=True)) In [36]: score = metrics.accuracy_score(iris_y_test, iris_predictions) In [37]: print("Accuracy: %f" % score) Accuracy: 0.933333 |
Construct a Deep Neural Network Classifier
We can continue use DNNClassifier in TF Learn to learning a deep neural network classifier:
# Build 3 layer DNN with 10, 20, 10 units respectively. In [38]: dnn_classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns, ....: hidden_units=[10, 20, 10], n_classes=3, model_dir="./tmp/iris_dnn_model") In [39]: dnn_classifier.fit(x=iris_X_train, y=iris_y_train, steps=2000) Out[39]: Estimator(params={'enable_centered_bias': True, 'activation_fn': <function relu at 0x114b4c5f0>, 'weight_column_name': None, 'hidden_units': [10, 20, 10], 'feature_columns': [_RealValuedColumn(column_name='', dimension=4, default_value=None, dtype=tf.float64, normalizer=None)], 'n_classes': 3, 'optimizer': 'Adagrad', 'dropout': None, 'gradient_clip_norm': None, 'num_ps_replicas': 0}) In [40]: accuracy_score = dnn_classifier.evaluate(x=iris_X_test, y=iris_y_test)["accuracy"] In [41]: print("Accuracy: {0:f}".format(accuracy_score)) Accuracy: 0.966667 |
It seems that dnn classifier got a better accuracy result for iris test data. Now we can use the classifier model to estimate any new data:
In [43]: new_samples = np.array( ....: [[6.4, 3.2, 4.5, 1.5], [5.8, 3.1, 5.0, 1.7]], dtype=float) In [44]: y = list(dnn_classifier.predict(new_samples, as_iterable=True)) In [45]: print('Predictions:{}'.format(str(y))) Predictions:[array([1]), array([2])] In [46]: y = list(linear_classifier.predict(new_samples, as_iterable=True)) In [48]: print('Predictions:{}'.format(str(y))) Predictions:[1, 2] |
Reference:
TF Learn
tf.contrib.learn Quickstart
Iris flower data set
UCI Iris Data Set
sklearn.datasets.load_iris
PCA example with Iris Data-set
Supervised learning: predicting an output variable from high-dimensional observations
Posted by TextMiner
Comments
Dive Into TensorFlow, Part VI: Beyond Deep Learning — No Comments