Shulou(Shulou.com)06/01 Report--

In this article Xiaobian for you to introduce in detail "Keras loading mnist dataset error how to solve", detailed content, clear steps, details handled properly, I hope that this "Keras loading mnist dataset error how to solve" article can help you solve doubts, the following follow the editor's ideas slowly in-depth, together to learn new knowledge.

1. Locate the mnist.py file in the local keras directory, directory:

F:python_enter_anaconda510Libsite-packages ensorflowpythonkerasdatasets

two。 Download the mnist.npz file locally at https://s3.amazonaws.com/img-datasets/mnist.npz3. Modify the mnist.py file to the following, and save from _ _ future__ import absolute_importfrom _ _ future__ import divisionfrom _ _ future__ import print_function from.. utils.data_utils import get_fileimport numpy as np def load_data (path='mnist.npz'): "" Loads the MNIST dataset. # Arguments path: path where to cache the dataset locally (relative to ~ / .keras / datasets). # Returns Tuple of Numpy arrays: `(x_train, y_train), (x_test, y_test)`. " The path here is the directory where you just prevented mnist.py. Pay attention to the slash f = np.load (path) x_train, y_train = f ['xtriple'], f ['yawntest`] x_test, y_test = f [' xroomtest`], f ['yprefertest`] f.close () return (x_train, y_train), (x_test, y_test)

Supplement: Keras MNIST handwritten numeral recognition data set

Download MNIST data

1 Import related modules

Import kerasimport numpy as npfrom keras.utils import np_utils import osfrom keras.datasets import mnist

2 download Mnist data for the first time

(X_train_image, y_train_image), (mnist.load_data ()

The first time the mnist.load_data () method is executed, the program will check whether the MNIST dataset file already exists in the user's directory, and if not, it will download it automatically. So it's slow to run the first time.

3 View the downloaded MNIST data files

4 View MNIST data

Print ('train data =', len (X_train_image)) # print ('test data =', len (X_test_image)) to view training data

1 the training set is composed of images and label. Images is a digital monochromatic digital image 28 x 28, and label is the decimal representation of the corresponding number of images.

2 display digital image

Import matplotlib.pyplot as pltdef plot_image (image): fig = plt.gcf () fig.set_size_inches (2) # set the size of the image plt.imshow (image,cmap='binary') # input image image,cmap parameter is set to binary, plt.show () is displayed in black and white grayscale ()

3 View the first data in the training data

Plot_image (x_train_image [0])

View the corresponding tag (real value)

Print (y_train_image [0])

Running result: 5

View multiple training data images and label

Above we only show an image of one set of data, and below we will show an image of several sets of handwritten numbers so that we can view the data.

Def plot_images_labels_prediction (images, labels, prediction, idx, num=10): fig = plt.gcf () fig.set_size_inches (12,14) # set size if num > 25: num= 25 for i in range (0, num): ax = plt.subplot (5,5,1 + I) # divided into 5 X 5 sub-images The third parameter represents the subgraph ax.imshow (images [idx], cmap='binary') title = "label=" + str (labels [IDX]) if len (prediction) > 0: # if there is a predicted value title + = ", predict=" + str (alternative [IDX]) ax.set_title (title Fontsize=10) ax.set_xticks ([]) ax.set_yticks ([]) idx + = 1 plt.show () plot_images_labels_prediction

View the top ten handwritten numbers in the test set

Data preprocessing of multilayer perceptron model based on plot_images_labels_prediction (X-ray testperception imageMagnetics image, [], 0Jing 10)

Feature (eigenvalue of digital image) data preprocessing can be divided into two steps:

(1) convert the original 288x28 digital image into an one-dimensional vector with reshape, whose length is 784, and convert it to float.

(2) Digital standardization of digital image image

1 View the shape of image

Print ("x_train_image:", len (x_train_image), x_train_image.shape) print ("y_train_label:", len (y_train_label), y_train_label.shape) # output: x_train_image: 60000 (60000, 28, 28) y_train_label: 60000 (60000,)

2 convert lmage to reshape

# convert image into reshape x_Train = x_train_image.reshape (60000784). Astype ('float32') x_Test = x_test_image.reshape (10000784). Astype (' float32') print ('x_Train:', x_Train.shape) print ('XeroTest', x_Test.shape)

3 Standardization

The digital standardization of images can improve the accuracy of the subsequent training model, because the number of images is from 0 to 255. it represents the gray level of each point in the graph.

# standardized x_Test_normalize = x_Test/255 x_Train_normalize = x_Train/255

4 View the standardized test set and training set image

Print (x_Train_normalize [0]) # Standardization of the first number in the training set x_train_image: 60000 (60000, 28, 28) y_train_label: 60000 (60000,) [0. 0. 0. 0. 0. 0. ... 0. 0. 0. 0. 0. 0. 0. 0.21568628 0.6745098 0.8862745 0.99215686 0.99215686 0.99215686 0.95686275 0.52156866 0.04313726 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.53333336 0.99215686 0.99215686 0.99215686 0.83137256 0.5294118 0.5176471 0.0627451 0. 0. 0. 0. Preprocessing of Label data

The label tag field is originally a number from 0 to 9, which must be converted into 10 combinations of 0 and 1 using One-hot Encoding unique hot coding. For example, 7 is passed through One-hot encoding.

The conversion to 0000000100 corresponds to exactly 10 neurons in the output layer.

# convert the training set and test set tags into single hot code y_TrainOneHot = np_utils.to_categorical (y_train_label) y_TestOneHot = np_utils.to_categorical (y_test_label) print (y_TrainOneHot [: 5]) # View the tags of the first 5 items [[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.] 5 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.] 0 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.] 4 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.] 1 [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.] introduction to the recognition of MNIST handwritten digital images by 9Keras multiperceptron

1 We will build the multilayer perceptron model as shown in the figure

After establishing model, model must be trained to predict (recognize) these handwritten numbers.

We have finished preprocessing the data. Includes standardization of data set inputs (digital images), one-hot encoding of label

Next we will build a model.

We will build a multilayer perceptron model with 784 neurons in the input layer, 256 neure in the hodden layer and 10 neurons in the output layer.

1 Import related modules

From keras.models import Sequentialfrom keras.layers import Dense

2 establish Sequence model

# build Sequential model model = Sequential ()

3 establish "input layer" and "hidden layer"

Use the model,add () method to add the Dense neural network layer.

The parameter model.add (Dense (units=256, input_dim = 784, keras_initializer='normal', activation='relu')) indicates that units=256 defines the number of neurons in the "hidden layer" as 256input_dim sets the number of neurons in the input layer to 784 kernels initialize the number of neurons in the input layer to 784 kernels using normally distributed random numbers to initialize weight and biasactivation to relu

4 establish the output layer

The parameter model.add (Dense (units=10, kernel_initializer='normal', activation='softmax')) indicates that the number of neurons in the "output layer" defined by units is 10 kernels initializers, which is the same as the activation='softmax activation function softmax.

5 View a summary of the model

Print (model.summary ())

Param's calculation is the last number of neurons * the number of neurons in this layer + the number of neurons in this layer.

Conduct training

1 define the training method

Model.compile (loss='categorical_crossentropy', optimizer='adam',metrics= ['accuracy'])

Loss (loss function): sets the loss function, where cross entropy is used.

Optimizer: the choice of optimizer can make training converge faster.

Metrics: the way to set the evaluation model is accuracy

Start training 2

Train_history = model.fit (epoch=10,batch_size=200,verbose=2, epoch=10,batch_size=200,verbose=2)

Using model.fit () for training, the training process is stored in the train_history variable.

(1) input training data parameters

X = x_Train_normalize

Y = y_TrainOneHot

(2) set the data ratio between training set and verification set

Validation_split=0.2 8: 2 = training set: validation set

(3) set the training cycle and the number of sub-items in each batch.

Epoch=10,batch_size=200

(4) display the training process

Verbose = 2

3. Establish show_train_history to display the training process.

Def show_train_history (train_history,train,validation): plt.plot (train_ train','validation'. Train) plt.plot (train_ approval. Validation]) plt.title ("Train_history") plt.ylabel (train) plt.xlabel ('Epoch') plt.legend ([' train','validation'], loc='upper left') plt.show ()

Accuracy of test data evaluation model

Scores = model.evaluate (print () print ('accuracy=',scores [1])

Accuracy= 0.9769

Make a prediction

Through the previous steps, we have established the model and completed the model training with an acceptable accuracy of 0.97. Next we will use this model for prediction.

1 perform prediction

Prediction = model.predict_classes (x_Test) print (prediction)

Result: [7 2 1... 4 5 6]

2 display 10 forecast results

Plot_images_labels_prediction (x-ray testwriting imageMagnesia yearly testwriting labelrewritingrewrittionrecoveryidxpen 340)

We can see that the first number label is 5 and the result is predicted to be 3.

Display obfuscation matrix

Above, when we predicted the number 5 in the 340th test set, it was mistakenly predicted to be 3. 5. If you want to further know which numbers in our model have higher prediction accuracy and which numbers will tolerate confusion.

Confusion matrix is also called error matrix.

Use Pandas to establish confusion matrix.

ShowMetrix = pd.crosstab (yearly testworthy labeljinghuangjiangjie name = ['label',] Rownames= ['predict']) print (showMetrix) label 01 2 3 4 5 6 7 8 9predict 0971 01 11 02 1 3 01 0 1124 4 00 12 0 4 02 5 0 1009 2 1 03 4 8 03 0 0 5 993 0 1 03 4 44 1 0 5 1 961 03 03 85 3 0 0 16 1 852 7 2 8 36 5 3 3 13 3 939 0 1 07 0 5 13 7 1 0 0 988 5 98 4 0 3 71 11 2 954 19 3 6 0 11 7 2 1 4 4 971

2 using DataFrame

Df = pd.DataFrame ({'label': y_test_label 'predict':prediction}) print (df) label predict0 7 71 2 22 1 13 0 04 4 45 1 16 4 47 9 98 5 59 9 100 011 6 612 9 913 0 014 1 115 5 516 9 917 7 718 3 319 4 420 9 921 6 622 6 623 5 524 4 425 0 026 7 727 4 428 0 029 1 1... ...... 9970 5 59971 2 29972 4 49973 9 99974 4 49975 3 39976 6 69977 4 49978 1 19979 7 79980 2 29981 6 69982 5 69983 0 09984 1 19985 2 29986 3 39987 4 49988 5 59989 6 69990 7 79991 8 89992 9 99993 0 09994 1 19995 2 29996 3 39997 4 49998 5 59999 the hidden layer of 66 increased to 1000 neurons model.add (Dense (units=1000) Input_dim=784, kernel_initializer='normal', activation='relu'))

With the increase of hidden layer neurons, the parameters also increased, so the training time of model became slower.

Add Dropout function to avoid over-fitting

# build Sequential model model = Sequential () model.add (Dense (units=1000, input_dim=784, kernel_initializer='normal', activation='relu')) model.add (Dropout (0.5)) # add Dropout model.add (Dense (units=10, kernel_initializer='normal', activation='softmax'))

The gap between the accuracy of training and the accuracy of verification becomes smaller.

The establishment of a multilayer perceptron model includes two hidden layers.

# build Sequential model model = Sequential () # input layer + "hidden layer" 1 model.add (Dense (units=1000, input_dim=784, kernel_initializer='normal', activation='relu')) model.add (Dropout (0.5)) # add Dropout# "hidden layer" 2model.add (Dense (units=1000, kernel_initializer='normal') Activation='relu')) model.add (Dropout (0.5)) # add Dropout# "output layer" model.add (Dense (units=10, kernel_initializer='normal', activation='softmax')) print (model.summary ())

Code:

Import tensorflow as tfimport kerasimport matplotlib.pyplot as pltimport numpy as npfrom keras.utils import np_utilsfrom keras.datasets import mnistfrom keras.models import Sequentialfrom keras.layers import Densefrom keras.layers import Dropoutimport pandas as pdimport os np.random.seed (10) os.environ ["CUDA_VISIBLE_DEVICES"] = "2" (x_train_image, y_train_label), (mnist.load_data () # # print ('train data =') Len (X_train_image)) # # print ('test data =', len (X_test_image)) def plot_image (image): fig = plt.gcf () fig.set_size_inches (2 plt.imshow 2) # set the size of the image plt.imshow (image,cmap='binary') # input image image,cmap parameter is set to binary to display plt.show () def plot_images_labels_prediction (images, labels) in black and white grayscale Prediction, idx, num=10): fig = plt.gcf () fig.set_size_inches (12,14) if num > 25: num= 25 for i in range (0, num): ax = plt.subplot (5,5,1 + I) # divided into 5 X 5 sub-images, the third parameter represents the number of sub-images ax.imshow (images [IDX]) Cmap='binary') title = "label=" + str (labels [IDX]) if len (prediction) > 0: title + = ", predict=" + str (substitution [IDX]) ax.set_title (title, fontsize=10) ax.set_xticks ([]) ax.set_yticks ([]) idx + = 1 plt.show () def show_train_history (train_history,train) Validation): plt.plot (train_ accuracy. Train]) plt.plot (train_ accuracy. Validation]) plt.title ("Train_history") plt.ylabel (train) plt.xlabel ('Epoch') plt.legend ([' train','validation'], loc='upper left') plt.show () # plot_images_labels_prediction (X-ray imaging imageMagnesia image, [] Print ("x_train_image:", len (x_train_image), x_train_image.shape) print ("y_train_label:", len (y_train_label)) Y_train_label.shape) # convert image into reshape x_Train = x_train_image.reshape (60000784). Astype ('float32') x_Test = x_test_image.reshape (10000784). Astype (' float32') # print ('x_Train:', x_Train.shape) # print ('Xerotest.' X_Test.shape) # Standardization x_Test_normalize = x_Test/255x_Train_normalize = x_Train/255 # print (x_Train_normalize [0]) # Standardization of the first digit in the training set # convert both the training set and the test set tags into a single hot code y_TrainOneHot = np_utils.to_categorical (y_train_label) y_TestOneHot = np_utils.to_categorical (y_test_label) print (y _ TrainOneHot [: 5]) # View the labels of the first five items # build a Sequential model model = Sequential () model.add (Dense (units=1000) Input_dim=784, kernel_initializer='normal', activation='relu') model.add (Dropout (0.5)) # add Dropout# "Hidden layer" 2model.add (Dense (units=1000, kernel_initializer='normal', activation='relu')) model.add (Dropout (0.5)) # add Dropout model.add (Dense (units=10) Kernel_initializer='normal', activation='softmax')) print (model.summary ()) # training method model.compile (loss='categorical_crossentropy', optimizer='adam',metrics= ['accuracy']) # start training train_history = model.fit (x=x_Train_normalize, yearly training Train OneHotMed splitters 0.2, epochs=10, batch_size=200 Verbose=2) show_train_history (train_history,'acc','val_acc') scores = model.evaluate (accuracy=',scores TestOneHot) print () print ('accuracy=',scores [1]) prediction = model.predict_classes (x_Test) print (prediction) plot_images_labels_prediction Rownames= ['predict']) print (showMetrix) df = pd.DataFrame ({' label': y_test_label, 'predict':prediction}) print (df) # plot_image (x_train_image [0]) # # print (y_train_image [0])

Code 2:

Import numpy as npfrom keras.models import Sequentialfrom keras.layers import Dense, Dropout, Deconv2Dfrom keras.utils import np_utilsfrom keras.datasets import mnistfrom keras.optimizers import SGDimport osos.environ ["CUDA_VISIBLE_DEVICES"] = "2" def load_data (): (XerotestReceiving yearly train), (XerotestForce yearly test) = mnist.load_data () number = 10000 x_train = x _ train [0: number] y_train = y _ train [0: number] x_train = x_train.reshape (number) 28 / 28) x_test = x_test.reshape (x_test.shape [0], 28 / 28) x_train = x_train.astype ('float32') x_test = x_test.astype (' float32') y_train = np_utils.to_categorical y_test = np_utils.to_categorical (Yantitest10) x_train = x_train/255 x_test = x_test / 255 return (x_train) Y_train), (Dense (units=689,activation='relu')) model.add (Dropout (0.5)) model.add (Dropout (0.5)) model.add (Dense (units=689)) Activation='relu') model.add (Dropout (0.5)) model.add (Dense (output_dim=10,activation='softmax')) model.compile (loss='categorical_crossentropy',optimizer='adam',metrics= ['accuracy']) model.fit (Xerox Encyclopedia 10000 epochsizeyard 20) res1 = model.evaluate (xanzhongyuzhong paddlesizeyard 10000) print ("Train Acc:", res1 [1]) res2 = model.evaluate (xroomtestparamagram batchsizeyard 10000) print ("Test Acc:" Res2 [1]) read this. This article "how to solve the problem of Keras loading mnist dataset" has been introduced. If you want to master the knowledge points of this article, you still need to practice and use it. If you want to know more about the article, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.