Using data from https://www.kaggle.com/c/digit-recognizer/data, the goal of this notebook is to use classifination tool, specifically KNN, to train and predict digits. All the modules used are as shown below, the main ones used are numpy, pandas and sklearn
In [1]:
import os
import numpy as np
import pandas as pd
import csv
import matplotlib.pyplot as plt
from sklearn import neighbors, metrics, grid_search, cross_validation
import matplotlib.pyplot as plt
##import seaborn as sns
import statsmodels.api as sm
import statsmodels.formula.api as smf
from sklearn import feature_selection, linear_model
import sklearn
pd.set_option('display.max_rows', 10)
pd.set_option('display.notebook_repr_html', True)
pd.set_option('display.max_columns', 10)
%matplotlib inline
plt.style.use('ggplot')
In [2]:
# reading in data from csv file
df = pd.read_csv(os.path.join('~/', 'downloads', 'train_digit.csv'))
df
Out[2]:
- Using only the first 10000 images as an initial analysis to get a general idea of the data set as well as saving time.
- Since the image is given in 28 x 28 pixels, it needed to be reshaped into a "matrix" in order to output the data into an image
In [3]:
#select the first 5000 rows of data, and split label and pixels into two different DataFrames
images = df.iloc[0:5000,1:]
labels = df.iloc[0:5000,:1]
images
Out[3]:
In [4]:
#reshaping and plotting pixels into "matrix" form
i=2
img=images.iloc[i].as_matrix()
img=img.reshape((28,28))
plt.imshow(img,cmap='gray')
plt.title(labels.iloc[i,0])
Out[4]:
In [5]:
# plotting the distribution of greyscale values for a single image
plt.hist(images.iloc[i])
Out[5]:
In [6]:
#since the values for pixels are in grey scale, 0-255, simplifying it to binary, black/white, would make the image more distinct
#thus easier for the classification
images[images>0]=1
img=images.iloc[i].as_matrix().reshape((28,28))
plt.imshow(img,cmap='binary')
plt.title(labels.iloc[i])
Out[6]:
KNN classification: first 10000 images¶
In [7]:
# KNN classification
# using the first 5000 images as training images to create the model
X = images
y = labels
model = neighbors.KNeighborsClassifier(n_neighbors = 5, weights = 'uniform')
model.fit(X,y)
print model.score(X,y)
In [8]:
# Use the next 5000 images as test images to evaluate the model previously created
test_images = df.iloc[5000:10000,1:]
test_labels = df.iloc[5000:10000,:1]
test_images[test_images>0]=1
model.score(test_images,test_labels)
Out[8]:
Further analysis using model created by first 5000 values¶
In [9]:
# Using the model to output predicted values and comparing it with original values
# check: 1 = correct, 0 = wrong
pred = model.predict(test_images)
check_df = test_labels
check_df['pred_labels'] = pred
check_df['check'] = 0
In [10]:
#creating a DataFrame with original label, predicted label and whether it matches
for x in range(5000,10000):
a = check_df.ix[x][0]
b = check_df.ix[x][1]
if ((a - b) == 0):
check_df.ix[x][2] = 1
In [11]:
check_df
Out[11]:
In [12]:
print "total correctly predicted values:", check_df.check.sum()
print "accuracy: ", float(check_df.check.sum())/float(5000)
In [13]:
# Looking at the accuracy of the model for each number, 1-9.
cont_df = pd.DataFrame({'label': range(0,10)})
cont_df['total'] = 0
cont_df['count'] = 0
cont_df['percent'] = float(0)
In [14]:
# Creating a DataFrame with digit, amount of digits presented, amount of predicted digits, and accuracy percentage
for x in range(0,10):
cont_df['total'].ix[x] = check_df[check_df.label == x]['check'].sum()
cont_df['count'].ix[x] = check_df[check_df.label == x]['label'].count()
cont_df['percent'].ix[x] = float(cont_df['total'].ix[x]) / float(cont_df['count'].ix[x])
In [15]:
cont_df
Out[15]:
In [16]:
plot_df = cont_df[['total','count']]
plot_df.plot(kind = 'bar')
plt.title('# predicted VS # OG')
Out[16]:
In [17]:
cont_df.plot(x = 'label',y = 'percent', kind = 'bar',ylim=0.7)
plt.title('% of correct prediction per label')
Out[17]:
From the two graphs above, values 4 and 8 are the ones that has the lowest percentage of correct predictions, while 0 and 1 has the highest
KNN classification: complete dataset¶
In [18]:
df = pd.read_csv(os.path.join('~/', 'downloads', 'train_digit.csv'))
train_df = df.sample(frac = .8, random_state = 0).sort()
test_df = df.drop(train_df.index)
train_images = train_df.iloc[0:,1:]
train_labels = train_df.iloc[0:,:1]
train_images[train_images>0]=1
test_images = test_df.iloc[0:,1:]
test_labels = test_df.iloc[0:,:1]
test_images[test_images>0]=1
In [19]:
X = train_images
y = train_labels
allmodel = neighbors.KNeighborsClassifier(n_neighbors = 5, weights = 'uniform')
allmodel.fit(X,y)
print allmodel.score(X,y)
In [20]:
pred = allmodel.predict(test_images)
check_df = test_labels
check_df['pred_labels'] = pred
check_df['check'] = 0
In [21]:
check_df = check_df.reset_index()
In [22]:
for x in range(len(check_df)):
a = check_df.ix[x][1]
b = check_df.ix[x][2]
if ((a - b) == 0):
check_df.ix[x][3] = 1
In [23]:
print "total correctly predicted values:", check_df.check.sum()
print "accuracy: ", float(check_df.check.sum())/float(check_df.pred_labels.count())
In [24]:
cont_df = pd.DataFrame({'label': range(0,10)})
cont_df['total'] = 0
cont_df['count'] = 0
cont_df['percent'] = float(0)
for x in range(0,10):
cont_df['total'].ix[x] = check_df[check_df.label == x]['check'].sum()
cont_df['count'].ix[x] = check_df[check_df.label == x]['label'].count()
cont_df['percent'].ix[x] = float(cont_df['total'].ix[x]) / float(cont_df['count'].ix[x])
In [25]:
plot_df = cont_df[['total','count']]
plot_df.plot(kind = 'bar')
plt.title('# predicted VS # OG')
Out[25]:
In [26]:
cont_df.plot(x = 'label',y = 'percent', kind = 'bar',ylim=0.7)
plt.title('% of correct prediction per label')
Out[26]:
- similar to the previous model created using 5000 images, the values with largest miss classification are 4 and 8
- increasing data amount increased accuracy by 2%, meaning more data leads to higher accuracy, however it's not perfect
In [35]:
# Reading the test dataset from kaggle, running it through the model and outputting it to a csv file
test_digit_df = pd.read_csv(os.path.join('~/', 'downloads', 'test_digit.csv'))
pred = allmodel.predict(test_digit_df)
subm = pd.DataFrame({'ImageId':[x for x in range(1, len(test_digit_df) + 1)], 'Label':pred})
subm.head()
subm.to_csv('sub.csv')
Out[35]:
- This version of the model scored 93% accuracy on kaggle's test dataset.
Different approaches to increasing accuracy¶
altering the K value for KNN algorithm¶
In [27]:
#using the first 5000 images to speed up testing time
df = pd.read_csv(os.path.join('~/', 'downloads', 'train_digit.csv'))
images = df.iloc[0:5000,1:]
labels = df.iloc[0:5000,:1]
images[images>0]=1
In [28]:
X = images
y = labels
for k in range(1,15):
model = neighbors.KNeighborsClassifier(n_neighbors = k, weights = 'uniform')
model.fit(X,y)
print "k: ", k, "score: ", model.score(X,y)
In [29]:
model = neighbors.KNeighborsClassifier(n_neighbors = 50, weights = 'uniform')
model.fit(X,y)
print "k: ", 50, "score: ", model.score(X,y)
As the K value increases, the score of the model slowly decreases. Thus the most suitable K value is 5.
Distance vs Uniform¶
First 10000 images¶
In [70]:
df = pd.read_csv(os.path.join('~/', 'downloads', 'train_digit.csv'))
images = df.iloc[0:5000,1:]
labels = df.iloc[0:5000,:1]
In [71]:
images[images>0]=1
X = images
y = labels
model = neighbors.KNeighborsClassifier(n_neighbors = 5, weights = 'distance')
model.fit(X,y)
print "score: ", model.score(X,y)
In [72]:
test_images = df.iloc[5000:10000,1:]
test_labels = df.iloc[5000:10000,:1]
test_images[test_images>0]=1
print model.score(test_images,test_labels)
In [73]:
pred = model.predict(test_images)
check_df = test_labels
check_df['pred_labels'] = pred
check_df['check'] = 0
check_df = check_df.reset_index()
for x in range(len(check_df)):
a = check_df.ix[x][1]
b = check_df.ix[x][2]
if ((a - b) == 0):
check_df.ix[x][3] = 1
cont_df = pd.DataFrame({'label': range(0,10)})
cont_df['total'] = 0
cont_df['count'] = 0
cont_df['percent'] = float(0)
for x in range(0,10):
cont_df['total'].ix[x] = check_df[check_df.label == x]['check'].sum()
cont_df['count'].ix[x] = check_df[check_df.label == x]['label'].count()
cont_df['percent'].ix[x] = float(cont_df['total'].ix[x]) / float(cont_df['count'].ix[x])
cont_df.plot(x = 'label',y = 'percent', kind = 'bar',ylim=0.7)
plt.title('% of correct prediction per label')
Out[73]:
- similar to the "Uniform" model, digits 4 and 8 gave low accuracy, however digits 3 and 5 are a lot higher
Complete Dataset¶
In [66]:
df = pd.read_csv(os.path.join('~/', 'downloads', 'train_digit.csv'))
images = df.iloc[0:,1:]
labels = df.iloc[0:,:1]
all_images[all_images>0]=1
X = images
y = labels
distmodel = neighbors.KNeighborsClassifier(n_neighbors = 5, weights = 'distance')
distmodel.fit(X,y)
unimodel = neighbors.KNeighborsClassifier(n_neighbors = 5, weights = 'uniform')
unimodel.fit(X,y)
print "distance score: ", distmodel.score(X,y)
print "uni score: ", unimodel.score(X,y)
In [67]:
pred = distmodel.predict(X)
check_df = y
check_df['pred_labels'] = pred
check_df['check'] = 0
check_df = check_df.reset_index()
for x in range(len(check_df)):
a = check_df.ix[x][1]
b = check_df.ix[x][2]
if ((a - b) == 0):
check_df.ix[x][3] = 1
cont_df = pd.DataFrame({'label': range(0,10)})
cont_df['total'] = 0
cont_df['count'] = 0
cont_df['percent'] = float(0)
for x in range(0,10):
cont_df['total'].ix[x] = check_df[check_df.label == x]['check'].sum()
cont_df['count'].ix[x] = check_df[check_df.label == x]['label'].count()
cont_df['percent'].ix[x] = float(cont_df['total'].ix[x]) / float(cont_df['count'].ix[x])
cont_df.plot(x = 'label',y = 'percent', kind = 'bar',ylim=0.7)
plt.title('% of correct prediction per label')
pred = unimodel.predict(X)
check_df = y
check_df['pred_labels'] = pred
check_df['check'] = 0
check_df = check_df.reset_index()
for x in range(len(check_df)):
a = check_df.ix[x][1]
b = check_df.ix[x][2]
if ((a - b) == 0):
check_df.ix[x][3] = 1
cont_df = pd.DataFrame({'label': range(0,10)})
cont_df['total'] = 0
cont_df['count'] = 0
cont_df['percent'] = float(0)
for x in range(0,10):
cont_df['total'].ix[x] = check_df[check_df.label == x]['check'].sum()
cont_df['count'].ix[x] = check_df[check_df.label == x]['label'].count()
cont_df['percent'].ix[x] = float(cont_df['total'].ix[x]) / float(cont_df['count'].ix[x])
cont_df.plot(x = 'label',y = 'percent', kind = 'bar',ylim=0.7)
plt.title('% of correct prediction per label')
Out[67]:
- Eventhough "Distance" model gave a score of 100, it doesn't translate to the model being 100% accurate.
- One reason why "Distance" gave a higher score than "Uniform" is that the model gave closer neighbors higher "importance", instead of viewing all neighbors equally. This leads to higher accuracy for values like 7 and 8
In [69]:
test_digit_df = pd.read_csv(os.path.join('~/', 'downloads', 'test_digit.csv'))
pred = distmodel.predict(test_digit_df)
subm = pd.DataFrame({'ImageId':[x for x in range(1, len(test_digit_df) + 1)], 'Label':pred})
subm.head()
subm.to_csv('distsub1.csv')
- "Distnace" model scored 96% accuracy on kaggle.