Tuesday, May 2, 2017

Building an Artificial Neural Network - Dropout



In [27]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
%config InlineBackend.figure_format = 'retina'

1 Import data

In [28]:
dataset = pd.read_csv("datasets/Churn_Modelling.csv")
In [29]:
dataset.head()
Out[29]:
RowNumber CustomerId Surname CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
0 1 15634602 Hargrave 619 France Female 42 2 0.00 1 1 1 101348.88 1
1 2 15647311 Hill 608 Spain Female 41 1 83807.86 1 0 1 112542.58 0
2 3 15619304 Onio 502 France Female 42 8 159660.80 3 1 0 113931.57 1
3 4 15701354 Boni 699 France Female 39 1 0.00 2 0 0 93826.63 0
4 5 15737888 Mitchell 850 Spain Female 43 2 125510.82 1 1 1 79084.10 0
In [30]:
X = dataset.iloc[:,3:13]
y = dataset.iloc[:,13]
In [31]:
X.head()
Out[31]:
CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary
0 619 France Female 42 2 0.00 1 1 1 101348.88
1 608 Spain Female 41 1 83807.86 1 0 1 112542.58
2 502 France Female 42 8 159660.80 3 1 0 113931.57
3 699 France Female 39 1 0.00 2 0 0 93826.63
4 850 Spain Female 43 2 125510.82 1 1 1 79084.10
In [32]:
y.head()
Out[32]:
0    1
1    0
2    1
3    0
4    0
Name: Exited, dtype: int64

2 Encode categorical independent variables

In [33]:
from sklearn.preprocessing import OneHotEncoder,LabelEncoder
label_encoder_geo = LabelEncoder()
X.iloc[:,1] = label_encoder_geo.fit_transform(X.iloc[:,1])
In [34]:
X.head()
Out[34]:
CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary
0 619 0 Female 42 2 0.00 1 1 1 101348.88
1 608 2 Female 41 1 83807.86 1 0 1 112542.58
2 502 0 Female 42 8 159660.80 3 1 0 113931.57
3 699 0 Female 39 1 0.00 2 0 0 93826.63
4 850 2 Female 43 2 125510.82 1 1 1 79084.10
In [35]:
label_encoder_sex = LabelEncoder()
X.iloc[:,2] = label_encoder_sex.fit_transform(X.iloc[:,2])
In [36]:
X.head()
Out[36]:
CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary
0 619 0 0 42 2 0.00 1 1 1 101348.88
1 608 2 0 41 1 83807.86 1 0 1 112542.58
2 502 0 0 42 8 159660.80 3 1 0 113931.57
3 699 0 0 39 1 0.00 2 0 0 93826.63
4 850 2 0 43 2 125510.82 1 1 1 79084.10
In [37]:
onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()
In [38]:
X
Out[38]:
array([[  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   1.00000000e+00,   1.01348880e+05],
       [  0.00000000e+00,   0.00000000e+00,   1.00000000e+00, ...,
          0.00000000e+00,   1.00000000e+00,   1.12542580e+05],
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   1.13931570e+05],
       ..., 
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          0.00000000e+00,   1.00000000e+00,   4.20855800e+04],
       [  0.00000000e+00,   1.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   9.28885200e+04],
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   3.81907800e+04]])
In [39]:
X[0]
Out[39]:
array([  1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
         6.19000000e+02,   0.00000000e+00,   4.20000000e+01,
         2.00000000e+00,   0.00000000e+00,   1.00000000e+00,
         1.00000000e+00,   1.00000000e+00,   1.01348880e+05])

Let's delete one categorical variable to avoid dummy variable trap

In [40]:
X = X[:,1:]
In [41]:
from sklearn.model_selection import train_test_split
In [42]:
X_train,X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 10)

Feature scaling is compulsory for ANN

In [43]:
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
In [44]:
X_test = sc.fit_transform(X_test)

Let's make ANN

In [45]:
import keras
In [46]:
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

Initialise ANN

In [47]:
classifier = Sequential()

Add the first hidden layer and input layer

In [48]:
classifier.add(Dense(output_dim = 6, init = 'uniform',activation = 'relu', input_dim = 11))

Adding Dropout

In [49]:
classifier.add(Dropout(p = 0.1))

Adding one more hidden layer

In [50]:
classifier.add(Dense(output_dim = 6, init = 'uniform',activation = 'relu'))

Adding another Dropout

In [51]:
classifier.add(Dropout(p = 0.1))

Adding output layer

We can get probability if we use sigmoid activtion function in the output layer
Use 'softmax' if its a multiclass
In [52]:
classifier.add(Dense(output_dim = 1, init = 'uniform',activation = 'sigmoid'))

Compiling ANN

adding stochastic gradient descent for calculating the weights. We use 'adam here
In [53]:
classifier.compile(optimizer='adam',loss ='binary_crossentropy', metrics = ['accuracy'])

Fitting the model

In [54]:
classifier.fit(X_train,y_train, batch_size=10, epochs=100)
Epoch 1/100
8000/8000 [==============================] - 2s - loss: 0.4917 - acc: 0.7980     
Epoch 2/100
8000/8000 [==============================] - 1s - loss: 0.4185 - acc: 0.8210     
Epoch 3/100
8000/8000 [==============================] - 2s - loss: 0.4102 - acc: 0.8237     
Epoch 4/100
8000/8000 [==============================] - 2s - loss: 0.4045 - acc: 0.8255     
Epoch 5/100
8000/8000 [==============================] - 2s - loss: 0.3985 - acc: 0.8270     
Epoch 6/100
8000/8000 [==============================] - 2s - loss: 0.4013 - acc: 0.8225     
Epoch 7/100
8000/8000 [==============================] - 2s - loss: 0.3988 - acc: 0.8267     
Epoch 8/100
8000/8000 [==============================] - 2s - loss: 0.4027 - acc: 0.8209     
Epoch 9/100
8000/8000 [==============================] - 2s - loss: 0.3997 - acc: 0.8244     
Epoch 10/100
8000/8000 [==============================] - 2s - loss: 0.3958 - acc: 0.8260     
Epoch 11/100
8000/8000 [==============================] - 2s - loss: 0.3973 - acc: 0.8256     
Epoch 12/100
8000/8000 [==============================] - 2s - loss: 0.4005 - acc: 0.8255     
Epoch 13/100
8000/8000 [==============================] - 2s - loss: 0.3945 - acc: 0.8234     
Epoch 14/100
8000/8000 [==============================] - 2s - loss: 0.3933 - acc: 0.8311     
Epoch 15/100
8000/8000 [==============================] - 2s - loss: 0.3855 - acc: 0.8370     
Epoch 16/100
8000/8000 [==============================] - 2s - loss: 0.3865 - acc: 0.8369     
Epoch 17/100
8000/8000 [==============================] - 2s - loss: 0.3875 - acc: 0.8361     
Epoch 18/100
8000/8000 [==============================] - 2s - loss: 0.3852 - acc: 0.8374     
Epoch 19/100
8000/8000 [==============================] - 2s - loss: 0.3832 - acc: 0.8410     
Epoch 20/100
8000/8000 [==============================] - 2s - loss: 0.3845 - acc: 0.8397     
Epoch 21/100
8000/8000 [==============================] - 2s - loss: 0.3825 - acc: 0.8367     
Epoch 22/100
8000/8000 [==============================] - 2s - loss: 0.3821 - acc: 0.8390     
Epoch 23/100
8000/8000 [==============================] - 2s - loss: 0.3796 - acc: 0.8425     
Epoch 24/100
8000/8000 [==============================] - 2s - loss: 0.3775 - acc: 0.8400     
Epoch 25/100
8000/8000 [==============================] - 2s - loss: 0.3803 - acc: 0.8411     
Epoch 26/100
8000/8000 [==============================] - 2s - loss: 0.3801 - acc: 0.8384     
Epoch 27/100
8000/8000 [==============================] - 2s - loss: 0.3808 - acc: 0.8430     
Epoch 28/100
8000/8000 [==============================] - 2s - loss: 0.3806 - acc: 0.8401     
Epoch 29/100
8000/8000 [==============================] - 2s - loss: 0.3781 - acc: 0.8417     
Epoch 30/100
8000/8000 [==============================] - 2s - loss: 0.3840 - acc: 0.8341     
Epoch 31/100
8000/8000 [==============================] - 2s - loss: 0.3767 - acc: 0.8420     
Epoch 32/100
8000/8000 [==============================] - 2s - loss: 0.3782 - acc: 0.8409     
Epoch 33/100
8000/8000 [==============================] - 2s - loss: 0.3777 - acc: 0.8385     
Epoch 34/100
8000/8000 [==============================] - 2s - loss: 0.3835 - acc: 0.8354     
Epoch 35/100
8000/8000 [==============================] - 2s - loss: 0.3763 - acc: 0.8402     
Epoch 36/100
8000/8000 [==============================] - 2s - loss: 0.3766 - acc: 0.8427     
Epoch 37/100
8000/8000 [==============================] - 2s - loss: 0.3749 - acc: 0.8404     
Epoch 38/100
8000/8000 [==============================] - 2s - loss: 0.3738 - acc: 0.8435     
Epoch 39/100
8000/8000 [==============================] - 2s - loss: 0.3774 - acc: 0.8392     
Epoch 40/100
8000/8000 [==============================] - 2s - loss: 0.3790 - acc: 0.8412     
Epoch 41/100
8000/8000 [==============================] - 2s - loss: 0.3754 - acc: 0.8410     
Epoch 42/100
8000/8000 [==============================] - 1s - loss: 0.3770 - acc: 0.8411     
Epoch 43/100
8000/8000 [==============================] - 2s - loss: 0.3811 - acc: 0.8387     
Epoch 44/100
8000/8000 [==============================] - 2s - loss: 0.3762 - acc: 0.8420     
Epoch 45/100
8000/8000 [==============================] - 2s - loss: 0.3760 - acc: 0.8427     
Epoch 46/100
8000/8000 [==============================] - 2s - loss: 0.3734 - acc: 0.8456     
Epoch 47/100
8000/8000 [==============================] - 2s - loss: 0.3768 - acc: 0.8395     
Epoch 48/100
8000/8000 [==============================] - 2s - loss: 0.3769 - acc: 0.8400     
Epoch 49/100
8000/8000 [==============================] - 2s - loss: 0.3735 - acc: 0.8400     
Epoch 50/100
8000/8000 [==============================] - 1s - loss: 0.3823 - acc: 0.8367     
Epoch 51/100
8000/8000 [==============================] - 1s - loss: 0.3747 - acc: 0.8411     
Epoch 52/100
8000/8000 [==============================] - 2s - loss: 0.3766 - acc: 0.8414     
Epoch 53/100
8000/8000 [==============================] - 1s - loss: 0.3765 - acc: 0.8369     
Epoch 54/100
8000/8000 [==============================] - 2s - loss: 0.3763 - acc: 0.8386     
Epoch 55/100
8000/8000 [==============================] - 2s - loss: 0.3747 - acc: 0.8391     
Epoch 56/100
8000/8000 [==============================] - 2s - loss: 0.3756 - acc: 0.8396     
Epoch 57/100
8000/8000 [==============================] - 1s - loss: 0.3768 - acc: 0.8400     
Epoch 58/100
8000/8000 [==============================] - 2s - loss: 0.3740 - acc: 0.8396     
Epoch 59/100
8000/8000 [==============================] - 2s - loss: 0.3699 - acc: 0.8430     
Epoch 60/100
8000/8000 [==============================] - 3s - loss: 0.3781 - acc: 0.8405     
Epoch 61/100
8000/8000 [==============================] - 2s - loss: 0.3754 - acc: 0.8411     
Epoch 62/100
8000/8000 [==============================] - 2s - loss: 0.3709 - acc: 0.8434     
Epoch 63/100
8000/8000 [==============================] - 2s - loss: 0.3773 - acc: 0.8410     
Epoch 64/100
8000/8000 [==============================] - 2s - loss: 0.3760 - acc: 0.8392     
Epoch 65/100
8000/8000 [==============================] - 2s - loss: 0.3724 - acc: 0.8432     
Epoch 66/100
8000/8000 [==============================] - 2s - loss: 0.3757 - acc: 0.8405     
Epoch 67/100
8000/8000 [==============================] - 2s - loss: 0.3707 - acc: 0.8436     
Epoch 68/100
8000/8000 [==============================] - 1s - loss: 0.3783 - acc: 0.8396     
Epoch 69/100
8000/8000 [==============================] - 1s - loss: 0.3747 - acc: 0.8400     
Epoch 70/100
8000/8000 [==============================] - 2s - loss: 0.3708 - acc: 0.8444     
Epoch 71/100
8000/8000 [==============================] - 2s - loss: 0.3743 - acc: 0.8421     
Epoch 72/100
8000/8000 [==============================] - 2s - loss: 0.3774 - acc: 0.8397     
Epoch 73/100
8000/8000 [==============================] - 2s - loss: 0.3744 - acc: 0.8395     
Epoch 74/100
8000/8000 [==============================] - 2s - loss: 0.3707 - acc: 0.8416     
Epoch 75/100
8000/8000 [==============================] - 2s - loss: 0.3744 - acc: 0.8410     
Epoch 76/100
8000/8000 [==============================] - 2s - loss: 0.3764 - acc: 0.8421     
Epoch 77/100
8000/8000 [==============================] - 2s - loss: 0.3729 - acc: 0.8397     
Epoch 78/100
8000/8000 [==============================] - 2s - loss: 0.3726 - acc: 0.8389     
Epoch 79/100
8000/8000 [==============================] - 1s - loss: 0.3716 - acc: 0.8421     
Epoch 80/100
8000/8000 [==============================] - 1s - loss: 0.3744 - acc: 0.8362     
Epoch 81/100
8000/8000 [==============================] - 1s - loss: 0.3714 - acc: 0.8370     
Epoch 82/100
8000/8000 [==============================] - 1s - loss: 0.3743 - acc: 0.8390     
Epoch 83/100
8000/8000 [==============================] - 1s - loss: 0.3748 - acc: 0.8361     
Epoch 84/100
8000/8000 [==============================] - 1s - loss: 0.3746 - acc: 0.8372     
Epoch 85/100
8000/8000 [==============================] - 1s - loss: 0.3744 - acc: 0.8332     
Epoch 86/100
8000/8000 [==============================] - 1s - loss: 0.3750 - acc: 0.8351     
Epoch 87/100
8000/8000 [==============================] - 1s - loss: 0.3743 - acc: 0.8392     
Epoch 88/100
8000/8000 [==============================] - 1s - loss: 0.3716 - acc: 0.8405     
Epoch 89/100
8000/8000 [==============================] - 1s - loss: 0.3738 - acc: 0.8399     
Epoch 90/100
8000/8000 [==============================] - 1s - loss: 0.3726 - acc: 0.8387     
Epoch 91/100
8000/8000 [==============================] - 1s - loss: 0.3755 - acc: 0.8391     
Epoch 92/100
8000/8000 [==============================] - 1s - loss: 0.3724 - acc: 0.8392     
Epoch 93/100
8000/8000 [==============================] - 1s - loss: 0.3731 - acc: 0.8452     
Epoch 94/100
8000/8000 [==============================] - 1s - loss: 0.3702 - acc: 0.8414     
Epoch 95/100
8000/8000 [==============================] - 1s - loss: 0.3731 - acc: 0.8417     
Epoch 96/100
8000/8000 [==============================] - 2s - loss: 0.3734 - acc: 0.8405     
Epoch 97/100
8000/8000 [==============================] - 2s - loss: 0.3743 - acc: 0.8417     
Epoch 98/100
8000/8000 [==============================] - 2s - loss: 0.3754 - acc: 0.8411     
Epoch 99/100
8000/8000 [==============================] - 2s - loss: 0.3737 - acc: 0.8392     
Epoch 100/100
8000/8000 [==============================] - 2s - loss: 0.3730 - acc: 0.8430     
Out[54]:
<keras.callbacks.History at 0x124b5aa58>
In [55]:
y_pred = classifier.predict(X_test)
In [56]:
y_pred[0:5]
Out[56]:
array([[ 0.4088383 ],
       [ 0.44399053],
       [ 0.54964072],
       [ 0.06084509],
       [ 0.64304215]], dtype=float32)
In [57]:
y_pred = y_pred >0.5
In [58]:
y_pred[0:5]
Out[58]:
array([[False],
       [False],
       [ True],
       [False],
       [ True]], dtype=bool)

Evaluate the results

In [59]:
from sklearn.metrics import confusion_matrix
In [60]:
confusion_matrix(y_test, y_pred)
Out[60]:
array([[1505,   73],
       [ 243,  179]])
In [63]:
(1505+179)/2000
Out[63]:
0.842

No comments :

Post a Comment