Building an Artificial Neural Network - Dropout

In [27]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
%config InlineBackend.figure_format = 'retina'

1 Import data¶

In [28]:

dataset = pd.read_csv("datasets/Churn_Modelling.csv")

In [29]:

dataset.head()

Out[29]:

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	1	15634602	Hargrave	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	2	15647311	Hill	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	3	15619304	Onio	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	4	15701354	Boni	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	5	15737888	Mitchell	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

In [30]:

X = dataset.iloc[:,3:13]
y = dataset.iloc[:,13]

In [31]:

X.head()

Out[31]:

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
0	619	France	Female	42	2	0.00	1	1	1	101348.88
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58
2	502	France	Female	42	8	159660.80	3	1	0	113931.57
3	699	France	Female	39	1	0.00	2	0	0	93826.63
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10

In [32]:

y.head()

Out[32]:

0    1
1    0
2    1
3    0
4    0
Name: Exited, dtype: int64

2 Encode categorical independent variables¶

In [33]:

from sklearn.preprocessing import OneHotEncoder,LabelEncoder
label_encoder_geo = LabelEncoder()
X.iloc[:,1] = label_encoder_geo.fit_transform(X.iloc[:,1])

In [34]:

X.head()

Out[34]:

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
0	619	0	Female	42	2	0.00	1	1	1	101348.88
1	608	2	Female	41	1	83807.86	1	0	1	112542.58
2	502	0	Female	42	8	159660.80	3	1	0	113931.57
3	699	0	Female	39	1	0.00	2	0	0	93826.63
4	850	2	Female	43	2	125510.82	1	1	1	79084.10

In [35]:

label_encoder_sex = LabelEncoder()
X.iloc[:,2] = label_encoder_sex.fit_transform(X.iloc[:,2])

In [36]:

X.head()

Out[36]:

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
0	619	0	0	42	2	0.00	1	1	1	101348.88
1	608	2	0	41	1	83807.86	1	0	1	112542.58
2	502	0	0	42	8	159660.80	3	1	0	113931.57
3	699	0	0	39	1	0.00	2	0	0	93826.63
4	850	2	0	43	2	125510.82	1	1	1	79084.10

In [37]:

onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()

In [38]:

X

Out[38]:

array([[  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   1.00000000e+00,   1.01348880e+05],
       [  0.00000000e+00,   0.00000000e+00,   1.00000000e+00, ...,
          0.00000000e+00,   1.00000000e+00,   1.12542580e+05],
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   1.13931570e+05],
       ..., 
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          0.00000000e+00,   1.00000000e+00,   4.20855800e+04],
       [  0.00000000e+00,   1.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   9.28885200e+04],
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   3.81907800e+04]])

In [39]:

X[0]

Out[39]:

array([  1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
         6.19000000e+02,   0.00000000e+00,   4.20000000e+01,
         2.00000000e+00,   0.00000000e+00,   1.00000000e+00,
         1.00000000e+00,   1.00000000e+00,   1.01348880e+05])

Let's delete one categorical variable to avoid dummy variable trap¶

In [40]:

X = X[:,1:]

In [41]:

from sklearn.model_selection import train_test_split

In [42]:

X_train,X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 10)

Feature scaling is compulsory for ANN¶

In [43]:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)

In [44]:

X_test = sc.fit_transform(X_test)

Let's make ANN¶

In [45]:

import keras

In [46]:

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

Initialise ANN¶

In [47]:

classifier = Sequential()

Add the first hidden layer and input layer¶

In [48]:

classifier.add(Dense(output_dim = 6, init = 'uniform',activation = 'relu', input_dim = 11))

Adding Dropout¶

In [49]:

classifier.add(Dropout(p = 0.1))

Adding one more hidden layer¶

In [50]:

classifier.add(Dense(output_dim = 6, init = 'uniform',activation = 'relu'))

Adding another Dropout¶

In [51]:

classifier.add(Dropout(p = 0.1))

Adding output layer¶

We can get probability if we use sigmoid activtion function in the output layer
Use 'softmax' if its a multiclass

In [52]:

classifier.add(Dense(output_dim = 1, init = 'uniform',activation = 'sigmoid'))

Compiling ANN¶

adding stochastic gradient descent for calculating the weights. We use 'adam here

In [53]:

classifier.compile(optimizer='adam',loss ='binary_crossentropy', metrics = ['accuracy'])

Fitting the model¶

In [54]:

classifier.fit(X_train,y_train, batch_size=10, epochs=100)

Epoch 1/100
8000/8000 [==============================] - 2s - loss: 0.4917 - acc: 0.7980     
Epoch 2/100
8000/8000 [==============================] - 1s - loss: 0.4185 - acc: 0.8210     
Epoch 3/100
8000/8000 [==============================] - 2s - loss: 0.4102 - acc: 0.8237     
Epoch 4/100
8000/8000 [==============================] - 2s - loss: 0.4045 - acc: 0.8255     
Epoch 5/100
8000/8000 [==============================] - 2s - loss: 0.3985 - acc: 0.8270     
Epoch 6/100
8000/8000 [==============================] - 2s - loss: 0.4013 - acc: 0.8225     
Epoch 7/100
8000/8000 [==============================] - 2s - loss: 0.3988 - acc: 0.8267     
Epoch 8/100
8000/8000 [==============================] - 2s - loss: 0.4027 - acc: 0.8209     
Epoch 9/100
8000/8000 [==============================] - 2s - loss: 0.3997 - acc: 0.8244     
Epoch 10/100
8000/8000 [==============================] - 2s - loss: 0.3958 - acc: 0.8260     
Epoch 11/100
8000/8000 [==============================] - 2s - loss: 0.3973 - acc: 0.8256     
Epoch 12/100
8000/8000 [==============================] - 2s - loss: 0.4005 - acc: 0.8255     
Epoch 13/100
8000/8000 [==============================] - 2s - loss: 0.3945 - acc: 0.8234     
Epoch 14/100
8000/8000 [==============================] - 2s - loss: 0.3933 - acc: 0.8311     
Epoch 15/100
8000/8000 [==============================] - 2s - loss: 0.3855 - acc: 0.8370     
Epoch 16/100
8000/8000 [==============================] - 2s - loss: 0.3865 - acc: 0.8369     
Epoch 17/100
8000/8000 [==============================] - 2s - loss: 0.3875 - acc: 0.8361     
Epoch 18/100
8000/8000 [==============================] - 2s - loss: 0.3852 - acc: 0.8374     
Epoch 19/100
8000/8000 [==============================] - 2s - loss: 0.3832 - acc: 0.8410     
Epoch 20/100
8000/8000 [==============================] - 2s - loss: 0.3845 - acc: 0.8397     
Epoch 21/100
8000/8000 [==============================] - 2s - loss: 0.3825 - acc: 0.8367     
Epoch 22/100
8000/8000 [==============================] - 2s - loss: 0.3821 - acc: 0.8390     
Epoch 23/100
8000/8000 [==============================] - 2s - loss: 0.3796 - acc: 0.8425     
Epoch 24/100
8000/8000 [==============================] - 2s - loss: 0.3775 - acc: 0.8400     
Epoch 25/100
8000/8000 [==============================] - 2s - loss: 0.3803 - acc: 0.8411     
Epoch 26/100
8000/8000 [==============================] - 2s - loss: 0.3801 - acc: 0.8384     
Epoch 27/100
8000/8000 [==============================] - 2s - loss: 0.3808 - acc: 0.8430     
Epoch 28/100
8000/8000 [==============================] - 2s - loss: 0.3806 - acc: 0.8401     
Epoch 29/100
8000/8000 [==============================] - 2s - loss: 0.3781 - acc: 0.8417     
Epoch 30/100
8000/8000 [==============================] - 2s - loss: 0.3840 - acc: 0.8341     
Epoch 31/100
8000/8000 [==============================] - 2s - loss: 0.3767 - acc: 0.8420     
Epoch 32/100
8000/8000 [==============================] - 2s - loss: 0.3782 - acc: 0.8409     
Epoch 33/100
8000/8000 [==============================] - 2s - loss: 0.3777 - acc: 0.8385     
Epoch 34/100
8000/8000 [==============================] - 2s - loss: 0.3835 - acc: 0.8354     
Epoch 35/100
8000/8000 [==============================] - 2s - loss: 0.3763 - acc: 0.8402     
Epoch 36/100
8000/8000 [==============================] - 2s - loss: 0.3766 - acc: 0.8427     
Epoch 37/100
8000/8000 [==============================] - 2s - loss: 0.3749 - acc: 0.8404     
Epoch 38/100
8000/8000 [==============================] - 2s - loss: 0.3738 - acc: 0.8435     
Epoch 39/100
8000/8000 [==============================] - 2s - loss: 0.3774 - acc: 0.8392     
Epoch 40/100
8000/8000 [==============================] - 2s - loss: 0.3790 - acc: 0.8412     
Epoch 41/100
8000/8000 [==============================] - 2s - loss: 0.3754 - acc: 0.8410     
Epoch 42/100
8000/8000 [==============================] - 1s - loss: 0.3770 - acc: 0.8411     
Epoch 43/100
8000/8000 [==============================] - 2s - loss: 0.3811 - acc: 0.8387     
Epoch 44/100
8000/8000 [==============================] - 2s - loss: 0.3762 - acc: 0.8420     
Epoch 45/100
8000/8000 [==============================] - 2s - loss: 0.3760 - acc: 0.8427     
Epoch 46/100
8000/8000 [==============================] - 2s - loss: 0.3734 - acc: 0.8456     
Epoch 47/100
8000/8000 [==============================] - 2s - loss: 0.3768 - acc: 0.8395     
Epoch 48/100
8000/8000 [==============================] - 2s - loss: 0.3769 - acc: 0.8400     
Epoch 49/100
8000/8000 [==============================] - 2s - loss: 0.3735 - acc: 0.8400     
Epoch 50/100
8000/8000 [==============================] - 1s - loss: 0.3823 - acc: 0.8367     
Epoch 51/100
8000/8000 [==============================] - 1s - loss: 0.3747 - acc: 0.8411     
Epoch 52/100
8000/8000 [==============================] - 2s - loss: 0.3766 - acc: 0.8414     
Epoch 53/100
8000/8000 [==============================] - 1s - loss: 0.3765 - acc: 0.8369     
Epoch 54/100
8000/8000 [==============================] - 2s - loss: 0.3763 - acc: 0.8386     
Epoch 55/100
8000/8000 [==============================] - 2s - loss: 0.3747 - acc: 0.8391     
Epoch 56/100
8000/8000 [==============================] - 2s - loss: 0.3756 - acc: 0.8396     
Epoch 57/100
8000/8000 [==============================] - 1s - loss: 0.3768 - acc: 0.8400     
Epoch 58/100
8000/8000 [==============================] - 2s - loss: 0.3740 - acc: 0.8396     
Epoch 59/100
8000/8000 [==============================] - 2s - loss: 0.3699 - acc: 0.8430     
Epoch 60/100
8000/8000 [==============================] - 3s - loss: 0.3781 - acc: 0.8405     
Epoch 61/100
8000/8000 [==============================] - 2s - loss: 0.3754 - acc: 0.8411     
Epoch 62/100
8000/8000 [==============================] - 2s - loss: 0.3709 - acc: 0.8434     
Epoch 63/100
8000/8000 [==============================] - 2s - loss: 0.3773 - acc: 0.8410     
Epoch 64/100
8000/8000 [==============================] - 2s - loss: 0.3760 - acc: 0.8392     
Epoch 65/100
8000/8000 [==============================] - 2s - loss: 0.3724 - acc: 0.8432     
Epoch 66/100
8000/8000 [==============================] - 2s - loss: 0.3757 - acc: 0.8405     
Epoch 67/100
8000/8000 [==============================] - 2s - loss: 0.3707 - acc: 0.8436     
Epoch 68/100
8000/8000 [==============================] - 1s - loss: 0.3783 - acc: 0.8396     
Epoch 69/100
8000/8000 [==============================] - 1s - loss: 0.3747 - acc: 0.8400     
Epoch 70/100
8000/8000 [==============================] - 2s - loss: 0.3708 - acc: 0.8444     
Epoch 71/100
8000/8000 [==============================] - 2s - loss: 0.3743 - acc: 0.8421     
Epoch 72/100
8000/8000 [==============================] - 2s - loss: 0.3774 - acc: 0.8397     
Epoch 73/100
8000/8000 [==============================] - 2s - loss: 0.3744 - acc: 0.8395     
Epoch 74/100
8000/8000 [==============================] - 2s - loss: 0.3707 - acc: 0.8416     
Epoch 75/100
8000/8000 [==============================] - 2s - loss: 0.3744 - acc: 0.8410     
Epoch 76/100
8000/8000 [==============================] - 2s - loss: 0.3764 - acc: 0.8421     
Epoch 77/100
8000/8000 [==============================] - 2s - loss: 0.3729 - acc: 0.8397     
Epoch 78/100
8000/8000 [==============================] - 2s - loss: 0.3726 - acc: 0.8389     
Epoch 79/100
8000/8000 [==============================] - 1s - loss: 0.3716 - acc: 0.8421     
Epoch 80/100
8000/8000 [==============================] - 1s - loss: 0.3744 - acc: 0.8362     
Epoch 81/100
8000/8000 [==============================] - 1s - loss: 0.3714 - acc: 0.8370     
Epoch 82/100
8000/8000 [==============================] - 1s - loss: 0.3743 - acc: 0.8390     
Epoch 83/100
8000/8000 [==============================] - 1s - loss: 0.3748 - acc: 0.8361     
Epoch 84/100
8000/8000 [==============================] - 1s - loss: 0.3746 - acc: 0.8372     
Epoch 85/100
8000/8000 [==============================] - 1s - loss: 0.3744 - acc: 0.8332     
Epoch 86/100
8000/8000 [==============================] - 1s - loss: 0.3750 - acc: 0.8351     
Epoch 87/100
8000/8000 [==============================] - 1s - loss: 0.3743 - acc: 0.8392     
Epoch 88/100
8000/8000 [==============================] - 1s - loss: 0.3716 - acc: 0.8405     
Epoch 89/100
8000/8000 [==============================] - 1s - loss: 0.3738 - acc: 0.8399     
Epoch 90/100
8000/8000 [==============================] - 1s - loss: 0.3726 - acc: 0.8387     
Epoch 91/100
8000/8000 [==============================] - 1s - loss: 0.3755 - acc: 0.8391     
Epoch 92/100
8000/8000 [==============================] - 1s - loss: 0.3724 - acc: 0.8392     
Epoch 93/100
8000/8000 [==============================] - 1s - loss: 0.3731 - acc: 0.8452     
Epoch 94/100
8000/8000 [==============================] - 1s - loss: 0.3702 - acc: 0.8414     
Epoch 95/100
8000/8000 [==============================] - 1s - loss: 0.3731 - acc: 0.8417     
Epoch 96/100
8000/8000 [==============================] - 2s - loss: 0.3734 - acc: 0.8405     
Epoch 97/100
8000/8000 [==============================] - 2s - loss: 0.3743 - acc: 0.8417     
Epoch 98/100
8000/8000 [==============================] - 2s - loss: 0.3754 - acc: 0.8411     
Epoch 99/100
8000/8000 [==============================] - 2s - loss: 0.3737 - acc: 0.8392     
Epoch 100/100
8000/8000 [==============================] - 2s - loss: 0.3730 - acc: 0.8430     

Out[54]:

<keras.callbacks.History at 0x124b5aa58>

In [55]:

y_pred = classifier.predict(X_test)

In [56]:

y_pred[0:5]

Out[56]:

array([[ 0.4088383 ],
       [ 0.44399053],
       [ 0.54964072],
       [ 0.06084509],
       [ 0.64304215]], dtype=float32)

In [57]:

y_pred = y_pred >0.5

In [58]:

y_pred[0:5]

Out[58]:

array([[False],
       [False],
       [ True],
       [False],
       [ True]], dtype=bool)

Evaluate the results¶

In [59]:

from sklearn.metrics import confusion_matrix

In [60]:

confusion_matrix(y_test, y_pred)

Out[60]:

array([[1505,   73],
       [ 243,  179]])

In [63]:

(1505+179)/2000

Out[63]:

0.842