Building an Artificial Neural Network

In [31]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
%config InlineBackend.figure_format = 'retina'

1 Import data¶

In [32]:

dataset = pd.read_csv("datasets/Churn_Modelling.csv")

In [33]:

dataset.head()

Out[33]:

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	1	15634602	Hargrave	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	2	15647311	Hill	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	3	15619304	Onio	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	4	15701354	Boni	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	5	15737888	Mitchell	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

In [34]:

X = dataset.iloc[:,3:13]
y = dataset.iloc[:,13]

In [35]:

X.head()

Out[35]:

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
0	619	France	Female	42	2	0.00	1	1	1	101348.88
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58
2	502	France	Female	42	8	159660.80	3	1	0	113931.57
3	699	France	Female	39	1	0.00	2	0	0	93826.63
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10

In [36]:

y.head()

Out[36]:

0    1
1    0
2    1
3    0
4    0
Name: Exited, dtype: int64

2 Encode categorical independent variables¶

In [37]:

from sklearn.preprocessing import OneHotEncoder,LabelEncoder
label_encoder_geo = LabelEncoder()
X.iloc[:,1] = label_encoder_geo.fit_transform(X.iloc[:,1])

In [38]:

X.head()

Out[38]:

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
0	619	0	Female	42	2	0.00	1	1	1	101348.88
1	608	2	Female	41	1	83807.86	1	0	1	112542.58
2	502	0	Female	42	8	159660.80	3	1	0	113931.57
3	699	0	Female	39	1	0.00	2	0	0	93826.63
4	850	2	Female	43	2	125510.82	1	1	1	79084.10

In [39]:

label_encoder_sex = LabelEncoder()
X.iloc[:,2] = label_encoder_sex.fit_transform(X.iloc[:,2])

In [40]:

X.head()

Out[40]:

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
0	619	0	0	42	2	0.00	1	1	1	101348.88
1	608	2	0	41	1	83807.86	1	0	1	112542.58
2	502	0	0	42	8	159660.80	3	1	0	113931.57
3	699	0	0	39	1	0.00	2	0	0	93826.63
4	850	2	0	43	2	125510.82	1	1	1	79084.10

In [41]:

onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()

In [42]:

X

Out[42]:

array([[  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   1.00000000e+00,   1.01348880e+05],
       [  0.00000000e+00,   0.00000000e+00,   1.00000000e+00, ...,
          0.00000000e+00,   1.00000000e+00,   1.12542580e+05],
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   1.13931570e+05],
       ..., 
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          0.00000000e+00,   1.00000000e+00,   4.20855800e+04],
       [  0.00000000e+00,   1.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   9.28885200e+04],
       [  1.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
          1.00000000e+00,   0.00000000e+00,   3.81907800e+04]])

In [43]:

X[0]

Out[43]:

array([  1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
         6.19000000e+02,   0.00000000e+00,   4.20000000e+01,
         2.00000000e+00,   0.00000000e+00,   1.00000000e+00,
         1.00000000e+00,   1.00000000e+00,   1.01348880e+05])

Let's delete one categorical variable to avoid dummy variable trap¶

In [44]:

X = X[:,1:]

In [45]:

from sklearn.model_selection import train_test_split

In [46]:

X_train,X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 10)

Feature scaling is compulsory for ANN¶

In [47]:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)

In [48]:

X_test = sc.fit_transform(X_test)

Let's make ANN¶

In [49]:

import keras

Using TensorFlow backend.

In [50]:

from keras.models import Sequential
from keras.layers import Dense

Initialise ANN¶

In [51]:

classifier = Sequential()

Add the first hidden layer and input layer¶

In [54]:

classifier.add(Dense(output_dim = 6, init = 'uniform',activation = 'relu', input_dim = 11))

Adding one more hidden layer¶

In [56]:

classifier.add(Dense(output_dim = 6, init = 'uniform',activation = 'relu'))

Adding output layer¶

We can get probability if we use sigmoid activtion function in the output layer
Use 'softmax' if its a multiclass

In [57]:

classifier.add(Dense(output_dim = 1, init = 'uniform',activation = 'sigmoid'))

Compiling ANN¶

adding stochastic gradient descent for calculating the weights. We use 'adam here

In [58]:

classifier.compile(optimizer='adam',loss ='binary_crossentropy', metrics = ['accuracy'])

Fitting the model¶

In [59]:

classifier.fit(X_train,y_train, batch_size=10, epochs=100)

Epoch 1/100
8000/8000 [==============================] - 1s - loss: 0.4815 - acc: 0.7979     
Epoch 2/100
8000/8000 [==============================] - 1s - loss: 0.4233 - acc: 0.7981     
Epoch 3/100
8000/8000 [==============================] - 1s - loss: 0.4173 - acc: 0.8059     
Epoch 4/100
8000/8000 [==============================] - 1s - loss: 0.4136 - acc: 0.8257     
Epoch 5/100
8000/8000 [==============================] - 1s - loss: 0.4111 - acc: 0.8291     
Epoch 6/100
8000/8000 [==============================] - 1s - loss: 0.4094 - acc: 0.8322     
Epoch 7/100
8000/8000 [==============================] - 1s - loss: 0.4081 - acc: 0.8337     
Epoch 8/100
8000/8000 [==============================] - 1s - loss: 0.4062 - acc: 0.8352     
Epoch 9/100
8000/8000 [==============================] - 1s - loss: 0.4057 - acc: 0.8356     
Epoch 10/100
8000/8000 [==============================] - 1s - loss: 0.4042 - acc: 0.8379     
Epoch 11/100
8000/8000 [==============================] - 1s - loss: 0.4033 - acc: 0.8367     
Epoch 12/100
8000/8000 [==============================] - 1s - loss: 0.4024 - acc: 0.8371     
Epoch 13/100
8000/8000 [==============================] - 1s - loss: 0.4019 - acc: 0.8354     
Epoch 14/100
8000/8000 [==============================] - 1s - loss: 0.4011 - acc: 0.8364     
Epoch 15/100
8000/8000 [==============================] - 1s - loss: 0.4007 - acc: 0.8350     
Epoch 16/100
8000/8000 [==============================] - 1s - loss: 0.4003 - acc: 0.8372     
Epoch 17/100
8000/8000 [==============================] - 1s - loss: 0.4001 - acc: 0.8367     
Epoch 18/100
8000/8000 [==============================] - 1s - loss: 0.3992 - acc: 0.8364     
Epoch 19/100
8000/8000 [==============================] - 1s - loss: 0.3988 - acc: 0.8374     
Epoch 20/100
8000/8000 [==============================] - 1s - loss: 0.3989 - acc: 0.8371     
Epoch 21/100
8000/8000 [==============================] - 1s - loss: 0.3982 - acc: 0.8364     
Epoch 22/100
8000/8000 [==============================] - 1s - loss: 0.3984 - acc: 0.8379     
Epoch 23/100
8000/8000 [==============================] - 1s - loss: 0.3979 - acc: 0.8377     
Epoch 24/100
8000/8000 [==============================] - 1s - loss: 0.3976 - acc: 0.8382     
Epoch 25/100
8000/8000 [==============================] - 1s - loss: 0.3971 - acc: 0.8391     
Epoch 26/100
8000/8000 [==============================] - 1s - loss: 0.3971 - acc: 0.8360     
Epoch 27/100
8000/8000 [==============================] - 1s - loss: 0.3966 - acc: 0.8381     
Epoch 28/100
8000/8000 [==============================] - 1s - loss: 0.3966 - acc: 0.8375     
Epoch 29/100
8000/8000 [==============================] - 1s - loss: 0.3961 - acc: 0.8365     
Epoch 30/100
8000/8000 [==============================] - 1s - loss: 0.3964 - acc: 0.8397     
Epoch 31/100
8000/8000 [==============================] - 1s - loss: 0.3964 - acc: 0.8365     
Epoch 32/100
8000/8000 [==============================] - 1s - loss: 0.3961 - acc: 0.8385     
Epoch 33/100
8000/8000 [==============================] - 1s - loss: 0.3964 - acc: 0.8369     
Epoch 34/100
8000/8000 [==============================] - 1s - loss: 0.3958 - acc: 0.8389     
Epoch 35/100
8000/8000 [==============================] - 1s - loss: 0.3958 - acc: 0.8386     
Epoch 36/100
8000/8000 [==============================] - 1s - loss: 0.3954 - acc: 0.8390     
Epoch 37/100
8000/8000 [==============================] - 1s - loss: 0.3960 - acc: 0.8379     
Epoch 38/100
8000/8000 [==============================] - 1s - loss: 0.3953 - acc: 0.8395     
Epoch 39/100
8000/8000 [==============================] - 1s - loss: 0.3959 - acc: 0.8385     
Epoch 40/100
8000/8000 [==============================] - 1s - loss: 0.3955 - acc: 0.8382     
Epoch 41/100
8000/8000 [==============================] - 1s - loss: 0.3957 - acc: 0.8389     
Epoch 42/100
8000/8000 [==============================] - 1s - loss: 0.3951 - acc: 0.8387     
Epoch 43/100
8000/8000 [==============================] - 1s - loss: 0.3949 - acc: 0.8387     
Epoch 44/100
8000/8000 [==============================] - 1s - loss: 0.3953 - acc: 0.8371     
Epoch 45/100
8000/8000 [==============================] - 1s - loss: 0.3950 - acc: 0.8387     
Epoch 46/100
8000/8000 [==============================] - 1s - loss: 0.3948 - acc: 0.8382     
Epoch 47/100
8000/8000 [==============================] - 1s - loss: 0.3948 - acc: 0.8389     
Epoch 48/100
8000/8000 [==============================] - 1s - loss: 0.3943 - acc: 0.8377     
Epoch 49/100
8000/8000 [==============================] - 1s - loss: 0.3946 - acc: 0.8400     
Epoch 50/100
8000/8000 [==============================] - 1s - loss: 0.3945 - acc: 0.8386     
Epoch 51/100
8000/8000 [==============================] - 1s - loss: 0.3943 - acc: 0.8387     
Epoch 52/100
8000/8000 [==============================] - 1s - loss: 0.3943 - acc: 0.8390     
Epoch 53/100
8000/8000 [==============================] - 1s - loss: 0.3944 - acc: 0.8397     
Epoch 54/100
8000/8000 [==============================] - 1s - loss: 0.3943 - acc: 0.8385     
Epoch 55/100
8000/8000 [==============================] - 1s - loss: 0.3941 - acc: 0.8389     
Epoch 56/100
8000/8000 [==============================] - 1s - loss: 0.3943 - acc: 0.8390     
Epoch 57/100
8000/8000 [==============================] - 1s - loss: 0.3939 - acc: 0.8399     
Epoch 58/100
8000/8000 [==============================] - ETA: 0s - loss: 0.3980 - acc: 0.837 - 1s - loss: 0.3940 - acc: 0.8395     
Epoch 59/100
8000/8000 [==============================] - 1s - loss: 0.3937 - acc: 0.8391     
Epoch 60/100
8000/8000 [==============================] - 1s - loss: 0.3941 - acc: 0.8389     
Epoch 61/100
8000/8000 [==============================] - 1s - loss: 0.3939 - acc: 0.8387     
Epoch 62/100
8000/8000 [==============================] - 1s - loss: 0.3933 - acc: 0.8382     
Epoch 63/100
8000/8000 [==============================] - 1s - loss: 0.3935 - acc: 0.8402     
Epoch 64/100
8000/8000 [==============================] - 1s - loss: 0.3935 - acc: 0.8397     
Epoch 65/100
8000/8000 [==============================] - 1s - loss: 0.3936 - acc: 0.8389     
Epoch 66/100
8000/8000 [==============================] - 1s - loss: 0.3938 - acc: 0.8390     
Epoch 67/100
8000/8000 [==============================] - 1s - loss: 0.3936 - acc: 0.8387     
Epoch 68/100
8000/8000 [==============================] - 1s - loss: 0.3934 - acc: 0.8367     
Epoch 69/100
8000/8000 [==============================] - 1s - loss: 0.3933 - acc: 0.8385     
Epoch 70/100
8000/8000 [==============================] - 1s - loss: 0.3934 - acc: 0.8392     
Epoch 71/100
8000/8000 [==============================] - 1s - loss: 0.3935 - acc: 0.8384     
Epoch 72/100
8000/8000 [==============================] - 1s - loss: 0.3931 - acc: 0.8396     
Epoch 73/100
8000/8000 [==============================] - 1s - loss: 0.3936 - acc: 0.8394     
Epoch 74/100
8000/8000 [==============================] - 1s - loss: 0.3932 - acc: 0.8380     
Epoch 75/100
8000/8000 [==============================] - 1s - loss: 0.3928 - acc: 0.8377     
Epoch 76/100
8000/8000 [==============================] - 1s - loss: 0.3935 - acc: 0.8392     
Epoch 77/100
8000/8000 [==============================] - 1s - loss: 0.3932 - acc: 0.8385     
Epoch 78/100
8000/8000 [==============================] - 1s - loss: 0.3930 - acc: 0.8384     
Epoch 79/100
8000/8000 [==============================] - 1s - loss: 0.3929 - acc: 0.8382     
Epoch 80/100
8000/8000 [==============================] - 1s - loss: 0.3934 - acc: 0.8395     
Epoch 81/100
8000/8000 [==============================] - 2s - loss: 0.3934 - acc: 0.8384     
Epoch 82/100
8000/8000 [==============================] - 1s - loss: 0.3930 - acc: 0.8392     
Epoch 83/100
8000/8000 [==============================] - 1s - loss: 0.3932 - acc: 0.8376     
Epoch 84/100
8000/8000 [==============================] - 1s - loss: 0.3932 - acc: 0.8387     
Epoch 85/100
8000/8000 [==============================] - 1s - loss: 0.3933 - acc: 0.8384     
Epoch 86/100
8000/8000 [==============================] - 1s - loss: 0.3931 - acc: 0.8381     
Epoch 87/100
8000/8000 [==============================] - 1s - loss: 0.3926 - acc: 0.8399     
Epoch 88/100
8000/8000 [==============================] - 1s - loss: 0.3931 - acc: 0.8377     
Epoch 89/100
8000/8000 [==============================] - 1s - loss: 0.3928 - acc: 0.8385     
Epoch 90/100
8000/8000 [==============================] - 1s - loss: 0.3926 - acc: 0.8389     
Epoch 91/100
8000/8000 [==============================] - 1s - loss: 0.3927 - acc: 0.8386     
Epoch 92/100
8000/8000 [==============================] - 1s - loss: 0.3930 - acc: 0.8387     
Epoch 93/100
8000/8000 [==============================] - 1s - loss: 0.3927 - acc: 0.8394     
Epoch 94/100
8000/8000 [==============================] - 1s - loss: 0.3930 - acc: 0.8395     
Epoch 95/100
8000/8000 [==============================] - 1s - loss: 0.3931 - acc: 0.8377     
Epoch 96/100
8000/8000 [==============================] - 1s - loss: 0.3925 - acc: 0.8375     
Epoch 97/100
8000/8000 [==============================] - 1s - loss: 0.3929 - acc: 0.8385     
Epoch 98/100
8000/8000 [==============================] - 1s - loss: 0.3926 - acc: 0.8372     
Epoch 99/100
8000/8000 [==============================] - 1s - loss: 0.3929 - acc: 0.8396     
Epoch 100/100
8000/8000 [==============================] - 1s - loss: 0.3925 - acc: 0.8381     

Out[59]:

<keras.callbacks.History at 0x11e8ca668>

In [60]:

y_pred = classifier.predict(X_test)

In [61]:

y_pred[0:5]

Out[61]:

array([[ 0.47014004],
       [ 0.27177387],
       [ 0.52922386],
       [ 0.12679352],
       [ 0.69866168]], dtype=float32)

In [62]:

y_pred = y_pred >0.5

In [63]:

y_pred[0:5]

Out[63]:

array([[False],
       [False],
       [ True],
       [False],
       [ True]], dtype=bool)

Evaluate the results¶

In [64]:

from sklearn.metrics import confusion_matrix

In [65]:

confusion_matrix(y_test, y_pred)

Out[65]:

array([[1514,   64],
       [ 270,  152]])

In [66]:

(1514+152)/2000

Out[66]:

0.833

In [ ]: