Coursera Ml - Implementing Regularized Logistic Regression Cost Function In Python

I am working through Andrew Ng's Machine Learning on Coursera by implementing all the code in python rather than MATLAB. In Programming Exercise 3, I implemented my regularized log

Solution 1:

If you recall from regularization, you do not regularize the bias coefficient. Not only do you set the gradient to zero when performing gradient descent but you do not include this in the cost function. You have a slight mistake where you are including this as part of the sum (see cell #18 on your notebook that you linked - the sum should start from j = 1 but you have it as j = 0). Therefore, you need to sum from the second element to the end for your theta, not the first. You can verify this on Page 9 of the ex2.pdf PDF assignment that is seen on your Github repo. This explains the inflated cost as you are including the bias unit as part of the regularization.

Therefore, when computing regularization in reg, index theta so that you start from the second element and onwards:

def compute_cost_regularized(theta, X, y, lda):
    reg =lda/(2*len(y)) * np.sum(theta[1:]**2) # Change here
    return 1/len(y) * np.sum(-y @ np.log(sigmoid(X@theta)) 
                         - (1-y) @ np.log(1-sigmoid(X@theta))) + reg

Once I do this, define your test values as well as define your sigmoid function, I get the right answer that you're expecting:

In [8]: def compute_cost_regularized(theta, X, y, lda):
   ...:     reg =lda/(2*len(y)) * np.sum(theta[1:]**2)
   ...:     return 1/len(y) * np.sum(-y @ np.log(sigmoid(X@theta))
   ...:                          - (1-y) @ np.log(1-sigmoid(X@theta))) + reg
   ...:

In [9]: def sigmoid(z):
   ...:     return 1 / (1 + np.exp(-z))
   ...:

In [10]: theta_test = np.array([-2,-1,1,2])
    ...: X_test = np.concatenate((np.ones((5,1)),
    ...:          np.fromiter((x/10 for x in range(1,16)), float).reshape((3,5)).T), axis = 1)
    ...: y_test = np.array([1,0,1,0,1])
    ...: lambda_test = 3
    ...:

In [11]: compute_cost_regularized(theta_test, X_test, y_test, lambda_test)
Out[11]: 2.5348193961097438