ML3.1 · LESSON 1 OF 6

Logistic regression

Compute logistic probabilities and evaluate the tradeoff introduced by a classification threshold.

THE IDEA

From a score to a probability

A linear model gives every example a score: z = w1·x1 + w2·x2 + b. That score can be any real number, large and negative or large and positive, but a probability must live between 0 and 1. The sigmoid bridges the two: p = 1 / (1 + e^(−z)). A score of 0 becomes 0.5 (no opinion). A score of 2 becomes 0.8808. A score of −2 becomes 0.1192. Large scores squeeze toward 1 or 0 but never reach them. This model is logistic regression. Despite the name, it classifies.

Move the line yourself

Below, 40 points form two clouds. The shading is the model's probability at every spot. The weights tilt the line and set how sharp the transition is, and the bias slides it. The lower strip shows the sigmoid with every point placed at its own score. Then press Gradient step: it nudges all three numbers downhill on the loss, and the loss graph shows what that buys you.

w10w20bias b0

mean loss 0.693accuracy 50%steps 0

Boundary: 0·x1 + 0·x2 + 0 = 0. Points with a red ring are on the wrong side. Orange shading means the model says positive, blue negative, white means unsure.

Try w2 around 3 to 5 first. Then scale w1 and w2 up and down together and compare accuracy with loss.

Predict first. You double both weights and the bias. Where does the decision boundary go, and what happens to the probabilities?

The common mistake: treating sigmoid output as a class

The output is a probability estimate, not a label. Turning it into a label needs a rule (usually p ≥ 0.5), which is a separate decision you will study in lesson ML3.3. Another mistake is computing the sigmoid naively in code. For z = −1000, np.exp(1000) overflows. The fix is to only ever exponentiate a non-positive number.

Naive sigmoid fails in numpy

>>> import numpy as np
>>> 1 / (1 + np.exp(-np.array([-1000.0])))
RuntimeWarning: overflow encountered in exp
array([0.])               # right by luck, with a warning. Under strict error settings it raises instead.

Chapter context

A classifier often estimates a score or probability before a decision rule turns it into a label. The decision threshold is part of the application policy; it is not the same thing as the fitted probability model.

TRY IT YOURSELF

Part 1. Do it with the controls

Start from all zeros: every probability is 0.5, the loss is ln 2 = 0.693. Tune the three sliders, then try the gradient step button. The tasks stay ticked once earned. The best weights on this grid are about 2.4, 5.3, -0.3 with loss 0.107, but find them yourself.

w10w20bias b0

mean loss 0.693accuracy 50%steps 0

Boundary: 0·x1 + 0·x2 + 0 = 0. Points with a red ring are on the wrong side. Orange shading means the model says positive, blue negative, white means unsure.

To do: Move the line so at least 90% of the 40 points fall on the correct side
To do: Also get the mean loss to 0.14 or lower. A line that separates is not enough: the loss also rewards sensible confidence
To do: Press Gradient step at least 10 times and watch the loss fall

Part 2. Do it in code

sigmoid(z) maps any real number (or numpy array) into (0, 1). It must not overflow for z = -1000 or z = 1000.
predict_proba(X, w, b) takes a table X (one row per example), a weight vector w and a bias b, and returns the probability of class 1 for every row.

import numpy as np

def sigmoid(z):
    # 1 / (1 + exp(-z)) is the formula, but np.exp(1000) overflows.
    # Make it safe for very negative and very positive z, and for arrays.
    pass

def predict_proba(X, w, b):
    # probability of class 1 for each row of X
    pass

X = np.array([[1.0, 2.0], [0.0, 0.0], [-1.0, 1.0]])
print(predict_proba(X, np.array([0.5, -0.25]), 0.1))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What probability does sigmoid(0) produce?

A0.5.B0.0, because a score of zero means no evidence for the classC1.0, because a score of zero is the boundary between the classes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\sigma(z)=\frac1{1+e^{-z}},\quad L=-y\log p-(1-y)\log(1-p).

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.

Define a policy for zero denominators rather than silently treating undefined metrics as universally meaningful.

Worked example

For actual labels [1,1,1,0,0] and predictions [1,1,0,1,0], TP=2, FP=1, FN=1 and TN=1. Accuracy is 3/5=0.6; precision, recall and F1 are all 2/3. Accuracy and F1 differ because they summarize different error patterns.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log
truth, pred = [1,1,1,0,0], [1,1,0,1,0]
tp = sum(a == b == 1 for a,b in zip(truth,pred))
fp = sum(a == 0 and b == 1 for a,b in zip(truth,pred))
fn = sum(a == 1 and b == 0 for a,b in zip(truth,pred))
precision, recall = tp/(tp+fp), tp/(tp+fn)
assert isclose(2*precision*recall/(precision+recall), 2/3)
assert sum(a == b for a,b in zip(truth,pred))/len(truth) == .6
assert isclose(-log(.8), .2231435513142097)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Accuracy can hide poor rare-class recall. A probability estimate does not choose the correct business threshold by itself. Ranking quality and calibration are separate properties.