ML6.2 · LESSON 2 OF 6

Scaling and metrics

Compare nearest-neighbor, naive Bayes and margin-based predictions using their actual assumptions.

THE IDEA

Distance adds up every feature, in its own units

Euclidean distance squares the gap in each feature and adds them. If one feature is measured in tens of thousands and another in tens, the big one decides everything. Take a query person aged 30 earning 52,000 dollars. Person A is 28 years older but earns 51,000. Person B is a year older and earns 58,000. Who is "closer"?

In dollars, A is 1000.3919 away and B is 6000.0001 away: the 28-year age gap is invisible next to a 1,000-dollar income gap. Express income in thousands and the same people give A at 28.0179 and B at 6.0828: now B is nearest. Nothing about the people changed, only a unit.

standardise (training mean and std)

Query: a person with income 52000 dollars and age 30. The bar shows what share of each squared distance comes from income (blue) and from age (orange).

rank	person	income	age	buys?	distance
1	A	51000	58	no	1000.3919
2	E	47000	45	no	5000.0225
3	B	58000	31	yes	6000.0001
4	F	66000	29	yes	14000
5	C	38000	27	yes	14000.0003
6	H	72000	41	yes	20000.003
7	G	30000	52	no	22000.011
8	D	90000	33	no	38000.0001

Nearest neighbour: A. The 3 nearest (shaded) vote: does not buy.

Switch the income unit, then tick the standardise box. The blue part of each bar is the share of the squared distance that comes from income, the orange part from age.

The fix: put features on one scale, using the training data only

Standardising rewrites each feature as how many standard deviations it sits from the training mean: z = (x − mean) / std. The mean income here is 56500 and the standard deviation 18096.9611; for age they are 39.5 and 10.6771. After this, a gap of one standard deviation counts the same whether it is in dollars or years, and the nearest neighbour is B (0.3445 away) whatever unit you started from. Try all three units with the box ticked.

Predict first. You convert a distance feature from metres to millimetres and leave the other features alone. Can the nearest neighbours change?

Other metrics change who is near

The scale is only half of the choice. Euclidean distance is the straight line. Manhattan distance adds the absolute gaps, like walking along a grid, so it does not favour diagonal moves. Cosine distance (1 minus the cosine of the angle) ignores how far from the origin a point is and only compares directions, which suits word-count vectors where length depends on document size. With the same six points and the same query, all three metrics pick a different neighbour:

The dotted purple line is the direction of the query from the origin.

rank	point	euclidean distance
1	c	1.4422
2	a	1.9
3	f	2.1213
4	d	3.6056
5	e	3.6056
6	b	4.4598

Nearest under euclidean: c

Pick a metric. The green link joins the query to its nearest point under that metric.

Euclidean picks c (1.4422), Manhattan picks a (1.9) and cosine picks b, which is far away but almost exactly in the query's direction.

The common mistake: leaking the test set into the scaler

The right order

mu = X_train.mean(axis=0)
sd = X_train.std(axis=0)          # a column with sd == 0 needs care
Z_train = (X_train - mu) / sd
Z_test  = (X_test  - mu) / sd     # the same mu and sd, never recomputed

Chapter context

Different model families turn different notions of similarity into predictions. A geometric distance, a conditional-independence assumption and a kernel are distinct modeling decisions rather than variations of the same formula.

TRY IT YOURSELF

Part 1. Do it on the plot

Both instruments share one state: the table (unit and standardise box) and the metric plot below it.

standardise (training mean and std)

Query: a person with income 52000 dollars and age 30. The bar shows what share of each squared distance comes from income (blue) and from age (orange).

rank	person	income	age	buys?	distance
1	B	58000	31	yes	6.0828
2	F	66000	29	yes	14.0357
3	C	38000	27	yes	14.3178
4	E	47000	45	no	15.8114
5	H	72000	41	yes	22.8254
6	A	51000	58	no	28.0179
7	G	30000	52	no	31.1127
8	D	90000	33	no	38.1182

Nearest neighbour: B. The 3 nearest (shaded) vote: buys.

The dotted purple line is the direction of the query from the origin.

rank	point	euclidean distance
1	c	1.4422
2	a	1.9
3	f	2.1213
4	d	3.6056
5	e	3.6056
6	b	4.4598

Nearest under euclidean: c

To do: Without standardising, make person A the nearest neighbour (income in dollars)
To do: Keep income in dollars but make person B the nearest neighbour
To do: With standardising on, visit all three income units and watch the ranking stay put
To do: On the metric plot, make point a the nearest neighbour of the query
To do: On the metric plot, make point b (far away, same direction) the nearest neighbour

Part 2. Do it in code

standardise(X_train, X_test) returns (Z_train, Z_test). Subtract each column's mean and divide by its standard deviation, both computed from X_train only (np.std with its default, dividing by n).
A column with zero variance in X_train must not produce nan or inf: divide it by 1 instead, so it just gets centred.
cosine_distance(a, b) returns 1 minus the cosine of the angle between two vectors. If either vector has length 0, return 1.0.

import numpy as np

def standardise(X_train, X_test):
    # return (Z_train, Z_test) using training statistics only
    pass

def cosine_distance(a, b):
    # 1 - cos(angle); 1.0 if either vector is all zeros
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Could a meter-to-millimeter unit change alter unscaled neighbors?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\|x-y\|_2^2=\sum_j(x_j-y_j)^2,\quad K(x,y)=\exp(-\gamma\|x-y\|^2),\;\gamma>0.

For SVM canonical boundary planes $w^Tx+b=\pm1$ , their separation is $2/\|w\|$ . A naive-Bayes posterior is proportional to $P(c)\prod_jP(x_j\mid c)$ .

A numerically convenient naive-Bayes decision rule is

\hat c=\arg\max_c\left[\log P(c)+\sum_j\log P(x_j\mid c)\right].

This avoids multiplying many small probabilities, but it does not remove the need to handle zero likelihoods or choose an appropriate feature distribution.

Worked example

For query [1,1], squared distances to [0,0], [2,1] and [3,3] are 2, 1 and 8. The nearest point is [2,1]. Separately, a canonical SVM with w=[3,4] has norm 5 and margin width 2/5=0.4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
q = [1,1]
points = [[0,0],[2,1],[3,3]]
distances = [sum((a-b)**2 for a,b in zip(q,p)) for p in points]
assert distances == [2,1,8]
assert distances.index(min(distances)) == 1
assert isclose(2/sqrt(3**2+4**2), .4)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Changing feature scale can change neighbors. Kernel similarity is not automatically calibrated probability. The SVM margin formula assumes a particular normalization of the separating planes.