ML6.6 · LESSON 6 OF 6

Model comparison

Compare nearest-neighbor, naive Bayes and margin-based predictions using their actual assumptions.

THE IDEA

One split is one sample

You have six candidates: a constant guess, a nearest class centre, Gaussian naive Bayes, two k-NN settings and an RBF kernel model. The honest way to compare is to give them the same data split: same training rows, same preprocessing, same validation rows. But even then the score depends on which rows happened to land in validation. With 50 validation points one point is worth 2 percentage points.

On the first seeded split, k-NN with k = 1 scores 88% and comes first, while naive Bayes scores 76%. Use the repeat buttons below to rerun the split with different seeds, up to 30 times.

Repeat the random split:

Each repeat shuffles the same 100 points into 50 for training and 50 for validation, with a fixed seed, and scores every model on the same split. Dots below show one score per repeat.

model	split 1	rank on split 1	mean	std	rank by mean	cost
Always the majority class	42%	6	42%	-	6	O(1) per query
Nearest class centre	78%	3	78%	-	3	O(d) per query
Gaussian naive Bayes	76%	4	76%	-	4	O(d) per query
k-NN, k = 15	76%	5	76%	-	5	O(n d) per query
k-NN, k = 1	88%	1	88%	-	1	O(n d) per query
RBF kernel ridge (gamma 1)	86%	2	86%	-	2	O(n d) per query

Compare with

Mean paired gap (first minus second): 0. With one split there is no way to estimate the spread, so nothing can be called.

Your verdict:

Final model Simplicity order, simplest first: Always the majority class, then Nearest class centre, then Gaussian naive Bayes, then k-NN, k = 15, then k-NN, k = 1, then RBF kernel ridge (gamma 1).

Rank on split 1 against rank by mean over repeats. The blue dots are single scores; the orange line is their mean.

Do not over-read small differences

Over 30 repeats the k = 1 model averages 78.5% and naive Bayes 81.5%, so the split-1 winner was only a lucky draw. The scores of a single model vary by about 4.9% (one standard deviation) from split to split. A gap smaller than that spread cannot be trusted.

A sturdier way: compare models paired, on the same splits, and look at the differences. Naive Bayes against k = 15 has an average gap of 0.0127 with a standard error of 0.0084: well inside twice the standard error, so too close to call. Naive Bayes against k = 1 has a gap of 0.03 with a standard error of 0.0119, which is above the two-standard-error line: a real difference on this data.

Predict first. Model X wins the first split by 4 points. Is X the best model?

Compare more than accuracy, and never tune on the test set

Accuracy is one column. The table also lists what each model costs to use: k-NN and the kernel model touch all stored points for every query, while a nearest-centre rule or naive Bayes touches a few numbers. When two models are within noise, choose the simpler or cheaper one. Also remember that a flexible model with the wrong notion of closeness (unscaled features, a wrong γ, an unsuitable metric) can lose to a trivial one.

Chapter context

Different model families turn different notions of similarity into predictions. A geometric distance, a conditional-independence assumption and a kernel are distinct modeling decisions rather than variations of the same formula.

TRY IT YOURSELF

Part 1. Do it on the plot

Use more repeats first, then judge pairs against the standard error, and finish by choosing the model you would ship.

Repeat the random split:

Each repeat shuffles the same 100 points into 50 for training and 50 for validation, with a fixed seed, and scores every model on the same split. Dots below show one score per repeat.

model	split 1	rank on split 1	mean	std	rank by mean	cost
Always the majority class	42%	6	42%	-	6	O(1) per query
Nearest class centre	78%	3	78%	-	3	O(d) per query
Gaussian naive Bayes	76%	4	76%	-	4	O(d) per query
k-NN, k = 15	76%	5	76%	-	5	O(n d) per query
k-NN, k = 1	88%	1	88%	-	1	O(n d) per query
RBF kernel ridge (gamma 1)	86%	2	86%	-	2	O(n d) per query

Compare with

Mean paired gap (first minus second): 0. With one split there is no way to estimate the spread, so nothing can be called.

Your verdict:

Final model Simplicity order, simplest first: Always the majority class, then Nearest class centre, then Gaussian naive Bayes, then k-NN, k = 15, then k-NN, k = 1, then RBF kernel ridge (gamma 1).

To do: Repeat the split at least 20 times, so one lucky split cannot decide
To do: Pick two different models whose gap is within noise, and mark the verdict as too close to call
To do: Pick two models where the first is clearly better than the second, and mark that verdict
To do: Choose the final model: the simplest one that is not clearly worse than the best

Part 2. Do it in code

kfold_indices(n, k) returns a list of k pairs (train_idx, val_idx) of numpy int arrays. The validation parts are consecutive blocks of 0..n-1; the first n % k blocks get one extra index. Each index is validation exactly once.
If k < 2 or k > n, raise ValueError.
compare_scores(a, b) takes two equal-length lists of scores from the same splits and returns "A", "B" or "tie". Let d = a - b. Return "A" if mean(d) > 2 * se, "B" if mean(d) < -2 * se, else "tie", where se = std(d, ddof=1) / sqrt(n). With fewer than 2 repeats return "tie".

import numpy as np

def kfold_indices(n, k):
    pass

def compare_scores(a, b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should test labels choose the kernel bandwidth?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\|x-y\|_2^2=\sum_j(x_j-y_j)^2,\quad K(x,y)=\exp(-\gamma\|x-y\|^2),\;\gamma>0.

For SVM canonical boundary planes $w^Tx+b=\pm1$ , their separation is $2/\|w\|$ . A naive-Bayes posterior is proportional to $P(c)\prod_jP(x_j\mid c)$ .

A numerically convenient naive-Bayes decision rule is

\hat c=\arg\max_c\left[\log P(c)+\sum_j\log P(x_j\mid c)\right].

This avoids multiplying many small probabilities, but it does not remove the need to handle zero likelihoods or choose an appropriate feature distribution.

Worked example

For query [1,1], squared distances to [0,0], [2,1] and [3,3] are 2, 1 and 8. The nearest point is [2,1]. Separately, a canonical SVM with w=[3,4] has norm 5 and margin width 2/5=0.4.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
q = [1,1]
points = [[0,0],[2,1],[3,3]]
distances = [sum((a-b)**2 for a,b in zip(q,p)) for p in points]
assert distances == [2,1,8]
assert distances.index(min(distances)) == 1
assert isclose(2/sqrt(3**2+4**2), .4)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Changing feature scale can change neighbors. Kernel similarity is not automatically calibrated probability. The SVM margin formula assumes a particular normalization of the separating planes.