As I mentioned at the previous posting, one of the purposes of this blog is to supplement the Github of my data science study. I will gradually post and present all the iPython notebooks or Mathematica notebooks.

I felt there’s no good Python tutorial for spectral clustering (at least from my search). Who can’t use scikit-learn among who is serious about machine learning. It was not difficult to find the theory of spectral clustering as well. However, it was hard to find a Python tutorial for the algorithm of spectral clustering.

The theory for clustering and soft k-means can be found in the book Information Theory, Inference, and Learning Algorithms of David Mackay. In particular, I have read chapter 20 ~ 22 and used the algorithms to make Python codes in various clusterings.


Today, the topic of our discussion is K-means clustering, and it is well-known clustering you’ll see at first if you start to learn clustering.

First, a good clustering has predictive power. Second, clusters can be a useful aid to communication because they allow lossy compression. A third reason for making a cluster model is that failures of the cluster model may highlight interesting objects that deserve special attention. A fourth reason for liking clustering algorithms is that they may serve as models of learning processes in neural systems.

What I want to show you is when it does not work well. And then I will also post how we avoid the troubles.


The K-means algorithm is an algorithm for putting N data points in an M-dimensional space into K clusters. Each cluster is parameterized by a vector ${\textbf m}^{(k)}$ called its mean.

First of all, set K means ${\textbf m}^{(k)}$ to random values. In the assignment step, each data point n is assigned to the nearest mean.

$$ \hat {k}~ ^ {(n)} = argmin_k d({\textbf m}^{(k)}, {\textbf x}^{(n)}) $$

We will make n-groups whose elements have the smallest distances.

# -*- coding: utf-8 -*-

from numpy import *
from matplotlib import pyplot

import random
import numpy as np
import matplotlib.pyplot as plt


data1=[[0.5+0.5*random.random(),0.5+0.5*random.random()] for i in range(N)]
data2=[[0.5*random.random(),0.5*random.random()] for i in range(N)]


data = np.concatenate((data1,data2))


def displaydata():

def findClosestCentroids( X, centroids ):
	K = shape( centroids )[0]
	m   = shape( X )[0]
	idx = zeros( (m, 1) )

	for i in range(0, m):
		lowest = 999
		lowest_index = 0

		for k in range( 0, K ):
			cost = X[i] - centroids[k]
			cost = cost )
			if cost < lowest:
				lowest_index = k
				lowest  = cost

		idx[i] = lowest_index
	return idx + 1 # add 1, since python's index starts at 0

def computeCentroidsLoop(X, idx, K):
	m, n = shape( X )
	centroids = zeros((K, n))

	for k in range(1, K+1):
		counter = 0
		cum_sum = 0
		for i in range( 0, m ):
			if idx[i] == k:
				cum_sum += X[i]
				counter += 1
		centroids[k-1] = cum_sum / counter
	return centroids

def computeCentroids( X, idx, K ):
	m, n = shape( X )
	centroids = zeros((K, n))

	data = c_[X, idx] # append the cluster index to the X

	for k in range( 1, K+1 ):
		temp = data[data[:, n] == k] # quickly extract X that falls into the cluster
		count = shape( temp )[0]		# count number of entries for that cluster

		for j in range( 0, n ):
			centroids[k-1, j] = sum(temp[:, j]) / count

	return centroids

def runkMeans( X, initial_centroids, max_iters, plot=False ):
	K = shape( initial_centroids )[0]
	centroids = copy( initial_centroids )
	idx = None

	for iteration in range( 0, max_iters ):
		idx = findClosestCentroids( X, centroids )
		centroids = computeCentroids( X, idx, K )

		if plot is True:
			data = c_[X, idx]

			# Extract data that falls in to cluster 1, 2, and 3 respectively, and plot them out
			data_1 = data[data[:, 2] == 1]
			pyplot.plot( data_1[:, 0], data_1[:, 1], 'ro', markersize=5 )

			data_2 = data[data[:, 2] == 2]
			pyplot.plot( data_2[:, 0], data_2[:, 1], 'go', markersize=5 )

			data_3 = data[data[:, 2] == 3]
			pyplot.plot( data_3[:, 0], data_3[:, 1], 'bo', markersize=5 )

			data_4 = data[data[:, 2] == 4]
			pyplot.plot( data_4[:, 0], data_4[:, 1], 'yo', markersize=5 )

			pyplot.plot( centroids[:, 0], centroids[:, 1], 'k*', markersize=14 )

			pyplot.ylim([-0.1,1.1]) block=True )

	return centroids, idx

def kMeansInitCentroids( X, K ):
	return np.random.permutation( X )[:K]

def findcentroids(X, centroids):

	K = 4

	initial_centroids = centroids

	idx = findClosestCentroids( X, initial_centroids )
	print idx[0:K]

	centroids = computeCentroids( X, idx, K )
	print centroids

def Kmeans(X,centroids):
	K = 4

	max_iters = 10

	runkMeans( X, centroids, max_iters, plot=True )

Here are the data points. I chose random 50 data from the two boxes with 0.5 lengthy edges. We know the proper K-number for K-means clustering is 2. However, when we set K > 2, the K-means clustering gives a ugly result. Besides, the clustering strongly depends on the initial choice of the 4 assignment points.

set_printoptions(precision=6, linewidth=200)


Choose the K-means for K = 4. The proper K should be 2 for the data points. However, what happens if K = 4? From the below plots, the program is not working well.

For this given example, $K = 2$ is obvious, but for the most realistic problems, we can not always set the correct $K$ number. Thus, our goal is make the machine itself find the $K$.

centroids = array([[random.random(), random.random()], [random.random(), random.random()], [random.random(), random.random()],[random.random(), random.random()]])
[[ 4.]
 [ 4.]
 [ 2.]
 [ 4.]]
[[ 0.199317  0.208257]
 [ 0.79478   0.780224]
 [ 0.53369   0.865798]
 [ 0.54678   0.551297]]











The K-means(cetroids) converge to the wrong points in the above figures.

Appendix : to convert iPython notebooks to markdown

$ ipython nbconvert Clustering.ipynb --to markdown


$ jupiter nbconvert Clustering.ipynb --to markdown

This make a simple file and a folder, Clustering_files. The folder includes the figures in the notebooks. You might have to change the name of the folders in case that you store the image files and notebooks in the specific folder rather than _post folder.