SLIDE 1
Clustering Algorithms
Dalya Baron (Tel Aviv University) XXX Winter School, November 2018
Clustering
Feature 1 Feature 2
Clustering Algorithms Dalya Baron (Tel Aviv University) XXX Winter - - PowerPoint PPT Presentation
Clustering Algorithms Dalya Baron (Tel Aviv University) XXX Winter - - PowerPoint PPT Presentation
Clustering Algorithms Dalya Baron (Tel Aviv University) XXX Winter School, November 2018 Clustering Feature 2 Feature 1 Clustering cluster #1 Feature 2 cluster #2 Feature 1 Clustering Why should we look for clusters? cluster #1 Feature
SLIDE 2
SLIDE 3
Clustering
Feature 1 Feature 2 cluster #1 cluster #2
SLIDE 4
Clustering
Feature 1 Feature 2 cluster #1 cluster #2 Why should we look for clusters?
SLIDE 5
Clustering
SLIDE 6
K-means
Input: measured features, and the number of clusters, k. The algorithm will classify all the objects in the sample into k clusters. Feature 1 Feature 2
SLIDE 7
K-means
(I) The algorithm places randomly k points that represent the centroids of the clusters. The algorithm performs several iterations, in each of them: (II) The algorithm associates each object with a single cluster, according to its distance from the cluster centroid. (III) The algorithm recalculates the cluster centroid according to the objects that are associated with it.
Feature 1 Feature 2
SLIDE 8
K-means
(I) The algorithm places randomly k points that represent the centroids of the clusters. The algorithm performs several iterations, in each of them: (II) The algorithm associates each object with a single cluster, according to its distance from the cluster centroid. (III) The algorithm recalculates the cluster centroid according to the objects that are associated with it.
Feature 1 Feature 2 Two centroids are randomly placed
SLIDE 9
K-means
(I) The algorithm places randomly k points that represent the centroids of the clusters. The algorithm performs several iterations, in each of them: (II) The algorithm associates each object with a single cluster, according to its distance from the cluster centroid. (III) The algorithm recalculates the cluster centroid according to the objects that are associated with it.
Feature 1 Feature 2 The objects are associated to the closest cluster centroid (Euclidean distance).
SLIDE 10
K-means
(I) The algorithm places randomly k points that represent the centroids of the clusters. The algorithm performs several iterations, in each of them: (II) The algorithm associates each object with a single cluster, according to its distance from the cluster centroid. (III) The algorithm recalculates the cluster centroid according to the objects that are associated with it.
Feature 1 Feature 2 New cluster centroids are computed using the average location of the cluster members.
SLIDE 11
K-means
(I) The algorithm places randomly k points that represent the centroids of the clusters. The algorithm performs several iterations, in each of them: (II) The algorithm associates each object with a single cluster, according to its distance from the cluster centroid. (III) The algorithm recalculates the cluster centroid according to the objects that are associated with it.
Feature 1 Feature 2 The objects are associated to the closest cluster centroid (Euclidean distance).
SLIDE 12
K-means
(I) The algorithm places randomly k points that represent the centroids of the clusters. The algorithm performs several iterations, in each of them: (II) The algorithm associates each object with a single cluster, according to its distance from the cluster centroid. (III) The algorithm recalculates the cluster centroid according to the objects that are associated with it.
Feature 1 Feature 2 The process stops when the objects that are associated with a given class do not change.
SLIDE 13
The anatomy of K-means
Internal choices and/or internal cost function: (I) Initial centroids are randomly selected from the set of examples. (II) The global cost function that is minimized by K-means:
Euclidean distance cluster centroids cluster members
SLIDE 14
The anatomy of K-means
Internal choices and/or internal cost function: (I) Initial centroids are randomly selected from the set of examples. (II) The global cost function that is minimized by K-means:
Euclidean distance cluster centroids cluster members
k=3, and two different random placements of centroids
SLIDE 15
The anatomy of K-means
Input dataset: a list of objects with measured features. For which datasets should we use K-means?
Feature 1 Feature 2 Feature 1 Feature 2
SLIDE 16
SLIDE 17
The anatomy of K-means
Input dataset: a list of objects with measured features. What happens when we have an outlier in the dataset?
Feature 1 Feature 2
- utlier!
SLIDE 18
The anatomy of K-means
Input dataset: a list of objects with measured features. What happens when we have an outlier in the dataset?
Feature 1 Feature 2
- utlier!
SLIDE 19
The anatomy of K-means
Input dataset: a list of objects with measured features. What happens when the features have different physical units?
input dataset K-means output
SLIDE 20
The anatomy of K-means
Input dataset: a list of objects with measured features. What happens when the features have different physical units?
input dataset K-means output How can we avoid this?
SLIDE 21
The anatomy of K-means
Hyper-parameters: the number of clusters, k. Can we find the optimal k using the cost function?
k=2 k=3 k=5
SLIDE 22
The anatomy of K-means
Hyper-parameters: the number of clusters, k. Can we find the optimal k using the cost function?
k=2 k=3 k=5
Number of clusters Minimal cost function
Elbow
SLIDE 23
Questions?
SLIDE 24
Hierarchal Clustering
Correa-Gallego+ 2016
- r, how to visualize complicated similarity measures
SLIDE 25
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2
SLIDE 26
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest.
SLIDE 27
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 28
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 29
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 30
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 31
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 32
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 33
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 Next: the algorithm merges the two closest clusters into a single cluster. Then, the algorithm re-calculates the distance of the newly-formed cluster to all the rest. distance Dendrogram
SLIDE 34
Hierarchal Clustering
Input: measured features, or a distance matrix that represents the pair-wise distances between the objects. Also, we must specify a linkage method. Initialization: each object is a cluster of size 1. Feature 1 Feature 2 The process stops when all the objects are merged into a single cluster distance Dendrogram
SLIDE 35
The anatomy of Hierarchal Clustering
Internal choices and/or internal cost function: The linkage method is used to define a distance between two newly formed
- clusters. Methods include: single (minimal), complete (maximal), average, etc.
Feature 1 Feature 2 distance single Dendrogram
SLIDE 36
Feature 1 Feature 2 distance complete Dendrogram
The anatomy of Hierarchal Clustering
Internal choices and/or internal cost function: The linkage method is used to define a distance between two newly formed
- clusters. Methods include: single (minimal), complete (maximal), average, etc.
SLIDE 37
Feature 1 Feature 2 distance average Dendrogram
The anatomy of Hierarchal Clustering
Internal choices and/or internal cost function: The linkage method is used to define a distance between two newly formed
- clusters. Methods include: single (minimal), complete (maximal), average, etc.
SLIDE 38
Feature 1 Feature 2 distance Dendrogram d
The anatomy of Hierarchal Clustering
Hyper-parameters: clusters are defined beneath a threshold d. Alternatively, we can select a threshold d that corresponds to the desired number of clusters, k.
SLIDE 39
Feature 1 Feature 2 distance Dendrogram d
The anatomy of Hierarchal Clustering
Hyper-parameters: clusters are defined beneath a threshold d. Alternatively, we can select a threshold d that corresponds to the desired number of clusters, k.
SLIDE 40
The anatomy of Hierarchal Clustering
Hyper-parameters: clusters are defined beneath a threshold d. Alternatively, we can select a threshold d that corresponds to the desired number of clusters, k. We can use the resulting dendrogram to choose a “good” threshold:
distance
SLIDE 41
The anatomy of Hierarchal Clustering
Hyper-parameters: clusters are defined beneath a threshold d. Alternatively, we can select a threshold d that corresponds to the desired number of clusters, k. We can use the resulting dendrogram to choose a “good” threshold:
distance
SLIDE 42
The anatomy of Hierarchal Clustering
Input dataset: can either be a list of objects with measured properties, or a distance matrix that represents pair-wise distances between objects. What happens if we have an outlier in the dataset?
SLIDE 43
The anatomy of Hierarchal Clustering
Input dataset: can either be a list of objects with measured properties, or a distance matrix that represents pair-wise distances between objects. What happens if the dataset does not have clear clusters?
distance
SLIDE 44
The anatomy of Hierarchal Clustering
Input dataset: can either be a list of objects with measured properties, or a distance matrix that represents pair-wise distances between objects. Different linkage methods are helpful with different datasets.
single linkage complete linkage average linkage
SLIDE 45
“Statistics, Data Mining, and Machine Learning in Astronomy”, by Ivezic, Connolly, Vanderplas, and Gray (2013).
Hierarchal Clustering in Astronomy
SLIDE 46
Visualizing similarity matrices with Hierarchical Clustering
Input: 10,000 emission line spectra, covering the wavelength range 300 - 700
- nm. There are ~90 emission lines in each spectrum, with an average SNR of 2-4.
wavelength (nm) wavelength (nm) normalized flux normalized flux
SLIDE 47
Visualizing similarity matrices with Hierarchical Clustering
We compute a correlation matrix of all the observed wavelengths. wavelength (nm) wavelength (nm) correlation coefficient
SLIDE 48
Visualizing similarity matrices with Hierarchical Clustering
We convert the correlation matrix to a distance matrix, and build a dendrogram
SLIDE 49
Visualizing similarity matrices with Hierarchical Clustering
We reorder the correlation matrix (the wavelengths) according to the resulting dendrogram. reordered axis
SLIDE 50
Visualizing similarity matrices with Hierarchical Clustering
de Souza et. al 2015
SLIDE 51
Questions?
SLIDE 52
Gaussian Mixture models
See: http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_covariances.html#sphx-glr-auto-examples-mixture-plot-gmm- covariances-py
SLIDE 53