| Title: | Distance Metrics for High-Dimensional Clustering |
|---|---|
| Description: | We provide three distance metrics for measuring the separation between two clusters in high-dimensional spaces. The first metric is the centroid distance, which calculates the Euclidean distance between the centers of the two groups. The second is a ridge Mahalanobis distance, which incorporates a ridge correction constant, alpha, to ensure that the covariance matrix is invertible. The third metric is the maximal data piling distance, which computes the orthogonal distance between the affine spaces spanned by each class. These three distances are asymptotically interconnected and are applicable in tasks such as discrimination, clustering, and outlier detection in high-dimensional settings. The package also provides a distance-based outlier detection procedure for high-dimensional, low-sample-size data, as described in Ahn, Lee and Lee (2018). |
| Authors: | Jung Ae Lee [aut, cre], Jeongyoun Ahn [aut] |
| Maintainer: | Jung Ae Lee <[email protected]> |
| License: | GPL (>= 2) |
| Version: | 1.4 |
| Built: | 2026-10-18 20:06:30 UTC |
| Source: | https://github.com/cran/distanceHD |
We provide three distance metrics for measuring the separation between two clusters in high-dimensional spaces. The first metric is the centroid distance, which calculates the Euclidean distance between the centers of the two groups. The second is a ridge Mahalanobis distance, which incorporates a ridge correction constant, alpha, to ensure that the covariance matrix is invertible. The third metric is the maximal data piling distance, which computes the orthogonal distance between the affine spaces spanned by each class. These three distances are asymptotically interconnected and are applicable in tasks such as discrimination, clustering, and outlier detection in high-dimensional settings. The package also provides a distance-based outlier detection procedure for high-dimensional, low-sample-size data, as described in Ahn, Lee and Lee (2018).
Jung Ae Lee <[email protected]>; Jeongyoun Ahn <[email protected]>
Maintainer: Jung Ae Lee <[email protected]>
1. Ahn J, Marron JS (2010). The maximal data piling direction for discrimination. Biometrika, 97(1):254-259.
2. Ahn J, Lee MH, Yoon YJ (2012). Clustering high dimension, low sample size data using the maximal data piling distance. Statistica Sinica, 22(2):443-464.
3. Ahn J, Lee MH, Lee JA (2019). Distance-based outlier detection for high dimension, low sample size data. Journal of Applied Statistics.46(1):13-29.
Calculate the Centroid (Euclidean) distance between two groups in a high-dimensional space (d > n), with support for a single-member cluster. It also works in low-dimensional settings.
dist_cen(x, group)dist_cen(x, group)
x |
|
group |
|
A numeric value of distance
Jung Ae Lee <[email protected]>
data(leukemia) group = leukemia$Y # 38 patients status with a value of 1 or 2 x = leukemia$X # 38 by 3051 genes # apply the function dist_cen(x, group) # 25.4data(leukemia) group = leukemia$Y # 38 patients status with a value of 1 or 2 x = leukemia$X # 38 by 3051 genes # apply the function dist_cen(x, group) # 25.4
Calculate the Mahalanobis distance between two groups in a high-dimensional space (d > n), using a ridge correction on the covariance matrix to ensure invertibility. The method also supports a single-member cluster and works in low-dimensional settings without a ridge correction.
dist_mah(x, group, alpha)dist_mah(x, group, alpha)
x |
|
group |
|
alpha |
|
A numeric value of distance
Jung Ae Lee <[email protected]>
data(leukemia) group = leukemia$Y # 38 patients status with a value of 1 or 2 x = leukemia$X # 38 x 3051 genes # apply the function dist_mah(x, group) # 26.8 # default alpha d = 3051; n = 38 alpha = sqrt(log(d)/n) dist_mah(x, group, alpha) # 26.8data(leukemia) group = leukemia$Y # 38 patients status with a value of 1 or 2 x = leukemia$X # 38 x 3051 genes # apply the function dist_mah(x, group) # 26.8 # default alpha d = 3051; n = 38 alpha = sqrt(log(d)/n) dist_mah(x, group, alpha) # 26.8
Calculate the MDP (maximal data piling) distance between two groups in a high-dimensional space (d > n), with support for a single-member cluster. It also works in low-dimensional settings.
dist_mdp(x, group)dist_mdp(x, group)
x |
|
group |
|
A numeric value of distance
Jeongyoun Ahn <[email protected]>
data(leukemia) group = leukemia$Y # 38 patients status with a value of 1 or 2 x = leukemia$X # 38 x 3051 genes # apply the function dist_mah(x, group) # 26.8data(leukemia) group = leukemia$Y # 38 patients status with a value of 1 or 2 x = leukemia$X # 38 x 3051 genes # apply the function dist_mah(x, group) # 26.8
Distance-based outlier detection for high dimension, low sample size data (Ahn, Lee and Lee, 2018). The observation farthest from the rest is flagged as a potential outlier and removed, one at a time, up to Mout observations. Each step is displayed by plotting the observed distances against the distances expected without outliers, estimated by a parametric bootstrap.
HDout(data, method = 1:3, Mout = 4, Nsim = 30, plotY = TRUE)HDout(data, method = 1:3, Mout = 4, Nsim = 30, plotY = TRUE)
data |
|
method |
choice of distance: |
Mout |
maximum number of outliers to test. The default is 4. |
Nsim |
number of bootstrap simulations. The default is 30; 10 is faster and 100 is more accurate. |
plotY |
if |
At each step (Nout = 1, ..., Mout), the leave-one-out distance of each remaining observation j is the distance between the single-member group {j} and the group of all other observations, computed with dist_cen, dist_mah or dist_mdp. For the rM distance, the ridge correction constant is , where m is the number of remaining observations. The observation with the largest distance is flagged as a potential outlier and removed.
The bootstrap samples are drawn from a normal distribution with the sample mean and covariance of the data without the flagged observation(s); when p >= n, a ridge term is added to the covariance. For each of the Nsim samples, the sorted leave-one-out distances are computed and then averaged.
The plot has one row per method and one column per step. Each panel plots the observed sorted distances (y-axis) against the average simulated sorted distances (x-axis); the flagged observation is shown as a filled point labelled with its column index. A flagged point lying far above the pattern of the others suggests an outlier.
A list (returned invisibly) with the following elements:
dj_c: sorted centroid distances at the last iteration (Mout).
dj_rM: sorted rM distances at the last iteration.
dj_MDP: sorted MDP distances at the last iteration.
outliers: a list of the column indices flagged at steps 1, ..., Mout, one vector per method.
Distances are 0 for methods that were not run.
Jung Ae Lee <[email protected]>
Ahn, J., Lee, M. H. and Lee, J. A. (2018). Distance-based outlier detection for high dimension, low sample size data, Journal of Applied Statistics. doi:10.1080/02664763.2018.1452901
# simulated data example: p = 1000 variables, n = 19 observations data(simdata) dim(simdata) # 1000 by 19 y = simdata # p by n matrix (p >= n) # first 200 variables, small Mout and Nsim keep the example fast; # HDout(y) with the defaults (Mout = 4, Nsim = 30) takes longer out = HDout(y[1:200, ], Mout = 2, Nsim = 5) out$outliers # observations flagged at each step, per method # leukemia data: first 200 genes, transposed to genes (p) by patients (n); # centroid distance only, without the plot data(leukemia) HDout(t(leukemia$X[, 1:200]), method = 1, Mout = 2, Nsim = 5, plotY = FALSE)$outliers# simulated data example: p = 1000 variables, n = 19 observations data(simdata) dim(simdata) # 1000 by 19 y = simdata # p by n matrix (p >= n) # first 200 variables, small Mout and Nsim keep the example fast; # HDout(y) with the defaults (Mout = 4, Nsim = 30) takes longer out = HDout(y[1:200, ], Mout = 2, Nsim = 5) out$outliers # observations flagged at each step, per method # leukemia data: first 200 genes, transposed to genes (p) by patients (n); # centroid distance only, without the plot data(leukemia) HDout(t(leukemia$X[, 1:200]), method = 1, Mout = 2, Nsim = 5, plotY = FALSE)$outliers
Gene expression data (3051 genes and 38 tumor mRNA samples) from the leukemia microarray study of Golub et al. (1999). )
data(leukemia)data(leukemia)
A list with the following elements:
X: a (38 by 3051) matrix giving the expression levels of 3051 genes for 38 leukemia patients. Each row corresponds to a patient, each column to a gene.
Y: a numeric vector of length 38 giving the cancer class of each patient.
gene.names: a matrix containing the names of the 3051 genes for the gene expression matrix X. The three columns correspond to the gene index, ID, and Name, respectively.
The data are described in Golub et al. (1999) and can be freely downloaded from http://www.broadinstitute.org/cgi-bin/cancer/publications/pub_paper.cgi?paper_id=43.
S. Dudoit, J. Fridlyand and T. P. Speed (2002). Comparison of discrimination methods for the classification of tumors using gene expression data, Journal of the American Statistical Association 97, 77-87.
Golub et al. (1999). Molecular classification of cancer: class discovery and class prediction by gene expression monitoring, Science 286, 531-537.
plsgenomics::leukemia
data(leukemia) # how many samples and genes? dim(leukemia$X) # how many samples of class 1 and 2, respectively? table(leukemia$Y)data(leukemia) # how many samples and genes? dim(leukemia$X) # how many samples of class 1 and 2, respectively? table(leukemia$Y)
A simulated high dimension, low sample size data set with 1000 variables and 19 observations, used to illustrate the outlier detection procedure HDout.
data(simdata)data(simdata)
A numeric matrix of dimension 1000 by 19 (p by n, p >= n). Each row corresponds to a variable and each column to an observation, which is the input format of HDout.
data(simdata) # how many variables (p) and observations (n)? dim(simdata) # 1000 x 19 y = simdata # p by n matrix (p >= n) out = HDout(y[1:200, ], Mout = 2, Nsim = 5) # first 200 variables, for speed out$outliers # observations flagged at each step, per methoddata(simdata) # how many variables (p) and observations (n)? dim(simdata) # 1000 x 19 y = simdata # p by n matrix (p >= n) out = HDout(y[1:200, ], Mout = 2, Nsim = 5) # first 200 variables, for speed out$outliers # observations flagged at each step, per method