| Title: | Amos Tanay's Group High Performance Statistical Utilities |
|---|---|
| Description: | A collection of high performance utilities to compute distance, correlation, auto correlation, clustering and other tasks. Contains graph clustering algorithm described in "MetaCell: analysis of single-cell RNA-seq data using K-nn graph partitions" (Yael Baran, Akhiad Bercovich, Arnau Sebe-Pedros, Yaniv Lubling, Amir Giladi, Elad Chomsky, Zohar Meir, Michael Hoichman, Aviezer Lifshitz & Amos Tanay, 2019 <doi:10.1186/s13059-019-1812-2>). |
| Authors: | Michael Hoichman [aut], Aviezer Lifshitz [aut, cre] |
| Maintainer: | Aviezer Lifshitz <[email protected]> |
| License: | GPL-2 |
| Version: | 2.5.0 |
| Built: | 2026-11-19 12:23:36 UTC |
| Source: | https://github.com/tanaylab/tgstat |
Performs a chi-squared test with optional Yates' continuity correction on each row of a two-column count matrix. For each row, a 2x2 contingency table is constructed using the row counts and the column sums, and the chi-squared statistic and p-value are computed.
tgs_chi2(x, yates = TRUE)tgs_chi2(x, yates = TRUE)
x |
a numeric matrix or sparse matrix of |
yates |
logical; if |
This function is useful for differential gene expression analysis, where each row represents a gene and the two columns represent UMI counts in two conditions. The test determines whether the gene's proportion differs significantly between conditions.
For each row i, the contingency table is:
| Condition 1 | Condition 2 | |
| Gene i | x[i,1] | x[i,2] |
| Other genes | colSum1 - x[i,1] | colSum2 - x[i,2] |
A numeric matrix with nrow(x) rows and 2 columns named
"chi2" and "pval". Row names from x are preserved.
When the denominator of the chi-squared formula is zero, the statistic
is set to 0 and the p-value to 1.
# Note: all the available CPU cores might be used set.seed(42) # Simulate UMI counts for 1000 genes in 2 conditions mat <- matrix(rpois(2000, lambda = 100), ncol = 2) rownames(mat) <- paste0("gene", 1:1000) colnames(mat) <- c("condition1", "condition2") result <- tgs_chi2(mat) head(result) # Without Yates' correction result_no_yates <- tgs_chi2(mat, yates = FALSE) # With sparse matrix sparse_mat <- Matrix::Matrix(mat, sparse = TRUE) result_sparse <- tgs_chi2(sparse_mat)# Note: all the available CPU cores might be used set.seed(42) # Simulate UMI counts for 1000 genes in 2 conditions mat <- matrix(rpois(2000, lambda = 100), ncol = 2) rownames(mat) <- paste0("gene", 1:1000) colnames(mat) <- c("condition1", "condition2") result <- tgs_chi2(mat) head(result) # Without Yates' correction result_no_yates <- tgs_chi2(mat, yates = FALSE) # With sparse matrix sparse_mat <- Matrix::Matrix(mat, sparse = TRUE) result_sparse <- tgs_chi2(sparse_mat)
Calculates correlation between two matrices columns or auto-correlation between a matrix columns.
tgs_cor( x, y = NULL, pairwise.complete.obs = FALSE, spearman = FALSE, tidy = FALSE, threshold = 0 ) tgs_cor_knn( x, y = NULL, knn, pairwise.complete.obs = FALSE, spearman = FALSE, threshold = 0 )tgs_cor( x, y = NULL, pairwise.complete.obs = FALSE, spearman = FALSE, tidy = FALSE, threshold = 0 ) tgs_cor_knn( x, y = NULL, knn, pairwise.complete.obs = FALSE, spearman = FALSE, threshold = 0 )
x |
numeric matrix or data frame |
y |
numeric matrix or data frame. In 'tgs_cor' also a numeric vector, which is treated as a matrix of one column, and then 'x' can be a vector too. |
pairwise.complete.obs |
see below |
spearman |
if 'TRUE' Spearman correlation is computed, otherwise Pearson |
tidy |
if 'TRUE' data is outputed in tidy format |
threshold |
absolute threshold above which values are outputed in tidy format |
knn |
the number of highest correlations returned per column |
'tgs_cor' is very similar to 'stats::cor'. Unlike the latter it uses all available CPU cores to compute the correlation in a much faster way. The basic implementation of 'pairwise.complete.obs' is also more efficient giving overall great run-time advantage.
Unlike 'stats::cor' 'tgs_cor' implements only two modes of treating data containing NA, which are equivalent to 'use="everything"' and 'use="pairwise.complete.obs". Please refer the documentation of this function for more details.
'tgs_cor(x, y, spearman = FALSE)' is equivalent to 'cor(x, y, method = "pearson")' 'tgs_cor(x, y, spearman = TRUE)' is equivalent to 'cor(x, y, method = "spearman")' 'tgs_cor(x, y, pairwise.complete.obs = TRUE, spearman = TRUE)' is equivalent to 'cor(x, y, use = "pairwise.complete.obs", method = "spearman")' 'tgs_cor(x, y, pairwise.complete.obs = TRUE, spearman = FALSE)' is equivalent to 'cor(x, y, use = "pairwise.complete.obs", method = "pearson")'
'tgs_cor' can output its result in "tidy" format: a data frame with three columns named 'col1', 'col2' and 'cor'. Only the correlation values which abs are equal or above the 'threshold' are reported. For auto-correlation (i.e. when 'y=NULL') a pair of columns numbered X and Y is reported only if X < Y.
'tgs_cor_knn' works similarly to 'tgs_cor'. Unlike the latter it returns only the highest 'knn' correlations for each column in 'x'. The result of 'tgs_cor_knn' is always outputed in "tidy" format.
One of the reasons to opt 'tgs_cor_knn' over a pair of calls to 'tgs_cor' and 'tgs_knn' is the reduced memory consumption of the former. For auto-correlation case (i.e. 'y=NULL') given that the number of columns NC exceeds the number of rows NR, then 'tgs_cor' memory consumption becomes a factor of NCxNC. In contrast 'tgs_cor_knn' would consume in the similar scenario a factor of max(NCxNR,NCxKNN). Similarly 'tgs_cor(x,y)' would consume memory as a factor of NCXxNCY, wherever 'tgs_cor_knn(x,y,knn)' would reduce that to max((NCX+NCY)xNR,NCXxKNN).
A constant column, and without 'pairwise.complete.obs' a column with missing values, has no correlations: 'tgs_cor' returns 'NA' for it, and in 'tgs_cor_knn' it has no neighbours and is not a neighbour of other columns. With 'pairwise.complete.obs' and missing values a column is treated as constant if all its values that are not missing are equal. A column that is constant only over the rows it shares with another column is not detected, and its correlation with that column is not reliable.
'tgs_cor_knn' or 'tgs_cor' with 'tidy=TRUE' return a data frame,
where each row represents correlation between two pairs of columns from 'x'
and 'y' (or two columns of 'x' itself if 'y==NULL'). 'tgs_cor' with the
'tidy=FALSE' returns a matrix of correlation values, where val[X,Y]
represents the correlation between columns X and Y of the input matrices (if
'y' is not 'NULL') or the correlation between columns X and Y of 'x' (if 'y'
is 'NULL').
# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r1 <- tgs_cor(m, spearman = FALSE) r2 <- tgs_cor(m, pairwise.complete.obs = TRUE, spearman = TRUE) r3 <- tgs_cor_knn(m, NULL, 5, spearman = FALSE)# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r1 <- tgs_cor(m, spearman = FALSE) r2 <- tgs_cor(m, pairwise.complete.obs = TRUE, spearman = TRUE) r3 <- tgs_cor_knn(m, NULL, 5, spearman = FALSE)
Calculates the euclidean distances between the rows of the correlation matrix of the columns of 'x'.
tgs_cor_dist(x, pairwise.complete.obs = FALSE, spearman = FALSE)tgs_cor_dist(x, pairwise.complete.obs = FALSE, spearman = FALSE)
x |
numeric matrix or data frame |
pairwise.complete.obs |
see 'tgs_cor' documentation |
spearman |
if 'TRUE' Spearman correlation is computed, otherwise Pearson |
'tgs_cor_dist(x, pairwise.complete.obs, spearman)' returns the same distances as 'tgs_dist(tgs_cor(x, pairwise.complete.obs = pairwise.complete.obs, spearman = spearman))' up to rounding errors, for well-conditioned data. (When the columns of 'x' have offsets that are large relative to their spread, 'tgs_cor' loses precision while 'tgs_cor_dist', which centers the columns first, does not, so the two differ.)
When 'x' has much fewer rows (G) than columns (N), 'tgs_cor_dist' does not compute the NxN correlation matrix. Instead it maps each column of 'x' to a point in G dimensions, such that the euclidean distances between the points equal the distances between the rows of the correlation matrix, and passes these points to 'tgs_dist'. This takes time proportional to G^3 + N^2 x G (an eigendecomposition of a GxG matrix and the distances) instead of N^2 x G + N^3, and is used when G < N / 3 with BLAS ('tgs_use.blas') and G < N / 2 without, which is where it was measured to be faster.
It is also used only when all the values of 'x' are finite and no column of 'x' is constant. Otherwise 'tgs_cor_dist' returns 'tgs_dist(tgs_cor(...))'.
An object of class 'dist' with the distances between the columns of 'x', similar to the output of 'tgs_dist'.
# Note: all the available CPU cores might be used set.seed(seed = 0) m <- matrix(rnorm(100 * 2000), nrow = 100) d <- tgs_cor_dist(m) hc <- hclust(d)# Note: all the available CPU cores might be used set.seed(seed = 0) m <- matrix(rnorm(100 * 2000), nrow = 100) d <- tgs_cor_dist(m) hc <- hclust(d)
Calculates distances between the matrix rows, or between the rows of two matrices.
tgs_dist( x, y = NULL, diag = FALSE, upper = FALSE, tidy = FALSE, threshold = Inf )tgs_dist( x, y = NULL, diag = FALSE, upper = FALSE, tidy = FALSE, threshold = Inf )
x |
numeric matrix or data frame |
y |
numeric matrix with the same number of columns as 'x', or 'NULL' |
diag |
see 'dist' documentation |
upper |
see 'dist' documentation |
tidy |
if 'TRUE' data is outputed in tidy format |
threshold |
threshold below which values are outputed in tidy format |
This function is very similar to 'package:stats::dist'. Unlike the latter it uses all available CPU cores to compute the distances in a much faster way.
Unlike 'package:stats::dist' 'tgs_dist' uses always "euclidean" metrics (see 'method' parameter of 'dist' function). Thus:
'tgs_dist(x)' is equivalent to 'dist(x, method = "euclidean")'
If 'y' is given, 'tgs_dist' returns the distances between each row of 'x'
and each row of 'y': 'tgs_dist(x, y)' is equivalent to
as.matrix(dist(rbind(x, y)))[1:nrow(x), nrow(x) + 1:nrow(y)], and
missing values are treated as in 'dist'. 'diag' and 'upper' are then ignored.
'tgs_dist' can output its result in "tidy" format: a data frame with three columns named 'row1', 'row2' and 'dist'. Only the distances that are less or equal than the 'threshold' are reported. Distance between row number X and Y is reported only if X < Y, unless 'y' is given ('row1' is then a row of 'x' and 'row2' a row of 'y'). 'diag' and 'upper' parameters are ignored when the result is returned in "tidy" format.
With BLAS ('tgs_use.blas') the squared distance between points x and y is computed as 'sum(x^2) + sum(y^2) - 2 * sum(x * y)', which loses precision for points that are close relative to their norms. A squared distance that is within the rounding error of this formula is computed directly, so identical points are at distance 0.
If 'tidy' is 'FALSE' - the output is similar to that of 'dist' function, or if 'y' is given, a matrix of the distances between the rows of 'x' (its rows) and the rows of 'y' (its columns). If 'tidy' is 'TRUE' - 'tgs_dist' returns a data frame, where each row represents distances between two pairs of original rows.
# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r <- tgs_dist(m) r2 <- tgs_dist(m, m[1:10, ])# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r <- tgs_dist(m) r2 <- tgs_dist(m, m[1:10, ])
For each row of 'x' finds the 'knn' nearest rows of 'y', or of 'x' itself if 'y' is 'NULL', by euclidean distance.
tgs_dist_knn(x, y = NULL, knn, threshold = Inf)tgs_dist_knn(x, y = NULL, knn, threshold = Inf)
x |
numeric matrix or data frame |
y |
numeric matrix with the same number of columns as 'x', or 'NULL' |
knn |
the number of nearest rows returned per row of 'x' |
threshold |
the maximal distance of a neighbour |
'tgs_dist_knn' returns the same neighbours as selecting the 'knn' smallest distances of each row of 'x' from 'tgs_dist(x, y)' (or from 'tgs_dist(x)'), but it never holds all the distances in memory: its memory consumption is a factor of max(NXxNC + NYxNC, NXxKNN), where NX and NY are the numbers of rows of 'x' and 'y' and NC their number of columns.
The distances, including the treatment of missing values, are those of 'tgs_dist'. If 'y' is 'NULL' a row is not a neighbour of itself. Ties are broken by the lower row number, a pair whose distance is 'NA' is never a neighbour, and only the distances that are less or equal than 'threshold' are reported, so a row may have fewer than 'knn' neighbours.
With BLAS ('tgs_use.blas') and data without missing values, the squared distance between points x and y is computed as 'sum(x^2) + sum(y^2) - 2 * sum(x * y)', which loses precision for points that are close relative to their norms.
A data frame with the columns 'row1' (a row of 'x'), 'row2' (a row of 'y', or of 'x' if 'y' is 'NULL'), 'dist' (their distance) and 'rank' (1 for the nearest row), sorted by 'row1' and 'rank'. 'row1' and 'row2' are factors if the matrices have row names.
tgs_dist(), tgs_cor_knn(), tgs_knn()
# Note: all the available CPU cores might be used set.seed(seed = 0) m <- matrix(rnorm(5000 * 20), nrow = 5000) centers <- m[sample(nrow(m), 10), ] nn <- tgs_dist_knn(m, knn = 10) nearest_center <- tgs_dist_knn(m, centers, knn = 1)# Note: all the available CPU cores might be used set.seed(seed = 0) m <- matrix(rnorm(5000 * 20), nrow = 5000) centers <- m[sample(nrow(m), 10), ] nn <- tgs_dist_knn(m, knn = 10) nearest_center <- tgs_dist_knn(m, centers, knn = 1)
Checks whether all the elements of the vector are finite.
tgs_finite(x)tgs_finite(x)
x |
numeric or integer vector or matrix |
'tgs_finite' returns 'TRUE' if all the elements of 'x' are finite numbers. (See: 'is.finite'.)
'TRUE' if all the elements of 'x' are finite, otherwise 'FALSE'.
tgs_finite(1:10) tgs_finite(c(1:10, NaN)) tgs_finite(c(1:10, Inf))tgs_finite(1:10) tgs_finite(c(1:10, NaN)) tgs_finite(c(1:10, Inf))
Builds directed graph of correlations where the nodes are the matrix columns.
tgs_graph(x, knn, k_expand, k_beta = 3)tgs_graph(x, knn, k_expand, k_beta = 3)
x |
see below |
knn |
maximal node degree |
k_expand |
see below |
k_beta |
see below |
This function builds a directed graph based on the edges in 'x' and their ranks.
'x' is a data frame containing 4 columns named: 'col1', 'col2', 'val', 'rank'. The third column ('val' can have a different name). The result in the compatible format is returned, for example, by 'tgs_knn' function.
'tgs_graph' prunes some of the edges in 'x' based on the following steps:
A pair of columns i, j that appears in 'x' in 'col1', 'col2' implies the edge in the graph from i to j: edge(i,j). Let the rank of i and j be rank(i,j).
Calculate symmetrised rank of i and j: sym_rank(i,j) = rank(i,j) * rank(j,i). If one of the ranks is missing from the previous result sym_rank is set to NA.
Prune the edges: remove edge(i,j) if sym_rank(i,j) == NA OR sym_rank(i,j) < knn * knn * k_expand
Prune excessive incoming edges: remove edge(i,j) if more than knn * k_beta edges of type edge(node,j) exist and sym_rank(i,j) is higher than sym_rank(node,j) for node != j.
Prune excessive outgoing edges: remove edge(i,j) if more than knn edges of type edge(i,node) exist and sym_rank(i,j) is higher than sym_rank(i,node) for node != i.
The graph edges are returned in a data frame, with the weight of each edge. edge(i,j) receives weight 1 if its sym_rank is the lowest among all edges of type edge(i,node). Formally defined: weight(i,j) = 1 - (place(i,j) - 1) / knn, where place(i,j) is the location of edge(i,j) within the sorted set of edges outgoing from i, i.e. edge(i,node). The sort is done by sym_rank of the edges.
# Note: all the available CPU cores might be used set.seed(seed = 1) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r1 <- tgs_cor(m, pairwise.complete.obs = FALSE, spearman = TRUE) r2 <- tgs_knn(r1, knn = 30, diag = FALSE) r3 <- tgs_graph(r2, knn = 3, k_expand = 10)# Note: all the available CPU cores might be used set.seed(seed = 1) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r1 <- tgs_cor(m, pairwise.complete.obs = FALSE, spearman = TRUE) r2 <- tgs_knn(r1, knn = 30, diag = FALSE) r3 <- tgs_graph(r2, knn = 3, k_expand = 10)
Clusters directed graph.
tgs_graph_cover(graph, min_cluster_size, cooling = 1.05, burn_in = 10)tgs_graph_cover(graph, min_cluster_size, cooling = 1.05, burn_in = 10)
graph |
directed graph in the format returned by tgs_graph |
min_cluster_size |
used to determine the candidates for seeding (= min weight) |
cooling |
factor that is used to gradually increase the chance of a node to stay in the cluster |
burn_in |
number of node reassignments after which cooling is applied |
The algorithm is explained in a "MetaCell: analysis of single-cell RNA-seq data using K-nn graph partitions" paper, published in "Genome Biology" #20: https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1812-2
Data frame that maps each node to its cluster.
# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r1 <- tgs_cor(m, pairwise.complete.obs = FALSE, spearman = TRUE) r2 <- tgs_knn(r1, knn = 30, diag = FALSE) r3 <- tgs_graph(r2, knn = 3, k_expand = 10) r4 <- tgs_graph_cover(r3, 5)# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r1 <- tgs_cor(m, pairwise.complete.obs = FALSE, spearman = TRUE) r2 <- tgs_knn(r1, knn = 30, diag = FALSE) r3 <- tgs_graph(r2, knn = 3, k_expand = 10) r4 <- tgs_graph_cover(r3, 5)
Clusters directed graph multiple times with randomized sample subset.
tgs_graph_cover_resample( graph, knn, min_cluster_size, cooling = 1.05, burn_in = 10, p_resamp = 0.75, n_resamp = 500, method = "hash" )tgs_graph_cover_resample( graph, knn, min_cluster_size, cooling = 1.05, burn_in = 10, p_resamp = 0.75, n_resamp = 500, method = "hash" )
graph |
directed graph in the format returned by tgs_graph |
knn |
maximal number of edges used per node for each sample subset |
min_cluster_size |
used to determine the candidates for seeding (= min weight) |
cooling |
factor that is used to gradually increase the chance of a node to stay in the cluster |
burn_in |
number of node reassignments after which cooling is applied |
p_resamp |
fraction of total number of nodes used in each sample subset |
n_resamp |
number iterations the clustering is run on different sample subsets |
method |
method for calculating co_cluster and co_sample; valid values: "hash", "full", "edges" |
The algorithm is explained in a "MetaCell: analysis of single-cell RNA-seq data using K-nn graph partitions" paper, published in "Genome Biology" #20: https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1812-2
If method == "hash", a list with two members. The first member is a data frame with 3 columns: "node1", "node2" and "cnt". "cnt" indicates the number of times "node1" and "node2" appeared in the same cluster. The second member of the list is a vector of number of nodes length reflecting how many times each node was used in the subset.
If method == "full", a list containing two matrices: co_cluster and co_sample.
If method == "edges", a list containing two data frames: co_cluster and co_sample.
# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 200 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) r1 <- tgs_cor(m, pairwise.complete.obs = FALSE, spearman = TRUE) r2 <- tgs_knn(r1, knn = 20, diag = FALSE) r3 <- tgs_graph(r2, knn = 3, k_expand = 10) r4 <- tgs_graph_cover_resample(r3, 10, 1)# Note: all the available CPU cores might be used set.seed(seed = 0) rows <- 100 cols <- 200 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) r1 <- tgs_cor(m, pairwise.complete.obs = FALSE, spearman = TRUE) r2 <- tgs_knn(r1, knn = 20, diag = FALSE) r3 <- tgs_graph(r2, knn = 3, k_expand = 10) r4 <- tgs_graph_cover_resample(r3, 10, 1)
Returns k highest values of each column.
tgs_knn(x, knn, diag = FALSE, threshold = 0)tgs_knn(x, knn, diag = FALSE, threshold = 0)
x |
numeric matrix or data frame (see below) |
knn |
the number of highest values returned per column |
diag |
if 'F' values of row 'i' and col 'j' are skipped for each i == j |
threshold |
filter out values lower than threshold |
'tgs_knn' returns the highest 'knn' values of each column of 'x' (if 'x' is a matrix). 'x' can be also a sparse matrix given in a data frame of 'col', 'row', 'value' format.
'NA' and 'Inf' values are skipped as well as the values below 'threshold'. If 'diag' is 'F' values of the diagonal (row == col) are skipped too.
A sparse matrix in a data frame format with 'col1', 'col2', 'val' and 'rank' columns. 'col1' and 'col2' represent the column and the row number of 'x'.
# Note: all the available CPU cores might be used set.seed(seed = 1) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r <- tgs_knn(m, 3)# Note: all the available CPU cores might be used set.seed(seed = 1) rows <- 100 cols <- 1000 vals <- sample(1:(rows * cols / 2), rows * cols, replace = TRUE) m <- matrix(vals, nrow = rows, ncol = cols) m[sample(1:(rows * cols), rows * cols / 1000)] <- NA r <- tgs_knn(m, 3)
Performs a two-sample Kolmogorov-Smirnov test on each column of a matrix,
comparing the rows labeled TRUE with the rows labeled FALSE.
tgs_ks(x, labels, alternative = c("two.sided", "less", "greater"))tgs_ks(x, labels, alternative = c("two.sided", "less", "greater"))
x |
a numeric matrix or a data frame of numeric columns. Rows are observations and columns are features. |
labels |
a logical vector of length |
alternative |
one of |
For each column j, the statistic and the p-value are equal to those of
ks.test(x[labels, j], x[!labels, j], alternative = alternative,
exact = FALSE): D is the maximal difference between the empirical
cumulative distribution functions of the two samples (two-sided), or the
maximal difference in the direction of alternative, and the p-value
is computed from the asymptotic (Kolmogorov) distribution of the statistic,
also when there are ties.
Missing values (NA, NaN) are dropped from each column
separately, and rows whose label is NA are ignored. Infinite values
are kept, as in ks.test.
A numeric matrix with ncol(x) rows and 2 columns named
"D" and "pval". Row names are the column names of x.
Columns where one of the samples has no non-missing values are NA.
# Note: all the available CPU cores might be used set.seed(42) x <- matrix(rnorm(2000 * 100), ncol = 100) labels <- rep(c(TRUE, FALSE), each = 1000) x[labels, 1:10] <- x[labels, 1:10] + 0.5 result <- tgs_ks(x, labels) head(result)# Note: all the available CPU cores might be used set.seed(42) x <- matrix(rnorm(2000 * 100), ncol = 100) labels <- rep(c(TRUE, FALSE), each = 1000) x[labels, 1:10] <- x[labels, 1:10] + 0.5 result <- tgs_ks(x, labels) head(result)
For each matrix row (or column) apply a function to each cell of a ragged array, that is to each (non-empty) group of values given by the levels of a factor.
tgs_matrix_tapply(x, index, fun, ..., margin = 2)tgs_matrix_tapply(x, index, fun, ..., margin = 2)
x |
a matrix or a sparse matrix of 'dgCMatrix' type |
index |
a factor (or a vector, coerced to a factor by 'as.factor') of the same length as the number of columns of 'x' (or rows, if 'margin' is 1) |
fun |
the function to be applied |
... |
optional arguments to 'fun' |
margin |
2 (the default) to group the columns of 'x' by 'index', 1 to group the rows. Can only be passed by name. |
'tgs_matrix_tapply(x, index, fun)' is essentialy an efficient implementation of 'apply(mat, 1, function(x) tapply(x, index, fun))'. 'tgs_matrix_tapply(x, index, fun, margin = 1)' groups the rows instead of the columns: it is the same as 'tgs_matrix_tapply(t(x), index, fun)' without transposing 'x', and for 'sum' the same as 'rowsum(x, index)'.
'sum', 'mean', 'max', 'min' and 'median' are computed in C++ when 'fun' is passed by name (e.g. 'fun = max', not 'fun = function(v) max(v)') and the only other argument is 'na.rm' (for 'sum' also the extra values to add, for 'mean' also 'trim = 0'). Any other function is called from R once for every group and row (or column), which is much slower.
The implicit zeros of a sparse matrix are values like any other, e.g. for 'mean', 'median', 'max' and 'min'.
A matrix with a row for each level of 'index' and a column for each
row of 'x' (or column, if 'margin' is 1). The [i, j] element is the
result of applying 'fun' to x[j, index == levels(index)[i]] (or to
x[index == levels(index)[i], j] if 'margin' is 1), and NA for a level
of 'index' that has no values.
Note that the return value is a dense matrix even when x is sparse.
# Note: all the available CPU cores might be used set.seed(seed = 1) nr <- 6 nc <- 10 mat <- matrix(sample(c(rep(0, 6), 1:3), nr * nc, replace = TRUE), nrow = nr, ncol = nc) index <- factor(rep_len(1:3, ncol(mat)), levels = 0:5) r1 <- apply(mat, 1, function(x) tapply(x, index, sum)) r2 <- tgs_matrix_tapply(mat, index, sum) r3 <- tgs_matrix_tapply(t(mat), index, max, margin = 1)# Note: all the available CPU cores might be used set.seed(seed = 1) nr <- 6 nc <- 10 mat <- matrix(sample(c(rep(0, 6), 1:3), nr * nc, replace = TRUE), nrow = nr, ncol = nc) index <- factor(rep_len(1:3, ncol(mat)), levels = 0:5) r1 <- apply(mat, 1, function(x) tapply(x, index, sum)) r2 <- tgs_matrix_tapply(mat, index, sum) r3 <- tgs_matrix_tapply(t(mat), index, max, margin = 1)
Performs a Wilcoxon rank sum (Mann-Whitney) test on each column of a
matrix, comparing the rows labeled TRUE with the rows labeled
FALSE, and computes the area under the ROC curve (AUC).
tgs_wilcox(x, labels, alternative = c("two.sided", "less", "greater"))tgs_wilcox(x, labels, alternative = c("two.sided", "less", "greater"))
x |
a numeric matrix or a data frame of numeric columns. Rows are observations and columns are features. |
labels |
a logical vector of length |
alternative |
one of |
For each column j, the statistic and the p-value are equal to those of
wilcox.test(x[labels, j], x[!labels, j], alternative = alternative,
exact = FALSE, correct = TRUE): W is the number of pairs of a
TRUE and a FALSE observation where the TRUE
observation is higher, with ties counted as 1/2, and the p-value is
computed from the normal approximation with correction for ties and
continuity correction. auc = W / (n1 * n2), where n1 and
n2 are the numbers of non-missing observations of the column in each
sample: the probability that a TRUE observation is higher than a
FALSE one.
Missing values (NA, NaN) are dropped from each column
separately, and rows whose label is NA are ignored. Infinite values
are kept, as in wilcox.test.
A numeric matrix with ncol(x) rows and 3 columns named
"W", "auc" and "pval". Row names are the column names
of x. Columns where one of the samples has no non-missing values are
NA. When the p-value is not defined (e.g. all values of the column
are tied and alternative is "two.sided"), it is NA.
# Note: all the available CPU cores might be used set.seed(42) x <- matrix(rnorm(2000 * 100), ncol = 100) labels <- rep(c(TRUE, FALSE), each = 1000) x[labels, 1:10] <- x[labels, 1:10] + 0.5 result <- tgs_wilcox(x, labels) head(result)# Note: all the available CPU cores might be used set.seed(42) x <- matrix(rnorm(2000 * 100), ncol = 100) labels <- rep(c(TRUE, FALSE), each = 1000) x[labels, 1:10] <- x[labels, 1:10] + 0.5 result <- tgs_wilcox(x, labels) head(result)