Semi-supervised consensus clustering for gene expression data analysis

Simple clustering methods such as agglomerative hierarchical clustering and k-means have been widely used on gene expression data analysis. However, individual clustering algorithms have their limitations in dealing with different datasets. For example, k-means is unable to capture clusters with complex structures, and selection of k value is somewhat challenge without subjectivity. Therefore, many studies used consensus clustering (also called cluster ensemble) to improve the robustness and quality of clustering results [1–4].

Consensus clustering solves a clustering problem in two steps. The first step, known as base clustering, takes a dataset as input and outputs an ensemble of clustering solutions. The second step takes the cluster ensemble as input and combines the solutions through a consensus function, and then produces final partitioning as the final output, known as final clustering. The consensus clustering algorithms differ in chosen algorithms for basic clustering, consensus function and final clustering. Monti et al. used hierarchical clustering(HC) or self-organizing map (SOM) as the base clustering to generate consensus matrix and either HC or SOM for final clustering [1]. Yu et al. used k-means as the base clustering on subspace datasets and graph-cut algorithms for the final clustering [2]. Kim used k-means as the base algorithm with random multiple number of clusters and applied a graph-cut algorithm for final clustering [3]. The base clustering generates diverse clustering solutions through: 1) generating subspace datasets using gene resampling [1, 2, 4]; 2) using a single clustering algorithm with random parameter initializations such as selecting a random number of clusters [3, 4]; 3) using different clustering algorithms for each base clustering [5]. Some consensus clustering methods used a pairwise similarity matrix of instances to combine multiple clustering solutions [1, 2], others used associations between instances and clusters in the consensus matrix [4]. These consensus clustering algorithms usually outperform single clustering algorithms on gene expression datasets [1–4].

Consensus clustering has been used for clustering samples to discover and classify cancer types in cancer microarray data [1–4, 6]. It achieved successes in capturing informative patterns from microarray data [1–3]. A well known consensus clustering algorithm, link-based cluster ensemble (LCE) was introduced in [4]. LCE outperforms 10 algorithms tested in [4], specifically, four simple clustering algorithms, three pairwise similarity based consensus clustering algorithms, and three graph-based cluster ensemble techniques. Consensus clustering is also used for clustering genes to identify biologically informative gene clusters [5].

Many studies used prior knowledge in clustering genes [7–13]. These methods are referred as semi-supervised clustering approaches. The results showed that using small amount of prior knowledge was able to significantly improve the clustering results; also the more specific prior knowledge used the better in improving the quality of clustering.

Consensus clustering itself can be considered as unsupervised and improves the robustness and quality of results. Semi-supervised clustering is partially supervised and improves the quality of results in domain knowledge directed fashion. Although there are many consensus clustering and semi-supervised clustering approaches, very few of them used prior knowledge in the consensus clustering. Yu et al. used prior knowledge in assessing the quality of each clustering solution and combining them in a consensus matrix [14]. In this paper, we propose to integrate semi-supervised clustering and consensus clustering, design a new semi-supervised consensus clustering algorithm, and compare it with consensus clustering and semi-supervised clustering algorithms, respectively. In our study, we evaluate the performance of semi-supervised consensus clustering, consensus clustering, semi-supervised clustering and single clustering algorithms using h-fold cross-validation. Prior knowledge was used on h-1 folds, but not in the testing data. We compared the performance of semi-supervised consensus clustering with other clustering methods.

Our semi-supervised consensus clustering algorithm (SSCC) includes a base clustering, consensus function, and final clustering. We use semi-supervised spectral clustering (SSC) as the base clustering, hybrid bipartite graph formulation (HBGF) as the consensus function, and spectral clustering (SC) as final clustering in the framework of consensus clustering in SSCC.

Semi-supervised spectral clustering

SSC uses prior knowledge in spectral clustering. It uses pairwise constraints from the domain knowledge. Pairwise constraints between two data points can be represented as must-links (in the same class) and cannot-links (in different classes). For each pair of must-link (i,j), assign s _{i j}  = s _{j i}  = 1, For each pair of cannot-link (i, j), assign s _{i j}  = s _{j i}  = 0.

If we use SSC for clustering samples in gene expression data using t-nearest neighbor graph representation, two samples with highly similar expression profiles are connected in the graph. Using cannot-links means to change the similarity between the pairs of samples into 0, which breaks edges between a pair of samples in the graph. Therefore, only must-links are applied in our study. The details of SSC algorithm is described in Algorithm 1. Given the data points x ₁, …, x _n , l pairwise constraints of must-link are generated. The similarity matrix S can be obtained using similarity function s _{i j}  = exp(-∥x _i  - x _j ∥²/2σ ²). σ is the scaling parameter for measuring when two points are considered similar, and was calculated according to [15]. Then S is modified to be a sparse matrix, only t nearest neighbors are kept for each data point in S. Then, l pairwise constraints are applied in S. Steps 5-10 follow normalized spectral clustering algorithm [16, 17].

Semi-supervised consensus clustering

To make a consensus clustering into a semi-supervised consensus clustering algorithm, prior knowledge can be applied in base clustering, consensus function, or final clustering. Final clustering is usually applied on the consensus matrix generated from base clustering. SSCC uses semi-supervised clustering algorithm SSC for base clustering, does not use prior knowledge either in consensus function or final clustering. Our experiment was performed using h-fold cross-validation. The dataset was split into training and testing sets, and the prior knowledge was added to the h-1 folds training set. After the final clustering result was obtained, it was evaluated on the testing set alone. The influence of prior knowledge could be assessed in a cross-validation framework.

Our semi-supervised consensus clustering algorithm is described in Algorithm 2. Similar to [4], for a given n × d dataset of n samples and d genes, a n × q data subspace (q < d) is generated by

$q=q_{\mathit{\text{min}}}+\lfloor \alpha (q_{\mathit{\text{max}}}-q_{\mathit{\text{min}}})\rfloor$

(1)

α ∈ [ 0,1] is a uniform random variable, q _{m i n} and q _{m a x} are the lower and upper bonds of the subspace. q _{m i n} and q _{m a x} are set to 0.75d and 0.85d. Let $\prod ={\pi }_1,\dots ,{\pi }_m$ be a cluster ensemble with m clustering solutions. SSC is applied on each subspace dataset to obtain clustering results. We use the fixed number of clusters k, each ${\pi }_i=C_1^i,\dots ,C_k^i$ is one clustering solution. A basic cluster-association matrix BM is generated at first based on the crisp associations between samples and clusters using HBGF, in which there are n samples and m×k clusters. If x _i belongs to a cluster C _j , B M(x _i ,C _j )=1,i=1,…,n; j=1,…,g, otherwise B M(x _i ,C _j )=0. Next, a refined cluster-association matrix RM is generated from BM by estimating new association values in R M(x _i ,C _j ) if B M(x _i ,C _j )=0. R M(x _i ,C _j ) is the similarity between C _j and other clusters to which x _i probably belongs. The similarity of any clusters in the cluster ensemble is obtained from a weighted graph of clusters. Finally, spectral clustering is applied on RM to obtain the final clustering solution.

We compared the performance of SSCC with SSC, LCE and k-means and each of our pairwise comparison provides information of the effect of either semi-supervision or consensus clustering. Specifically, comparing LCE with k-means reveals the effectiveness of ensemble strategy since k-means is used as the base clustering in LCE. Similarly, in comparing SSC with SSCC, we used the same amount of prior knowledge, so actually we compared spectral clustering with consensus clustering. The comparison between SSCC and LCE reveals the effect of semi-supervision under the consensus clustering paradigm.

SSCC significantly outperforms SSC with or without prior knowledge. This clearly shows that consensus clustering algorithms outperform single clustering algorithms in the gene expression datasets. This observation is consistent with [1–4].

We compared SSCC with LCE using the same datasets and same parameter settings. Without considering prior knowledge, the difference between SSCC and LCE is in base clustering, SSCC uses spectral clustering but LCE uses k-means. They both use spectral clustering for final clustering (Table 1). Without prior knowledge, SSC becomes SC, and SC outperforms k-means in all 8 datasets (Figures 1, 2 and Table 3). This indicates the performance of base clustering has significant influence on results of consensus clustering.

Since n>m, n>k, d>n, d>l, and d>t in our experiments, the bottle neck of SSCC is to find t nearest neighbors with computational time O(m n ² d). The implementation of spectral clustering is a parallel algorithm [20], so the majority of computational time of SSCC can be reduce to $O\left(\frac{m{n}^{2}d}{p^{\prime }}\right)$, where p ^′ is the number of parallel threads. SSCC is limited to large data set due to the computational complexity of spectral clustering. SSCC can be improved by adopting faster spectral clustering algorithms, which are applicable for data sets with thousands of instances.

Our study provided an insight into the contribution of consensus clustering and semi-supervised clustering to the clustering results. To our knowledge, the Knowledge based Cluster Ensemble (KCE) [14] is the only algorithm using prior knowledge in consensus clustering paradigm for gene expression datasets. Unfortunately, we are unable to directly compare SSCC with KCE because of the unavailability of the software.

Our study uses SSCC for clustering samples. Since the optimal number of clusters (k in k-means algorithm) and the class label of each sample are known, the prior knowledge is derived from the given class structure. A must-link constraint is given to a pair of samples if they are from the same class. For many real applications, we might not know the whole class structure, but most likely we know whether some of samples are in the same class (cluster). We can generate must-links between these samples, and prior knowledge is derived from these samples. In these cancer gene expression datasets, we validate the performance of SSCC with the labeled data. The next step would be to apply SSCC for clustering genes for gene function prediction. However, the performance on clustering genes might vary due to two reasons: the quality of prior knowledge and the optimal number of clusters. Pairwise constraints in this study have been generated from class labels of samples in the cancer gene expression datasets and they are true prior knowledge. Prior knowledge in clustering of genes will be known gene functions, and they are partial domain knowledge. A gene may have multiple functions; some functions are inclusive to others as well. For example, a level 6 gene ontology term apoptotic process (GO:0006915) has over ten thousands of gene products and under which at level 7, there are 21 GO terms. Our earlier work shows that more specific (higher level) GO term contribute better to semi-supervised clustering result [13]. Also the description of a certain gene function is based on current knowledge in the domain field. Such domain knowledge is often subject to change. For example, current knowledge of certain existing gene is limited and will gradually be enriched. Therefore, the generated prior knowledge from a pair of genes most likely contains certain noise and subsequently influence the results. The optimal number of clusters is often unknown and a different distance measure would generate a different optimum number of clusters. Therefore, for comparison of semi-supervised clustering algorithms, it is better to use defined prior knowledge, such as the sample labels we used in this paper. When an algorithm considered to be superior over the others, such an algorithm can be used to cluster genes.

In reality, obtaining large amount of prior knowledge for gene expression datasets is difficult. Designing algorithms which work best with a small amount of prior knowledge, such as less than 20 pairwise constraints, will be very useful for clustering microarray data. A study on semi-supervised clustering shows that with small amounts of prior knowledge, search-based approach tends to outperform similarity-based [31]. With larger amounts of labeled data, similarity-based tends to perform better. Combining both approaches outperforms respective individual approaches. SSC is a similarity-based semi-supervised clustering algorithm. The results in Figures 1, 2 show that the performance of SSCC and SSC is slightly improved with small numbers of constraints and significantly improved with increasing numbers of constraints. Our SSCC method presented in this paper is applicable not only to gene expression data, but also to other types of data as long as prior knowledge is provided.

In this study, we proposed a new semi-supervised consensus clustering method, designed an algorithm, and compared it with another semi-supervised clustering algorithm, a consensus clustering algorithm and a simple clustering algorithm on eight real cancer gene expression datasets. In general, using prior knowledge improves the performance of clustering in gene expression datasets. Consensus clustering is able to reach the goal of maximizing intra-cluster similarity and minimizing inter-cluster similarity. Also, using prior knowledge enhances the high consistency between data partitioning and domain knowledge. A combination of both significantly improves the quality of clustering. SSCC outperforms the semi-supervised clustering algorithm SSC and consensus clustering algorithm LCE in most datasets over various parameter settings, ensemble size and type, with or without prior knowledge. This study demonstrates that SSCC is an effective and robust semi-supervised consensus clustering algorithm with prior knowledge, and also a superior consensus clustering algorithm without prior knowledge.