Functional coherence metrics in protein families
© The Author(s) 2016
Received: 8 September 2015
Accepted: 17 May 2016
Published: 23 June 2016
Biological sequences, such as proteins, have been provided with annotations that assign functional information. These functional annotations are associations of proteins (or other biological sequences) with descriptors characterizing their biological roles. However, not all proteins are fully (or even at all) annotated. This annotation incompleteness limits our ability to make sound assertions about the functional coherence within sets of proteins. Annotation incompleteness is a problematic issue when measuring semantic functional similarity of biological sequences since they can only capture a limited amount of all the semantic aspects the sequences may encompass.
Instead of relying uniquely on single (reductive) metrics, this work proposes a comprehensive approach for assessing functional coherence within protein sets. The approach entails using visualization and term enrichment techniques anchored in specific domain knowledge, such as a protein family. For that purpose we evaluate two novel functional coherence metrics, mUI and mGIC that combine aspects of semantic similarity measures and term enrichment.
These metrics were used to effectively capture and measure the local similarity cores within protein sets. Hence, these metrics coupled with visualization tools allow an improved grasp on three important functional annotation aspects: completeness, agreement and coherence.
Measuring the functional similarity between proteins based on their annotations is a non trivial task. Several metrics exist but due both to characteristics intrinsic to the nature of graphs and extrinsic natures related to the process of annotation each measure can only capture certain functional annotation aspects of proteins. Hence, when trying to measure the functional coherence of a set of proteins a single metric is too reductive. Therefore, it is valuable to be aware of how each employed similarity metric works and what similarity aspects it can best capture. Here we test the behaviour and resilience of some similarity metrics.
Over the last two decades functional annotation systems have been providing annotations for numerous proteins as well as other gene products. One of the most common steps used in functional annotation is the use of sequence alignment algorithms to compare sequences and find homologies from which functions can be extrapolated. Usually, lists of proteins (or other gene products) result from the output of many high-throughput technologies. Therefore, not only is it important to identify common functions in those sets of proteins but also to quantify how functionally related the proteins are in order to increase understanding of the involvement of biological systems [1, 2].
The Gene Ontology (GO) project aims to provide generically consistent descriptions for the molecular phenomena in which gene products are involved . For over a decade the increasing popularity and consequent growth of GO has led to its adoption and prevalent use in annotation projects. Consequently, this pervasiveness has enabled and motivated the development of several semantic similarity metrics [4–6]. Semantic similarity can be defined as the quantity that reflects the closeness in meaning of two concepts in an ontology. However, the semantic similarity between two proteins, which can be annotated with several GO terms is commonly called “functional similarity” since it is the functional annotation terms that are being compared. More recently, several metrics focusing specifically on measuring the functional cohesiveness of a set of proteins (or gene products) through their annotations have been developed. These metrics for the assessment of functional coherence using annotations are commonly based on the previously developed groupwise semantic similarity approaches.
One of those metrics, GS2 , uses a set-based approach and was developed with computational efficiency in mind to measure gene set functional similarity based on GO terms. The GS2 algorithm ranks annotation terms using a simple gene counting method and then compares each gene with the remaining genes with respect to the distribution of functional annotations. This simple measure can only capture similarity trends within gene sets and can not precisely assess similarity. Despite that, GS2 has performed well when compared with the semantic similarity pairwise measure of .
On the other hand, another set of three different metrics: average seed degree, total length and relative seed degree were developed by , for the assessment of functional coherence in gene sets based on the topological properties of GO-derived graphs. The procedure leading to these metrics relies on building GO subgraphs that subsume each gene set annotation (for each GO aspect), whereas each node is a GO term and each edge is an is_a relationship between terms. Subsequently, those graphs are further enriched by adding genes, as a new type of node, associated to the original GO nodes, and additional new edges are created between GO terms whenever these share gene annotations. The original term-to-term edges are weighted using the Information Content  difference between both terms while the new edges created after addition of the gene nodes to the graph are statistically weighted based on the total number of edges in the graph and the number of supporting genes for each particular edge. Hence, this approach handles the issue at hand both from an annotation enrichment perspective and an annotation relationship perspective. Steiner trees are then extracted from the graphs and the sum of all edge lengths is minimized for all possible subgraphs. The aforementioned three metrics are then applied to these trees. The average seed degree averages, for a full tree, the counts of the number of genes associated to the seed terms thus reflecting a global measure of enrichment. On the other hand the total length metric reflects the overall relatedness of functions by performing the sum of the length of all edges in a tree. The relative seed degree metric combines the aspects described above as a ratio. The methodology performs well, but like other GO-evaluation methodologies, its metrics are dependent on the gene annotation state.
The GO-based functional dissimilarity (GFD) metric  approaches the problem of functional coherence in gene sets by considering that each gene can encode several proteins with different functions. In this metric, for each gene set, only the most common and specific function is chosen as being the most globally cohesive function. In this approach, genes are represented as sets to which a simple counting edge-based measure ratio is applied and that aims at equating both gene relatedness and specificity. The actual GFD is then the minimum of dissimilarity possible for all representations of a given set of genes. Like the previous metrics this one also depends on the completeness of the annotations used in order to provide accurate measurements. Furthermore, by considering only the most common and specific function in a gene set the authors are effectively discarding potential non-related functions that would cause noise, however at the cost of disregarding multi-functional associations in gene sets.
Furthermore, and despite not being exactly a system for measuring functional coherence in gene sets, RuleGO  provides a service that statistically compares and characterizes two disjointed gene sets. Underneath it runs a rule-based system that incrementally iterates the list of GO terms annotating the two input gene sets and verifies at each step if a new co-occurrence rule can be created. Much like the typical gene enrichment systems, this system also performs over-representation tests on the rules created and only rules corresponding to a p-value below a given statistical significance threshold (after multiple testing correction) are considered. This process results in multi-attribute rules containing annotation terms and respective support indexes and evaluation parameters that can be used in the characterization of the disjointed gene sets. In this methodology rules are evaluated by length (number of genes in a rule premise) representing support, by depth (normalized sum of the GO graph levels where terms in the rule appear) representing specificity and by an additional quality measure.
A different approach is taken by  where functional coherence in gene sets is assessed with the help of the biological literature. Here, term-by-gene matrices are constructed with entries derived from weighted frequencies of the terms across a collection of abstracts (biological literature). The genes are then represented as vectors and the similarity between them is calculated as the cosine of the vector angles. Thus, a pair of genes would have a cosine score of 1.0 if they shared the exact same abstracts in the collection. Gene sets in this method were deemed functionally coherent when cosine values above a given threshold (0.6) were often found with significances measured by a statistical test (Fisher’s exact test). This threshold was chosen based on the distribution of similarity cosine scores in 1,000 random gene sets. Hence, functional coherence here is derived essentially from the supporting literature, thus making the method sensitive to the quality of the document corpus used. Regardless, the method was used to obtain results similar to those produced by another literature-based functional coherence assessing method .
Since functional annotation quality is paramount,  developed a system to provide an annotation confidence score for genome annotations. The system operates on the basis of a genome comparison approach whereby annotations in a target genome are scored in comparison with a reference genome. The gene alignments across genomes are made via the BLAST tool with adjustments for expected number of genes (different organisms have different gene counts) and phylogenetic distance (closer genomes typically share more genes than distant ones). However, actual annotation similarity is derived from free-text annotations which are converted into word vectors that enable the calculation of a simple cosine similarity measure. Both sequence similarity and annotation similarity are combined into a single metric by applying statistical techniques.
Despite the existence of these types of metrics the protein annotation landscape is often very heterogeneous in terms of quality, specificity and completeness. Annotation quality is related with the annotation method and source used, e.g. defined by the different GO evidence codes associated to each annotation. Annotation specificity relates to how specific or general an annotation term is, and when in a protein set there is a clear disproportion between general term annotations and specific annotations, that set can be said to suffer from annotation incompleteness.
In this work we concern ourselves mostly with the aspects of annotation completeness and specificity. Given that functional similarity is derived from semantic similarity approaches over the annotation terms, it is also relevant to define the concept of annotation agreement as a measure of annotation homogeneity for a given set of proteins. This metric, will naively measure the coherence of a given set based on the fraction of shared annotation terms between all proteins in the set, and thus will be highly susceptible to the lack of annotation completeness. We use this measure as a baseline whereas we introduce other metrics to characterize the state of known functional similarity of a given set and gauge the potential state of annotation incompleteness. Hence, in this work functional coherence is defined as a measure of functional closeness (similarity) among all proteins in a set given the current functional annotations within that protein set.
A functional annotation is defined as a pairing between a gene product (protein) identifier and a term providing some functional description. In this study, only the molecular function term annotations from GO were considered because the aim of this work lies closer to studying one-dimensional annotation (as proposed by ) at the molecular functional level in enzymes. Ideally, the functional annotations over a given protein set should allow us to infer biological relationships within the set. In order to achieve that, it is convenient to have metrics that enable us to compare how similar (or dissimilar) annotations are within a given protein set. However, considering the GO DAG structure it becomes apparent that measuring functional relatedness via annotation is not a trivial matter. Therefore, in order to help make such assertions regarding functional relatedness, three main annotation aspects were considered: completeness, agreement and coherence.
This kind of annotation incompleteness can derive from the fact that different protein annotation methods are used, which provide different degrees of annotation confidence. Therefore, annotation heterogeneity is created accordingly to the annotation confidence level given by each annotation method. For instance, a majority of the automatic annotation methods typically create more generic annotations. On the other hand, manual curation is more likely to lead to more highly specific annotations. Additionally, the inherent research bias towards more intensively studied model organisms and biological processes can also help further this state of incompleteness.
with x i as the number of annotations for a term i, N as the total number of proteins annotated and t the total number of distinct annotation terms. Therefore, the greater the amount of shared annotations the greater is the annotation agreement. Figure 1 b) illustrates a hypothetical full annotation agreement situation. In this situation, each one of the one hundred proteins is annotated to the same exact annotation term set and thus that hypothetical set achieves maximum or total annotation agreement. However, this is a naive metric that is also overly sensitive to annotation incompleteness and even small amounts of noise.
Naturally, a set of proteins having a total annotation agreement is also functionally similar, to the extent of its most specific annotation terms. On the other hand, functional similarity may not need to be so strictly defined. Additionally, due to the above mentioned incompleteness issue and the multi-functional nature of proteins, when measuring functional similarity through annotation, it may be useful to consider just some of the annotations as being functionally characteristic of a given protein set. Furthermore, for the purposes of this work, the concept of annotation coherence is further refined and defined as the fraction of shared annotations that define the core of the functional activities that is common and most relevant and thus able to characterize a given protein set, as a functional cohesive group. Figure 1 c) illustrates a hypothetical full annotation coherence situation, where the grey shaded nodes represent the functionally more relevant terms, or the central functional cohesiveness of that set. However, a single metric is too reductive in assessing these (and other) different aspects of annotation that can dictate the functional coherence of the annotation space in protein sets. Therefore, in this work, we use a set of metrics and respective interpretation strategies relating to these three aspects of annotation described above in order to explore protein (enzyme) annotation spaces.
Fisher exact test’s 2x2 contingency table
Coherence resilience assays
Number of protein UniProt identifiers (size) in each of the classes in the CAZy database (ver. c7-2011)
List of the protein families belonging to the PL class in the CAZy database and their respective size (in number of UniProt identifiers)
For this work two novel functional coherence metrics, mUI and mGIC were developed. They are based on the combination of semantic similarity metrics simUI and simGIC and a term-for-term enrichment analysis as described by the following algorithm:
INIT annotationGraph and annotationGraph’
FOR each term IN annotationGraph
EXECUTE enrichment analysis of term
IF term enriched
annotationGraph’ < - term
mUI < - compute simUI of annotationGraph’
mGIC < - compute simGIC of annotationGraph’
The annotation graph for a protein set (family) being measured is generated (line 1). For each term in the annotation graph (line 2) enrichment analysis using a term-for-term (with Elim adjustment) strategy is performed as previously described. If a term is found to be statistically enriched (line 4) it is added to a derived annotation graph (line 5). When both annotation graphs are processed (line 6) the simUI and simGIC are applied to the shadow graph (annotationGraph’) resulting in the values for the mUI and mGIC metrics, respectively (line 7 and 8).
Results and discussion
From the analysis of Fig. 3 it can be seen, as expected, that the similarity reported by each metric generally decreases as noise (in the form of random proteins) is increasingly added (replacing the original proteins) in each of the tested PL families. In this study, each of considered metrics is scaled on a [0, 1] theoretical range. The aim of our protein family degeneration assays is to observe two main aspects for each of the metrics, noise resilience and resolution. With noise resilience we check by how much the reported values can vary given the same amount of noise. As for resolution we register the difference between the maximum and minimum values it can actually report during our assays.
The Agreement metric is the least noise resilient metric, as can be seen by both the generally low values it reports and the steep declines after adding small amounts of noise to family sets with previously high agreement. This property is most evident in mono-functional families like PL5, PL16 and PL17 and also PL12 where the introduction of 10 % random proteins produces a sharp decline in the reported values. This occurs because this naive metric only equates the average of annotation term frequencies in each protein family (or set). This metric was chosen and used as the overall baseline.
Difference between maximum and minimum values reported for each tested metric (Agreement, simUI, simGIC, mUI, mGIC, GS2) against each PL family and iterations of derived respective sets created by insertion of increasing amounts of random proteins (from CAZy) into the original families
The mUI and mGIC (such as the metrics they are derived from) also display, as expected, similar behaviours to each other. Their results measure the enrichment contribution relative to the original semantic similarity metrics. In fact, for most of the tested PL families and their respective degenerate sets the reported values are very similar. However, unlike the other tested metrics mUI and mGIC are resilient to noise (replacement with random proteins). That is evident from the gradual curves in Fig. 3 which in most families plateau until higher levels of randomization and typically only fall abruptly after addition of 90 % random proteins. This resilience to noise is conferred by the term enrichment step which pre-selects only the subset of proteins that are annotated with the terms found to be statistically significant by the enrichment procedure. Thus, this is an important factor to consider when analysing the results provided by these two metrics. As they were engineered to capture local (subset) functional coherence, for a comprehensive evaluation they should only be used in an analysis that also simultaneously considers the annotation coverage within the analysed set. This also explains the observed peaks at high noise levels in some of the families (PL2, PL6, PL9, PL11) where a small number of terms annotates a small subset of proteins and thus creates a local similarity effect. That is, at high levels of random protein replacement the original families are greatly degenerated because they lose the proteins that were characteristic for the identity of that family while, on the other hand, randomly gaining less related proteins. Hence, if a couple of random proteins being introduced happen to be very similar in terms of annotations and those terms are also found to be statistically enriched, then a new similarity core is introduced which results in the appearance of those peaks of high similarity. However, for this work this behaviour is advantageous because the underlying assumption is that each protein family shares core annotations that define the group role of that set of proteins. Thus, by using a term enrichment technique the purpose is to target and select these core annotation terms. The proteins annotated by these identified core annotation terms can then, for instance, be used for annotation extension within that set as previously proposed . Thus, according to that proposal, for an hypothetical partially annotated protein set (with an expected degree of functional relatedness) the mUI/mGIC metrics can be used to identify the functional core of that set while reporting its functional similarity. If that core, reports a high similarity value and also provides enough statistical power (number of associated protein sequences) it can be used to create, for instance, a Hidden Markov Model profile model. Subsequently, that model can potentially be used as a classifier in order to extend annotations from the core to the sub-annotated sequences in the original measured protein set.
For the case of the PL10 family set portrayed in Fig. 4 the “bottleneck annotation” is on the term “lyase activity”. Domain knowledge indicates that this term should annotate each protein in this family (e.g. the PL10 family is part of the Polysaccharide Lyases). However, this annotation term is relatively generic and considering the proportion of proteins not annotated with children of this term (as can be easily seen from the graph) it is fair to assume substantial annotation incompleteness. Additionally, considering the plot in Fig. 3 that represents the degeneration of the PL10 set, it can be seen that the values for mUI and mGIC actually increase along with the degeneration of the set. As previously explained the enrichment process of the mUI/mGIC algorithm considers only a protein subset of the target set being measured. Hence, it is important to consider other metrics (for instance the parent metrics simUI/simGIC) in tandem with these novel metrics for a global assessment of functional coherence in a set. Nevertheless, these novel metrics allow the identification of core activities which can potentially be extended to more sequences within the original set.
Measuring the functional similarity between proteins based on their annotations is a non trivial task. Several metrics exist but due to characteristics both intrinsic to the nature of graphs and extrinsic natures related to the process of annotation each measure can only capture certain functional annotation aspects of proteins. Hence, when trying to measure the functional coherence of a set of proteins a single metric is too reductive. Therefore, it is valuable to be aware of how each employed similarity metric works and what similarity aspects it can best capture. Here we test the behaviour and resilience of some similarity metrics.
Additionally, we propose a comprehensive approach at determining functional coherence in protein sets (families) based not only on metrics but also statistics (term enrichment) and visualization coupled with domain knowledge-based empirical assessments.
Furthermore, we propose two novel metrics mUI and mGIC that combine two of the above mentioned approaches, semantic similarity metrics and term enrichment. The goal is to capture protein subsets within families (or other functionally related sets) that characterize that family (or set), which can subsequently be used for annotation extension for potentially sub-annotated proteins within the same family (or set).
The proposed approach is modular and can be integrated with other annotation methodologies mostly as a pre-processing step. In the future, we will be implementing both mUI and mGIC (along with other) metrics into our web application GRYFUN. This will more easily capture the annotation functional cores in protein sets and pipeline them to a custom annotation extension module based on HMM profiles that we are currently developing.
This work was supported by the Portuguese Fundação para a Ciência e Tecnologia through a PhD research grant [PhD grant ref. SFRH/BD/48035/2008 to H.P.B] and three research centers Strategic Projects funding [project UID/MAT/00006/2013 (CEAUL) to L.S. & UID/MULTI/04046/2013 (BioISI) to L.A.C. & UID/CEC/00408/2013 (LaSIGE) to H.P.B. and F.M.C.] and by the European Commission [BiobankCloud project under the Seventh Framework Programme grant #317871 to F.M.C.]. Furthermore we like to thank Faculdade de Ciências da Universidade de Lisboa for the tuition support for H.P.B.
HPB implemented all metrics and performed the evaluation, and the remainder authors supervised the work. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Clark WT, Radivojac P. Information-theoretic evaluation of predicted ontological annotations. Bioinformatics. 2013; 29(13):53–61.View ArticleGoogle Scholar
- Jiang Y, Clark WT, Friedberg I, Radivojac P. The impact of incomplete knowledge on the evaluation of protein function prediction: a structured-output learning perspective. Bioinformatics. 2014; 30(17):609–16.View ArticleGoogle Scholar
- Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, Davis AP, Dolinski K, Dwight SS, Eppig JT, Harris MA, Hill DP, Issel-Tarver L, Kasarskis A, Lewis S, Matese JC, Richardson JE, Ringwald M, Rubin GM, Sherlock G. Gene ontology: tool for the unification of biology. the gene ontology consortium. Nat Genet. 2000; 25(1):25–9.View ArticleGoogle Scholar
- Pesquita C, Faria D, Bastos H, Ferreira A, Falcão A, Couto F. Metrics for go based protein semantic similarity: a systematic evaluation. BMC Bioinformatics. 2008; 9(Suppl 5):4. doi:http://dx.doi.org/10.1186/1471-2105-9-S5-S4.View ArticleGoogle Scholar
- Gan M, Dou X, Jiang R. From ontology to semantic similarity: Calculation of ontology-based semantic similarity. Sci World J. 2013; 2013:793091. doi:http://dx.doi.org/10.1155/2013/793091.View ArticleGoogle Scholar
- Wu X, Pang E, Lin K, Pei ZM. Improving the measurement of semantic similarity between gene ontology terms and gene products: Insights from an edge- and ic-based hybrid method. PLoS ONE. 2013; 8(5):1–11. doi:http://dx.doi.org/10.1371/journal.pone.0066745.View ArticleGoogle Scholar
- Ruths T, Ruths D, Nakhleh L. Gs2: an efficiently computable measure of go-based similarity of gene sets. Bioinformatics. 2009; 25(9):1178–84. http://bioinformatics.oxfordjournals.org/content/25/9/1178.full.pdf+html.View ArticleGoogle Scholar
- Wang JZ, Du Z, Payattakool R, Yu PS, Chen CF. A new method to measure the semantic similarity of go terms. Bioinformatics. 2007; 23(10):1274–81. http://bioinformatics.oxfordjournals.org/content/23/10/1274.full.pdf+html.View ArticleGoogle Scholar
- Richards AJ, Muller B, Shotwell M, Cowart LA, Rohrer B, Lu X. Assessing the functional coherence of gene sets with metrics based on the gene ontology graph. Bioinformatics. 2010; 26(12):79–87. http://bioinformatics.oxfordjournals.org/content/26/12/i79.full.pdf+html.View ArticleGoogle Scholar
- Resnik P. Using information content to evaluate semantic similarity in a taxonomy. In: In Proceedings of the 14th International Joint Conference on Artificial Intelligence: 1995. p. 448–53.
- Diaz-Diaz N, Aguilar-Ruiz J. Go-based functional dissimilarity of gene sets. BMC Bioinformatics. 2011; 12(1):360.View ArticleGoogle Scholar
- Gruca A, Sikora M, Polanski A. Rulego: a logical rules-based tool for description of gene groups by means of gene ontology. Nucleic Acids Res. 2011; 39(suppl 2):293–301.View ArticleGoogle Scholar
- Xu L, Furlotte N, Lin Y, Heinrich K, Berry MW, George EO, Homayouni R. Functional cohesion of gene sets determined by latent semantic indexing of pubmed abstracts. PLoS ONE. 2011; 6(4):18851.View ArticleGoogle Scholar
- Raychaudhuri S. A literature-based method for assessing the functional coherence of a gene group. Bioinformatics. 2003; 19(3):396–401.View ArticleGoogle Scholar
- Yang Y, Gilbert D, Kim S. Annotation confidence score for genome annotation: a genome comparison approach. Bioinformatics. 2010; 26(1):22–9. http://bioinformatics.oxfordjournals.org/content/26/1/22.full.pdf+html.View ArticleGoogle Scholar
- Reed JL, Famili I, Thiele I, Palsson BO. Towards multidimensional genome annotation. Nat Rev Genet. 2006; 7(2):130–41. doi:http://dx.doi.org/10.1038/nrg1769.View ArticleGoogle Scholar
- Pesquita C, Faria D, Falcão AO, Lord P, Couto FM. Semantic Similarity in Biomedical Ontologies. PLoS Comput Biol. 2009; 5(7):1000443. doi:http://dx.doi.org/10.1371/journal.pcbi.1000443.MathSciNetView ArticleGoogle Scholar
- Gentleman R. Visualizing and Distances Using GO. Technical report. 2005. http://www.bioconductor.org/docs/vignettes.html.
- Pesquita C, Faria D, Bastos H, Falcao AO, Couto F. Evaluating go-based semantic similarity measures. In: BioOntologies SIG at ISMB/ECCB - 15th Annual International Conference on Intelligent Systems for Molecular Biology (ISMB): 2007.
- Alexa A, Rahnenführer J, Lengauer T. Improved scoring of functional groups from gene expression data by decorrelating go graph structure. Bioinformatics. 2006; 22(13):1600–7.View ArticleGoogle Scholar
- Cantarel BL, Coutinho PM, Rancurel C, Bernard T, Lombard V, Henrissat B. The carbohydrate-active enzymes database (cazy): an expert resource for glycogenomics. Nucleic Acids Res. 2009; 37(suppl 1):233–8.View ArticleGoogle Scholar
- Barrell D, Dimmer E, Huntley RP, Binns D, O’Donovan C, Apweiler R. The goa database in 2009 - an integrated gene ontology annotation resource. Nucleic Acids Res. 2009; 37(Database-Issue):396–403.View ArticleGoogle Scholar
- Bastos HP, Clarke LA, Couto F. Annotation extension through protein family annotation coherence metrics. Front Genet. 2013; 4(201). doi:http://dx.doi.org/10.3389/fgene.2013.00201.
- Bastos HP, Sousa L, Clarke LA, Couto FM. Gryfun: A web application for go term annotation visualization and analysis in protein sets. PLoS ONE. 2015; 10(3):0119631. doi:http://dx.doi.org/10.1371/journal.pone.0119631.View ArticleGoogle Scholar