On evaluating brain tissue classifiers without a ground truth

S. Bouix, M. Martin-Fernandez, L. Ungar, M. Nakamura M.-S. Koo, R. W. McCarley, M. E. Shenton
NeuroImage
Volume 36, Pages 1207-1224
2007

Download full paper

Abstract

In this paper, we present a set of techniques for the evaluation of brain tissue classifiers on a large data set of MR images of the head. Due to the difficulty of establishing a gold standard for this type of data, we focus our attention on methods which do not require a ground truth, but instead rely on a common agreement principle. Three different techniques are presented: the Williams’ index, a measure of common agreement; STAPLE, an Expectation Maximization algorithm which simultaneously estimates performance parameters and constructs an estimated reference standard; and Multidimensional Scaling, a visualization technique to explore similarity data. We apply these different evaluation methodologies to a set of eleven different segmentation algorithms on forty MR images. We then validate our evaluation pipeline by building a ground truth based on human expert tracings. The evaluations with and without a ground truth are compared. Our findings show that comparing classifiers without a gold standard can provide a lot of interesting information. In particular, outliers can be easily detected, strongly consistent or highly variable techniques can be readily discriminated, and the overall similarity between different techniques can be assessed. On the other hand, we also find that some information present in the expert segmentations is not captured by the automatic classifiers, suggesting that common agreement alone may not be sufficient for a precise performance evaluation of brain tissue classifiers.

Sub-regions of the brain selected for manual labeling.

Reference

Bouix S, Martin-Fernandez M, Ungar L, Koo MNMS, McCarley RW, Shenton ME. On evaluating brain tissue classifiers without a ground truth. NeuroImage 2007;36:1207-1224.

Bibtex entry

@Article{bouixNI07,
  author         = {S. Bouix and M. Martin-Fernandez and L. Ungar and M.       
                   Nakamura M.-S. Koo and R. W. McCarley and M. E. Shenton},   
  title          = {On evaluating brain tissue classifiers without a ground    
                   truth},                                                     
  journal        = {NeuroImage},                                               
  year           = {2007},                                                     
  volume         = {36},                                                       
  pages          = {1207--1224}
}

Grants

NIH K02-MH01110, NIH R01-MH50747, NIH P41-RR13218 (NAC), NIH U54-EB005149 (NAMIC), NIH R01-MH40799

Copyright Information

© Elsevier. Copyrights to this PDF document are held by Elsevier B.V.. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the Elsevier Publishing.

This material is presented electronically to ensure timely dissemination of scholarly and technical work. Certain rights are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the author and/or copyright holder.