2024-02-21 : 33174 EMDB map entries, 19014 PDB coordinate entries RCSB PDB | PDBe

2010 EM Validation Task Force

The initial meeting of the Electron Microscopy Validation Task Force (EM VTF) was held on September 28-29, 2010. Twenty highly regarded EM scientists from around the world assembled at Rutgers University in New Brunswick NJ to consider possible validation standards for 3D maps and models produced by electron microscopy reconstruction methods. The 2-day workshop was co-chaired by Richard Henderson (MRC, Cambridge) for map validation and Andrej Sali (UCSF) for model validation. Initial recommendations are now published. The full citation is:

Richard Henderson, Andrej Sali, Matthew L. Baker, Bridget Carragher, Batsal Devkota, Kenneth H. Downing, Edward H. Egelman, Zukang Feng, Joachim Frank, Nikolaus Grigorieff, Wen Jiang, Steven J. Ludtke, Ohad Medalia, Pawel A. Penczek, Peter B. Rosenthal, Michael G. Rossmann, Michael F. Schmid, Gunnar F. Schröder, Alasdair C. Steven, David L. Stokes, John D. Westbrook, Willy Wriggers, Huanwang Yang, Jasmine Young, Helen M. Berman, Wah Chiu, Gerard J. Kleywegt, Catherine L. Lawson, Outcome of the First Electron Microscopy Validation Task Force Meeting, Structure, Volume 20, Issue 2, 8 February 2012, Pages 205-214, ISSN 0969-2126, 10.1016/j.str.2011.12.014.

Agenda

Tuesday, September 28, 2010

CABM, Rutgers University (Room 010)

Morning and Afternoon Sessions are open to the Rutgers/UMDNJ Community (flyer)

9:00am Welcome Wah Chiu
Introduction Helen Berman
Validation of X-ray structures overview Gerard Kleywegt
10:00am Break
10:15am
Overview of Charge/Questions to Committees Cathy Lawson
10:30am Map Validation Session Richard Henderson, Chair
10:30 Richard Henderson 10:45 Niko Grigorieff 11:00 Pawel Penczek

11:15 Steve Ludtke 11:30 Wen Jiang 11:45 Ed Egelman

12:00 Ohad Medalia 12:15 Alasdair Steven
12:30pm Lunch
1:40pm Model Validation Session Andrej Sali, Chair

1:45 Andrej Sali 2:00 Michael Rossmann 2:15 Matt Baker

2:30 Gunnar Schroeder 2:45 Willy Wriggers
3:00pm Coffee Break
3:15pm Additional Presentations, Discussion
5:00pm Meeting Adjourns - Transportation back to the Heldrich
7:00pm Dinner at Soho on George

Wednesday, September 29, 2010

The Heldrich Hotel (Rooms Neilson and Dickson)

09:00am Presentations and discussions to prepare report for full group
Maps: Room Neilson Richard Henderson, Chair
Models: Room Dickson Andrej Sali, Chair
11:45am Lunch
01:00pm Full group discussions of Maps and Models in Room Neilson
03:30pm Full group presentations in Room Neilson
05:00pm Meeting adjourns
07:00pm Dinner at Christopher's (Heldrich Hotel)
revised 27 September 2010

Attendees

EMDataBank Organizing Committee

Dr. Helen M. Berman
Department of Chemistry and Chemical Biology
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

Dr. Wah Chiu
Verna and Marrs McLean
Department of Biochem. & Mol. Biology
Baylor College of Medicine
One Baylor Plaza
Houston, TX 77030

Dr. Gerard Kleywegt
EMBL-EBI
Wellcome Trust Genome Campus
Hinxton, Cambridge, CB10 1SD
United Kingdom

Dr. Cathy Lawson
Department of Chemistry and Chemical Biology
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

EM Models Discussion Group EM Map Discussion Group

Dr. Andrej Sali (Chair)
Department of Pharmaceutical Chemistry
University of California at San Francisco
MC 2552
Mission Bay, Byers Hall
1700 4th Street, Suite 503B
San Francisco, CA 94158-2330

Dr. Matthew Baker
Verna and Marrs McLean Department of Biochem. & Mol. Biology
Baylor College of Medicine
One Baylor Plaza
Houston, TX 77030

Dr. Batsal Devkota
Department of Chemistry and Chemical Biology
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

Dr. Peter Rosenthal
Divison of Physical Biochemistry
The National Institute for Medical Research
The Ridgeway
Mill Hill
London, NW7 1AA, UK

Dr. Michael Rossmann
Department of Biological Science
Purdue University
915 W. State Street
Lilly Hall of Life Science
West Lafayette, IN 47907

Dr. Gunnar Schroeder
Forschungszentrum Jülich
Institute of Structural Biology and Biophysics
ISB-3: Structural Biochemistry
52425 Jülich
Germany

Dr. Willy Wriggers
D.E. Shaw Research
39th Floor
120 W. 45th St.
New York, New York 10036

__

Dr. Richard Henderson (Chair)
MRC Laboratory of Molecular Biology
Hills Road
Cambridge CB2 0QH
44 12 2340 2215
rh15-at-mrc-lmb.cam.ac.uk

Dr. Bridget Carragher
The Scripps Research Institute
MC CB129
10550 North Torrey Pines Road
La Jolla, CA 92037

Dr. Kenneth Downing
Life Sciences Division
Lawrence Berkeley Laboratory
One Cyclotron Road
Mailstop: DONNER
Berkeley, CA 94720

Dr. Edward H. Egelman
Department of Biochemistry and Molecular Genetics
University of Virginia
Box 800733
Charlottesville, VA 22908-0733

Dr. Joachim Frank
Department of Biochemistry & Molecular Biophysics
Department of Biological Sciences
Columbia University Medical Center
650 West 168th Street
Black Building, Room 2-221
New York, NY 10032

Dr. Nikolaus Grigorieff
MS 029, Rosenstiel Center
Brandeis University
415 South St.
Waltham MA 02454-9110

Dr. Wen Jiang
Dept. of Biological Sciences
Purdue University
915 W. State Street
Lilly Hall of Life Science
West Lafayette, IN 47907

Dr. Steven J. Ludtke
Verna and Marrs McLean Department of
Biochemistry and Molecular Biology
One Baylor Plaza
MS: BCM125
Houston, TX 77030

Dr. Ohad Medalia
Department of Life Sciences and the NIBN
The Ben-Gurion University
84120 Beer-Sheva, Israel

Dr. Pawel Penczek
Department of Biochemistry and Molecular Biology
University of Texas-Houston Medical School
P.O. Box 20708
Houston, Texas 77225

Dr. Michael Schmid
Verna and Marrs McLean Department of Biochemistry and Molecular Biology
One Baylor Plaza
MS: BCM125
Houston, TX 77030

Dr. Alasdair C. Steven
Laboratory Chief
Laboratory of Structural Biology Research
National Institute of Arthritis and Musculoskeletal and Skin Diseases
Building 50, Room 1517
Bethesda, MD 20892

Other Participants

Dr. Margaret Gabanyi
PSI Structural Biology Knowledgebase
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

Dr. Zukang Feng
RCSB Protein Data Bank
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

Dr. John Westbrook
RCSB Protein Data Bank
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

Dr. Huanwang Yang
RCSB Protein Data Bank
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

Dr. Jasmine Young
RCSB Protein Data Bank
Rutgers, The State University of New Jersey
610 Taylor Road
Piscataway, NJ 08854-8087

EM VALIDATION TASK FORCE CHARGE AND QUESTIONS

Charge to the Committee

The EM Validation Task Force (EM-VTF) is charged with advising members of EMDataBank.org on approaches to validate maps deposited to EMDB and models deposited to PDB that are obtained from cryoEM data.

Specifically, the EM-VTF is to provide recommendations for EM structure validation criteria and tools. Such validation tools can be based either on existing or proposed software; these tools would become freely available for use by individual laboratories. Validation tools will not be used by EMDB or PDB as a basis to “reject” maps or models, but rather to flag potential problems for a depositor or user to be aware of.

The assignment for the 1st meeting is to discuss assigned questions (see below), decide on priorities and develop plans for creating the recommendations. Recommendations will be assembled into "white papers" that can be used to seek support for development and implementation of the validation tools, with aim explicitly to publish a combined "EM Validation" White Paper in a journal.

Questions

The questions below were drafted by the EM-VTF organizing committee and discussion group leaders for consideration by the map and model discussion groups, and are intended to stimulate discussion. All EM-VTF members are welcome to address any of the questions.

Maps

(1) How can map accuracy (both noise level and overall “correctness”) be assessed? What statistical values provide an indication of the quality of a reconstruction? What tests can be recommended that can be used to validate image processing parameters? In modelling in general, there are 3 quality criteria of a “model”, such as a density map or an explicit structural model. There is in general an ensemble of maps (or structural models) that are consistent with raw data, thus there is the precision of the ensemble, the accuracy of a particular map (or model), and the degree of the coverage of the measured system by the map (or a model). Is there any utility in this perspective for describing the quality of a map, not only model(s) based on the map?

(2) How should map resolution be reported? Currently it is self-reported by depositors as a single value with text description, e.g., “determined by FSC at 0.5 cutoff”. Should EMDB require the complete Fourier Shell Correlation information from depositors? Are there other useful indicators/tools?

(3) What density manipulation/filtering procedures should be specified for the deposited map densities? Should any procedures be disallowed? Examples: negative density truncation, profile fitting, low or high pass filtering, B-factor sharpening, cropping, masking, FOM weighting.

(4) Would it be desirable to have a tool to validate map point group/helical symmetry and define orientation and position with respect to a standard frame?

(5) What parameters should be used to indicate reconstruction quality, noise level and signal-to-noise ratio from 3D tomogram and sub-tomogram averages?

Models

(1) What kind of structural models (eg, atomic, coarse-grained, single models, ensembles of models) do we expect based on EM maps?

(2) What general criteria in principle describe the quality of a “model”, including an explicit macromolecular structural model based on an EM map (eg, precision, accuracy, coverage)?

(3) What can we learn from other assessment efforts (eg, X-ray, NMR, homology and ab initio protein structure modelling, SAXS, integrated modelling, EM map) and should we strive for a common language and framework across these fields, allowing us to understand the commonalities and differences, thus benefit from concepts and tools developed by a much larger community?

(4) How should the fit of an atomic model into an EM map be evaluated? Are there suitable local (e.g., per residue, per secondary-structure element, or per domain) measures? Are there suitable global fit measures? (How) would assessment vary with map resolution, or with fitting protocol (e.g., rigid-body vs. annealing)?

(5) How can we evaluate (esp. for tomograms) that a fitted model is the correct one (it could conceivably be the wrong molecule!) or that the solution is unique/optimal? Is there any measure of quality-of-fit or signal-to-noise for (a) selecting a model to fit to a feature and (b) the selected orientation/position?

(6) We know how to validate the stereochemistry/geometry of atomic models, but how should we "value" the results when applied to EM models? How should we handle cases were rigid-body placement/refinement of a high-resolution X-ray structure has been carried out, the validation statistics will say nothing about the quality of the EM model (except for clashes)? And how should we handle cases where refinement/model-building has been done (there may be thousands of outliers even in a good model just because the resolution is so low)? And (how) should we use clashes?

(7) The precise taxonomy/protein sequence of an EM sample (a) may not be known or (b) the only high-resolution structures available for fitting may be from a species homolog. How should PDB handle the following situations: model deposited with wrong sequence(s), model deposited based on homology-modelling?

(8) How are errors in EM maps and errors in models coupled? This interdependence is possibly going to be increasingly explicitly mutual (eg, via iterative map / model calculations). How to take into account the quality of maps when estimating the quality of models based on these maps?

posted 8/24/10 CL

Materials for Review

Related Workshops

Other Validation Task Forces

    Example Validation Reports (X-ray)

    EM Metadata Dictionaries

    Source: vtf.emdataresource.org archived content. Outdated links have been removed, April 2022



    Prior Support from
    National Institutes of Health
    National Institute of General Medical Sciences

    © 2007-2024 EMDataResource