Performance Evaluation and Benchmarking of Intelligent Systems

1. Front Matter

   Pages i-xix

   PDF

2. Metrics for Multiagent Systems

   • Robert N. Lass, Evan A. Sultanik, William C. Regli

   Pages 1-19

3. Evaluation Criteria for Human-Automation Performance Metrics

   • Birsen Donmez, Patricia E. Pina, M.L. Cummings

   Pages 21-40

4. Performance Evaluation Methods for Assistive Robotic Technology

   • Katherine M. Tsui, David J. Feil-Seifer, Maja J. Matarić, Holly A. Yanco

   Pages 41-66

5. Issues in Applying Bio-Inspiration, Cognitive Critical Mass and Developmental-Inspired Principles to Advanced Intelligent Systems

   • Gary Berg-Cross, Alexei V. Samsonovich

   Pages 67-92

6. Evaluating Situation Awareness of Autonomous Systems

   • Jan D. Gehrke

   Pages 93-111

7. From Simulation to Real Robots with Predictable Results: Methods and Examples

   • S. Balakirsky, S. Carpin, G. Dimitoglou, B. Balaguer

   Pages 113-137

8. Cognitive Systems Platforms using Open Source

   • Patrick Courtney, Olivier Michel, Angelo Cangelosi, Vadim Tikhanoff, Giorgio Metta, Lorenzo Natale et al.

   Pages 139-168

9. Assessing Coordination Demand in Cooperating Robots

   • Michael Lewis, Jijun Wang

   Pages 169-186

10. Measurements to Support Performance Evaluation of Wireless Communications in Tunnels for Urban Search and Rescue Robots

    • Kate A. Remley, George Hough, Galen Koepke, Dennis Camell

    Pages 187-219

11. Quantitative Assessment of Robot-Generated Maps

    • C. Scrapper, R. Madhavan, R. Lakaemper, A. Censi, A. Godil, A. Wagan et al.

    Pages 221-248

12. Mobile Robotic Surveying Performance for Planetary Surface Site Characterization

    • Edward Tunstel, John M. Dolan, Terrence Fong, Debra Schreckenghost

    Pages 249-268

13. Performance Evaluation and Metrics for Perception in Intelligent Manufacturing

    • Roger Eastman, Tsai Hong, Jane Shi, Tobias Hanning, Bala Muralikrishnan, S. Susan Young et al.

    Pages 269-310

14. Quantification of Line Tracking Solutions for Automotive Applications

    • Jane Shi, Rick F. Rourke, Dave Groll, Peter W. Tavora

    Pages 311-338

To design and develop capable, dependable, and affordable intelligent systems, their performance must be measurable. Scienti?c methodologies for standardization and benchmarking are crucial for quantitatively evaluating the performance of eme- ing robotic and intelligent systems’ technologies. There is currently no accepted standard for quantitatively measuring the performance of these systems against user-de?ned requirements; and furthermore, there is no consensus on what obj- tive evaluation procedures need to be followed to understand the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communic- ing results, inter-comparing system performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. Currently, this lack of cohesion in the community hinders progress in many domains, such as m- ufacturing, service, healthcare, and security. By providing the research community with access to standardized tools, reference data sets, and open source libraries of solutions, researchers and consumers will be able to evaluate the cost and be- ?ts associated with intelligent systems and associated technologies. In this vein, the edited book volume addresses performance evaluation and metrics for intel- gent systems, in general, while emphasizing the need and solutions for standardized methods. To the knowledge of the editors, there is not a single book on the market that is solely dedicated to the subject of performance evaluation and benchmarking of intelligent systems.

• Tracking
• automation
• communication
• currentjm
• intelligent systems
• linear optimization
• measurement
• perception
• performance
• performance evaluation
• performance metrics
• quantitative performance evaluation
• robot
• robotics
• simulation

From the reviews: “This book is the result of the 2008 performance Metrics for Intelligent Systems Workshop at the National Institute of Standards and Technology (NIST). It investigates intelligent systems, starting with multiagent systems … . presents an impressive quantity of data, diagrams, proposals, experimental methods, and open topics, and also includes current solutions and relevant literature. … Intelligent systems developers and users will be able to build on this book’s material.” (G. Gini, ACM Computing Reviews, December, 2009)

• Computer Science & Mathematics Div., Oak Ridge National Laboratory, Oak Ridge, U.S.A.

  Raj Madhavan

• Applied Physics Lab., Johns Hopkins University, Laurel, U.S.A.

  Edward Tunstel

• Technology (NIST), National Institute of Standards &, Gaithersburg, U.S.A.

  Elena Messina

Policies and ethics