We are proposing to HRI'08 (March 12-15) a workshop on Metrics for Human-Robot Interaction to be held on March 12 2008(see draft proposal below). To convince the HRI program committee of the need
for the workshop, we would like to get a count of the number of
If you feel that you or your student may find this workshop
may wish to attend or actively participate, please let us know at
We would also appreciate any comments you may have about the draft
for the workshop and its suggested format.
Catherina Burghart and Aaron Steinfeld
------------------- Draft proposal below -------------------
Workshop Proposal: Metrics for Human-Robot Interaction
1 Purpose and Scope
The evaluation of interactions between robots and humans in field studies as well as the
comparison of different types of interaction, robot and human behavior, and results require
adequate measures and guidelines. These measures have to take into account objective robot
characteristics on the one hand as well as metrics for social interaction on the other hand.
These metrics could code behaviours, ways of interaction and many more. There are metrics
that can be acquired using objective measuring tools, others depend on the personal
interpretation of the staff conducting and analysing experiments. When human beings are
concerned metrics for social human-robot interaction are of utmost interest in order to
achieve robotic systems that can be intuitively handled by people without causing frustration
This workshop aims at setting up a set of guidelines for the performance and analysis of
experiments in human-robot interaction. This workshop also intends to concoct a handbook
of metrics for HRI which will then be made available to the community. Additionally,
a glossary and unification of terminology is intended.
1.1 Key Issues for Discussion
The following key questions will be raised to motivate the workshop
discussion, with the goal of providing answers (or at least steps
towards answers) within they workshop:
o Which guidelines have to be followed for careful experimentation in HRI?
o Are there objective metrics applicable to HRI?
o Are there social metrics applicable to HRI?
o What is the relevance of subjective criteria for evaluation?
o Can subjectively categorized criteria be used to form objective metrics for social HRI?
o How can benchmarks (standardized tasks) be used to
evaluate human-robot interactions?
1.2 Impact: Usable Guidelines for Evaluation
The goal of the workshop is to propose guidelines for the analysis of
of Human robot experiments and to create a handbook of metrics, that
would be acceptable to the HRI community, and allow
researchers both to evaluate their own work, and to better assess the
progress of others. The format of the workshop (see below) will focus
on developing guidelines for evaluating features in HRI,
conduct comparative studies, and prove objective properties addressing
all issues of HRI. The
format will take into account the experiences of all attendees. The
intention is to produce a citeable source of metrics and evaluation guidelines,
which will have significant impact on designers of robots and investigators of
human-robot interaction. Such guidelines facilitate objective and
reproducible evidence of a robot's capability to successfully interact with people in the intended manner.
To do this, we intend to publish the results in a special issue of an
international journal. We will set up a web site with the presentation
slides and explanatory material. This online resource will serve as a
portal to evaluation methodology, much like similar pages have in the
past been used to promote standards or appropriate use of empirical
methods. We will also consider making an edited version of the
workshop video recordings available.
2 Proposed Format
There are many researchers investigating interaction between
humans and robots, but
surprisingly little published work on metrics and experimental
evaluation methodology involving persons in HRI. As the
aim of the proposed workshop is to produce the evaluation guidelines,
and a handbook on applicable metrics, we choose a setting enabling a good
working atmosphere. The format combines information about different
metrics and evaluation methods given in session talks and invited talks,
panels, and moderated group discussions.
We believe the workshop will run for a whole day, but this depends on
interest and number of contributed talks. We will record all
proceedings. The number of attendees will not be limited, but
registration will be required.
3 Post-workshop Documentation
All the panel results, video recordings and presentation materials
will be collected for the web site. After the workshop, we will
solicit articles from the different groups for the special issue, and
for the workshop web page.
4 Organizers and Relevant Expertise
The workshop will be co-chaired by Catherina
Burghart and Aaron Steinfeld. The organization committee includes other members (listed
below) who have all expressed interest in participating and
The co-chairs are supported by a distinguished list of organizers,
all of whom
significant experience both in organization and in relevant
research. The list
TBD (To Be Determined)
University of Karlsruhe
Institute of Process Control and Robotics
76128 Karlsruhe, Germany
Carnegie Mellon University
Pittsburggh, PA 15213