Augmented Cognition Mitigation Strategies
Next Generation Concepts
Summary Report
_________________________________
October 14, 2005
Kelly S. Hale, Sven Fuchs, Kay Stanney
Design Interactive, Inc.
897 Kensington Gardens Court
Oviedo, Florida 32765
Joseph Juhnke
Tanagram Partners
125 North Halsted, Suite 400
Chicago, IL 60661
Prepared For
CDR Dylan Schmorrow
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Table Of Contents
1
Introduction ................................................................................................................. 3
2
Current Mitigation Strategies...................................................................................... 4
2.1
Context-Sensitive Help ....................................................................................... 4
2.2
Cueing ................................................................................................................. 5
2.3
Decluttering (Reduce amount of info) ................................................................ 7
2.4
Delegation ........................................................................................................... 9
2.5
Modality Augmentation .................................................................................... 10
2.6
Pacing/Scheduling (Reduce rate of info) .......................................................... 11
2.7
Sequencing ........................................................................................................ 13
2.8
Task Sharing ..................................................................................................... 14
2.9
Transposition..................................................................................................... 14
2.10
Summary ........................................................................................................... 15
3
Conceptual Framework for Mitigation Strategy Selection ....................................... 17
4
Innovative Mitigation Strategies ............................................................................... 22
4.1
Innovative Strategies to Alleviate Bottlenecks ................................................. 22
4.2
Innovative Strategies to Enhance Situation Awareness (SA) ........................... 26
5
Mitigation Strategies Summary ................................................................................ 28
6
Operational Example: Real-Time Mitigation Strategies for Tactical Action Officer29
6.1
Operational Scenario ......................................................................................... 29
7
Conclusions and Future Directions ........................................................................... 33
8
References ................................................................................................................. 34
Design Interactive, Inc.
2
Augmented Cognition Mitigation Strategies
Next Generation Concepts
1 Introduction
DARPA’s Improving Warfighter Information Intake Under Stress (i.e., Augmented
Cognition [AugCog]) program aims to extend the information management capacity of
human-system dyads in operational environments through real-time system mitigation
strategies driven by physiological sensors monitoring operator cognitive state. A general
premise of the current AugCog effort is that human information processing (HIP)
capabilities are fundamentally the weak link in the symbiotic relationship between
humans and computers. Within human information processing there are several
‘bottlenecks’ or points of limited processing capacity, including sensory memory,
working memory (WM), attention, and executive function. It has been suggested that
through mitigation strategies, the costs of HIP bottlenecks (e.g., degraded human
performance due to overload, underload, stress) can be overcome. As physiological
sensors become more robust and capable of characterizing cognitive states beyond
bottlenecks (e.g., situation awareness metrics), the focus of real-time mitigation may
expand beyond alleviating overload to optimizing operator state (i.e., predictive
mitigation to avoid overload) to measurably improve performance of the human-system
dyad. The design of real-time mitigation strategies must thus be carefully addressed.
To date, system mitigation has been applied in a brute force manner and thus tends to
cause context switching (Baldonado, Woodruff & Kuchinsky, 2000), as well as loss in
situational awareness (Boiney, 2005); there has generally been an observed cost
associated with getting context back. Thus, there appears to be a need to identify more
effective mitigation strategies. This effort focused on summarizing current mitigation
strategies used in human-systems integration (HSI) as well as techniques utilized in other
media (e.g., Fine Arts, Photography, Film, Theatre) that may be adapted to enhance
human-system performance within dynamic, information-rich, stressful environments.
The goal was to identify theory-driven mitigation strategies from HSI, and leverage the
Arts to achieve more effective and innovative, next-generation concepts for mitigation
strategies, i.e., think outside the box that WIMPs (Windows, Icons, Menus, Pointing
devices) have us bounded by. Through the current effort, a framework was developed to
aid real-time mitigation strategy selection. In addition, the repertoire of mitigation
strategies was expanded in terms of both the breadth of strategies available and the
manner in which each is designed and implemented within varying operational settings.
When the project commenced and the scientific literature review was underway, a critical
gap in the literature associated with how mitigation strategies are conceptualized was
uncovered. Thus in this effort we undertook filling this gap which involved devising a
theoretical foundation from which to conceptualize mitigation strategies; specifically a
conceptual framework based on Norman’s (1988) ”Seven Stages of Action” has been
developed which relates adaptive mitigation activity to a model of the human action cycle
of intent, execution and evaluation. The importance of this conceptual model is that it
structures and systematically organizes the space of mitigation strategies. Thus the
current effort focused more on building a theoretical foundation for mitigation strategies
rather than the proposed empirical studies which will be pursued in the follow-on efforts.
Design Interactive, Inc.
3
Augmented Cognition Mitigation Strategies
Next Generation Concepts
2 Current Mitigation Strategies
The current effort focused on identifying mitigation strategies that may be applied to (1)
reduce overload/bottlenecks as identified via psycho-physiological sensors or (2) enhance
situation awareness (SA). Bottlenecks that have been the focus of AugCog research thus
far include sensory, attention, working memory and executive function bottlenecks.
Future research is planned to develop psycho-physiological metrics for SA, as
optimization of SA is critical within complex, information-rich environments such as
military Command, Control, Communications, Computers, Intelligence, Surveillance and
Recognizance (C4ISR) operations. This section describes mitigation strategies that have
been developed and are currently used in various interactive systems. Each strategy
described may be selectively implemented in real-time (following theoretical mitigation
selection plans as outlined in this report) to either alleviate processing bottlenecks and/or
enhance SA, thereby enhancing performance for an individual operator within a complex,
information rich environment.
2.1 Context-Sensitive Help
By changing the physical layout, useful affordances of both a physical and cognitive
nature can be brought closer to where users need them and at the time they need them –
providing context-sensitive help.
Strengths
The simple fact that physical and cognitive affordances are available at the right moment
can help users notice possibilities they might otherwise overlook (Kirsh, 2000). Contextsensitive help systems take away the task of locating a desired information snippet inside
today’s extensive help systems. This is important, as we are apt to lose the thread of our
composition (Kirsh was commenting on a word editing task) the deeper we have to go
outside of our current environment of activity (Kirsh, 2000).
Weaknesses
Automated help systems, however, can be perceived as helpful and non-intrusive (e.g.,
Microsoft® Word’s auto-correction feature) or highly annoying (“Clippy,” the
Microsoft® Office Assistant). Careful consideration of interruption strategies is therefore
needed. However, offering help may be generally inappropriate in high-load
environments, as it causes distraction from the core task and may disturb the operator’s
pace. If external resources are available and can be accurately identified, it may be better
to delegate the task.
Specific Execution Methods
In today’s computer systems, context-sensitive help is implemented in many different
ways, mostly in close distance to where it is needed. The right-click context menu, which
can be considered a quasi-standard for Windows® environments, adheres to the Gestalt
principle of proximity and Fitts’ Law (describes the relationship between distance and
size of a pointing target, Fitts, 1954), making the interface more efficient. Context-aware
environments (e.g. Microsoft® Visual Basic®) automatically provide help for the
Design Interactive, Inc.
4
Augmented Cognition Mitigation Strategies
Next Generation Concepts
function at the current cursor position, e.g. code reference for a highlighted keyword, or
they provide a list of available options (e.g. attributes in an HTML tag).
Software agents can automatically provide assistance in the background. They monitor
and analyze user activity and trigger events if particular user action is detected.
2.2 Cueing
Cueing is a common way to compensate for weaknesses in information rich
environments, where people receive data at such a rapid rate that it may not be
assimilated (Sheridan and Ferrell, 1974). In such cases, users develop and apply
prioritizing strategies. However, these strategies are not always accurate. Limited
attentional resources also make users vulnerable to the attentional spotlight effect
(Posner, Snyder & Davidson, 1980; Townsend, 1971) which decreases their event
awareness (signal detection) capabilities outside of their attentional focus, and may leave
a high priority event undetected (“miss”). Cues provided by the system notify users of
events by directing/redirection attention.
Strengths
Research has been conducted on the design of cues, e.g. which modality provides the
fastest response or what properties make a cue more distractive. Information can be
interpreted with minimal cognitive effort if presented in the appropriate modality
(Wickens, 2002), and response time to signals differs among modalities, with auditory
and haptic cues resulting in faster response times compared to visual cues (ETSI, 2002).
The more intrusive a cue is, the more awareness it will create for the cued event, and the
more effective it will be in shifting attention. Arroyo, Selker and Stouffs (2002) found
that the least used modalities in computer interfaces (i.e. smell and vibration) have bigger
disruptive effects, probably because of their novelty.
Weaknesses
Arroyo, et al. (2002) findings suggest that people’s backgrounds and worldview
determine the effectiveness of various modalities for interruption. Also, if a cue is given
at an inopportune moment, slower task performance (e.g., Czerwinski, Curtell & Horvitz,
2000; McFarlane, 1999), more errors (Kreifeldt and McCarthy, 1981), and worse
decisions (Speier, Valacich & Vessey, 1999) may be observed. All of these factors may
be critical to safety and mission success in C4ISR environments.
To mitigate the disruptive effects of interruption, researchers are investigating systems
that reason about when (Horvitz and Apaciple, 2003; Horvitz, Jacobs & Hovel, 1999;
Hudson, Fogarty, Atkenson, et al., 2003) and how to interrupt users. McFarlane
(2002) evaluated four strategies for coordinating interruption; immediate (with no respect
to the user’s current task), negotiated (people have choices about whether to allow
interruptions and how and when to handle them), mediated (an attempt to predict
people’s interruptibility), and scheduled (provide a degree of reliable expectation about
when interruptions will occur). These approaches depend on some model of sensory
attention when deciding how and when to interrupt. According to Norman’s (1988) taskflow model, each task consists of a planning, evaluation and execution phase.
Design Interactive, Inc.
5
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Interrupting these phases may be costly (Zijlstra, Roe, Leonova & Krediet, 1999; Monk,
Boehm-Davis & Trafton, 2002). Since every subtask is a task itself, high-level tasks (i.e.,
containing subtasks) have a recursive structure that imposes more cognitive load as more
levels of subtasks are present. Therefore, task completion is desirable in order to release
these cognitive resources. If this cannot be accomplished, Trafton, Altmann, Brock, and
Mintz (2003) suggest a rehearsal strategy to help resume a task after an interruption.
Zacks and colleagues conducted studies that provide insights into the ways in which tasks
are decomposed hierarchically in the mind (Zacks, Braver, Sheridan, et.al., 2001; Zacks
& Tversky, 2001; Zacks, Tversky and Iyer, 2001). These decompositions are linked to
distinct patterns of brain activity. Based on this research, Adamczyk and Bailey (2004)
suggest a task model that reveals the best timing for an interruption. They also suggest
that, besides task status, an interruption manager should dynamically factor in
notification relevance, i.e. apply strategies to prioritize subtasks.
Specific Execution Methods
Cueing techniques are in use for all modalities. The appropriate modality for a cueing
event may or may not be chosen to interfere with the sensory channels of the current task,
based on priority assessment and task status. The level of intrusion should relate to the
priority of the cued information, however, as cues may interrupt users in their current task
causing loss of SA and disorientation when resuming from processing the cue. For
example, in a computer application, an auditory signal does not intrude visual workspace
but provides a cue that something happened in the background (see Bailey et al., 2000 for
more on balancing information awareness and intrusion).
Visual cues may rely on the visual pop-out effect which is implemented by a difference
in color, shape, or intensity. Intrusiveness of visual cues is also highly dependent on
position: For high levels of intrusion (i.e. cueing of high-priority events), cues should be
placed within foveal vision, low priority events may be cued by changes in the peripheral
or ambient environment. Auditory cues vary by their nature. They come as generic
sounds of different complexity (with more complex sounds being more disruptive),
earcons (i.e., real-world sounds that carry a metaphorical meaning), or speech, and can
vary in volume or pitch. In more advanced systems, audio output may also be spatialized
in order to distinguish information channels or carry spatial information. Haptic cueing is
commonly implemented through vibration devices, such as those implemented in
pointing devices (e.g., haptic mouse) and cell phones (e.g., meeting setting that gives
vibration cue for incoming call) serving as the most common examples. Haptic cues may
vary in intensity and pattern, and can be used to provide spatial data (if more than one
vibration source is present).
Sometimes, different modalities are combined to create a system of cues. One common
application is alarm systems that increase the alarm level step by step, starting with a
visual cue (e.g. a flashing light). If this cue is not responded to, the system increases its
intrusiveness by adding auditory signals (bells and whistles). Advanced car alarm
systems even address suspected intruders by speech before setting off a public alarm (e.g.
entreprix SWAT II, http://entreprix.com/swat2.html).
Design Interactive, Inc.
6
Augmented Cognition Mitigation Strategies
Next Generation Concepts
2.3 Decluttering (Reduce amount of info)
Clutter is created when screen real estate or channel capacity is occupied by numerous
unrelated entities, or if these entities are not well-arranged. Decluttering refers to
reducing the amount of information to be displayed. Systems may follow design rules
like the Gestalt Principles (i.e., the sum of the whole is greater than its parts; Koffka,
1935) that, correctly applied, can declutter displays and these can facilitate the user’s
interpretation of interface state by increasing interface consistency. Some examples are:
-
The Gestalt Law of Proximity: Elements that are closer together will be perceived as
a coherent object
The Gestalt Law of Similarity: Elements that look similar will be perceived as part of
the same form.
The Gestalt Law of Good Continuation: Humans tend to continue contours whenever
the elements of the pattern establish an implied direction.
The Gestalt Law of Closure: Humans tend to enclose a space by completing a contour
and ignoring gaps in the figure.
A second approach towards decluttering is level-of-detail manipulation, where layers of
content may be hidden to simplify the current display to highlight critical information.
Strategies include adaptive menus (where options that are rarely used by the operator are
hidden), zoom (where users are able to zoom in on specific context area), fisheye views
(where a very wide-angle lens that shows an area of interest quite large and with detail
with the remainder of the graph/image successively smaller and in less detail; Sakar &
Brown, 1992) and preview techniques (where an incomplete, yet representative subset of
an environment’s content is presented).
Strengths
Interface consistency is helpful to avoid clutter, as it increases learnability and reduces
errors (Nielsen, 1989). Consistency should support human perception and cognitive
processes such as visual scanning, learning, and remembering (Mahjan & Shneiderman,
1997). Once an interface is learned, cognitive load – necessary to locate and identify
elements – is decreased. Consistent color schemes and positions also facilitate orientation
and help to reduce clutter. However, as operators deal with increasing amounts of data,
unified appearance caused by consistency efforts, may inhibit the richness of information
(Gentner & Nielsen, 1996) and therefore reduce the number of orientation cues.
Level-of-detail manipulations can be implemented to take away non-relevant information
or advanced features that may not be utilized on a regular basis. These strategies can be
used to simplify displays for novice operators, where advanced features would distract
from learning basic system features. Preview techniques reduce efforts of assessment and
selection of data. Schweiger (2001) used link comments in hypertext structures (mousesensitive, popped up next to the mouse cursor) that contained a summary of one or two
sentences about the content of the linked page. Similar previews enhanced knowledge
acquisition and supported intentional and incidental learning (Cress and Knabel, 2003) by
improving the link selection and reducing serendipity effects. Users with link comments
Design Interactive, Inc.
7
Augmented Cognition Mitigation Strategies
Next Generation Concepts
opened significant fewer pages than those without the added information. Link comments
also reduced the frequency of backward navigation.
Weaknesses
Zooming incorporates a trade-off between detail level and awareness of the global
picture. One attempt to solve the problem is a second-view approach, where the global
view is presented in a second display, highlighting the zoom area (e.g. Adobe
Photoshop). But this solution is still non-optimal, as it requires additional screen realestate and forces the user to switch between views and mentally integrate the information
of both (Sakar & Brown, 1992). Non-linear magnification implemented with ‘fisheye’
lenses is used to achieve a balance between expansion and compression of the data,
depending on the user’s focus point (Gutwin & Skopik, 2003). This non-linearity,
however, may cause distortion in proportions of objects it is applied to which may lead to
decreased speed and an increased need for accuracy in pointing tasks (Gutwin, 2002).
When implementing preview techniques, careful decisions are needed about how to
reduce the data, as taking away context also eliminates cues that may be helpful to the
user.
Specific Execution Methods
Today’s interfaces use various approaches of visual level-of-detail manipulation.
Adaptive menus are used in Microsoft® Office, where options that are rarely needed by
the user are hidden. In graphic software, layers of content can be shown and hidden as
appropriate to not obscure the elements that are currently worked on. Another classic
approach to level-of-detail manipulation is zooming – where “zooming in” provides local
detail while “zooming out” allows for a high-level overview. This technique is common
in electronic maps, such as MapQuest™ (www.mapquest.com).
A fisheye view (Furnas, 1986) has been used to balance scope and detail of information
presentation. Gutwin & Skopik (2003) compared a set of magnification techniques and
stated that, although the fisheye lens performed best overall, some tasks may be better off
with a ‘flat lens’ (maintains relative sizes and positions in a certain area; e.g. on maps,
where distance and directions have a meaning) or a panning view (performed best on
pointing tasks in high magnification/high accuracy environments; e.g., point-and-click
tasks).
Preview techniques such as those implemented in Mini Player mode in Apple®’s
iTunes® aim at reducing detail by limiting the amounts of data to be processed by
presenting an incomplete, yet representative subset of an environment’s content. Only
upon request a detailed view or extended functionality is displayed.
Currently, level-of-detail manipulations are not implemented for auditory and haptic
modalities. Although one could imagine systems that increase or decrease context based
on situation (tone/earcon vs. spoken word vs. spoken sentence), to date most auditory
content is statically implemented. The same is true for haptics, except for advanced
gaming or force feedback devices.
Design Interactive, Inc.
8
Augmented Cognition Mitigation Strategies
Next Generation Concepts
2.4 Delegation
Delegation involves offloading some aspect of current tasking to another operator. Mixed
Initiative refers to tasks that are shared between an operator and an intelligent system,
where the system takes complete control over certain aspects of a task. The term initiative
implies an intelligent system that can actively adapt its status to environmental factors.
This technique has been used extensively to decrease operator overload by dynamically
appointing tasks to other, less taxed, users or systems.
Strengths
Delegation utilizes the capabilities of operators that are better suited for this particular
task or parts thereof and can be used when misinterpretation inhibits task performance.
The real strength of the delegation is its dynamic workflow optimization schema. When
an operator is overloaded, delegation to system or another operator may reduce
bottlenecks by reducing the amount of tasking a single operator must complete.
Weaknesses
Delegation should be applied with caution, particularly if the global goal is to increase
situation awareness. By delegating tasking to outside resources, the current operator
reduces his/her awareness of offloaded tasks. This may result in decreased awareness of
current state. In addition, automated delegation is difficult as the delegating agent must
not only accurately understand the workload of the primary (the person the task was
originally sent to) and potential delegates, it must also know which of its potential
delegates is best equipped to address this particular situation with the skill sets and
situational awareness required. In addition, delegation eliminates the operator’s need for
learning. If a task is taken away from an operator as soon as he/she does not perform
optimally, he/she will never have the opportunity to improve individual performance on
the given task.
Specific Execution Methods
Manual delegation is a common practice in the human work place and would seem an
intuitive addition for an information flow system. One example is the auto-pilot system in
aviation that is turned on by the operator when no events are expected which would
require human involvement. However, these systems are mostly passive: The operator
initiates the system to take control of certain aspects of the task but is still in charge, as
the system only maintains the current state – sometimes capable of limited adaptive
actions supported by sensory technology. The user is still required to monitor and make
adjustments as needed. Cruise control is another example for delegating parts of a task to
a system.
Mixed Initiative is a specific form of delegation, where intelligent systems are assigned
responsibility for a task or task component, are already common in current computing
environments, car navigation serving as an example: The system detects the current
position of the vehicle and adjusts its output accordingly, offloading the task of wayfinding from the driver. Also, some car manufacturers have rain sensors which control
the wipers and their speed as needed, thus taking away this responsibility from the driver
completely.
Design Interactive, Inc.
9
Augmented Cognition Mitigation Strategies
Next Generation Concepts
2.5 Modality Augmentation
Modality Augmentation can occur in two forms: modality redundancy and modality
substitution/switching. Modality redundancy may provide the same information in
multiple modalities (e.g., flashing light and auditory beep as a warning), and/or may
provide complimentary information (i.e., additional information; e.g., flashing light with
auditory speech message stating the problem status) in a second modality. Modality
substitution/switching replaces one sensory modality with another (e.g., change flashing
light to auditory beep as warning signal).
Strengths
According to Wicken’s (2002) Multiple Resource Theory (MRT), separate sensory and
cognitive resources are available to process information from different modalities.
Therefore, simultaneous processing of competing tasks can be supported by strategically
allocating data streams to various multimodal sensory systems. If resources are depleted
for one modality, there may still be cognitive capacity left if the information is presented
through other sensory channels.
Weaknesses
Caution is advised when it comes to modality substitution/switching, as researchers
increasingly report evidence that knowledge is grounded in modality-specific systems
(Barsalou, Simmons, Barbey & Wilson, 2003), and there is no direct empirical evidence
for amodal symbols. Thus, the presentation modality is proposed to influence encoding
which may cause association problems when attempting to display the same information
in different modalities (as modality switching does) if previous knowledge or
interpretation is required for the task. For example, if operators are presented with a
visual indication of status, and later presented with an auditory indication of status for the
same system, interpreting this second cue as status information may require more
processing load due to switched modality. In addition, Pecher, Zeelenberg and Barsalou
(2003) report a study supporting the hypothesis that perceptual simulation underlies
conceptual processing and that switching from one modality to another during perceptual
processing incurs a processing cost, implying effects on operator performance. Also,
Arnell and Larson (2002) showed that, under particular circumstances, multimodal
stimuli presentation is subject to the attentional blink effect, where the second of two
successive stimuli is likely to be missed when falling into a timeframe of about 500ms
after the first.
Specific Execution Methods: Redundancy
Most existing modality augmentation approaches target the visual channel (e.g. HMDs,
HUDs, Starner, Mann, Rhodes & Levine, 1997; Billinghurst, Kato & Poupyrev, 2001).
For example, image and pattern recognition technologies facilitate surveillance tasks by
identifying and highlighting areas of interest in a video stream, thus enhancing the visual
channel with additional, system-generated content. Such approaches are currently
implemented in jet fighter cockpits (target tracking), navigation and vision-aid systems
for car drivers, and security camera systems. Furthermore, crossmodal systems have been
developed (e.g., Lyons, Gandy & Starner, 2000) that enrich visual real-world
Design Interactive, Inc.
10
Augmented Cognition Mitigation Strategies
Next Generation Concepts
environment with artificial auditory information. Audio tour guides in museums may be
seen as predecessors of these crossmodal Augmented Reality applications.
Specific Execution Methods: Switching/Substitution
Bell, Boye, Gustafson and Wiren (2000) proposed a system that had the ability to predict
and prevent the occurrence of longer error sequences during interaction. E.g., a low
confidence score from a speech recognizer or an error indication from another part of the
system could be used as a signal to encourage a user to switch from dictating to writing.
Also, system activity is currently used as indicator for switching: Microsoft® Outlook®
2003 provides a visual cue for incoming e-mail only if there is no keyboard or mouse
activity, assuming that the user is busy with a visual task if engaged in these actions.
Only an auditory cue is triggered if the system perceives the user is involved in a visual
task (via keyboard/mouse activity).
2.6 Pacing/Scheduling (Reduce rate of info)
Pacing or scheduling strategies must be applied to conform to the speed of incoming
information streams. This enhances the concept of time beyond its common descriptive
character: A functional role is added in which time is relevant as a work requirement or
constraint, as information in a control decision, as the outcome of such a decision, or as a
property of a task, interface or agent (Hildebrandt, Dix & Meyer, 2004). Scheduling
strategies or timesharing skills are one important determinant of performance in multitask situations (Wickens, 1992).
One pacing strategy is the use of checkpoints, whereby the user monitors elapsed time at
predetermined points and compares it to an existing plan. Based on this comparison, the
user may opt to speed up, slow down, or maintain the current pace. Another approach in
sequential tasking may be to adjust speed of information output to a user’s individual
comfort level or abilities. In a study of visual pacing cues, Mamykina, Mynatt and Terry
(2001) found that providing a high-level task overview that allows for planning of
resource allocation may be equally important as continuous pacing suggestions. They
conclude that pacing systems should be intelligent enough to recover from a loss of pace
and still provide the user with valuable information, perhaps suggesting recovery
strategies. Sensing technology was also suggested to help detect situations potentially
dangerous for maintaining a pace.
Goals for paced systems are to minimize cognitive demands for assessing a current pace,
provide information on task progress, display ambient cues that can be quickly
understood without incurring significant interruption from the current task or demanding
more effort than the tasks they are designed to augment, and place knowledge in the
world to flexibly support different strategies for managing the pace of a timed task
(Mamykina, et al., 2001). According to Hildebrandt, et al. (2004), designers should
consider time design strategies when time-related issues occur in a system, such as
temporal validity of information, interruption scheduling, temporal reference systems
synchronization, multi-tasking, the regularity, periodicity and interleavability of tasks.
Particularly highlighted are computer systems that are used in safety-critical systems with
hard real-time requirements, and environments where the sequential structure of tasks
Design Interactive, Inc.
11
Augmented Cognition Mitigation Strategies
Next Generation Concepts
becomes increasingly complex, e.g. multi-tasking and interleaving. Therefore, timedesign is a highly relevant issue in C4ISR environments.
Strengths
A properly executed pacing mitigation would, based on cognitive load, slow incoming
information during peak times and speed up information during slower periods. It should
have the end goal of getting the user back to a real time status as quickly as possible. This
requires the system to understand the user’s cognitive state, task completion rate and
incoming task rate.
Weaknesses
Using checkpoints requires an undisturbed a priori plan for minimal cognitive demands
of pacing (Mamykina, et al., 2001). Additionally, when people are faced with the need to
pace themselves while simultaneously performing a cognitively demanding task, pacing
is conceptualized as a secondary task that nonetheless requires cognitive resources
(Casini & Macar, 1999). With increasing cognitive demands of the primary task, selfpacing abilities decrease as users’ time estimations become inaccurate. These phenomena
arise from a competition between temporal and non-temporal processors for limited
attention resources (Predebon, 1999; Casini & Macar, 1999; Zakay & Block, 1997).
Therefore, people often place cues in their physical environment to support internal
mental calculations (Kirsh, 1995), e.g. alarm clocks on the lectern when giving a
presentation.
With many pacing systems, particularly for closely coupled systems (i.e. when user and
system are highly integrated), users are paced by the machine (Meyer & Hildebrandt,
2002). For example, if System Response Times (SRT) are very short, users will try to
keep up with the computer’s rapid work speed (Shneiderman, 1984). While it has been
previously argued that an inverse relationship exists between system delay and user
productivity (Teal & Rudnicky, 1992), a more recent memorization study with a
browsing environment (Meyer & Hildebrandt, 2002) indicates that performance was best
for intermediate SRT. It can be argued that short SRTs do not provide sufficient time for
proper encoding of information, while long SRTs increase annoyance and external
distraction.
Unforeseeable events may not allow humans to keep a predetermined pace in an eventdriven scenario. Without any level of awareness, the system keeps on cueing, therefore
creating more cognitive load. In such cases, continued notification of a permanently lost
pace was found to increase the anxiety level of the user which negatively correlated with
performance (Mamykina, et al., 2001).
Specific Execution Methods
Current operating systems are event-driven, i.e. they wait for an event and react to it.
However, systems do not have active awareness of task status or interpretation
capabilities. Paced by predetermined settings (statically implemented or based on user
preferences), they solely rely on the user to pace task execution. Sometimes, cues
(reminders) are offered if time constraints have to be met.
Design Interactive, Inc.
12
Augmented Cognition Mitigation Strategies
Next Generation Concepts
2.7 Sequencing
Complex systems often require execution of many tasks at the same time (multitasking).
If all tasks present their data at the same time, sensory stores and working memory of the
operator may be quickly overloaded (Wickens, 2002). One way to alleviate overload is to
present tasks in sequences. Sequencing refers to creating a linear order in which tasks are
presented to the user in multi-tasking environments. To avoid cognitive bottlenecks,
information is presented one chunk at a time rather than simultaneously. However, in
information-rich environments, it is common that multiple events requiring operator
response occur at the same time. In this case, sequencing involves decomposition of tasks
into smaller portions, and subsequent presentation of subtasks. After one subtask is
completed, a subtask of another event may be presented.
Strengths
Sequencing may be implemented to ensure critical information is responded to in a
timely manner by re-ordering incoming information based on priority (i.e., high
priority/critical information is presented before lower priority information). When
sequencing is implemented, depiction of task timeline and current status has been found
to facilitate operator judgment and performance in selecting appropriate response
strategies (St John & Osga, 1999).
Weaknesses
Switching tasks requires cognitive resources: Upon exiting a task we must store its state
and upon entering another task context, we must recover its task state, i.e. the global task
goal, as well as the current point of interruption, have to be kept in mind. If recovering is
not successful, users may have to start over if returning cues are not provided (Donmez,
Boyle & Lee, 2003). Hence, sequencing may help prevent attentional resources and
sensory channels from overload, but its interruption and task-switching patterns can
increase working memory load.
When attempting to implement automatic task switching, the trade-off between task
priority and cognitive load imposed by task interruption needs to be considered. It is
advised to finish tasks if possible to release the associated cognitive capacities, and to
interrupt at opportune moments (i.e. between planning, execution and evaluation stages).
Place-keeping strategies assist the task-switching process by providing cues of where the
interruption occurred – the cursor is the classic example for place-keeping: It
permanently indicates the current position in a text. To some degree, the commonly used
principles of location consistency and mapping may guide users in remembering and
learning where interface elements are located. A clear or even predictable order of events
(such as step-by-step organization of a task) may, as well, facilitate reorientation. Kirsh
(2000) suggests that we may reduce the number of interruptions we encounter, or at least
their disruptiveness, if we can accomplish our tasks in fewer “environments,” i.e. with
less attention-shifting. This implies that task-switching (and therefore the need for placekeeping) should be kept at a minimum, which is supported by Miyata and Norman’s,
(1986) notion about cognitive load in hierarchical tasks. These principles may also assist
in creating an easier task model, i.e., a clear definition of a task and its subelements
Design Interactive, Inc.
13
Augmented Cognition Mitigation Strategies
Next Generation Concepts
which allows for a more economical use of cognitive resources and may facilitate
opportune moments for cueing/interruptions (Adamczyk & Bailey, 2004).
Specific Execution Methods
One main implication of today’s WIMPs interface paradigm is that the user can perform
every task at any point of time by switching windows and applications, therefore creating
one’s own sequence of actions, based on individual priority assessments and preferences.
2.8 Task Sharing
Task sharing distributes cognitive effort across a team of cooperating users distribute
cognitive effort, thereby reducing individual stress (Kirsh, 2000).
Strengths
Operators often turn to task sharing when information displays present conflicting or
confusing information. In such cases, colleagues are brought into the task to help resolve
ambiguities and/or clarify the situation for the current operator who is unclear of current
status. Thus, operators are able to resolve conflict by utilizing external resources from
team members. Unlike delegation, with task sharing the operator stays in the loop – only
parts of a task are delegated to another user or the system.
Weaknesses
When attempting to implement an automatic task-sharing tool, the restrictions identified
for delegation systems apply as well: The tool would need to be informed about potential
candidates to share the task with and their current cognitive state.
Specific Execution Methods
Task sharing can occur through pulling in external resources for assistance, e.g. asking
colleagues for help or advice without giving the task completely out of hands (i.e.
delegation). Cruise control is an example for task sharing with a system: The driver
initiates the system to take control of certain aspects of the driving task but is still in
charge, as the system only maintains the current state. The user needs to monitor and
make adjustments as needed.
2.9 Transposition
The human brain perceives and processes different types of information differently.
Verbal, spatial, geometric, and musical are some examples of the different cognitive
categories (Berz, 1995; Deutsch, 1970; Sulzen, 2001; Wickens, 1984). Studies have
shown that while individuals have different limits for each of these cognitive abilities
(Sulzen, 2001), all can perceive at least some multiple types of information
simultaneously (Wickens, 1984). This provides designers the opportunity to increase the
amount of information that can be processed by an individual by monitoring activity for
each capability and routing information via untaxed capabilities. Transposition, the
cognitive equivalent to Modality Switching, refers to switching the cognitive processing
demands (e.g., from verbal to spatial) of a task while maintaining the same input
modality. For example, an in-car navigation system may provide a visual map display
outlining a given route while a driver is stationary. As the car starts to move (i.e., driver
Design Interactive, Inc.
14
Augmented Cognition Mitigation Strategies
Next Generation Concepts
is involved in highly spatial task of driving), the navigation system may switch to provide
route knowledge via verbal information (i.e., text). Often, transposition and modality
augmentation compliment one another, and may occur in concert to achieve desired
results. In the car navigation system example, the best alternative while driving may be to
present route directions via verbal speech (as opposed to text; this will offload both the
visual sensory and spatial cognitive channel that are tapped during driving).
Strengths
Transposition allows information to be presented in various formats that may enhance
information throughput if the operator is overloaded in one processing area (e.g., spatial)
but not the other (e.g., verbal). While the transposed information may not be presented in
the optimal modality, the critical information is provided to the operator in a secondary
form.
Weaknesses
Designers must carefully consider how transposition affects cognitive load. Operators
may experience a higher cognitive load when processing information presented in a nonoptimal format. In addition, context may be altered during transposition that may impact
understanding.
2.10 Summary
Table 1 provides a summary of current mitigation strategies organized alphabetically. For
each identified strategy, specific execution methods (i.e., examples) are provided that
outline how each is currently implemented to achieve two main goals: (1) alleviate
overload and (2) enhance situation awareness. While some mitigation strategy
implementations are consistent across these two global goals (e.g., cueing), others have
specific applications for one goal only (e.g., context-sensitive help).
Table 1: Summary of Current Mitigation Strategies
Mitigation Strategy
Context-sensitive Help
-
Cueing
-
Decluttering
Design Interactive, Inc.
-
Execution Methods to
Alleviate Overload
N/A
Execution Methods to
Enhance SA
- Provide additional context on
request (and in close proximity to
current user focus).
- Intelligent agents that monitor user
activity and offer context when
appropriate
- Interpret system state and present
available options
Select intrusion level proportional to priority of the cued information.
Auditory cues vary by sound complexity, volume and pitch, and can be
categorized into sounds, earcons and speech.
Haptic cues are currently implemented through vibration.
Visual cues rely on visual popout, and vary in color, shape, intensity, and
position.
Modalities may be combined, e.g. crossmodal cueing: Auditory cue used to
indicate incoming visual stimuli (e.g., tone to indicate incoming email).
Use consistency and Gestalt principles to facilitate interpretation.
Manipulate level of detail by zooming or with preview techniques.
15
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Table 1 (Con’t)
Mitigation Strategy
Delegation
-
Modality Augmentation
-
-
Pacing/Scheduling
-
Sequencing
Task Sharing
-
Transposition
-
Design Interactive, Inc.
Execution Methods to
Execution Methods to
Alleviate Overload
Enhance SA
Currently no automatic
- N/A
implementations of delegation.
Manual delegation based on
subjective assessment of load
and delegation targets.
o Delegate to other human
o Delegate to intelligent
agent (mixed initiative)
o Get external task support
(task sharing)
Redundant Information: Supporting the original perception with additional,
identical information in either the same (e.g. HUD) or another channel (e.g.,
auditory pulsing tone added to visual blinking light warning)
Complementary Information: Supporting original perception with additional
information in either same (e.g., ??) or another channel (e.g. cueing
incoming email with a letter symbol and a tone)
Modality Switching/Substitution: Changing the sensory channel (i.e.,
presentation mode) to alleviate sensory bottlenecks.
Pacing is not used in current systems; applications react to system or user
events, or follow manually created a priori schedules.
Automatic pacing requires dynamic adjustment to accommodate loss of
pace due to unforeseeable events
Not used in current systems; contradictory to today’s interface paradigm.
User assesses system and task state to create an appropriate sequence.
N/A
- Pulling in external resource to assist
or perform aspects of the task.
Changing the cognitive processing content from verbal to spatial or vice
versa to alleviate cognitive bottlenecks (e.g., change visual spatial map
[showing route] to visual verbal text [written directions])
16
Augmented Cognition Mitigation Strategies
Next Generation Concepts
3 Conceptual Framework for Mitigation Strategy Selection
Mitigation strategies to date have been applied in a brute force manner, and thus tend to
cause context switching (Baldonado, et. al.,.2000), as well as loss in situational awareness
(Boiney, 2005); there has generally been an observed cost associated with getting context
back. A framework for real-time system mitigation is needed to provide guidance as to
when and how current mitigation strategies should be implemented into a system in realtime to enhance operator performance.
Recent advances in cognitive science and neuroscience have led to development of
various physiological and neurological approaches to measuring cognitive load in realtime (e.g., eye movements, pupil diameter, galvanic skin response, EEG, functional
magnetic resonance imaging [fMRI]) that may be used to trigger mitigation strategies.
While there are various methods available, “EEG is the only physiological signal that has
been shown to reflect subtle shifts in alertness, attention and workload that can be
identified and quantified on a second-by-second basis” (Berka, Levendowski, Ramsey, et
al., 2005, p.90). Thus, EEG metrics have been developed that can accurately identify
when an operator experiences a processing bottleneck (i.e., overload), whether that
bottleneck be within sensory, attention, working memory or executive function
processing. In the near future, EEG sensors that evaluate operator SA in real-time may be
developed, which may also be used to trigger mitigation strategies. Thus, while global
mitigation goals may differ, a conceptual framework that outlines a process flow is
required to direct the order in which mitigation plans and strategies should be
implemented.
Building from this knowledge, the conceptual framework for mitigation strategy
selection developed under the current effort identifies how real-time psychophysiological measures can be used to drive cognitive mitigation through interface
adaptation. Once psycho-physiological measures indicate less-than-optimal performance
(e.g., processing bottleneck, poor SA), a diagnosis agent can determine the character of
the problem and human-system interaction can be modified to mitigate this specific
performance gap. The outcome of this action should then be reflected in subsequent
measurements of psycho-physiological indicators, thus creating a continuous, iterative
process to improve human-system interaction. In order to further characterize this closedloop system, it can be mapped to models developed for the human action cycle, such as
Norman’s (1988) “Seven Stages of Action” that describe how people accomplish goals.
Design Interactive, Inc.
17
Augmented Cognition Mitigation Strategies
Next Generation Concepts
1. Perceive
psychophysiological
measures
7. Apply the
interface
adaptation
2. Interpret
cognitive state
If no
Problem
detected
6. Select
interface
adaptation
3. Evaluation/
Problem
detection
4. Define
mitigation
objective to
counteract
problem
5. Select a
mitigation
strategy
Figure 1: Norman’s “Seven Stages of Action” modified for cognitive mitigation.
Mitigation strategies are triggered in stages 1-3 of the model by perceiving the
physiological or behavioral activity of the user through automated measures which are
interpreted and evaluated by a mitigation managing application (if performance is on
target, no mitigation is triggered and stages 1-3 are recursive). Mitigation covers steps
four to six (see Figure 2) in the “Seven Stages” model: At step 4, a mitigation goal is
formulated in correspondence to any problematic cognitive state detected in step 3. In
many cases this mitigation objective will aim to reverse the problem (e.g. “avoid visual
overload” if visual channel capacity is found to be exceeded). Step 5 comprises a
collection of identified mitigation plans and strategies (Norman’s “intention to act”),
each of which addresses one or more objectives, which then lead into a corresponding set
of concrete adaptive actions (i.e. modifications of the interface, task reallocation,
triggering of instructional strategy). As the adaptation is applied, updated cognitive state
metrics become available. The system can thus evaluate the success of the mitigation and
initiate further mitigating actions if problems persist.
In selecting a mitigation strategy, various constraints would be considered. These include
system state constraints (e.g., what information is currently being presented across the
system and how), environmental constraints (e.g., environmental noise), user state
constraints (e.g., current processing load), and global user constraints (e.g., individual
abilities/preferences).
1
Given a global goal (e.g., reducing operator processing bottlenecks [sensory,
attention, working memory, executive function], enhancing SA), the proposed
mitigation strategy framework includes five general plans of action (Figure 2). These
plans should be considered in the order presented as potential mitigation strategies. If
a solution is found that resolves the identified bottleneck while meeting all
Design Interactive, Inc.
18
Augmented Cognition Mitigation Strategies
Next Generation Concepts
constraints, that mitigation strategy
will be selected and the cycle will
proceed to step 6 (Figure 1).
2
The mitigation strategy selection
framework first examines whether
all operator sensory and processing
(i.e., working memory) resources
are optimized. Using data from
psycho-physiological sensors (from
step 1 in Figure 1), the system can
determine if all resources (i.e.,
sensory
modalities,
cognitive
processing areas) are being used
efficiently. If additional resource
capacity is available, the mitigation
selection system should augment or
transpose some information that
contributes to the identified
bottleneck (most likely based on
priority; i.e., highest priority
information should be presented in
optimal modality and/or through
redundant cues; lower priority
information may be presented in
non-optimal modality). The next
option in the mitigation selection
framework is to use cueing and
intelligent sequencing strategies to
direct
attention
to
relevant
information.
5. MITIGATION STRATEGY
Optimize Individual Processing Capacity
Modality Augmentation
Transposition
(If not successful)
Direct Attention to Critical Information
Cueing
Intelligent Sequencing
(If not successful)
Optimize Communication Requirements
Declutter
Context-sensitive Help
(If not successful)
Optimize Communication Rate
Pacing
(If not successful)
3
4
If, given the current situation,
cueing strategies are deemed
ineffective
(e.g.,
too
much
information is being displayed), the
mitigation selection framework
suggests optimizing communication
requirements through declutter
techniques and/or adding contextsensitive help.
The next mitigation plan optimizes
the rate of information by
implementing
pacing/scheduling
strategies (e.g., slow down or
Design Interactive, Inc.
Increase External Resources
Delegation
Task Sharing
Figure 2: Mitigation Selection Framework
(Step 5 of Figure 1)
19
Augmented Cognition Mitigation Strategies
Next Generation Concepts
temporarily stop incoming non-critical chat messages).
5
Finally, if the above plans are deemed ineffective in achieving the global goal, the
mitigation selection framework directs delegation of tasks to either an intelligent
agent or additional operator or task sharing. Delegation and task sharing are seen as a
last resort in meeting global goal of optimizing individual performance within the
human-system dyad (i.e., all mitigation strategies to optimize individual performance
with the current system have been considered prior to delegation).
4. MITIGATION OBJECTIVE
5. MITIGATION STRATEGY
Optimize Individual Processing Capacity
Modality Augmentation
Alleviate Bottlenecks
Transposition
(If not successful)
Direct Attention to Critical Information
Cueing
Sensory
Bottleneck
Intelligent Sequencing
Attention
Bottleneck
Working
Memory
Bottleneck
(If not successful)
Optimize Communication Requirements
Declutter
Executive
Function
Bottleneck
Context-sensitive Help
(If not successful)
Optimize Communication Rate
Pacing
(If not successful)
Increase External Resources
Delegation
Task Sharing
Figure 3A: Specific Mitigation Selection Model (Step 4-5 of Figure 1) for Alleviating
Bottlenecks
Design Interactive, Inc.
20
Augmented Cognition Mitigation Strategies
Next Generation Concepts
4. MITIGATION OBJECTIVE
5. MITIGATION STRATEGY
Optimize Individual Processing Capacity
Modality Augmentation
Alleviate Bottlenecks
Transposition
(If not successful)
Direct Attention to Critical Information
Cueing
Intelligent Sequencing
Enhance
perception of
information
Enhance
comprehension
of information
Enhance
prediction
ability
(If not successful)
Optimize Communication Requirements
Declutter
Context-sensitive Help
(If not successful)
Optimize Communication Rate
Pacing
(If not successful)
Increase External Resources
Delegation
Task Sharing
Figure 3B: Specific Mitigation Selection Model (Step 4-5 of Figure 1) for Increasing
Situation Awareness
Design Interactive, Inc.
21
Augmented Cognition Mitigation Strategies
Next Generation Concepts
4 Innovative Mitigation Strategies
Studies have shown the need for flexible interaction from one user to the next (Langley,
1999). Similarly, the successful future of mitigation strategies lies within the power of
individual uniqueness; this is why information display strategies of tomorrow must be
inquisitive, adaptive, configurable, and considerate.
To date, augmentation strategies implemented in real-time for optimized HSI have been
limited. Further, display strategies for real-time mitigation that have been empirically
validated are limited in number and creativity (Schmorrow, Stanney, Wilson, & Young,
2005; Dorneich, Whitlow, Ververs, et al., 2004). Through review of techniques and best
practices from other media, innovative information display strategies for dynamic,
complex, information rich systems may be developed that can be used to alleviate
overload conditions, or existing mitigation strategies may be enhanced and optimized.
The Arts and communications were identified as likely sources for techniques that may
be exploited for this purpose. For example, movies and theatre use a linear timeline, but
often include multiple strands of action. These individual storylines are decomposed into
scenes and then rearranged (Howley, 2004) to create variety and tension. The rapid scene
transitions impose a need for continuous reorientation on the part of the audience in order
to reconnect the pieces and build a global picture of the plot (Mancini & Buckingham,
2001). This same phenomenon may occur during task switching in sequenced
environments. Having evolved beyond traditional desktop paradigms (e.g., WIMP
interfaces), the gaming industry can also effectively change context in space or time
without affecting one’s situation awareness (Billinghurst, et al., 2001; Szalavari, 1998).
This section lists innovative mitigation approaches and techniques that may be used to
dynamically update information presentation schemas to achieve one of two global goals
within an information rich, complex operational environment (e.g., military command
and control). Section 4.1 addresses the goal of alleviating operator processing
bottlenecks. Section 4.2 addresses the goal of enhancing SA. These approaches and
techniques are taken from areas outside human-computer interaction such as theatre,
movies, gaming, and mass media. After reviewing various strategies used in other media,
each strategy was evaluated for its applicability for military command and control
environments. These new execution strategies were then categorized into the mitigation
selection framework (Figure 2), thus expanding mitigation execution options available
under each mitigation strategy category.
4.1 Innovative Strategies to Alleviate Bottlenecks
The framework for mitigation strategy selection suggests that all current mitigation
strategies can be implemented to alleviate processing bottlenecks with the exceptions of
context-sensitive help and task sharing. Context-sensitive help inherently involves adding
information, thereby increasing cognitive demand. Task sharing does not necessarily
reduce cognitive bottlenecks as operators are still required to be ‘in the loop’ and
participate in tasking along with team members. The remaining mitigation strategies
identified across the five mitigation plans (Figure 2) can be effective in meeting the
Design Interactive, Inc.
22
Augmented Cognition Mitigation Strategies
Next Generation Concepts
objective of decreasing operator overload. The challenge to designers is to identify
innovative ways to mitigation system display in real-time that efficiently optimizes
operator state and measurably improves performance. Table 2 lists innovative approaches
and techniques for relevant mitigation strategies that may be applied within a complex
C4ISR environment to alleviate bottlenecks and enhance HSI within complex,
information rich environments. Innovative strategies for pacing and delegation were not
found, and thus these strategies are not included in table 2.
Table 2: Innovative Approaches/Techniques for Real-Time Mitigation of Bottlenecks
Mitigation Plan
Optimize
Individual
Processing
Capacity
Mitigation
Strategy
Modality
Augmentation
Innovative Approaches/Techniques
Aural-Spatial
Representation
Ambient Sound
Transposition
Design Interactive, Inc.
High-Key/LowKey Lighting
Based on echo and reverberations, people
can ‘hear’ how big a room is.
This way of encoding spatial data into
auditory information may open new
implementation strategies for multimodality.
Background sounds added in postproduction phase of a movie are perceived
as incidental but in fact function to enhance
the drama.
Embedding information into the natural
environment so that it is not noticed as such
(e.g. one notices the weather and acts
appropriately without actively attending to it)
may allow delivery of information without
distraction.
Lighting style where the scene is evenly lit,
suggests a familiar world containing few
surprises or mysteries, whereas strongly
contrasted areas of light and shadow create
a sense of mystery.
This principle may be transformed into
dynamic color schemes that contain
information about system status. A scheme
of harmonic colors might indicate a stable
operating environment, whereas strongly
contrasted colors may indicate that special
attention is needed.
23
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Table 2 (Con’t)
Mitigation Plan
Direct Attention
Mitigation
Strategy
Cueing
Innovative Approaches/Techniques
Environmental
Cues
Induction of
Physical
Response
GalvanicVestibular
Stimulation
(GVS)
Tilt Shot
Screen Flash
Music
Style/Tempo
Design Interactive, Inc.
People place cues in their environment
(e.g., sticky notes) to increases affordance
of information or to serve as a reminder for
tasks that will be pursued at a later time.
This concept of letting the user “park”
information where it is needed allows the
operator to temporarily decrease cognitive
load.
To date, haptic cues are usually transmitted
by vibration devices. However, lowfrequency sound produces a tactile
sensation that may be used for cueing in
certain environments.
Adding more dimensions to a modality may
reduce the bottleneck and increase
information throughput
This technique simulates the sensation of
movement by applying electromagnetic
signals directly to the inner ear.
GVS may open entirely new ways of cueing
by exploiting a new channel.
A tilted camera suggests a reaction to a
scene or object, usually involving
strangeness, imbalance, tension, or the
unexpected. E.g., the camera can have an
unusual angle after an accident, or shake
during an earthquake.
Shaking or tilting visual content may be
used as ambient cue without distorting or
manipulating the content.
A technique implemented in ego-shooter
games to indicate hits, where the screen
turns red for an instant, but fades to normal
rapidly so no detail is lost.
Since this cue appears in the foveal focus,
it does not require eye gazing or attention
shifting and is more likely to be noticed than
a peripheral cue.
Music is often used to create atmosphere in
a film.
Similar to lighting, background music may
be used as an ambient cueing tool. E.g.,
systems have been developed that indicate
stock activities to brokers by changes in
music style or tempo. Furthermore, music
has been found to affect emotional state
which may be utilized to affect stress levels
or other parameters of operators.
24
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Table 2 (Con’t)
Mitigation Plan
Direct Attention
Mitigation
Strategy
Cueing
Innovative Approaches/Techniques
Environmental
Sound
Shift in constant tone draws attention.
Experts can qualify sounds or changes of
sound, e.g. mechanics may hear problems
with the engine and even be able to
diagnose based on these sounds, or
computer experts may listen to the
frequency of the cooling fan to detect a
system problem.
Sequencing
Transitions
In cinematography, a Cut signifies a shorter
time lapse or merely a scene change. The
use of Out-/Inpoints (e.g. ending one shot
and starting the next shot with the same
element) makes the cut less abrupt. A Fade
can be used to suggest a passage of time,
a journey, or a new location. A Dissolve is
used to suggest a special relationship
between the scenes that dissolve into one
another (a relationship closer than
suggested by a fade).
Transition techniques may be adapted for
sequencing to alleviate the effects of task
switching.
Decreased ambient light focuses attention,
e.g. theaters use lighting to focus attention
and notify (cue) upcoming events.
Sequencing may be implemented by fading
out irrelevant content, therefore focusing
the user’s attention to the current task.
If two people share a common level of
knowledge, this reduces the amount of
information needed to convey a message.
The same applies to human-computer
interaction, allowing the use of codes,
abbreviations, shortcuts, etc., thus reducing
the amount of information to be processed
and therefore alleviating bottlenecks.
Looking at macro events (pattern
recognition) may reduce the need to attend
to details, thus reducing the amount of
information to be processed and therefore
alleviating bottlenecks.
Ambient
Lighting
Optimize
Communication
Requirements
Decluttering
Learned
Efficiency
Pattern
Monitoring
Design Interactive, Inc.
25
Augmented Cognition Mitigation Strategies
Next Generation Concepts
4.2 Innovative Strategies to Enhance Situation Awareness (SA)
The framework for mitigation strategy selection suggests that all current mitigation
strategies can be implemented to enhance SA with the exception of delegation. This
strategy involves offloading tasking to another operator/system, thereby limiting access
to all available information and potentially limiting SA. The remaining mitigation
strategies identified across the five mitigation plans (Figure 2) can be effective in meeting
the objective of decreasing operator overload. The challenge to designers is to identify
innovative ways to mitigation system display in real-time that efficiently optimizes
operator SA and measurably improves performance. Table 3 lists innovative approaches
and techniques for each identified mitigation strategy that may be applied within a
complex C4ISR environment to enhance SA within complex, information rich
environments. Innovative strategies for pacing and task sharing were not found, and thus
these strategies are not included in Table 3.
Table 3: Innovative Approaches/Techniques for Real-Time Mitigation to Enhance SA
Mitigation Plan
Optimize Individual
Processing
Capacity
Direct Attention
Mitigation
Strategy
Modality
Augmentation
Innovative Approaches/Techniques
Transposition
Cueing
See Table 2
Control of
Information
Stream
Validation
Feedback
Cutaway
Sequencing
Optimize
Communication
Requirements
See Table 2
Decluttering
Establishing
Shot
ContextSensitive
Help
Error Correction
Change-inState Indicator
Transmission
Failure
Reconciliation
Design Interactive, Inc.
People use fill words to maintain attention
during an interruption of the information
stream (“well”, “uhm”, …)
When communicating, people expect cues
for transmission success (“uh huh”).
A shot briefly interrupting one action to
provide a glimpse of another also taking
place, and then returning to the first action.
Can provide additional knowledge without
permanently losing focus on the primary
task.
A long shot giving an overview of a scene
so the audience is not confused about what
is happening and where.
Overviews/different perspectives may
enhance Situation Awareness.
Even if signal is improperly transmitted,
people can often “fill the gaps” with
assumptions based on context.
If the system can identify these signal gaps
it could provide resolution options to the
user.
Animators use “onion-skin” technique to
track changes from previous frames.
Display that/how an object has been
manipulated.
If a communication signal is not responded
to as expected, the sender will verify proper
transmission.
26
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Table 3 (Con’t)
Mitigation Plan
Optimize
Communication
Rate
Mitigation
Strategy
Pacing
Innovative Approaches/Techniques
Cross Cutting
Slow/Fast
Editing
Design Interactive, Inc.
Rapid sequence of shots between two
different locations, used to create tension.
The sequence builds to a climax and ends
with two things coming together.
Applied to pacing, cross cutting can be used
to create arousal if fatigue or underload is
detected.
Utilize cutting speed (i.e. shot length) to
generate excitement and anticipation (e.g.
chase sequence) with short shots, or
appear calming and relaxing (e.g. love
scenes) with long shots.
Relevance for Mitigation: Could be applied
to influence the user’s emotional state
through sequencing and pacing.
27
Augmented Cognition Mitigation Strategies
Next Generation Concepts
5 Mitigation Strategies Summary
Through review of current mitigation strategies, a conceptual framework outlining how
mitigation plans and associated strategies may be implemented in an ordered fashion was
developed as a first step in creating a real-time human-system closed loop system where
physiological metrics can drive innovative, seamless, adaptable system displays. Building
from the framework created in this report which outlines how mitigation strategies should
be implemented to achieve one of two global goals (alleviate bottlenecks, enhance SA),
current and innovative techniques have been described that may be considered based on
system and operator state and constraints (i.e., information being presented at any given
time, operator load across various channels). Our review of the Arts and other media
resulted in various innovative approaches/techniques that may be applied to C4ISR
environments to measurably improve operator performance. These approaches were
categorized into current mitigation strategies from Figure 3, and may be implemented as
alternative mitigation techniques.
To implement the conceptual mitigation selection framework presented here in real-time,
each specific execution method must be assigned given IF-THEN constraints to allow
automatic evaluation of effectiveness under specific conditions. These constraints must
consider all factors that may influence applicability of the specific strategy which may
include system state constraints (e.g., what information is currently being presented
across the system and how), environment constraints (e.g., environmental noise), user
state constraints (e.g., current processing load), and global user constraints (e.g.,
individual abilities/preferences). The next section provides an example highlighting how
the mitigation selection framework would be implemented in an operationally relevant
context.
Design Interactive, Inc.
28
Augmented Cognition Mitigation Strategies
Next Generation Concepts
6 Operational Example: Real-Time Mitigation Strategies for
Tactical Action Officer
During high intensity operations, the Tactical Flag Command Center (TFCC) provides
intelligence, communications and modifications to the operational plan for the Carrier
Strike Group (CSG). The Battle Watch Captain (BWC) is present in the TFCC during
high tempo; high alert times and is the center of the shared situational awareness. In
times when the BWC is not present in the TFCC, the Force Tactical Action Officer
(FTAO) assumes the responsibilities of the BWC; in fact, often the FTAO is the BWC.
The BWC/FTAO is one of the primary roles within the TFCC and has direct authority
from the (CSG) Commander in the decision making process. (Strausser, Kollmorgen &
Juhnke, 2005). Thus, the FTAO’s tasks involve monitoring the ship’s area of
responsibility for potential threats, and deciding upon an appropriate course of action
should a potential threat be identified.
In a complex, stressful environment such as this, physiological measures applied to an
FTAO may detect operator processing bottlenecks. Building from the framework
developed in this report, a processing bottleneck would trigger the mitigation selection
framework (Figure 2). Figure 4 demonstrates how this framework may be applied in an
operational context by specifying specific IF-THEN conditions that lead to
implementation of specific mitigation strategies.
6.1 Operational Scenario
This scenario illustrates a “typical” day in the service of Flag Tactical Action Officer
(FTAO), Phil Stevenson. References have been placed throughout the story to illustrate
relevance to the IF-THEN mitigation selection model (Figure 4). While this scenario is
focused on bottleneck resolution, there are also examples of SA improvement mitigations
at work.
“Another day, another tussle.” Those words were becoming Phil Stevenson’s mantra
lately. It seemed like this conflict was never going to end. Phil and his crew were ranked
among the highest performing teams in the fleet; a feat Phil knew had much to do with
his augie rig. Phil’s crew had won the “lottery” and been outfitted with some new, just
being tested, gear about six months ago. He smirks when he recalls the joke he and his
crew made about the hassle it was going to be. That’s all water under the bridge; he’s a
convert now.
His watch begins like any other day. He reviews the last watch summary on his personal
viewing system while he sips his morning coffee. A process that used to take several
hours is compressed into a matter of 15 minutes using the multimodal presentation that is
paced to optimize his current state of retention. Displaying the last watch’s progress
through task based mission plan by crosscutting information and key decision points from
the last watch gives Phil a high level view of occurrences and a complete understanding
of mission objectives he has yet to achieve. He even chuckles when he recognizes one of
Randy’s signature moves. Randy’s crew is right behind Phil’s on metrics, but it looks like
Phil has to do a little catch-up work for him again.
Design Interactive, Inc.
29
Augmented Cognition Mitigation Strategies
Next Generation Concepts
1a) Modality Augmentation
KEY
Visual resources are not overloaded
IMPROVED SA
Augment high priority visual communications with additional
BOTTLENECK RESOLUTION
visual information
If auditory saturation > Substitute auditory communication to visual
If auditory saturation > Substitute auditory status representations to visual
If auditory saturation > Substitute auditory cues to visual
If haptic saturation > Substitute haptic communications to visual
If haptic saturation > Substitute haptic status representations to visual
If haptic saturation > Substitute haptic cues to visual
Auditory resources are not overloaded
Augment high priority visual communications with additional auditory information
If visual saturation > Substitute visual communication to auditory
If visual saturation > Substitute visual status representations to auditory
If visual saturation > Substitute visual cues to auditory
If visual saturation > Substitute haptic communication to auditory
If haptic saturation > Substitute haptic status representations to auditory
If haptic saturation > Substitute haptic cues to auditory
Haptic resources are not overloaded
Augment high priority visual communications with additional haptic information
If visual saturation > Substitute visual status representations to haptic
If visual saturation > Substitute visual cues to haptic
If auditory saturation > Substitute auditory status representations to haptic
If auditory saturation > Substitute auditory cues to haptic
1b) Transposition
Verbal resources are not overloaded
Augment verbal resources with additional verbal detail
If spatial saturation > Substitute spatial communication to verbal
If spatial saturation > Substitute spatial status representations to verbal
If spatial saturation > Substitute spatial cues to verbal
Spatial resources are not overloaded
Augment spatial resources with additional spatial detail
If verbal saturation > Substitute verbal communication to spatial
If verbal saturation > Substitute verbal status representations to spatial
If verbal saturation > Substitute verbal cues to spatial
2a) Cueing
Critical communications are not being acknowledged
If no response to critical status > Cue status in same mode
If no response to critical status > Cue status in on different least intrusive unsaturated
mode
If no response to critical status > Cue status in on different next least intrusive
unsaturated mode. Repeat until acknowledgment
2b) Intelligent Sequencing
Executive function is overloaded
Present communications in order of priority (Defer lower priority communications)
3a) Declutter
Executive function is overloaded
Reduce information density to minimal amount required to maintain decision-making (all
modes)
4a) Pacing
Executive function is overloaded
Reduce the rate all communications are presented
5a) Mixed Initiative
Executive function is overloaded
Automate tasks to assist decision-making
5b) Delegate
Executive function is overloaded
Redirect communications to available (not overloaded) resources
Figure 4: Operationally Relevant Mitigation Strategy Selection Process
Design Interactive, Inc.
30
Augmented Cognition Mitigation Strategies
Next Generation Concepts
“Let’s heat ‘em up team!” Phil announces over group comm. Projection screens are
already displaying summaries of team activities and status [1a]. The task for today is to
secure a sector that is heavily trafficked by commercial vessels. There’s a good chance
that some of the commercial vessels are terrorist platforms hiding amongst the civilians
in their typical cowardly manner.
Dynamically assigned their Areas of Interest (AOI)[1a] Phil’s track team begins
scanning, and validating traffic. The 5-man team is strong except for Rudy; he’s still
pretty green and isn’t adjusting to the predictive load models as well as the others. [2b]
Phil notes that Randy’s AOI has already been reduced because of overload. [3a] He
stores the view of Randy’s reduced environment into his memory log for later
consideration.
Ironically it’s Rudy who finds the first “pheasant,” a carrier class freighter that has a
questionable ownership history. “Immediately the rest of the team responds to Rudy’s
find. That’s when things get interesting. Sam, one of Phil’s brightest exclaims, “Oh dear,
those are dots I didn’t want to connect!” over group comm. A radio transmission comes
in from the Nemesis but Phil doesn’t hear it because his attention is focused rightly on
Sam’s concerns. “If she’s concerned, I’m concerned,” he thinks. He notes the radio
transmission has been added to his text log for later review [2b]. It must not have been
important. The computer cues [2a] Phil that it is transferring 3 of the lower priority radio
channels to text logs [1a], converting a radio based status report to a graphic
representation on his dashboard [1b], and redistributing the high priority channels in
different places “in his head.” [1a] Phil really loves spatialized radio comms. They are so
much easier to separate.
Before he can speak, the computer has already requested further details regarding Sam’s
concern. [5a] Her analysis of traffic patterns, port summaries, and ship logs has shown
that 4 of the maybe 200 ships in the AOI have been traveling the same route over the past
20 days. Traffic logs show their actual progress to be normal except for the same 40-mile
diversion in the middle of ocean and then back on course. There were no records of any
platforms or vessels in that location, so it would be easy to consider the diversion a
course deviation to avoid an obstacle, except they all stopped at the same location for 6
hours and then continued back to their planned course. [3a] Phil notes the room has
dimmed quite a bit; group cog-load must be high. [3a] A spot over Rudy flashes and
catches Phil’s attention. [2a]“Rudy, what’s your status?” “I need auth for a boarding
party sir” Rudy responds. The computer whispers in Phil’s head that there is a 10%
probability of concern for Rudy’s track. “Rudy, let’s defer for a few minutes. I think
we’re going to need our resources.” Phil points at the big screen. Confirmed “civie”
vessels have been reduced to mere dots on the screen while the 4 ships Sam has flagged
have their routes, and PIMs plotted. [3a] “We need to run these numbers people” Phil
announces. The computer has already offloaded two of the vessels to James for analysis
and is summarizing determinations on the big screen. [5a][5b] It becomes evident that
these four ships, even though they are on different timetables will be converging on the
fleet at the same time. The computer whispers statistics in Phil’s head and it becomes
evident he needs to react now. They are an hour away from an attack.
Design Interactive, Inc.
31
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Phil orders the weapons teams on alert and cues the rest of the crews on watch of his
findings. Communications jump exponentially. The entire fleet is hot now. Comms. are
coming in so fast Phil has to rely heavily on the computer’s automated prioritizing and
response tools. [5a] At one point a commander sends Phil an updated mission plan and
the computer generously offers to allow Phil to dictate his responses and queries. During
a few points of very intense communication Phil notices the world seems to slow down.
He knows it’s the computer pacing his comms.[4a] to help him concentrate. He looks
forward to the cue that he is back on real-time. [3a] Time-late is not Phil’s favorite place.
Sometime during the conflict the computer assumed weapon allocation and assignment
and Lloyd a “heckovaguy” stationed in Ohio was brought online to assume portions of
Phil’s load. [5b] Lloyd’s performance rating was exemplary. Phil knew he had the best
help he could get.
Eventually the conflict resolved, no lives were lost and all four of the enemy vessels were
captured. It was no surprise that their cargo included an assortment of “dirty” torpedoes.
A mess had definitely been averted today.
Design Interactive, Inc.
32
Augmented Cognition Mitigation Strategies
Next Generation Concepts
7 Conclusions and Future Directions
This paper reviewed mitigation strategies that are currently implemented to enhance
operator performance within traditional human computer interaction systems. Each of
these strategies may be implemented within a complex, information rich environment
such as military C4ISR environments to measurably improve a number of objectives,
including alleviating human processing bottlenecks and enhancing situation awareness.
Although this review uncovered a number of mitigation strategies that are available at
present, there was no conceptual model to aid designers in determining which mitigation
strategy is best under a given set of conditions.
To address this void, a conceptual mitigation selection framework was developed that
outlines five mitigation selection plans (optimize information processing capacity, direct
attention, optimize communication requirements, optimize communication rate, increase
external resources) that may be considered in order to address various global mitigation
goals, including alleviating operator processing bottlenecks and enhancing operator
situation awareness. In addition, media, communication, and the Arts were embraced in
order to identify innovative ways of mitigation that have yet been undiscovered.
Although no innovative strategies were found, numerous techniques that may
considerably enhance the way mitigation strategies are implemented were extracted and
added to the mitigation selection framework. An example of how this conceptual
framework may be applied in an operational setting was provided. In follow-on efforts,
empirical studies will be conducted to validate the mitigation selection framework within
an operational environment.
The conceptual model and associated mitigation strategies (current and innovative
approaches to enhancing system design) outlined in this report provide a starting point for
realizing the goal of real-time mitigation within complex C4ISR environments by
offering a structured approach for strategy selection. Other considerations that need to be
addressed in future work include the development of exit policies for each mitigation
strategy (i.e., when is mitigation no longer required, e.g. when to return from down-paced
presentation to regular speed), how to coordinate the mitigation cycles for different
objectives (i.e. reduce overload and enhance situation awareness), and how to apply
mitigation framework to team environments. In addition, the conceptual model outlined
here may be expanded to mitigate various human state conditions beyond processing
bottlenecks and situation awareness. If such conditions can be captured in real-time via
psycho-physiological sensors, there may be an opportunity to enhance operator
performance, e.g. maximize focus and increase motivation.
Design Interactive, Inc.
33
Augmented Cognition Mitigation Strategies
Next Generation Concepts
8 References
Adamczyk, P, Bailey B.P. (2004). If not now, when?: the effects of interruption at different
moments within task execution, Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems, 271 – 278.
Arnell, K.M., Larson, J.M. (2002). Cross-modality attentional blinks without preparatory task-set
switching. Psychonomic Bulletin & Review, 9(3), 497-506.
Arroyo, E., Selker, T. and Stouffs, A. (2002) Interruptions as multimodal outputs: Which are the
less disruptive? ICMI'02, 479-483.
Bailey, B.P., Konstan, J.A. and Carlis, J.V. (2000) Adjusting Windows - Balancing Information
Awareness with Intrusion. Proceedings of the 6th Conference on Human Factors and the
Web.
Baldonado, M.Q.W., Woodruff, A., & Kuchinsky, A. (2000). Guidelines for using multiple views
in information visualization. In Proceedings of AVI2000, Palermo, Italy (pp. 110-119).
Retrieved, July 2005, from http://www2.parc.com/istl/projects/uir/pubs/items/UIR-2000-05Baldonado-AVI2000-MultipleViews.pdf
Barsalou, L.W., Simmons W.K., Barbey, A.K., Wilson, C.D. (2003). Grounding conceptual
knowledge in modality-specific systems. TRENDS in Cognitive Sciences, 7(2), 84-91.
Bell, L., Boye, J, Gustafson, J and Wirén, M. Modality Convergence in a Multimodal Dialogue
System. Proceedings of Götalog 2000, Fourth Workshop on the Semantics and Pragmatics of
Dialogue, 29-34.
Berka, C., Levendowski, D.J., Ramsey, C.K., Davis, G., Lumicao, M.N., Stanney, K., Reeves, L.
Harkness Regli, S., Tremoulet, P.D., Stibler, K. (2005). Evaluation of an EEG-Workload
model in an Aegis simulation: Biomonitoring for physiological and cognitive performance
during Military Operations. Proceedings of the International Society for Optical
Engineering. 5797, 90 - 99.
Berz, W. L. (1995). Working memory in music: A theoretical model. Music Perception, (12),
353-364.
Billinghurst, M, Kato, H., & Poupyrev, I. (2001). The Magic Book: Moving seamlessly between
reality and virtuality. IEEE Computer Graphics and Applications, 21(3), 2-4.
Boiney, L. (2005). Team decision making in time-sensitive environments. Presented at the 10th
International Command and Control Research and Technology Symposium: The Future of
C2. Retrieved, July 12, 2005, from
http://www.dodccrp.org/events/2005/10th/CD/papers/175.pdfRetrieved, July 12, 2005, from
http://www.dodccrp.org/events/2005/10th/CD/papers/175.pdf
Casini, L. & Macar, F. (1999) Multiple Approaches to investigate the Existence of an Internal
Clock Using Attentional Resources, Behavioral Processes 45, 73-85.
Cress, U. & Knabel, O.B. (2003). Previews in hypertexts: effects on navigation and knowledge
acquisition. Journal of Computer Assisted Learning, 19, 517-527.
Czerwinski, M., E., Cutrell & Horvitz, E. (2000). Instant Messaging and Interruption: Influence
of Task Type on Performance. Proceedings of OZCHI, 356-361.
Donmez, B., Boyle, L., Lee, J.D. (2003). Taxonomy of Mitigation Strategies for Driver
Distraction. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting,
1865-1869.
Dorneich, M., Whitlow, S., Ververs, P.M., Mathan, S., Raj, A., Muth, E., Hoover, A.,
DuRousseau, D., Parra, L., & Sajda, P. (2004) The role of peripheral feedback in emotional
experience with music. Music Perception, 22(1), 79-115). DARPA improving warfighter
information intake under stress – Augmented Cognition: concept validation experiment
Design Interactive, Inc.
34
Augmented Cognition Mitigation Strategies
Next Generation Concepts
(CVE) analysis report for the Honeywell team. Prepared under contract: DAAD16-03-C0054.
ETSI [European Telecommunications Standards Institute]. (2002). Human factors (HF);
Guidelines on the multimodality of icons, symbols, and pictograms. (Report No. ETSI EG
202 048 v 1.1.1 (2002-08). Sophia Antipolis, France: ETSI.
Furnas, G. (1986) Generalized Fisheye Views. ACM SIGCHI Bulletin, 17(4), 16-23.
Gentner D. and Nielsen, J. (1996). The Anti-Mac Interface. Communications of the ACM, 30(8),
70-82.
Gutwin C. and Skopik, A. (2003) Fisheye Views are Good for Large Steeering Tasks.
Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 201-208.
Gutwin, C. (2002). Improving Focus Targeting in Interactive Fisheye Views, Proceedings of CHI
2002, 267-274.
Hildebrandt, M.A., Dix, A.J., Meyer, H.A. (2004). Time Design. CHI '04 extended abstracts on
Human Factors in Computing Systems, 1737-1738.
Horvitz, E. & Apacible, J. (2003). Learning and Reasoning About Interruption. Proceedings of
the Fifth ACM International Conference on Multimodal Interfaces, 20-27.
Horvitz, E., Jacobs, A. & Hovel, D. (1999). Attention-Sensitive Alerting. Proc. Uncertainty in
Artificial Intelligence, 305-313.
Howley, K. (2004) Breaking, Making, and Killing Time in Pulp Fiction, Scope – an online
journal of film studies. May 2004. Retrieved July 13, 2005, from
http://www.nottingham.ac.uk/film/journal/articles/making-breaking-and-killing.htm.
Hudson, S.E., Fogarty, J., Atkeson, C.G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J.C., &
Yang, J. (2003). Predicting Human Interruptibility with Sensors: A Wizard of Oz Feasibility
Study. Proceedings of the SIGCHI conference on Human factors in computing
systems, 257-264.
Kirsh, D. (1995). The Intelligent Use of Space. Artificial Intelligence, 73, 31-68.
Kirsh, D. (2000). A Few Thoughts on Cognitive Overload. Intellectica, 30, 19-51.
Koffka, K. (1935). Principles of Gestalt Psychology. Lund Humphries: London.. Retrieved
October 13, 2005, from
http://www.marxists.org/reference/subject/philosophy/works/ge/koffka.htm.
Kreifeldt, J.G. & McCarthy, M.E. (1981). Interruption as a Test of the User-Computer Interface.
Proceedings of the 17th Annual Conference on Manual Control, Jet Propulsion Laboratory,
California Institute of Technology, JPL Publication 81-95, 655-667.
Langley, P. (1999). User modeling in adaptive interfaces. In Proceedings of the Seventh
International Conference on User Modeling, (p. 357-370). Banff, Alberta: Springer.
Retrieved, July 12, 2005, from
http://www.cs.utah.edu/classes/cs5350/handouts/adapt.um99.pdf
Lyons, K., Gandy, M., & Starner, T. (2000) Guided by Voices: An Audio Augmented Reality
System. Proceedings of International Conference on Auditory Display (ICAD) 2000. April
2000.
Mahjan, R. & Shneiderman, B. (1997). Visual and Text Consistency Checking Tools for
Graphical User Interfaces. IEEE Transactions on Software Engineering, 23(11), 722-735.
Mamykina, L., Mynatt, E.,Terry, M.A. (2001). Time Aura: interfaces for pacing, Proceedings of
the SIGCHI conference on Human Factors in Computing Systems, 144-151.
Mancini, C. & Buckingham, S.S. (2001) Cognitive coherence relations and hypertext: from
cinematic patterns to scholarly discourse. Proceedings of the twelfth ACM conference on
Hypertext and Hypermedia, 165-174.
McFarlane, D.C. (1999). Coordinating the Interruption of People in Human-Computer
Interaction. INTERACT, 295-303.
McFarlane, D. C. (2002). Comparison of four primary methods for coordinating the interruption
of people in human-computer interaction, Human-Computer Interaction, 17(1), 63-139.
Design Interactive, Inc.
35
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Meyer, H.A. & Hildebrandt, M. (2002). Towards Time Design: Pacing of hypertext navigation by
system response times. Ext. Abstracts CHI2002, ACM Press, 824-825.
Miyata, Y. & Norman, D. (1986). Psychological issues in support of multiple activities. In:
Norman, D.A. & Draper, S.W. (Eds.), User Centered Systems Design: New Perspectives on
Human-Computer Interaction. Hillsdale: Lawrence Erlbaum Associates, 265-284.
Monk, C. A., Boehm-Davis, D. A. & Trafton, J. G. (2002). The attentional costs of interrupting
task performance at various stages. Proceedings of the Human Factors and Ergonomics
Society 2002, 1824-1828.
Nielsen, J. (1989). Coordinating user interfaces for consistency. ACM SIGCHI Bulletin, 20(3),
63-65.
Norman, D.A. (1988). The Psychology of Everyday Things. Basic Books, New York.
Pecher, D., Zeelenberg, R., Barsalou, L.W. (2003). Verifying different-modality properties for
concepts produces switching costs. Psychological Science, 14(2), 119.
Posner, M. I., Snyder, C. R. R. & Davidson, B. J. (1980). Attention and the detection of signals.
Journal of Experimental Psychology: General, (109), 160-174.
Predebon J. (1999) Time Judgments as a Function of Clock Duration: Effects of temporal
paradigm and an attention-demanding nontemporal task, Perceptual and Motor Skills, 88,
1251-1254.
Sakar, M. & Brown M.H. (1992). Graphical Fisheye Views of Graphs. Proceedings of ACM on
Human Factors in Computing Systems, 83-91.
Schmorrow, D., Stanney, K.M., Wilson, G., & Young, P. (2005). Augmented cognition in
human-system interaction. In G. Salvendy (Ed.), Handbook of human factors and
ergonomics (3rd edition). New York: John Wiley.
Schweiger, W. (2001). Hypermedien im Internet. Verlag Reinhard Fischer, München.
Sheridan T. and Ferrell W. (1974). Man-Machine Systems: Information, Control, and Decision
Models of Human Performance. MIT Press, Cambridge, Massachusetts, USA.
Shneiderman, B. (1984). Response time and display rate in human performance with computers.
Computing Surveys, 16, 3, 265-285.
Speier, C., Valacich, J.S. & Vessey. I. (1999). The Influence of Task Interruption on Individual
Decision Making: An Information Overload Perspective. Decision Sciences, 30(2), 337-360.
St John, M. and Osga, G. (1999). Supervision of Concurrent Tasks Using a Dynamic Task Status
Display. Proceedings of the 43rd Human Factors & Ergonomics Society Annual Meeting,
168-172.
Starner, T., Mann, S., Rhodes, B., & Levine, J. (1997). Augmented Reality Through Wearable
Computing. M.I.T Media Laboratory Perceptual Computing Section Technical Report No.
397.
Strausser, J., Kollmorgen, G., and Juhnke, J. (2005). DARPA Small Business Innovation
Research (SBIR) Program Phase I Report for the Period July 14, 2004 to January 14, 2005.
Contract ID W31P4Q-04-C-R253.
Sulzen, J. (2001). Modality based working memory. School of Education, Stanford
University. Retrieved, February 5, 2003 from http://ldt.stanford.edu/~jsulzen/jamessulzen-portfolio/classes/PSY205/modality-project/paper/modality-expt-paper.PDF.
Szalavári, Z., Eckstein, E., Gervautz, M. (1998) Collaborative gaming in augmented
reality. Proceedings of the ACM symposium on Virtual reality software and
technology, 195-204.
Teal S.L., Rudnicky, A.I. (1992). A performance model of system delay and user strategy
selection. Proceedings of the SIGCHI conference on Human factors in computing systems,
295-305.
Townsend, J.T. (1971). A note on the identifiability of parallel and serial processes. Perception
and Psychophysics, 10, 161-163.
Design Interactive, Inc.
36
Augmented Cognition Mitigation Strategies
Next Generation Concepts
Trafton, J. G., Altmann, E. M., Brock, D. P. and Mintz, F. E. (2003).Preparing to resume an
interrupted task: Effects of prospective goal encoding and retrospective rehearsal,
International Journal of Human-ComputerStudies, 58, 583-603.
Wickens, C. D. (1984). Processing resources in attention. In R. Parasurraman & R. Davies (eds.),
Varieties of Attention (pp. 63-101). New York: Academic Press.
Wickens, C. D. (1992). Engineering Psychology and Human Performance (2nd Edition). New
York, NY: Harper Collins.
Wickens, C.D. (2002). Multiple resources and performance prediction. Theoretical Issues in
Ergonomics Science, 3(2), 159-177.
Zacks, J., & Tversky, B. (2001). Event structure in perception and cognition. Psychological
Bulletin, 127(1), 3-21.
Zacks, J.M., Braver, T.S., Sheridan, M.A., Donaldson, D.I., Snyder, A.Z., Ollinger, J.M.,
Buckner, R.L. & Raichle, M.E. (2001) Human brain activity time-locked to perceptual event
boundaries. Nature Neuroscience 4, 651 – 655.
Zacks, J., Tversky, B., & Iyer, G. (2001). Perceiving, remembering, and communicating structure
in events, Journal of Experimental Psychology: General, 130(1), 29-58.
Zakay, D. & Block, R.A. (1997). Temporal Cognition, Current Directions in Psychological
Science, 6,(1), 12-16.
Zijlstra, F. R. H., Roe R. A., Leonova, A. B. & Krediet, I. (1999). Temporal factors in mental
work: Effects of interrupted activities, Journal of Occupational and Organizational
Psychology, 72, 163-185.
Design Interactive, Inc.
37