CHAPTER 7:
Direct Manipulation and
Immersive Environments
Designing the User Interface:
Strategies for Effective Human-Computer Interaction
Sixth Edition
Ben Shneiderman, Catherine Plaisant,
Maxine S. Cohen, Steven M. Jacobs, and Niklas Elmqvist
in collaboration with
Nicholas Diakopoulos
Addison Wesley is
an imprint of
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
Direct Manipulation and
Immersive Environments
Topics
1. Introduction
2. What is Direct Manipulation?
3. Some examples of Direct Manipulation
4. 2D and 3D Interfaces
5. Teleoperation and Presence
6. Augmented and Virtual Reality
2
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-2
Introduction
• Positive feelings associated with good user
interfaces
– Mastery of the interface
– Competence in performing tasks
– Ease in learning the system originally
and in assimilating advanced features
– Confidence in the capacity to retain mastery over time
– Enjoyment in using the system
– Eagerness to show the system off to novices
– Desire to explore more powerful aspects of the system
3
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-3
Principles of Direct Manipulation
1. Continuous representations of the objects and
actions of interest with meaningful visual
metaphors.
2. Physical actions or presses of labeled buttons,
instead of complex syntax.
3. Rapid, incremental, reversible actions whose
effects on the objects of interest are visible
immediately.
4
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-4
Attributes of Direct Manipulation
• Novices can learn basic functionality quickly, usually through a
demonstration by a more experienced user
• Experts can work rapidly to carry out a wide range of tasks, even
defining new functions and features
– Knowledgeable intermittent users can retain operational concepts
– Error messages are rarely needed
• Users
– Immediately see whether their actions are furthering their goals, and, if the actions
are counterproductive, they can simply change the direction of their activity
– Experience less anxiety because the interface is comprehensible and because
actions can be reversed easily
– Gain a sense of confidence and mastery because they are the initiators of action,
they feel in control, and they can predict the interface’s responses
5
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-5
Attributes of Direct Manipulation
(continued)
• One way of trying to understand and categorize
the direct manipulation metaphor is by looking at
the translational distance between users and the
representation of the metaphor, which will be
referred to as strength
– Strength can be perceived along a continuum from
weak to immersed
– This can be further described as the level of
indirectness between the user’s physical actions and
the actions in the virtual space
6
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-6
Attributes of Direct Manipulation
Systems (continued)
• Multi-touch allows new actions to be assigned to
various combinations of finger touches
– The 2-finger actions like zoom in/out are
intuitive, but others must be learned and take
longer to discover
– This accounts for why a young child can easily
learn to tap, change screens, and touch on a
tablet (the intuitive actions), but does not have
the skills to re-arrange the icons on the screen
(the learned actions)
7
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-7
Attributes of Direct Manipulation
Systems (concluded)
• Examples of Translational Distances
– Weak – early video game controllers
– Medium – touch screens, multi-touch
– Strong – data glove, gesturing, manipulating
tangible objects
– Immersive – virtual reality, oculus rift
https://www.oculus.com/en-us/press-kit-hardware/
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
8
7-8
Direct Manipulation Systems
(example)
• Three users working concurrently on a large
tabletop touch device.
– They can use their hands and fingers to manipulate
the objects on the device
– Note the use of the different hand gestures
9
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-9
Direct Manipulation Systems
(example)
• A tangible user interface for molecular biology, developed
in Art Olson’s Laboratory at The Scripps Research
Institute
– Utilizes auto-fabricated molecular models tracked with the
Augmented Reality Toolkit (from the University of Washington
Human Interface Technology Lab)
– The video camera on the laptop captures the molecule’s position
and orientation, enabling the molecular modeling software to
display information such as the attractive/repulsive forces
surrounding the molecule
10
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-10
Discussion of Direct Manipulation
Problems with direct manipulation
• Spatial or visual representations can be too spread out
• High-level flowcharts and database-schema can become
confusing
• Designs may force valuable information off of the screen
• Users must learn the graphical representations
• The visual representation may be misleading
• Typing commands with the keyboard may be faster
11
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-11
Direct Manipulation Systems
(example)
• GPS solutions
– This is a screenshot from Google StreetView of the inside of the University Center
at Nova Southeastern University in Florida
– On the bottom is a scrollable image of other views on campus. In the left hand
corner is a more conventional static map showing the physical street location of
the campus
– Users can move the “person” to a different location on campus and the views will
change accordingly
12
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-12
Direct Manipulation Systems (another
example)
• Video games
– Woman playing World of Warcraft, using both her keyboard and mouse,
while hearing sounds of the game via her headset
13
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-13
Direct Manipulation Systems (another
example)
• 3D Printing
– This shows astronaut (Bruce Wilmore) onboard the International Space
Station with the ratchet wrench that was created with Made in Space’s
3D printer
– This device was designed, qualified, tested, and printed in space in less
than one week
14
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-14
2D and 3D Interfaces
• “Pure” 3D interfaces have strong utility in some contexts,
for example:
– Medical
– Product design
– Scientific visualization
• In some situations, 2D may actually be preferable to
simplify interactions
• The power of 3D interfaces lies in applying them in the
appropriate domain or context where the added
dimension provides more understanding and improves
task outcomes
15
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-15
3D Interfaces
• By using a medical simulation inserted into a large scale
visualization (using CAVE technology), physicians were
able to find a solution, that would not have been possible
without doing the actual surgery
16
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-16
3D Interfaces (continued)
Features for effective 3D
–
–
–
–
–
–
–
–
–
Use occlusion, shadows, perspective, and other 3D techniques carefully
Minimize the number of navigation steps for users to accomplish their
tasks
Keep text readable
Avoid unnecessary visual clutter, distraction, contrast shifts, and
reflections
Simplify user movement
Prevent errors
Simplify object movement
Organize groups of items in aligned structures to allow rapid visual
search
Enable users to construct visual groups to support spatial recall
17
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-17
3D Interfaces (continued)
Guidelines for inclusion of enhanced 3D features:
– Provide overviews so users can see the big picture
– Allow teleoperation
– Offer X-ray vision so users can see into or beyond
objects
– Provide history keeping
– Permit rich user actions on objects
– Enable remote collaboration
– Give users control over explanatory text and let users
select for details on demand
– Offer tools to select, mark, and measure
18
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-18
3D Interfaces (concluded)
Guidelines for inclusion of enhanced 3D features
(continued):
– Implement dynamic queries to rapidly filter out
unneeded items
– Support semantic zooming and movement
– Enable landmarks to show themselves even at a
distance
– Allow multiple coordinated views
– Develop novel 3D icons to represent concepts that are
more recognizable and memorable
19
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-19
Teleoperation and Presence
• Two “parents”:
– direct manipulation in personal computers
– process control in complex environments
• Physical operation is remote
• Complicating factors in the architecture of remote
environments:
– Time delays
• transmission delays
• operation delays
– Incomplete feedback
– Feedback from multiple sources
– Unanticipated interferences
20
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-20
Teleoperation and Presence
(example)
• Nurse demonstrates using the Telehealth cart otoscope to conduct a
real-time tympanic (ear) membrane exam
• On screen is Physician Assistant who, from a remote location, can
see and evaluate the patient and provide an appropriate plan of care
21
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-21
Teleoperation and Presence
(example)
• ImmerseBoard allows two users to be co-located and
work on the same shared screen
22
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-22
Augmented and Virtual Reality
• Virtual reality breaks the physical limitations of space and
allow users to act as though they were somewhere else
• Augmented reality shows the real world with an overlay of
additional overlay
• Situational awareness shows information about the real
world that surrounds you by tracking your movements in a
computer model
• Augmented reality is an important variant
– Enables users to see the real world with an overlay of
additional interaction.
23
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-23
Augmented and Virtual Reality
(example)
• Virtual reality might be used to help surgeons or their assistants
during surgery, by showing pertinent information superimposed on a
view of the real world. http://augmentarium.umiacs.umd.edu
24
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-24
Augmented and Virtual Reality
(another example)
• Using augmented reality overlays, various points of interest can be
shown on a mobile phone
• Icons represent the type of place (food, shopping, etc.) and distances
from the current location
• Links are provided to user reviews
25
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-25
Augmented and Virtual Reality
(another example)
• Customers can use their personal mobile devices to pull
up objects from the IKEA Catalog and see how the
various items would look in their own house
26
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-26
Augmented and Virtual Reality
(another example)
• Image of a virtual meditative world that users can use for
engaging in meditation activities
– The virtual world has sounds
– They change with each chakra (stage) of the meditation process
– This is an application of positive computing
27
© 2017 Pearson Education, Inc., Hoboken, NJ. All rights reserved.
7-27
Purchase answer to see full
attachment