VMWARE
W H WHITE
I T E PPAPER
A P E R
Virtualization Overview
1
VMWARE WHITE PAPER
Table of Contents
Introduction .............................................................................................................................................. 3
Virtualization in a Nutshell ................................................................................................................... 3
Virtualization Approaches .................................................................................................................... 4
Virtualization for Server Consolidation and Containment ........................................................... 7
How Virtualization Complements New-Generation Hardware .................................................. 8
Para-virtualization ................................................................................................................................... 8
VMware’s Virtualization Portfolio ........................................................................................................ 9
Glossary ..................................................................................................................................................... 10
2
VMWARE WHITE PAPER
Virtualization Overview
Introduction
Virtualization in a Nutshell
Among the leading business challenges confronting CIOs and
IT managers today are: cost-effective utilization of IT infrastructure; responsiveness in supporting new business initiatives;
and flexibility in adapting to organizational changes. Driving
an additional sense of urgency is the continued climate of IT
budget constraints and more stringent regulatory requirements.
Virtualization is a fundamental technological innovation that
allows skilled IT managers to deploy creative solutions to such
business challenges.
Simply put, virtualization is an idea whose time has come.
The term virtualization broadly describes the separation of a
resource or request for a service from the underlying physical
delivery of that service. With virtual memory, for example,
computer software gains access to more memory than is
physically installed, via the background swapping of data to
disk storage. Similarly, virtualization techniques can be applied
to other IT infrastructure layers - including networks, storage,
laptop or server hardware, operating systems and applications.
This blend of virtualization technologies - or virtual infrastructure - provides a layer of abstraction between computing,
storage and networking hardware, and the applications running
on it (see Figure 1). The deployment of virtual infrastructure
is non-disruptive, since the user experiences are largely
unchanged. However, virtual infrastructure gives administrators
the advantage of managing pooled resources across the enterprise, allowing IT managers to be more responsive to dynamic
organizational needs and to better leverage infrastructure
investments.
�����������
���
�����������
�����������
����������������
����������������
����������������
���������������������������
����������������
����������������
������
���
����
Before Virtualization:
• Single OS image per machine
���
������
���
����
After Virtualization:
• Software and hardware tightly coupled
• Hardware-independence of operating
system and applications
• Running multiple applications on same machine
often creates conflict
• Virtual machines can be provisioned to any
system
• Underutilized resources
• Can manage OS and application as a single
unit by encapsulating them into virtual
machines
• Inflexible and costly infrastructure
Figure 1: Virtualization
3
VMWARE WHITE PAPER
Virtualization Approaches
Using virtual infrastructure solutions such as those from
VMware, enterprise IT managers can address challenges that
include:
While virtualization has been a part of the IT landscape for
decades, it is only recently (in 1998) that VMware delivered
the benefits of virtualization to industry-standard x86-based
platforms, which now form the majority of desktop, laptop and
server shipments. A key benefit of virtualization is the ability to
run multiple operating systems on a single physical system and
share the underlying hardware resources – known as partitioning.
• Server Consolidation and Containment – Eliminating ‘server
sprawl’ via deployment of systems as virtual machines (VMs)
that can run safely and move transparently across shared
hardware, and increase server utilization rates from 5-15%
to 60-80%.
• Test and Development Optimization – Rapidly provisioning
test and development servers by reusing pre-configured
systems, enhancing developer collaboration and standardizing development environments.
Today, virtualization can apply to a range of system layers,
including hardware-level virtualization, operating systemlevel virtualization, and high-level language virtual machines.
Hardware-level virtualization was pioneered on IBM mainframes
in the 1970s, and then more recently Unix/RISC system vendors
began with hardware-based partitioning capabilities before
moving on to software-based partitioning.
• Business Continuity – Reducing the cost and complexity of
business continuity (high availability and disaster recovery
solutions) by encapsulating entire systems into single files
that can be replicated and restored on any target server,
thus minimizing downtime.
For Unix/RISC and industry-standard x86 systems, the two
approaches typically used with software-based partitioning are
hosted and hypervisor architectures (See Figure 2). A hosted
approach provides partitioning services on top of a standard
operating system and supports the broadest range of hardware
configurations. In contrast, a hypervisor architecture is the first
layer of software installed on a clean x86-based system (hence
it is often referred to as a “bare metal” approach). Since it has
direct access to the hardware resources, a hypervisor is more
efficient than hosted architectures, enabling greater scalability,
robustness and performance.
• Enterprise Desktop – Securing unmanaged PCs, workstations and laptops without compromising end user
autonomy by layering a security policy in software around
desktop virtual machines.
�����������
�����������
���
���
���
���
����������
��������
����������
������
����������
������
����������
������
����������
������
�������
�������
����������������������
��������������������
���������������������������
���������������������
����������������
����������������
���
������
���
���
Hosted Architecture
���
����
Bare-Metal (Hypervisor) Architecture
• Installs and runs as an application
• Lean virtualization-centric kernel
• Relies on host OS for device support
and physical resource management
• Service Console for agents and helper
applications
Figure 2: Virtualization Architectures
4
������
����
VMWARE WHITE PAPER
Hypervisors can be designed to be tightly coupled with operating systems or can be agnostic to operating systems. The latter
approach provides customers with the capability to implement
an OS-neutral management paradigm, thereby providing
further rationalization of the data center.
Application-level partitioning is another approach, whereby
many applications share a single operating system, but this
offers less isolation (and higher risk) than hardware or software
partitioning, and limited support for legacy applications or
heterogeneous environments. However, various partitioning
techniques can be combined, albeit with increased complexity.
Hence, virtualization is a broad IT initiative, of which partitioning
is just one facet. Other benefits include the isolation of virtual
machines and the hardware-independence that results from
the virtualization process. Virtual machines are highly portable,
and can be moved or copied to any industry-standard (x86based) hardware platform, regardless of the make or model.
Thus, virtualization facilitates adaptive IT resource management,
and greater responsiveness to changing business conditions
(see Figures 3-5).
To provide advantages beyond partitioning, several system
resources must be virtualized and managed, including CPUs,
main memory, and I/O, in addition to having an inter-partition
resource management capability. While partitioning is a useful
capability for IT organizations, true virtual infrastructure delivers
business value well beyond that.
�����������
�����������
�����������
����������������
����������������
����������������
����������������
����������������
����������������
�������
�������
�������
�������
�������
�������
�������
Figure 3: Traditional Infrastructure
5
VMWARE WHITE PAPER
Hardware/Software Separation
�����������
��������
���
���
������
�������������
����������
������
������
�����
�����
������
����������
������
����������������������
�������
�������
�������
Figure 4: Virtual Infrastructure
Infrastructure is what
connects resources to your
business.
�����������
�����������
�����������
����������������
����������������
����������������
���������������
���������������
���������������
Result: decreased costs and
increased efficiencies and
responsiveness
����������������������
�������
�������
�������
�������
�������
Transforms farms of individual x86 servers, storage, and
networking into a pool of computing resources
Figure 5: VMware Virtual Infrastructure
6
Virtual Infrastructure is a
dynamic mapping of your
resources to your business.
�������
VMWARE WHITE PAPER
Virtualization for Server Consolidation and
Containment
higher management costs. Virtual infrastructure enables
more effective optimization of IT resources, through the
standardization of data center elements that need to be
managed.
Virtual infrastructure initiatives often spring from data center
server consolidation projects, which focus on reducing existing
infrastructure “box count”, retiring older hardware or life-extending legacy applications. Server consolidation benefits result
from a reduction in the overall number of systems and related
recurring costs (power, cooling, rack space, etc.)
Partitioning alone does not deliver server consolidation or
containment, and in turn consolidation does not equate to
full virtual infrastructure management. Beyond partitioning and basic component-level resource management, a
core set of systems management capabilities are required
to effectively implement realistic data center solutions (see
Figure 6). These management capabilities should include
comprehensive system resource monitoring (of metrics such
as CPU activity, disk access, memory utilization and network
bandwidth), automated provisioning, high availability and
workload migration support.
While server consolidation addresses the reduction of existing
infrastructure, server containment takes a more strategic view,
leading to a goal of infrastructure unification. Server containment uses an incremental approach to workload virtualization,
whereby new projects are provisioned with virtual machines
rather than physical servers, thus deferring hardware purchases.
It is important to note that neither consolidation nor containment should be viewed as standalone exercise. In either case,
the most significant benefits result from adopting a total costof-ownership (TCO) perspective, with a focus on the ongoing,
recurring support and management costs, in addition to onetime, up-front costs. Data center environments are becoming
more complex and heterogeneous, with correspondingly
VM
VM
VM
Management
and
Distributed
Virtualization
Services
VirtualCenter
VMotion
VMM
VM
VMM
VMM
VMM
Consolidated Backup
Resource
Management
DRS
DAS
CPU
Virtualization
Distributed
Services
VMFS
Virtual
Networking
MPIO
Other Enterprise
Features
Enterprise-Class
Features
MMU
Virtualization
I/O
Virtualization
Hypervisor
ESX Server
Blade Hardware
CPU
Monitor
Provisioning
Memor y
NIC
Disk
Other Hardware
Hardware
Certification
Hardware
Figure 6: Virtual Infrastructure Management
7
VMWARE WHITE PAPER
How Virtualization Complements NewGeneration Hardware
Extensive ‘scale-out’ and multi-tier application architectures are
becoming increasingly common, and the adoption of smaller
form-factor blade servers is growing dramatically. Since the
transition to blade architectures is generally driven by a desire
for physical consolidation of IT resources, virtualization is an
ideal complement for blade servers, delivering benefits such as
resource optimization, operational efficiency and rapid provisioning.
The latest generation of x86-based systems feature processors
with 64-bit extensions supporting very large memory capacities. This enhances their ability to host large, memory-intensive
applications, as well as allowing many more virtual machines to
be hosted by a physical server deployed within a virtual infrastructure. The continual decrease in memory costs will further
accelerate this trend.
Likewise, the forthcoming dual-core processor technology
significantly benefits IT organizations by dramatically lowering
the costs of increased performance. Compared to traditional
single-core systems, systems utilizing dual-core processors will
be less expensive, since only half the number of sockets will be
required for the same number of CPUs. By significantly lowering
the cost of multi-processor systems, dual-core technology will
accelerate data center consolidation and virtual infrastructure
projects,
Beyond these enhancements, VMware is also working closely
with both Intel and AMD to ensure that new processor technology features are exploited by virtual infrastructure to the fullest
extent. In particular, the new virtualization hardware assist
enhancements (Intel’s “VT” and AMD’s “Pacifica”) will enable
robust virtualization of the CPU functionality. Such hardware
virtualization support does not replace virtual infrastructure, but
allows it to run more efficiently.
Para-virtualization
Although virtualization is rapidly becoming mainstream technology, the concept has attracted a huge amount of interest,
and enhancements continue to be investigated. One of these is
para-virtualization, whereby operating system compatibility is
traded off against performance for certain CPU-bound applications running on systems without virtualization hardware assist
(see Figure 7). The para-virtualized model offers potential performance benefits when a guest operating system or application
is ‘aware’ that it is running within a virtualized environment,
and has been modified to exploit this. One potential downside
of this approach is that such modified guests cannot ever be
migrated back to run on physical hardware.
In addition to requiring modified guest operating systems, paravirtualization leverages a hypervisor for the underlying technology. In the case of Linux distributions, this approach requires
extensive changes to an operating system kernel so that it can
coexist with the hypervisor. Accordingly, mainstream Linux
distributions (such as Red Hat or SUSE) cannot be run in a paravirtualized mode without some level of modification. Likewise,
Microsoft has suggested that a future version of the Windows
operating system will be developed that can coexist with a new
hypervisor offering from Microsoft.
Yet para-virtualization is not an entirely new concept. For
example, VMware has employed it by making available as
an option enhanced device drivers (packaged as VMware
Tools) that increase the efficiency of guest operating systems.
Furthermore, if and when para-virtualization optimizations are
eventually built into commercial enterprise Linux distributions,
VMware’s hypervisor will support those, as it does all mainstream operating systems.
�����������
�����������
���������������������
����������������
���������������������
����������������
��������������
������������
Figure 7: Para-virtualization
8
VMWARE WHITE PAPER
VMware’s Virtualization Portfolio
VMware is the only provider of high-performance virtualization
products that give customers a real choice in operating systems.
VMware supports: Windows 95/98/NT/2K/2003/XP/3.1/MS-DOS
6; Linux (Red Hat, SUSE, Mandrake, Caldera); FreeBSD (3.x, 4.04.9); Novell (NetWare 4,5,6); Sun Solaris 9 and 10 (experimental).
VMware pioneered x86-based virtualization in 1998 and
continues to be the innovator in that market, providing the
fundamental virtualization technology for all leading x86based hardware suppliers. The company offers a variety of
software-based partitioning approaches, utilizing both hosted
(Workstation and VMware Server) and hypervisor (ESX Server)
architectures. (see Figure 8)
VMware’s virtual machine (VM) approach creates a uniform
hardware image – implemented in software – on which operating systems and applications run. On top of this platform,
VMware’s VirtualCenter provides management and provisioning
of virtual machines, continuous workload consolidation across
physical servers and VMotion™ technology for virtual machine
mobility.
VirtualCenter is virtual infrastructure management software that
centrally manages an enterprise’s virtual machines as a single,
logical pool of resources. With VirtualCenter, an administrator can manage thousands of Windows NT, Windows 2000,
Windows 2003, Linux and NetWare servers from a single point
of control.
VMware is designed from the ground up to ensure compatibility with customers’ existing software infrastructure investments.
This includes not just operating systems, but also software for
management, high availability, clustering, replication, multipathing, and so on.
VMware’s hypervisor-based products and solutions have been
running at customer sites since 2001, with more than 75% of
customers running ESX Server in production deployments. As
the clear x86 virtualization market leader, VMware is uniquely
positioned to continue providing robust, supportable, highperformance virtual infrastructure for real-world, enterprise data
center applications.
Unique to VMware is the VMotion technology, whereby live,
running virtual machines can be moved from one physical
system to another while maintaining continuous service availability. VMotion thus allows fast reconfiguration and optimization of resources across the virtual infrastructure.
App
Ap
pp
App
App
OS
OS
OS
OS
CONSISTENT VIRTUAL HARDWARE PLATFORM
Open Interfaces
System Architecture
& Highlights
VMware
Infrastructure
ACE
Workstation
VMware Server
ESX Server
Secured Enterprise
Desktop
Technical
Desktop
Departmental
Computing
Enterprise
Computing
Hosted on Windows
Hosted on Windows
or Linux
Hosted on Windows
or Linux
Bare Metal
V-SMP Option
Mgmt Server,
Console & APIs
VMotion
Figure 8: Single Virtual Platform Desktop to Enterprise
9
VMWARE WHITE PAPER
Glossary
Virtual Machine
A representation of a real machine using software that provides
an operating environment which can run or host a guest operating system.
Guest Operating System
An operating system running in a virtual machine environment
that would otherwise run directly on a separate physical system.
Virtual Machine Monitor
Software that runs in a layer between a hypervisor or host operating system and one or more virtual machines that provides
the virtual machine abstraction to the guest operating systems.
With full virtualization, the virtual machine monitor exports a
virtual machine abstraction identical to a physical machine, so
that standard operating systems (e.g., Windows 2000, Windows
Server 2003, Linux, etc.) can run just as they would on physical
hardware.
Hypervisor
A thin layer of software that generally provides virtual partitioning capabilities which runs directly on hardware, but underneath higher-level virtualization services. Sometimes referred to
as a “bare metal” approach.
Hosted Virtualization
A virtualization approach where partitioning and virtualization
services run on top of a standard operating system (the host).
In this approach, the virtualization software relies on the host
operating system to provide the services to talk directly to the
underlying hardware.
Para-virtualization
A virtualization approach that exports a modified hardware
abstraction which requires operating systems to be explicitly
modified and ported to run.
Virtualization Hardware Support
Industry standard servers will provide improved hardware
support for virtualization. Initial hardware support includes
processor extensions to address CPU and some memory
virtualization. Future support will include I/O virtualization, and
eventually more complex memory virtualization management.
Hardware-level virtualization
Here the virtualization layer sits right on top of the hardware
exporting the virtual machine abstraction. Because the virtual
machine looks like the hardware, all the software written for it
will run in the virtual machine.
10
Operating system–level virtualization
In this case the virtualization layer sits between the operating
system and the application programs that run on the operating
system. The virtual machine runs applications, or sets of applications, that are written for the particular operating system being
virtualized.
High-level language virtual machines
In high-level language virtual machines, the virtualization layer
sits as an application program on top of an operating system.
The layer exports an abstraction of the virtual machine that can
run programs written and compiled to the particular abstract
machine definition. Any program written in the high-level
language and compiled for this virtual machine will run in it.
For more information:
http://www.vmware.com
http://www.vmware.com/solutions/
http://www.vmware.com/vinfrastructure/
�������������������������������������������������������������������������������������������������������
�� ������������� ����� ���� ������� ���������� ���������� ��� ���� ��� ����� ��� ����� ������� ����� ����������� ����������� �����������
����������� ����������� ����������� ����������� ����������� ����������� ����������� ����������� ���������� ���� ����������� ��������
���������������������������������������������������������������������������������������������������������������������
����������� ����� ��� ���� ������� ������� ������� ������ ��������������� ���� ������ ������ ���� ������ ���������� ������� ���� ���
�����������������������������������������
doi:10.1145/1866739 . 1 8 6 6 7 5 4
Article development led by
queue.acm.org
Managing virtualization at a large scale is
fraught with hidden challenges.
by Evangelos Kotsovinos
Virtualization:
Blessing
or Curse?
touted as the solution
to many challenging problems, from resource
underutilization to data-center optimization and
carbon emission reduction. However, the hidden costs
of virtualization, largely stemming from the complex
and difficult system administration challenges it
V ir t ua lizati on is o f t en
poses, are often overlooked. Reaping
the fruits of virtualization requires
the enterprise to navigate scalability
limitations, revamp traditional operational practices, manage performance,
and achieve unprecedented cross-silo
collaboration. Virtualization is not a
curse: it can bring material benefits,
but only to the prepared.
Al Goodman once said, “The perfect
computer has been invented. You just
feed in your problems and they never
come out again.” This is how virtualization has come to be perceived in recent
years: as a panacea for a host of IT problems. Bringing virtualization into the
enterprise is often about reducing costs
without compromising quality of service. Running the same workloads as
virtual machines (VMs) on fewer servers can improve server utilization and,
perhaps more importantly, allow the
deferral of data-center build-outs—the
same data-center space can now last
longer.
Virtualization is also meant to enhance the manageability of the enterprise infrastructure. As virtual servers
and desktops can be live-migrated with
no downtime, coordinating hardware
upgrades with users or negotiating
work windows is no longer necessary—
upgrades can happen at any time with
no user impact. In addition, high availability and dynamic load-balancing solutions provided by virtualization product families can monitor and optimize
the virtualized environment with little
manual involvement. Supporting the
same capabilities in a nonvirtualized
world would require a large amount of
operational effort.
Furthermore, enterprises use virtualization to provide IaaS (Infrastruc-
ja n ua ry 2 0 1 1 | vo l . 5 4 | n o. 1 | c o m m u n i c at i o n s o f t he acm
61
practice
ture as a Service) cloud offerings that
give users access to computing resources on demand in the form of VMs.
This can improve developer productivity and reduce time to market, which is
key in today’s fast-moving business
environment. Since rolling out an application sooner can provide first-mover advantage, virtualization can help
boost the business.
The Practice
Although virtualization is a 50-year-old
technology,3 it reached broad popularity only as it became available for the
x86 platform from 2001 onward—and
most large enterprises have been using the technology for fewer than five
years.1,4 As such, it is a relatively new
technology, which, unsurprisingly, carries a number of less-well-understood
system administration challenges.
Old Assumptions. It is not, strictly
speaking, virtualization’s fault, but
many systems in an enterprise infrastructure are built on the assumption
of running on real, physical hardware.
The design of operating systems is
often based on the principle that the
hard disk is local, and therefore reading from and writing to it is fast and
low cost. Thus, they use the disk generously in a number of ways, such as
caching, buffering, and logging. This,
of course, is perfectly fair in a nonvirtualized world.
With virtualization added to the
mix, many such assumptions are
turned on their heads. VMs often use
shared storage, instead of local disks,
to take advantage of high availability
and load-balancing solutions—a VM
with its data on the local disk is a lot
more difficult to migrate, and doomed
if the local disk fails. With virtualization, each read and write operation
travels to shared storage over the network or Fiber Channel, adding load
to the network interface controllers
(NICs), switches, and shared storage
systems. In addition, as a result of consolidation, the network and storage
infrastructure has to cope with a potentially much higher number of systems,
compounding this effect. It will take
years for the entire ecosystem to adapt
fully to virtualization.
System Sprawl. Conventional wisdom has it that the operational workload of managing a virtualized server
62
communications of th e ac m
running multiple VMs is similar to that
of managing a physical, nonvirtualized
server. Therefore, as dozens of VMs can
run on one virtualized server, consolidation can reduce operational workload. Not so: the workload of managing a physical, nonvirtualized server is
comparable to that of managing a VM,
not the underlying virtualized server.
The fruits of common, standardized
management—such as centrally held
configuration and image-based provisioning—have already been reaped by
enterprises, as this is how they manage
their physical environments. Therefore, managing 20 VMs that share a
virtualized server requires the same
amount of work as managing 20 physical servers. Add to that the overhead of
managing the hypervisor and associated services, and it is easy to see that
operational workload will be higher.
More importantly, there is evidence
that virtualization leads to an increase
in the number of systems—now running in VMs—instead of simply consolidating existing workloads.2,5 Making it easy to get access to computing
capacity in the form of a VM, as IaaS
clouds do, has the side effect of leading
to a proliferation of barely used VMs,
since developers forget to return the
VMs they do not use to the pool after
the end of a project. As the number of
VMs increases, so does the load placed
on administrators and on shared infrastructure such as storage, Dynamic
Host Configuration Protocol (DHCP),
and boot servers.
Most enterprise users of virtualization implement their own VM reclamation systems. Some solutions are
straightforward and borderline simplistic: if nobody has logged on for
more than three months, then notify
and subsequently reclaim if nobody
objects. Some solutions are elaborate
and carry the distinctive odor of overengineering: analyze resource utilization over a period of time based on
heuristics; determine level of usage;
and act accordingly. Surprising as it
may be there is a lack of generic and
broadly applicable VM reclamation
solutions to address sprawl challenges. In addition, services that are common to all VMs sharing a host—such
as virus scanning, firewalls, and backups—should become part of the virtualization layer itself. This has already
| ja n ua ry 2 0 1 1 | vo l . 5 4 | n o. 1
started happening with such services
entering the hypervisor, and it has the
potential to reduce operational workload substantially.
Scale. Enterprises have spent years
improving and streamlining their management tools and processes to handle
scale. They have invested in a backbone of configuration management
and provisioning systems, operational
tools, and monitoring solutions that
can handle building and managing
tens or even hundreds of thousands of
systems. Thanks to this—largely home-
grown—tooling, massively parallel operational tasks, such as the build-out
of thousands of servers, daily operating
system checkouts, and planned datacenter power-downs, are routine and
straightforward for operational teams.
Enter virtualization: most vendor
solutions are not built for the large enterprise when it comes to scale, particularly with respect to their management
frameworks. Their scale limitations
are orders of magnitude below those
of enterprise systems, often because
of fundamental design flaws—such as
overreliance on central components or
data sources. In addition, they often do
not scale out; running more instances
of the vendor solution will not fully address the scaling issue, as the instances
will not talk to each other. This chal-
practice
lenge is not unique to virtualization. An
enterprise faces similar issues when it
introduces a new operating system to its
environment. Scaling difficulties, however, are particularly important when it
comes to virtualization for two reasons:
first, virtualization increases the number of systems that must be managed,
as discussed in the section on system
sprawl; second, one of the main benefits
of virtualization is central management
of the infrastructure, which cannot be
achieved without a suitably scalable
management framework.
tion and size of operational teams.
Interoperability. Many enterprises
have achieved a good level of integration between their backbone systems.
The addition of a server in the configuration-management system allows
it to get an IP address and host name.
The tool that executes a power-down
draws its data about what to power
off seamlessly from the configurationmanagement system. A change in a
server’s configuration will automatically change the checkout logic applied
to it. This uniformity and tight integra-
physical infrastructure.
To be sure, some enterprises are fortunate enough to have a homogeneous
environment, managed by a product
suite for which solid virtualization extensions already exist. In a heterogeneous infrastructure, however, with
more than one virtualization platform,
with virtualized and nonvirtualized
parts, and with a multitude of tightly
integrated homegrown systems, the
introduction of virtualization leads to
administration islands—parts of the
infrastructure that are managed differ-
As a result, enterprises are left with
a choice: either they live with a multitude of frameworks with which to
manage the infrastructure, which increases operational complexity; or they
must engineer their own solutions that
work around those limitations—for example, the now open source Aquilon
framework extending the Quattor toolkit (http://www.quattor.org). Another
option is for enterprises to wait until
the vendor ecosystem catches up with
enterprise-scale requirements before
they virtualize. The right answer depends on a number of factors, including the enterprise’s size, business
requirements, existing backbone of
systems and tools, size of virtualized
and virtualizable infrastructure, engineering capabilities, and sophistica-
tion massively simplifies operational
and administrative work.
Virtualization often seems like an
awkward guest in this tightly integrated enterprise environment. Each virtualization platform comes with its own
APIs, ways of configuring, describing,
and provisioning VMs, as well as its
own management tooling. The vendor ecosystem is gradually catching
up, providing increased integration
between backbone services and virtualization management. Solutions are
lacking, however, that fulfill all three
of the following conditions:
˲˲ They can be relatively easily integrated with homegrown systems.
˲˲ They can handle multiple virtualization platforms.
˲˲ They can manage virtual as well as
ently from everything else. This breaks
the integration and uniformity of the
enterprise environment, and increases
operational complexity.
Many enterprises will feel like they
have been here before—for example,
when they engineered their systems to
be able to provision and manage multiple operating systems using the same
frameworks. Once again, customers
face the “build versus suffer” choice.
Should they live with the added operational complexity of administration
islands until standardization and convergence emerge in the marketplace,
or should they invest in substantial
engineering and integration work to
ensure hypervisor agnosticism and integration with the existing backbone?
Troubleshooting. Contrary to con-
ja n ua ry 2 0 1 1 | vo l . 5 4 | n o. 1 | c o m m u n i c at i o n s o f t he acm
63
practice
ventional wisdom, virtualized environments do not really consolidate three
physical machines into one physical
machine; they consolidate three physical machines onto several physical subsystems, including the shared server,
the storage system, and the network.
Finding the cause of slowness in
a physical computer is often a case of
glancing at a few log files on the local
disk and potentially investigating local
hardware issues. The amount of data
that needs to be looked at is relatively
small, contained, and easily found.
Monitoring performance and diagnosing a problem of a virtual desktop,
on the other hand, requires trawling
through logs and data from a number
of sources including the desktop operating system, the hypervisor, the storage system, and the network.
In addition, this large volume of
disparate data must be aggregated or
linked; the administrator should be
able to obtain information easily from
all relevant systems for a given time period, or to trace the progress of a specific packet through the storage and
network stack. Because of this multisource and multilayer obfuscation,
resolution will be significantly slower
if administrators have to look at several screens and manually identify bits
of data and log files that are related, in
terms of either time or causality. New
paradigms are needed for storing, retrieving, and linking logs and performance data from multiple sources.
Experience from fields such as Web
search can be vital in this endeavor.
Silos? What Silos? In a nonvirtualized enterprise environment, responsibilities for running different parts of
the infrastructure are neatly divided
among operational teams, such as
Unix, Windows, network, and storage operations. Each team has a clear
scope of responsibility, communication among teams is limited, and apportioning credit, responsibility, and
accountability for infrastructure issues
is straightforward.
Virtualization bulldozes these silo
walls. Operational issues that involve
more than one operational team—and,
in some cases, all—become far more
common than issues that can be resolved entirely within a silo. As such,
cross-silo collaboration and communication are of paramount importance,
64
communications of th e ac m
Virtualization
holds promise
as a solution for
many challenging
problems.
Expectations
are running high.
Can virtualization
deliver?
| ja n ua ry 2 0 1 1 | vo l . 5 4 | n o. 1
requiring a true mentality shift in the
way enterprise infrastructure organizations operate—as well as, potentially,
organizational changes to adapt to this
requirement.
Impact of Changes. Enterprises
have spent a long time and invested
substantial resources into understanding the impact of changes to different
parts of the infrastructure. Changemanagement processes and policies
are well oiled and time tested, ensuring
that every change to the environment is
assessed and its impact documented.
Once again, virtualization brings
fundamental change. Sharing the infrastructure comes with centralization
and, therefore, with potential bottlenecks that are not as well understood.
Rolling out a new service pack that increases disk utilization by 5IOPS (input/output operations per second) on
each host will have very little impact in
a nonvirtualized environment—each
host will be using its disk a little more
often. In a virtualized environment, an
increase of disk usage by 5IOPS per VM
will result in an increase of 10,000IOPS
on a storage system shared by 2,000
VMs, with potentially devastating consequences. It will also place increased
load on the shared host, as more
packets will have to travel through the
hypervisor, as well as the network infrastructure. We have seen antivirus
updates and operating-system patches
resulting in increases in CPU utilization on the order of 40% across the
virtualized plant—changes that would
have a negligible effect when applied to
physical systems.
Similarly, large-scale reboots can
impact shared infrastructure components in ways that are radically different from the nonvirtualized past.
Testing and change management processes need to change to account for
effects that may be much broader than
before.
Contention. Virtualization platforms do a decent job of isolating VMs
on a shared physical host and managing resources on that host (such as CPU
and memory). In a complex enterprise
environment, however, this is only part
of the picture. A large number of VMs
will be sharing a network switch, and
an even larger number of VMs will be
sharing a storage system. Contention
on those parts of the virtualized stack
practice
can have as much impact as contention
on a shared host, or more. Consider
the case where a rogue VM overloads
shared storage: hundreds or thousands
of VMs will be slowed down.
Functionality that allows isolating and managing contention when it
comes to networking and storage elements is only now reaching maturity
and entering the mainstream virtualization scene. Designing a virtualization technology stack that can take
advantage of such features requires
engineering work and a good amount
of networking and storage expertise
on behalf of the enterprise customer.
Some do that, combining exotic network adapters that provide the right
cocktail of I/O virtualization in hardware with custom rack, storage, and
network designs. Some opt for the
riskier but easier route of doing nothing special, hoping that system administrators will cope with any contention
issues as they arise.
GUIs. Graphical user interfaces
work well when managing an email
inbox, data folder, or even the desktop
of a personal computer. In general, it
is well understood in the human-computer interaction research community
that GUIs work well for handling a relatively small number of elements. If that
number gets large, GUIs can overload
the user, which often results in poor
decision making.7 Agents and automation have been proposed as solutions
to reduce information overload.6
Virtualization solutions tend to
come with GUI-based management
frameworks. That works well for managing 100 VMs, but it breaks down in
an enterprise with 100,000 VMs. What
is really needed is more intelligence
and automation; if the storage of a virtualized server is disconnected, automatically reconnecting it is a lot more
effective than displaying a little yellow
triangle with an exclamation mark in
a GUI that contains thousands of elements. What is also needed is interoperability with enterprise backbones
and other systems, as mentioned previously.
In addition, administrators who are
accustomed to the piecemeal systems
management of the previrtualization
era—managing a server here and a
storage element there—will discover
they will have to adapt. Virtualiza-
tion brings unprecedented integration and hard dependencies among
components—a storage outage could
mean that thousands of users cannot
use their desktops. Enterprises need
to ensure that their operational teams
across all silos are comfortable with
managing a massively interconnected
large-scale system, rather than a collection of individual and independent
components, without GUIs.
Conclusion
Virtualization holds promise as a solution for many challenging problems. It
can help reduce infrastructure costs,
delay data-center build-outs, improve
our ability to respond to fast-moving
business needs, allow a massive-scale
infrastructure to be managed in a more
flexible and automated way, and even
help reduce carbon emissions. Expectations are running high.
Can virtualization deliver? It absolutely can, but not out of the box. For
virtualization to deliver on its promise,
both vendors and enterprises need to
adapt in a number of ways. Vendors
must place strategic emphasis on enterprise requirements for scale, ensuring that their products can gracefully handle managing hundreds of
thousands or even millions of VMs.
Public cloud service providers do this
very successfully. Standardization,
automation, and integration are key;
eye-pleasing GUIs are less important.
Solutions that help manage resource
contention end to end, rather than only
on the shared hosts themselves, will
significantly simplify the adoption of
virtualization. In addition, the industry’s ecosystem needs to consider the
fundamental redesign of components
that perform suboptimally with virtualization, and it must provide better ways
to collect, aggregate, and interpret logs
and performance data from disparate
sources.
Enterprises that decide to virtualize strategically and at a large scale
need to be prepared for the substantial
engineering investment that will be
required to achieve the desired levels
of scalability, interoperability, and operational uniformity. The alternative
is increased operational complexity
and cost. In addition, enterprises that
are serious about virtualization need a
way to break the old dividing lines, fos-
ter cross-silo collaboration, and instill
an end-to-end mentality in their staff.
Controls to prevent VM sprawl are key,
and new processes and policies for
change management are needed, as
virtualization multiplies the effect of
changes that would previously be of
minimal impact.
Virtualization can bring significant
benefits to the enterprise, but it can
also bite the hand that feeds it. It is no
curse, but, like luck, it favors the prepared.
Acknowledgments
Many thanks to Mostafa Afifi, Neil Allen, Rob Dunn, Chris Edmonds, Robbie Eichberger, Anthony Golia, Allison Gorman Nachtigal, and Martin
Vazquez for their invaluable feedback
and suggestions. I am also grateful to
John Stanik and the ACM Queue Editorial Board for their feedback and guidance in completing this article.
Related articles
on queue.acm.org
Beyond Server Consolidation
Werner Vogels
http://queue.acm.org/detail.cfm?id=1348590
CTO Roundtable: Virtualization
http://queue.acm.org/detail.cfm?id=1508219
The Cost of Virtualization
Ulrich Drepper
http://queue.acm.org/detail.cfm?id=1348591
References
1. Bailey, M., Eastwood, M., Gillen, A., Gupta, D.
Server virtualization market forecast and analysis,
2005–2010. IDC, 2006.
2. Brodkin, J. Virtual server sprawl kills cost savings,
experts warn. NetworkWorld. Dec. 5, 2008.
3. Goldberg, R.P. Survey of virtual machine research.
IEEE Computer Magazine 7, 6 (1974), 34–45.
4. Humphreys, J. Worldwide virtual machine software
2005 vendor shares. IDC, 2005.
5. IDC. Virtualization market accelerates out of the
recession as users adopt “Virtualize First” mentality;
2010.
6. Maes, P. Agents that reduce work and information
overload. Commun. ACM 37, 7 (1994), 30–40.
7. Schwartz, B. The Paradox of Choice. HarperCollins, NY,
2005.
Evangelos Kotsovinos is a vice president at Morgan
Stanley, where he leads virtualization and cloudcomputing engineering. His areas of interest include
massive-scale provisioning, predictive monitoring,
scalable storage for virtualization, and operational tooling
for efficiently managing a global cloud. He also serves
as the chief strategy officer at Virtual Trip, an ecosystem
of dynamic start-up companies, and is on the Board
of Directors of NewCred Ltd. Previously, Kotsovinos
was a senior research scientist at T-Labs, where he
helped develop a cloud-computing R&D project into a
VC-funded Internet start-up. A pioneer in the field of
cloud computing, he led the XenoServers project, which
produced one of the first cloud-computing blueprints.
© 2011 ACM 0001-0782/11/0100 $10.00
ja n ua ry 2 0 1 1 | vo l . 5 4 | n o. 1 | c o m m u n i c at i o n s o f t he acm
65
Purchase answer to see full
attachment