Snapshots are not backups! or VDP and YOU

Ominous words would be echoed in the meeting… “You do have a BACKUP right?”


Working in production environments the constant challenge of maintaining uptime aka ‘steady-state’ but at the same slowly or as quick as feasible move forward with changing demands of the business.

Change can came in many forms. It is a driver for your organization.

A simple response to a vulnerability; patching is a necessity.

New features are required. Upgrades will be needed.

And more importantly disaster avoidance. The idea is to prepare in advance avoid disaster. It is akin to shift and dodge BEFORE some bump comes in the road. There are many approaches to this like having a stretched geo-location metro cluster.

Whatever the driver you have to have a fallback plan. If the post-change activity fails, if there is an unforeseen after-effect.. Things do not always work 100% as planned. What is your fallback plan? What? You have a VMware environment. You did click the snapshot button.. Well, that does work but it isn’t a full backup

From KB 1025279

  • Snapshots are not backups. A snapshot file is only a change log of the original virtual disk.
  • Snapshots are not complete copies of the original vmdk disk files….it only copies the delta disks. The change log in the snapshot file combines with the original disk files to make up the current state of the virtual machine. If the base disks are deleted, the snapshot files are useless.
  • Delta files can grow to the same size as the original base disk file, which is why the provisioned storage size of a virtual machine increases by an amount up to the original size of the virtual machine multiplied by the number of snapshots on the virtual machine.
  • The maximum supported amount of snapshots in a chain is 32. However, VMware recommends that you use only 2-3 snapshots in a chain. — [ed The reason is there is a performance hit]

In fact VMware recommendation is to setup an alarm in vcenter if the VM is running from a snapshot to avoid this condition

See KB 1018029 “Configuring VMware vCenter Server to send alarms when virtual machines are running from snapshots”

Now the question still remains.. What options do you have?

Well there is good news!! VMware as of March 1, 2015. “VMware vSphere Data Protection Advanced will be consolidated into VMware vSphere Data Protection (available through vSphere Essentials Plus Kit or higher vSphere editions, all vSphere with Operations Management editions and all vCloud Suite editions) and will no longer require purchase of a separate license. All functionality available with vSphere Data Protection Advanced, previously available as a standalone product, is now included in VMware vSphere Data Protection 6.0 – See more at: Announcement


Why is this cool? There are many reasons but to sum things up.

VMware Data Protection Advanced (VDP) is very cool. It is based on modern backup solutions.

  • There are no tapes
  • There is deduplication – Variable length up
  • There is replication
  • File recovery
  • VM recovery
  • Application aware backups
  • Efficient, bandwidth throttling
  • Changed Block Tracking (CBT) Restore
    vSphere Data Protection uses Changed Block Tracking (CBT) during image-level backups. CBT is also utilized with image-level restores in some cases to improve speed and efficiency
  • It can plug into something really big (Data Domain and Avamar)
    • Data Domain allows for “Consolidate backup, archive, and disaster recovery with high-speed deduplication”
    • Avamar  DEDUPLICATION BACKUP SOFTWARE AND SYSTEM — VDP is based on Avamar. See the announcement.

It is super easy to install and use. I did say easy and it is, because you can even configure VDP to allow for.

  • Linux-based virtual appliance: Easily install and configure backups.
  • Self-Service File Level Recovery: Enable guest OS administrators to restore individual files and folders.
  • Wizard-driven backup policies: Assign backup jobs to individual virtual machines or larger containers such as a cluster or resource pool, with specific schedules and retention policies.
  • There is no need for agents in the VM for normal backups.
  • Application aware backups. Backup agents for Microsoft SQL Server, Exchange, and SharePoint. The agents enable application consistent backup and recovery of these applications on virtual and physical machines

– See more at:

Some tips about VDP deployment.

Do not put the all your eggs in the same basket

— Don’t setup your backup volumes in the same datastore your VMs reside in. The option to use Data Domain is great option! Data Domain can be data backup target.

DNS. Have it working!

It is fast enough? Avoid problems and run a performance test before your backups. Make sure your Data backup targets are validated for performance.

Initial configuration is deployment via OVF. 

Log on via the https://ip-address-assigned/:8543/vdp-configure/

Here you log in as root/changeme


BUT if you need to ssh in later via IP or hostname. You cannot use the root account. You must use the admin account, which has the same password then you su to root.

“Currently, users can access the VDP appliance command line using the vSphere Client console, SSH, or Putty sessions. With the VDP 5.8 and later releases, the ability to use SSH or Putty to log on to the VDP appliance with the root user has been removed.” — Administration Guide

and lastly

VAMI is your friend and so is the log.

The VAMI is: Virtual Appliance Management Infrastructure (VAMI). VAMI provides end‐users of virtual appliances with a Web console and command line interface that can:

  • „Configure network settings
  • „Check for updates and install them, manually or automatically„
  • Review basic system information for the virtual appliance
  • Stop or restart the virtual appliance

Where is the magical vami?

From the command line you can find it here: /opt/vmware/share/vami/


and if you run into problems..

Log in via ssh to the vdp appliance. Run the following while you attempt the action where you see the error.

root@vdp01:~/#: tail -f /usr/local/avamar/var/vdr/server_logs/vdr-server.log

Then watch the log and try to reproduce the error.

Additional Resources:

Here is a great overview from the VMware HOL team!

VDP overview install and backups! VDP DEMO

and more VDP feature walk through DEMO

and learn how to:

  1. Creating a Virtual Machine Backup Job
  2. Creating a Replication Job
  3. Creating an Application Backup Job
  4. File Level Restore
  5. Restoring a Virtual Machine
  6. Restoring an Application


part 2/2 Troubleshooting VSAN errors. VSAN misconfiguration??

VSAN cluster isn’t 100% there is some problems with see all the storage. 
1. Cannot write to the VSAN Datastore
2. Correct amount of capacity isn’t present
How to resolve with ESXCLI
Log into each host via ssh. You do remember the root password right?? This example is a three node VSAN cluster.
node 1::::::
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-31T01:11:33Z
Local Node UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Backup UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member UUIDs: 551374b5-03f9-7bd6-6257-a0369f58b8e8, 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster Membership UUID: d6da1955-e2f8-38eb-d7f0-a0369f58b8e8
NODE 2:::::
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-31T00:12:02Z
   Local Node UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
   Local Node State: MASTER  << a different master for a different UUID
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4  <<< That is a different UUID!
   Sub-Cluster Backup UUID:
Sub-Cluster UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member UUIDs: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Membership UUID: 60df1955-b9f1-d685-13b0-a0369f58b8e4
~ # esxcli vsan cluster leave
~ # esxcli vsan cluster join -u 551374b5-03f9-7bd6-6257-a0369f58b8e8   <<<- join the correct UUID (cluster)
Validate on NODE 2
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-31T00:12:52Z
Local Node UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Backup UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member UUIDs: 551374b5-03f9-7bd6-6257-a0369f58b8e8, 54f9dc6f-8674-f412-364d-a0369f58b5a8, 55197cee-f530-4966-5ea6-a0369f58b8e4  <<< three members
Sub-Cluster Membership UUID: d6da1955-e2f8-38eb-d7f0-a0369f58b8e8
~ # esxcli vsan network list
VmkNic Name: vmk1
IP Protocol: IPv4
Interface UUID: 17dd1955-0bdf-abac-aba6-a0369f58b8e4
Agent Group Multicast Address:
Agent Group Multicast Port: 23451
   Master Group Multicast Address:
Master Group Multicast Port: 12345
Multicast TTL: 5
Valdiate on NODE 3 and 1
NODE 3::::::
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-31T00:14:33Z
Local Node UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Backup UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member UUIDs: 551374b5-03f9-7bd6-6257-a0369f58b8e8, 54f9dc6f-8674-f412-364d-a0369f58b5a8, 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Membership UUID: d6da1955-e2f8-38eb-d7f0-a0369f58b8e8
~ # esxcli vsan network list
VmkNic Name: vmk1
IP Protocol: IPv4
Interface UUID: f16e1355-c174-11f6-2602-a0369f58b5a8
Agent Group Multicast Address:
Agent Group Multicast Port: 23451
Master Group Multicast Address:
Master Group Multicast Port: 12345
   Multicast TTL: 5
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-31T01:12:22Z
Local Node UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Backup UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
   Sub-Cluster Membership Entry Revision: 2
   Sub-Cluster Member UUIDs: 551374b5-03f9-7bd6-6257-a0369f58b8e8, 54f9dc6f-8674-f412-364d-a0369f58b5a8, 55197cee-f530-4966-5ea6-a0369f58b8e4 <<— all three members!
   Sub-Cluster Membership UUID: d6da1955-e2f8-38eb-d7f0-a0369f58b8e8
~ # esxcli vsan network list
VmkNic Name: vmk1
IP Protocol: IPv4
Interface UUID: 3b5e1955-5eb6-2bbc-57bc-a0369f58b8e8
Agent Group Multicast Address:
   Agent Group Multicast Port: 23451
   Master Group Multicast Address: <<< all the same Multicast address
   Master Group Multicast Port: 12345
Multicast TTL: 5
~ #

Technical notes on IP Based Storage Guide for EMC VNX and VMware 5.5

When it comes to IP based storage there are two major choices for use in VMware environments. NFS and iSCSI

This article will discuss iSCSI options.

NFS is a very valid design choice with very flexible options for deployment. NFS for use with VMware is Fast and Flexible in other words a solid choice. But NFS is for another time and a different discussion.

Why IP Based storage??

In a single word: SPEED and Flexibility. Okay two words. Core networking speeds are no longer limited by 10baseT, 100baseT but 10 Gigabit Ethernet is more so the standard. You already have IP based network, why not try to leverage what is installed?

I have seen greater interest in deployment of iSCSI based storage for VMware lately. Not just 1 gb but 10 ‎Gigabit Ethernet is gaining more of a foothold for DataCenters, as customer upgrade their core networking from 1 Gb to 10Gb. The potential for performance gains is really a good thing, but fundamental to deployment is to consider a solid network design.

What I mean is end-to-end. What kind of switch are you using? Are you limited by a certain number of 10 gb ports? How many switches do you have. Do you have a single point of failure? This is more critical as you will be really leveraging your network for “double duty”. Instead of a discrete network designated for storage such as FC can provide you will now run storage information across your IP network. Ideally two separate physical switches are ideal. BUT at a minimum use VLANS for logical separation. And let me go ahead and say it… “VLAN 0 (zero) is a not really a scalable enterprise option!” That is a huge red flag you will require more network analysis and work to deploy a IP based storage solution.

There are many considerations for a successful iSCSI implementation.

1) Gather Customer Requirements 2) Design  3) Verify 4) Implement/ Deploy 5) TEST User Acceptance Testing

Ideally having two 10Gb switches for redundancy is a good thing! Be careful in the selection of a Enterprise grade switch. Have seen horrible experience when improper features are not enabled. i.e. flow control can be a good thing!

Software based iSCSI initiators. Don’t forget about Delayed ACK. See Sean’s excellent article here: Delayed ACK setting with VNX. Read more VMware details about how to implement Delayed ACK from KB1002598

“Modify the delayed ACK setting on a discovery address (recommended):

  1. On a discovery address, click the Dynamic Discovery tab.
  2. Click the Server Address tab.
  3. Click Settings > Advanced.”

Double, no TRIPLE check the manufacture recommended CABLE TYPE and LENGTH. For example: Does your 10GbE use fiber optic cable? Do you have the correct type? What about the SFP?  And if you are not using fiber optic, but choose to use TwinAX cabling. Do you have the correct cable as per manufacture requirements?

For example: Meraki, only makes and supports a 1 Meter 10g Passive Copper Cable for their switches.  If you look at any normal Cisco Business Class switch they support 1, 3, 5 meter passive and 7, 10 meter active cables on their switches. 

Active cables are generally more expensive, but could be a requirement depending on your datacenter and or colocation layout.

I try to approach the solution from both ends. Storage to the Host and the reverse Host to the storage.  Examine end-to-end dependancies. Even though your area of responsibility isn’t the network, you will be reliant on the network services and any misconfiguration will impact your ability to meet the design requirements stated. You may not have bought or had any input to the existing infrastructure but you will be impacted by what is there currently in use. How will you Keyword: Interoperability how each independent system will interact with another system. Upstream and downstream dependencies.

Other considerations:

For example: vmkernel port binding: The diagram below is from VMware KB 2038869 “Considerations for iSCSI Port Binding”

Port binding is used in iSCSI when multiple VMkernel ports for iSCSI reside in the same broadcast domain and IP subnet to allow multiple paths to an iSCSI array that broadcasts a single IP address. When using port binding, you must remember that:

  • Array Target iSCSI ports must reside in the same broadcast domain and IP subnet as the VMkernel port.
  • All VMkernel ports used for iSCSI connectivity must reside in the same broadcast domain and IP subnet.
  • All VMkernel ports used for iSCSI connectivity must reside in the same vSwitch.
  • Currently, port binding does not support network routing.”


While there isn’t FC zoning for IP based storage there will be a requirement for subletting and VLAN separation.

For VNX here are some design considerations for iSCSI design:

The following points are best practices for connecting iSCSI hosts to a CLARiiON or VNX:

  • iSCSI subnets must not overlap the management port or Service LAN (128.221.252.x).

  • For iSCSI, there is no zoning (unlike an FC SAN) so separate subnets are used to provide redundant paths to the iSCSI ports on the CLARiiON array. For iSCSI you should have mini-SANs (VLANs) with only one HBA per host in each VLAN with one port per storage processor (SP) (for example, A0 and  B0 in one VLAN, A1 and  B1 in another).  All connections from a single server to a single storage system must use the same interface type, either NIC or HBA, but not both.

  • It is a good practice to create a separate, isolated IP network/VLAN for the iSCSI subnet. This is because the iSCSI data is unencrypted and also having an iSCSI-only network makes troubleshooting easier.

  • If the host has only a single NIC/HBA, then it should connect to only one port per SP. If there are more NICs or HBAs in the host, then each NIC/HBA can connect to one port from SP A and one port from SP B. Connecting more SP ports to a single NIC can lead to discarded frames due to the NIC being overloaded.

  • In the iSCSI initiator, set a different “Source IP” value for each iSCSI connection to an SP.  In other words, make sure that each NIC IP address only appears twice in the host’s list of iSCSI Source IP addresses: once for a port on SP A and once for a port on SP B.

  • Make sure that the Storage Process Management ports do not use the same subnets as the iSCSI ports – see [Link Error:UrlName “emc235739-Changing-configuration-on-one-iSCSI-port-may-cause-I-O-interruption-to-all-iSCSI-ports-on-this-storage-processor-SP-if-using-IP-addresses-from-same-subnet” not found] for more information.

  • It is also a best practice to use a different IP switch for the second iSCSI port on each SP. This is to prevent the IP switch being a single point of failure. In this way, were one IP switch to completely fail, the host can failover (via PowerPath) to the paths on the other IP switch. In the same way, it would be advisable to use different switches for multiple IP connections in the host.

  • Gateways can be used, but the ideal configuration is for HBA to be on the same subnet as one SP A port and one SP B port, without using the gateway.

For example, a typical configuration for the iSCSI ports on a CLARiiON, with two iSCSI ports per SP would be:

A0: (Subnet mask
A1: (Subnet mask
B0: (Subnet mask
B1: (Subnet mask

A host with two NICs should have its connections configured similar to the following in the iSCSI initiator to allow for load balancing and failover:

NIC1 (for example, – SP A0 and SP B0 iSCSI connections
NIC2 (for example, – SP A1 and SP B1 iSCSI connections

Similarly, if there were four iSCSI ports per SP, four subnets would be used. Half of the hosts with two HBA would then use the first two subnets, and the rest would use the other two.

The management ports should also not overlap the iSCSI ports. As the iSCSI network is normally separated from the LAN used to manage the SP, this is rarely an issue, but to follow the example iSCSI addresses above, the other IP used by the array could be as following examples:

VNX Control Station 1:
VNX Control Station 2:
SP A management IP address:
SP B management IP address:

The High Availability Validation Tool will log an HAVT warning if it detects that a host is connected via a single iSCSI Initiator. Even if the initiator has a path to both SP’s it is still at HA risk from a host connectivity view. You will also see this if using unlicensed PowerPath.

Caution! Do not use the IP address range 192.168.1.x because this is used by the serial port PPP connection

Oh.. I haven’t even discussed VMware storage path policy, as that would really be dependent on your array. However VNX is ALUA 4 and RoundRobin works really well! If you don’t have or want PowerPath as an option!


VMware Storage Guide 5.5 (PDF)

VMware Storage Guide 6.0 (PDF)

“Best Practices for Running VMware vSphere on iSCSI” (TECHNICAL MARKETING DOCUMENTATION v 2.0A)

“Using VNX Storage with VMware vSphere” EMC TechBook

How to “fix” VCSA IP settings from command line.

More and more often customers are looking for an easier method to deploy their vsphere management.

Vcenter traditionally has been an application loaded on top of Windows. .. but “the times they are a changing”

There are more use cases that the business requirements will allow for deployment of vcenter appliance.

But here is a quick post to help you “fix” your IP configuration for your appliance. Sometimes during the deploy of the VCSA OVA there is a miss communication or fat finger incident.. Here is how to address that.

It also allows you to change hostname, DNS, default gateway and proxy.


Open a console session of the VCSA
Login as: root
Default password is: vmware
Execute the following command: /opt/vmware/share/vami/vami_config_net


 Main Menu

0)    Show Current Configuration (scroll with Shift-PgUp/PgDown)
1)    Exit this program
2)    Default Gateway
3)    Hostname
4)    DNS
5)    Proxy Server
6)    IP Address Allocation for eth0

After executing the command, a menu is displayed. Within the menu It is possible to change the IP address, hostname, DNS, Default gateway and proxy server.
After allocating a static IP Address to the VCSA, the post configuration can be done by using the following URL:



VCSA was powered on.

ping was not responsive

Verified IP address

cat /etc/sysconfig/networking/devices/ifcfg-eth0 showed

cat /etc/sysconfig/networking/devices/ifcfg-eth0

VExpert 2015 announced


vExpert 2015

VMware has announced the vExpert list for 2015. Each year I read the FANTASTIC information shared by all the vExperts and I am always learning something new. This year I made a huge effort to share more and do more to vocalize all things VMware to educate my customers and wow.. I am truly honored that I was included in the list this year.

Thank you everyone for all your support!


VFRC sizing considerations SSD quandry

Sure you know about it.. I am sure you have read about it.. but you even know how to enable it.
But are you sizing it correctly??
Recently a customer I was working with wanted to boot their ESX blades from RAID1 SSD drives!
Wow cool, but not cool. Let me explain…

ESXi is tiny relative to any other OS. Since this hypervisor is so small and loads entirely into memory when it boots, I explained that using the SSD for other options would be more beneficial. Using the SSD for boot would not really be taking advantage of what SSD can do for your Virtualized environment.

I didn’t go as far as to suggest using a stateless auto deploy boot system.. but something a bit easier to manage. Of course they could have used boot from SAN, but based on the reliability of their environment they wanted something a bit more independent.. flash devices fit their use case perfectly.

Now about those SSD drives.. what to do.. what to do..?

This particular customer has FC backed storage and blades so a hyper-converged solution like VSAN wasn’t a good fit. Yes 10G was available but the question was how to maximize the use of the SSD. Disclaimer I didn’t design the components of the compute nodes and local storage, but I wanted to make the most of it..
Queue in…. Enterprise Plus licensing and its additional licensed features.

“vSphere Flash Read Cache”

Some quick bullet items to summarize.

  • Hypervisor-based software defined flash storage tier solution
    • Cache is a high-speed memory that can be either a reserved section of main memory or a storage device
    • Supports Write-Through Cache Mode
    • Improve VMs performance by leveraging local flash devices
    • Ability to virtualize business critical applications
  • Aggregates local flash devices to provide a clustered flash resource for VM and vSphere host consumption
  • Leverages local flash devices as cache
  • Integrated with vCenter, HA, DRS and vMotion
    • VFFS Volume is created
    • Fine-grained control via per VMDK caching
    • Block Size configuration capabilities Min 4KB – Max 1024KB
    • Virtual Flash Host Swap Cache
      • Provides the ability to use virtual flash resource for memory swapping
      • Legacy support for Swap-to-SSD
    • Scale-out Storage Capability: 32 Nodes
    • Requirements
      • vSphere Server 5.5
      • vSphere Web Client
      • VM Hardware version 10
      • License – vSphere Enterprise Plus

In a nutshell 20% of VMDK size is used to determine size of VFRC

Interesting white paper analysis of VFRC

There are many articles of how to use, how to implement but few articles of benchmarks

Advanced professional services would go beyond recommending use of VFRC but actually creating a test bed for the customer to determine the best size of and which block size to use.
default block size is 8kb

Assessment tool would be

Notes to share..


The answer is…. VMware EVO:RAIL ?? VMworld2014

VMworld 2014 has just wrapped up. WOW!
There is SO MUCH to share and so much to learn.
One of the biggest things to make public is EVO:RAIL.

That is just one of many.. other items to be discussed later…


What is EVO:RAIL
An overview demo from Duncan Epping.

EVO:RAIL is considered a HCIA, Hyper Converged Infrastructure Appliance

Some points to consider. “With an EVO:RAIL SYSTEM from power on within 15 minutes you can start provisioning VMs” WOW. Think about the decrease in complexity for the implementation. That is not just converged but HYPER-CONVERGED.

Like the wiki definition states; this is a single optimized computing package. You have virtualized compute, virtualized storage. All underneath an optimized management layer. vCenter, and log insight is the backbone but you have an optimized HTML5 INTERFACE.

This is the core fundamentals of SDDC. Software Defined Data Center.

EVO: RAIL Management
EVO: RAIL enables deployment, configuration, and management through a new, intuitive HTML5-based user
interface showcased in the next section. EVO: RAIL provides new non-disruptive updates for VMware software with zero downtime and automatic scale-out of EVO: RAIL appliances.

Software components of EVO:RAIL
• EVO: RAIL Deployment, Configuration, and Management
• VMware vSphere® Enterprise Plus, including ESXi for compute
• Virtual SAN for storage
• vCenter Server™
• vCenter Log Insight™

What does one EVO:RAIL Provide?
Virtual Machine Density
• EVO: RAIL is sized to run approximately 100 average-sized, general-purpose, data center VMs. Actual capacity varies by VM size and workload. There are no restrictions on application type. EVO: RAIL supports any application that a customer would run on vSphere.

(General-purpose VM profile: 2 vCPU, 4GB vMEM, 60GB of vDisk, with redundancy)

But, but .. I need more power!!
There will be a followup to EVO:RAIL… EVO:RACK!!!

Highly Resilient by Design

A highly resilient HCIA design starting with four independent server nodes within a 2U footprint from our qualified EVO:RAIL partners. Each node is running vSphere and Virtual SAN, configured as a single vSphere cluster with a single distributed Virtual SAN datastore; add into the mix VMotion, HA and DRS for additional resiliency, you now have all the key ingredients to facilitate zero VM downtime during planned maintenance or during disk, or a host failure.

Some key notes from Duncan’s post:
Customer Choice

EVO:RAIL is delivered as a fully integrated HCIA offering via a single SKU to the customer. There are two important things to note:

EVO:RAIL is not a reference architecture
A customer cannot purchase the EVO:RAIL software standalone and attempt to build their own HCIA on an EVO:RAIL Partners qualified and optimized hardware or non qualified server hardware

This is just the beginning of many conversations. Some questions to followup on would be to define the use case for EVO:RAIL.
How does it integrate into a brownfield environment? How to scale out this solution? Can you use other storage in addition to VSAN? What are licensing cost?

It’s alive!!! Marvin lives…


From wikipedia:
“Converged infrastructure operates by grouping multiple information technology (IT) components into a single, optimized computing package. Components of a converged infrastructure may include servers, data-storage devices, networking equipment and software for IT infrastructure management, automation and orchestration.”


What is EVO:RAIL



What is Converged Infrastructure?

Software Defined Storage: VMware VVOL

Closer to reality… Closer to GA

Yesterday VMware announced public beta2 for vsphere6 and VVOL.

Why separate Beta programs?

While vSphere 6 can be leveraged ontop of most existing hardware that supports vSphere 5.5; not all storage vendors will be ready for VVOL.

All major storage vendors will be supporting the new VVOL feature. And the impact of this is very big.

But let’s start with some background information….

What is VVOL

**please note the capital V 🙂

VVOL is a new paradigm — a standard, a model, template for storage. You won’t be doing storage like you have in the past; well not 100% the same. Sure you will have the traditional tasks a storage admin will have to do for fabric based storage arrays.. but what makes things really interesting is what I call the integration point.

Currently Storage arrays are not VM aware; VVOL changes that. I guess you can call it adding more “intelligence” to your storage.

The conversation is storage meet this VM/Application. It isn’t about just consuming a LUN, but more about making a VMDK a native object on the storage system.

This point/ intersection is where the conversation should grab your attention.

I won’t try to recap what the status quo is today for storage and vmware, but know this: it is complex. Time consuming at a high operational cost and requires specialized training to ensure availability, management of SLA: performance and capacity.

Very important reference reading: (much more detailed information)

Cormac Hogan’s blog:

Duncan Epping:

Chad Sakac


VMworld 2013 VVOL Session

VMworld 2012 VVOL Session


And when you are ready… go and DO THE BETA! (or at least join the beta community and learn all you can!!)

Rawlinson shows you all about the beta and examples of storage vendors using VVOL


Learning VMware… where to start on your journey to Virtualization Mastery.

Ok, so I have been asked this question many times as a consultant. This topic often is brought up  when interacting with their customer.

And the answer is what a consultant most often says: “It depends…”

You have to understand what the purpose of the question is.

Meaning who is asking you the question? What is the motivation to ask the question?

Is it for themselves? it is for their direct reports? It is for their management?

Maybe they would like some justification to provide career growth? Maybe they would like to have career growth? Or, often they have this new task called *”virtualization” that now is a Area of Responsibility they have been charged to manage.

*or virtualisation — for other parts of the world 😉 


Anyhow you slice it you cannot avoid Virtualization technologies and the impact is has to you.

There are several aspects of Virtualization and it depends on your job function to determine what is material to you.


There are different areas of concern (aka AOR Areas of Responsibility) that impact your what and how you get things done!

A PC technician. How will Virtualization impact my job? Can I do things easier?

The Business Unit Manager How will this impact my budget?

The System Admin How can I do things more efficiently?

The Storage admin How will I provide storage to this thing call ESX?

The Security Team How are we maintaining compliance with HIPAA? SOX? PCI?


This isn’t a post around the different options for Virtualization and selection of hypervisors. Vendors and mindset of each vendor strategy. Type 1 vs Type 2.


Typically the technician (technical staff member) will want to have a deep depth of understanding, but you cannot have depth without breadth. I know that having a deep understanding of one area of expertise can be beneficial, but what about the next level? How does what you do INTERACT with the other areas. What about what I call INTEGRATION POINTS?

What you do; you will need to do well; exceptionally well! But case in point: What you deploy will require you to understand how it interacts with the IT ecosystem.

ecosystem |ˈekōˌsistəmˈēkō-|

noun Ecology

a biological community of interacting organisms and their physical environment.

• (in general use) a complex network or interconnected system


Since we are not discussing the flora and fauna of our data center or the native plants in the region that provide drought resistance or provide genetic diversity to the seed bank for the area… but I digress.  😉


The complexity of the network, the interconnected system. How does Virtualization add to the picture? Does it impact your consumption of the resources normally provided to you? Yes

Does it impact the Level of Service you can provide to downstream users?  Does it make things easier?

Those are thought provoking questions.

But first where are you in your path to understanding and mastery of virtualization?


There are several areas to gaining mastery of VMWARE virtualization.

Area of Expertise


Data Center Virutalization

End-User Computing – aka EUC or Desktop

Network (NSX) << more on that one later



Okay you have now defined your breadth what about depth?

“A journey of a thousand miles.. starts with one footstep.” There is no short-cut.. but there is a path to guide you.



Advanced Professional


Where to start?

Choose your path!


Data Center Virtualization

End-User Computing


So you want to learn?

Where to really start?

Find some free training from the comfort of (aka home local loop-back) From the comfort of home you can start your journey or if you are in the industry.. accelerate your proficiency today

VMWARE online videos 


FREE self-paced learning online 

Webinars – (Web + seminar) = learning delivered to you! –

“New to the VMware free training library is our ViewPoint Webinar Series where we connect you directly with VMware’s own expert training team, making it easier to take full advantage of VMware® products, from data center virtualization and hybrid cloud to software-defined data center.”

Here is a list of the self-paced learning available for free:

BUT. But …. But I learn by doing!!!

Ok some people are kinetic learners and well learn by doing.

VMWARE Hands On Lab 


And then there is more!!

You can access the documentation for the 2013 HOL labs right here

Ares of study for HOL Labs

SDDC (Software Defined Data Center)


Hybrid Cloud

Partner Solutions


So you want to get certified.. here is a great source to sum things up and review


Study Guides
link – The Official VCP5 Certification Guide (VMware Press)

kindle version – The Official VCP5 Certification Guide (VMware Press)


Essential Reading

link Mastering vSphere 5.5 by Scott Lowe

Open Source tool: Vagrant + vmware Fusion +EMC ScaleIO = ?

Open Source tool: Vagrant OPENSOURCE. Learn it. Use it. Software tools are great and sharing tools and supporting the OpenSource community is a good thing. This is part of a multi-part post. I will share my experience setting up ScaleIO in my vmware Fusion Lab. First find a tool to make provisioning quicker, more consistent, and automated. Tool: Vagrant From the vagrant website:

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.”

more later..   vagrant and more about


Intro to ScaleIO


EMC ScaleIO at a Glance

EMC ScaleIO is a software-only solution that uses application hosts’ local disks to realize a virtual SAN that is comparable to or better than external SAN storage, but at a fraction of the cost and complexity. ScaleIO makes a convergence of the storage and application layers possible, ending up with a wall-to-wall single layer of hosts. The lightweight software components of ScaleIO are installed on the application hosts alongside applications like databases and hypervisors.

Breaking traditional barriers of storage scalability, ScaleIO scales out to hundreds and thousands of nodes. ScaleIO’s performance scales linearly with the number of application servers and disks. With ScaleIO, any administrator can add, move, or remove servers and capacity on demand during I/O operations. ScaleIO helps ensure the highest level of enterprise-grade resilience while maintaining maximum storage performance.

ScaleIO natively supports all the leading Linux distributions, Windows Server and hypervisors and works agnostically with any SSD, HDD, and network. The product includes encryption at rest and quality of service (QoS) of performance. ScaleIO can be managed from both a command-line interface (CLI) and an intuitive graphical user interface (GUI). Deploying ScaleIO in both greenfield and existing data center environments is a simple process and takes only a few minutes.”