Design, Storage, Uncategorized, Virtualization, vmware, VSAN

Core Knowledge vSAN HBA

The fundamentals cannot be over-emphasized. You need to ensure that the key components of your vSAN host is configured per recommendations.

Just a reminder of the HBA controller configuration.

  1. Make sure the device is on the Hardware Compatibility Guide (HCG) 
  2. And verify the firmware is up-to-date.

I have seen first hand what impact different firmware can have on your environment.

Example: Dell Perc H310

Controller queue depth impacts the rebuild/resync times. A low controller queue depth may impact the availability of your production VMs during rebuild/resync. A minimum queue depth of 256 is required in vSAN. Some vSAN Ready Node profiles require minimum queue depth of 512, All Flash configs.

For more details see this: vSAN Hardware Quick Reference Guide

The availability of vSAN and VMFS can be vying for the same resource; the HBA.

Do NOT mix Disk Access modes to your Host Bus Adapter (HBA) also called an I/O Controller. Pass through configuration is preferred, but RAID-0 can work. vSAN prefers to have a more direct access to the device attached to the I/O Controller.  So for example if the HBA is setup with some logic configuration the groups all the devices together before presenting to the ESXi host then you have some prep work to do. Several array controllers do not support pass through mode,  to use this type of controller for vSAN, we need to create a single disk RAID-0 group for every SSD and HDD.



Dell PERC 740


  • RAID levels access for the devices attached.
  • vSAN and VMFS devices on same HBA.

From the VMware KB:

  • Do not mix the controller mode for vSAN and non-vSAN disks.
    • If the vSAN disks are in pass-through/JBOD mode, the non-vSAN disks must also be in pass-through/JBOD mode.
    • If the vSAN disks are in RAID mode, the non-vSAN disks must also be in RAID mode.
    • Mixing the controller mode will mean that various disks will be handled in different ways by the storage controller. This introduces the possibility that issues affecting one configuration could also affect the other, with possible negative consequences for vSAN.

If you absolutely must use the same HBA:

  1. limit the use of the VMFS that is sharing the HBA with vSAN.
  2. AND DO NOT USE RDM for that shared device/HBA
  3. DO NOT have the boot device on the same vSAN controller
  • If the non-vSAN disks are in use for VMFS, the VMFS datastore should be used only for scratch, logging and coredumps.
    • Virtual machines should not be running from a disk or RAID group that shares its controller with vSAN disks or RAID groups.
    • ESXi host installation is permitted on non-vSAN disks attached to same controller.
  • Do not pass through non-vSAN disks to virtual machine guests as Raw Device Mappings (RDMs).

The number and type of drives plus their disk group configuration is not covered here but another topic of important discussion!




Design, servers, Storage, Uncategorized, Virtualization, vmware

vSphere Content Libraries (CL)


The introduction of the Content Libraries feature came with vSphere 6. The goal is to reduce the complexity in management of VM templates, vApps, ISO images, and scripts that your virtual environment needs for day to day operations. Content libraries are container objects.
The Content library can be

  1. Local to the vCenter your create it in.
  2. Published externally to other vCenters with password authentication
  3. Subscribed Content Library to another library

The flexibility of the content library topology availability will enable your organization to maximize your operational efficiencies. How? Here are some scenarios that Administrator face.
“What Template did you use to build this VM?”
“Is it patched? Is it the latest one?”

Now imagine this conversation across the business units that span across geographic regions, time zone etc.
What and Where?
Some key things that a CL will help prevent is the bad practice of building workflow and processes around a single person. Increase efficiency in your organization, by using a central repository of essentials files you can avoid using the “wrong” vm template. That answers the what version is the latest? You can increase efficiency of answering the question of where is the latest version?

How do you setup a CL?

  1. In the vSphere Web Client navigator, select vCenter Inventory Lists > Content Libraries.
  2. Click the Objects tab.
  3. Click the Create a New Library icon (create a content library).
  4. Enter a name for the content library, and in the Notes text box, enter a description for the library and click Next.
  5. Select the type of content library that you want to create.



Local content library

A local content library is accessible only in the vCenter Server instance where you create it.

Published content library

Select Publish externally to make the content of the library available to other vCenter Server instances.

If you want the users to use a password when accessing the library, select Enable authentication and set a password.

Optimized published content library

Select Optimize for syncing over HTTP to create an optimized published library.

This library is optimized to ensure lower CPU usage and faster streaming of the content over HTTP. Use this library as a main content depot for your subscribed libraries. You cannot deploy virtual machines from an optimized library. Use optimized published content library when the subscribed libraries reside on a remote vCenter Serversystem and enhanced linked mode is not used.

Subscribed content library

Creates a content library that is subscribed to a published content library. You can sync the subscribed library with the published library to see up-to-date content, but you cannot add or remove content from the subscribed library. Only an administrator of the published library can add, modify, and remove contents from the published library.

Provide the following settings to subscribe to a library:

  1. In the Subscription URL text box, enter the URL address of the published library.

  2. If authentication is enabled on the published library, enter the publisher password.

  3. Select a download method for the contents of the subscribed library.

    • If you want to download a local copy of all the items in the published library immediately after subscribing to it, select Download all library content immediately.

    • If you want to save storage space, select Download library content only when needed. You download only the metadata for the items in the published library.

      If you need to use an item, you can synchronize it to download its content.

  4. When prompted, accept the SSL certificate thumbprint.

    The SSL certificate thumbprint is stored on your system until you delete the subscribed content library from the inventory.

6. Click Next.
7. Select a datastore, or enter the path to a remote storage location where to keep the contents of this library.



Enter an SMB or an NFS server and path

If you use avCenter Server instance that runs on a Windows system, enter the SMB machine and share name.

If you use vCenter Server Appliance, enter a path to an NFS storage. You can store your templates on an NFS storage that is mounted to the appliance. After the create a new library operation is complete, the vCenter Server Appliance mounts the shared storage to the host OS.

Select a datastore

Select a datastore from your vSphere inventory.

vSAN Datastore will appear here as a choice

8. Review the information on the Ready to Complete page and click Finish.

Great now you have a Content library.. what next?

ADD CONTENT to your Content Library.
You can:
Clone the VM as a template into your Content Library (Right click the VM choose
Actions–> Clone –> Clone to Template in Library


Now for another time saver!
So, you already realize the importance of a repository and you have a single folder on datastore that says /iso-templates. Now what? You need to be able to copy all of that to your new Content Library. So you can publish the CL and enable other vCenter’s to Subscribe.
The tricky option is to deal with ISO images.

Sure Templates and VM’s can be handled with cloning VM to Template actions but here is a option for existing templates in your datastore. This will save you a bit of time in re-copying the ISO back into the content library.


When I first started to use the CL I didn’t see an option the the CL to add ISO files. I reached out to Roman Konarev and he provided this excellent guide.


How to import your ISOs from DS:
Get a URL to your ISO file that you want to import to Content library. The structure of that URL is the following: [DataStore url]/[ISOs folder]/[file_name].

Here is my ISOs folder:
Here is my DS url:

So, the final URL will be the following: ds:///vmfs/volumes/56cd1758-86602854-5166-020019640efe/RK_ISOs/small_ISO.iso

2)    Open a standard “Import library item” wizard and paste the URL above there:


** vSphere 6.5 update **

** Update to vSphere 6.5 and make it easier! **

What a difference a version makes!


  1. In the vSphere Web Client navigator, select vCenter Inventory Lists > Content Libraries.
  2. Right-click a content library and select Import Item.

    The Import Library Item dialog box opens.

  3. Under Source section, select the option to import an item from a local file. Click Browse to navigate to the file that you want to import from your local system. You can use the drop-down menu to filter files in your local system.
  4. Under Destination section, enter a name and description for the item, and click OK.

Content Libraries can even extend into the Cloud!

Create a content library that is subscribed to the content library you published from your on-premises data center. Content is synchronized from your on-premises data center to your SDDC in VMware Cloud on AWS.

Design, Troubleshooting, Virtualization, vmware

vSphere Web Client cool feature! Topology maps

Anyone who has to work with, administer VMware sphere needs to have to top down view. You can review uplink settings, uplinks per host. How each Distributed port group is related to the VM defined. VMKernel ports (vmk) IP addresses — all of them at a glance.. Very helpful to see what is online or offline etc.

To access a Topology map of the Distributed vSwitch and Virtual Machine Networking.

There are also advanced features to check out for you uses.

example: filter and save views!


  1. Navigate to the vSphere distributed switch in the vSphere Web Client.
  2. On the Configure tab, expand Settings and select Topology.

Troubleshooting, Uncategorized, Virtualization, vmware, VSAN

VMWARE Virtual SAN networking

VSAN networking can be a bit tricky to troubleshoot. Before I go deeper into the topic here is a very important concept to remember about VSAN clusters.

Given any VSAN cluster remember the following:

** “Introduction to Virtual SAN Networking

Before getting into network in detail, it is important to understand the roles that nodes/hosts can play in Virtual SAN. There are three roles in Virtual SAN: master, agent and backup. There is one master that is responsible for getting CMMDS (clustering service) updates from all nodes, and distributing these updates to agents. Roles are applied during cluster discovery, when all nodes participating in Virtual SAN elect a master. A vSphere administrator has no control over roles.”

** from Cormac’s troubleshooting guide

That is a lot to digest but if break it down you can see some key principles about a VSAN cluster to remember.

The roles in VSAN:
A master
B agent
C backup.

There is one master.
If you see more than one master there is something not quite right with you VSAN CLUSTER.

The VSAN admin does not control which node will be the master.

Log into each node of a three node VSAN. The normal pre-req for troubleshooting make sure ssh is enabled.

Run the following command on each node:
~ # esxcli vsan cluster get

Cluster Information will output below.

Cluster Information
Enabled: true

Current Local Time: 2015-03-30T22:38:38Z
Local Node UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Backup UUID:
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member UUIDs: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Membership UUID: a5ce1955-f5e5-5663-d338-a0369f58b8e4

Node 2
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-30T22:38:38Z
Local Node UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Backup UUID:
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member UUIDs: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Membership UUID: a5ce1955-f5e5-5663-d338-a0369f58b8e4

Node 3
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-30T22:56:46Z
Local Node UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Backup UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member UUIDs: 551374b5-03f9-7bd6-6257-a0369f58b8e8, 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster Membership UUID: d6da1955-e2f8-38eb-d7f0-a0369f58b8e8

See the image below for the error seen in the web client.

From the output above can you see the problem?



SDDC, Virtualization, vmware

The answer is…. VMware EVO:RAIL ?? VMworld2014

VMworld 2014 has just wrapped up. WOW!
There is SO MUCH to share and so much to learn.
One of the biggest things to make public is EVO:RAIL.

That is just one of many.. other items to be discussed later…


What is EVO:RAIL
An overview demo from Duncan Epping.

EVO:RAIL is considered a HCIA, Hyper Converged Infrastructure Appliance

Some points to consider. “With an EVO:RAIL SYSTEM from power on within 15 minutes you can start provisioning VMs” WOW. Think about the decrease in complexity for the implementation. That is not just converged but HYPER-CONVERGED.

Like the wiki definition states; this is a single optimized computing package. You have virtualized compute, virtualized storage. All underneath an optimized management layer. vCenter, and log insight is the backbone but you have an optimized HTML5 INTERFACE.

This is the core fundamentals of SDDC. Software Defined Data Center.

EVO: RAIL Management
EVO: RAIL enables deployment, configuration, and management through a new, intuitive HTML5-based user
interface showcased in the next section. EVO: RAIL provides new non-disruptive updates for VMware software with zero downtime and automatic scale-out of EVO: RAIL appliances.

Software components of EVO:RAIL
• EVO: RAIL Deployment, Configuration, and Management
• VMware vSphere® Enterprise Plus, including ESXi for compute
• Virtual SAN for storage
• vCenter Server™
• vCenter Log Insight™

What does one EVO:RAIL Provide?
Virtual Machine Density
• EVO: RAIL is sized to run approximately 100 average-sized, general-purpose, data center VMs. Actual capacity varies by VM size and workload. There are no restrictions on application type. EVO: RAIL supports any application that a customer would run on vSphere.

(General-purpose VM profile: 2 vCPU, 4GB vMEM, 60GB of vDisk, with redundancy)

But, but .. I need more power!!
There will be a followup to EVO:RAIL… EVO:RACK!!!

Highly Resilient by Design

A highly resilient HCIA design starting with four independent server nodes within a 2U footprint from our qualified EVO:RAIL partners. Each node is running vSphere and Virtual SAN, configured as a single vSphere cluster with a single distributed Virtual SAN datastore; add into the mix VMotion, HA and DRS for additional resiliency, you now have all the key ingredients to facilitate zero VM downtime during planned maintenance or during disk, or a host failure.

Some key notes from Duncan’s post:
Customer Choice

EVO:RAIL is delivered as a fully integrated HCIA offering via a single SKU to the customer. There are two important things to note:

EVO:RAIL is not a reference architecture
A customer cannot purchase the EVO:RAIL software standalone and attempt to build their own HCIA on an EVO:RAIL Partners qualified and optimized hardware or non qualified server hardware

This is just the beginning of many conversations. Some questions to followup on would be to define the use case for EVO:RAIL.
How does it integrate into a brownfield environment? How to scale out this solution? Can you use other storage in addition to VSAN? What are licensing cost?

It’s alive!!! Marvin lives…


From wikipedia:
“Converged infrastructure operates by grouping multiple information technology (IT) components into a single, optimized computing package. Components of a converged infrastructure may include servers, data-storage devices, networking equipment and software for IT infrastructure management, automation and orchestration.”


What is EVO:RAIL



What is Converged Infrastructure?

EMC, ScaleIO, Storage, Uncategorized, Virtualization, vmware

Open Source tool: Vagrant OPENSOURCE. Learn it. Use it. Software tools are great and sharing tools and supporting the OpenSource community is a good thing. This is part of a multi-part post. I will share my experience setting up ScaleIO in my vmware Fusion Lab. First find a tool to make provisioning quicker, more consistent, and automated. Tool: Vagrant From the vagrant website:

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.”

more later..   vagrant and more about


Intro to ScaleIO


EMC ScaleIO at a Glance

EMC ScaleIO is a software-only solution that uses application hosts’ local disks to realize a virtual SAN that is comparable to or better than external SAN storage, but at a fraction of the cost and complexity. ScaleIO makes a convergence of the storage and application layers possible, ending up with a wall-to-wall single layer of hosts. The lightweight software components of ScaleIO are installed on the application hosts alongside applications like databases and hypervisors.

Breaking traditional barriers of storage scalability, ScaleIO scales out to hundreds and thousands of nodes. ScaleIO’s performance scales linearly with the number of application servers and disks. With ScaleIO, any administrator can add, move, or remove servers and capacity on demand during I/O operations. ScaleIO helps ensure the highest level of enterprise-grade resilience while maintaining maximum storage performance.

ScaleIO natively supports all the leading Linux distributions, Windows Server and hypervisors and works agnostically with any SSD, HDD, and network. The product includes encryption at rest and quality of service (QoS) of performance. ScaleIO can be managed from both a command-line interface (CLI) and an intuitive graphical user interface (GUI). Deploying ScaleIO in both greenfield and existing data center environments is a simple process and takes only a few minutes.”

Storage, Virtualization, vmware, vsom

VMware vSphere with Operations Management

What a title! Operations Management  How do you define Operations management?

Can it be one system (ESX host or 1000?) How many VMs are running? How many vCenters do you run? How much storage do you have?

What: But more important is: What are you charged with? What is your responsibilities? How can you answer questions respective to your area(s) of responsibility? Can you answer questions related to your infrastructure?

OK. What about the next level of questions related to the above?  You may know what you run (hopefully) but how is it running? Is it running with maximum efficiency? Can you improve how things are running? Where are your bottlenecks? At which level? VM? Storage? Network? Host? You can determine that in about 60-90 seconds for  a single VM, a single host, a few DataStores.

What tools are in your toolkit? perfmon, esxtop, vcenter performance tab, vsom (vcops) other???


Now things are getting interesting… Data collecting is good. That is step one.

Now sorting and analyzing data is the next level. What about data correlation? Data analysis?

Root Cause Analysis (Why, why, why, why, why, why, why? aka “The 7 Why’s” … to drill down and find the answer) RCA is exhausting and time consuming. It burns up your man hours aka OpEX. How large is your team/ staff to address this question? And then the next quadrant.. Data Trending? Can you be proactive? Can you make more than a SWAG? How accurate is your predictive prowess? What is your strategy to handle these questions? I work with many customers looking to answer these questions. Scope and scale of responsibility can be overwhelming. I do know that VSOM does help and more often than not does answer the above questions and MORE.


Operations Management

 VSOM  Badges   Badges..

We don’t need “Badges? We don’t need no stinkin’ badges!” Actually you do.

When you understand the mindset Health Risk and Efficency or just for short H.e.R.E.   The approach to the vsphere management is very visual. Very quick and very dynamic.

*** Badges are not Metrics. Badges are MORE.


Performance, Capacity and Configuration INFORMATION – A single view. QUICK and Informative.
Analytic algorithms – beyond normal parameters… RED is bad!
VSOM. .Deploy the vApp…!
You’ve now employed a FTE (Full Time Employee) That just keeps working and working for you.

Health  — Operations

Immediate issues – Anomonilies & Faults

Risk — Short Term health
Forward looking issues, may come up if not addressed soon
Efficiency  — Long Term Health
Where improvements can be made
VSOM uses a multi-dimensional analysis. There are more facets, properties of an object monitored than a single dimension. i.e. is it on or off. How long it it on. What kind of health does it have? Can it access the resources defined for it? Can it meet the demand the object wants for the associated resources it needs? i.e. disk access? network access? compute access? Is this being throttled by a bottleneck somewhere?
How is your Virtual Environment? I call this the compute ecosystem.
Score card
“immediate Issues”
“Future Issues”
– Time remaining
  • when resources exhausted
-Capacity remaining
# VMs deployed
# VMs powered on.
# VMs remaining can be deployed
Longer time period vs workload (instantaneous)
Reclaimable Wastes “Right-sizing” of VMs. Maximize. Run Hotter.
Where do you want your operations to be be set. Are you running at the level you can manage? Are your end users consuming resources effectively??
  • above or below optimum
  • reduce VM sizing

Learn more vmware education Class 

Read a Book or for your online reference



How is your digital kungfu?? 🙂

More later..