Uncategorized, vnx2

Change Control Station IP for VNX

The installation of a VNX2

The first day is the toughest and easiest. Like baking a cake you need to make sure all the ingredients are there and ready to go. You verify what  was ordered is installed and is in working order. Sounds simple enough?

As it turns out the initial VNX “Rack and Stack” was correct.

  • All cables were plugged in correctly
  • No amber lights on the system.
  • The DAE (Disk Assembly Enclosures) were recognized.
  • No faults in Unisphere

Ok. Good to go? No? Why. Mr. Customer provided a duplicate IP for one of the control stations. The correct IP is the same subnet and mask but needs to be changed.

Here are the steps

  1. Log onto Unisphere confirm IP information for Control Station
  2. Log onto Control station via ssh

Use root/nasadmin (default)

3. Verify status of Control station. You can make changes to the primary not standby.

In this case the primary Control station was IP’d correctly. I need to change the secondary Control station, the standby.

/nasmcd/getreason  <<– run this command as root

Here is what you will see:

10 – slot_0 primary control station
11 – slot_1 secondary control station
 5 – slot_2 contacted
 5 – slot_3 contacted
4. Failover the Control station with this command:
    /nasmcd/sbin/cs_standby -failover
About 5 minutes later you will reboot the primary and failover to the secondary. 
5. Run this: /nasmcd/getreason
11 – slot_0 secondary control station
10 – slot_1 primary control station
 5 – slot_2 contacted
 5 – slot_3 contacted

6. Log into Unisphere

Normally you login to the VNX with using the primary Control Station IP. However since it is in failover mode use the secondary IP. In this case I am using the duplicate IP and will change it to the correct IP by using the Unisphere GUI

Screen Shot 2016-02-19 at 5.10.39 PM

Log into the Unisphere GUI as root. Make sure you choose scope “local” not Global

Navigate to the “Control station Properties Tab”

Screen Shot 2016-02-19 at 5.11.43 PM

Change the IP of the Control Station

Screen Shot 2016-02-19 at 5.12.20 PM


Click Apply.

In Confirm Action, click OK.


Modifying the Control Station hostname, IP address, subnet mask, or gateway may disrupt the Unisphere software connection and any other client access to the Control Station. It may be necessary to reconnect to continue administrative activities. If you make a mistake changing the network, the Control Station may no longer be reachable remotely.

7. ssh to the primary Control station and failback. Verify that the primary control station is in slot 0

#  /nasmcd/getreason
10 – slot_0 primary control station
11 – slot_1 secondary control station
 5 – slot_2 contacted
 5 – slot_3 contacted

That is it.


Remember when you login to Unisphere you connect via the Primary Control Station IP, if you are failed over you will connect to the secondary Control Station.


Thanks for reading! If you have a more efficient way please share. Yes, you can ssh in and do all of this via cli. 😉




EMC, vmware, VSAN

Is your IT infrastructure an Oil Tanker?

I had the opportunity to attend an Avnet/EMC/VMware/Brocade sponsored for channel partners EVO-RAIL VSPEX Blue BootCamp.

In a nutshell it was all you can drink information from a firehose— about EVO-RAIL specifically the EMC VSPEX BLUE.

EVO RAIL is a new beast of an animal. It is a different breed. No, not in and single dimension you measure. The combination of technology presented is a synergy. Definition: Synergy is the creation of a whole that is greater than the simple sum of its parts.

Yes; you can get the the form factor for compute separately. You can also do the same for VSAN and ESXi vSphere 5.5 and networking. but you cannot get what the entire VSPEX BLUE offering of EVO RAIL provides TOGETHER.

I get ahead of myself.

You have to have perspective to understand where we are today. To me that means if you don’t know where you come from you  cannot know where are you today and where you will be tomorrow. It is all relative.

EVO RAIL is a clustered system. It is a Datacenter in a 2U form factor. Not just compute but a modern hyper converged solution.


Sure. Another IT buzzword. Is it just talk? I would say no.

Everyone is talking about it but only a few are “doing it”… more often than not the IT industry is a buzz with the new technology of the day.
In this case it is Hyper-converged.
To put it simply storage is local to compute. What a minute how is that different than 15 years ago when Client-Server model was the norm and storage was already local to the compute. Compute meaning the processing of the server CPU. Well lots has changed.
How is it better. A snapshot of what is the current available technology:
  • Compute is way, way faster and more dense.
  • Networks are 10 gigabit vs Fast Ethernet 100 Base-T or FDDI Optical rings are no longer the only viable choice.
  • Storage is IOPs crazy.
  • And add to that the agility of VMware Virtualization!
So look at the speed of compute, network and storage. Technology will continue to get faster and better (lower cost for the return on investment).
But what really hasn’t changed much is the complexity of the solution. There are many moving parts but how do you delivery your solution today to support legacy applications and have the agility to respond to changing business objectives.
The old phrase “turning an oil tanker on a dime”. Is IT today an Oil tanker? Does your private cloud have the agility your business requires? How will the current toolset respond? How will your staff? Oh what was that “IT staffing has been reduced and that is a trend that hasn’t gone away”
IT organizations are forced to do more with little… queue in viable alternatives…
Do you outsource.. the simplest short term gain, but not always the best long term investment.
I view EVO rail as a datacenter in a box. EVO-RAIL VPSEX Blue is Not the Cluster in a box solution but much much more.
This reminds me of the forerunning of MCSC cluster in a box. Like I said.. perspective. Where has the IT industry been before relative to where it is today. I recall back in the day when cluster in a box was a viable (and the best solution at the time) era 2000. That was only a high-availablity MSCS cluster with one node active at a time. 
Yes, I deployed and supported a few of these solutions. It was cutting edge back then.
Adjusted for inflation:
$1000.00 USD in 2000 is $1395.20
so the CL1850 was $27,864.00 (2000 $)
or $38,569 in 2015 $.
But what did you get for your IT dollar?
Each “node” in the CL1850
  • 1 GB RAM (128-MB 100-MHz registered ECC SDRAM memory)
  • 2 Pentium III processors @550MHz
  • 3 NICS (one dedicate for internode communication) 100 BaseT
  • 2 RAID controllers
Shared Storage System:
  • 218.4 GB (6 x 36.4-GB 1′′ Ultra3 10,000 drives)
Form Factor 10U
Fast forward to 2015….What does EVO-RAIL VSPEX BLUE PROVIDE:
Each EMC VSPEX BLUE appliance includes:
  • 4 nodes of integrated compute and storage, including flash (SSD) and HDD — 2U
  • VMware EVO:RAIL software including VMware Virtual SAN (VSAN), Log Insight
        12 cores @ 2.1GHz per node
  • 128 or 192 GB memory per node
  • Choice of 10 Gigabit Ethernet network connectivity: SFP+ or RJ45
  • Drives: up to 16 (four per node)
  • Drives per node: 1 x 2.5” SSD, 3 x 2.5” HDD
  • Drive capacities
    • HDD: 1.2TB (max total 14.4TB)
    • SSD for caching: 400GB (max total 1.6TB)
               14.4TB capacity RAW
Additional Software Solutions – Exclusive to VSPEX-BLUE
  • EMC VSPEX BLUE Manager providing a system health dashboard and support portal
  • EMC CloudArray to expand storage capacity into the cloud (license for 1 TB cache and 10 TB cloud storage included)
  • VMware vSphere Data Protection Advanced (VDPA) for centralized backup and recovery
  • EMC RecoverPoint for Virtual Machines for continuous data protection of VMs. Includes licenses for 15 VMs.

The question still remains. Is your IT infrastructure an Oil Tanker? OR Can you turn on a dime??

How does your IT respond the the ever changing business demands?

Is EVO RAIL for everyone? There are a lot of use cases that EVO RAIL VSPEX BLUE will work perfect for. But,  No it isn’t for everyone.  BUT what it does is usher in a new consumption of IT that is different manner. You will not have to be provision your datacenter in the same piece meal function. You can commoditize that to a pre-validated solution that is supported by a single vendor.


This graphic has a lot more details than this single blog post can explain! I will try to explain each section that helps to make VSPEX BLUE a different redefined EVO RAIL solution.


– VSPEX BLUE Manager users can conveniently access electronic services, such as the EMC knowledge base articles, access to the VSPEX Community for online and real-time information and EMC VSPEX BLUE best practices.


– ESRS is a two-way, secure remote connection between your EMC environment and EMC Customer Service that enables remote monitoring, diagnosis, and repair – assuring availability and optimization of your EMC products






– Built in Deduplicated Backups, powered by EMC Avamar


– Block and File upto 10 TB FREE. “EMC CloudArray software provided scalable cloud based storage with your choice of many leading cloud providers enabling limitless Network Attached Storage, offsite backup and disaster recovery and the ability to support both Block and File simply”


– Built into the VSPEX MANAGER dashboard. This unique feature enables customers to browse complementary EMC and 3rd party products that easily extend the capabilities of the appliance.

Again there are a lot of take aways for the VSPEX BLUE – EVO RAIL solution. Contact me if you more information.
EMC, Storage



What is it: 

“The vVNX Community Edition (also referred to as vVNX) provides a flexible storage alternative for test and development environments.”

SDS = Software Defined Storage!!
Download a full-featured version of vVNX available for non-production use without any time limits.”
File type: *.ova
File Size: 2.1 GB
Release date: 5/4/2015
Version: 3.1.2

Download here! https://www.emc.com/products-solutions/trial-software-download/vvnx.htm

Architecture and functionality:

White Paper details: vVNX Community Edition https://community.emc.com/docs/DOC-44552

Installation Guide: https://community.emc.com/docs/DOC-42029


Before you can obtain and activate the trial vVNX license, you must have completed the following tasks:

  1. Registered to create a product support account.
  2. Downloaded the vVNX software.
  3. Installed vVNX.
  4. Launched Unisphere.
The Configuration  runs when you log in to Unisphere for the first time.
  1. Note the vVNX system UUID provided on the License dialog in the Configuration Wizard.
  2. Go to the Electronic License Management System (ELMS) download page at www.emc.com/auth/elmeval.htm
  3. Click Obtain evaluation license for vVNX.
  4. Enter the vVNX system UUID and select vVNX as the product type. 
  5. Click Download to save the license to your local system.
    Note: An email confirming that you have successfully obtained the evaluation license is sent to the email address you provided when you registered.
  6. Return to the License dialog in the Configuration Wizard and click Install License File
  7. Locate the license file, select it, and click Upload to install and activate it. 

Note: Do not repeat this procedure once you have saved the license and received the confirmation email. If you try to enter the vVNX system UUID again, you will receive a “Duplicate UUID” error message.

Design, EMC, iSCSI, vmware

Technical notes on IP Based Storage Guide for EMC VNX and VMware 5.5

When it comes to IP based storage there are two major choices for use in VMware environments. NFS and iSCSI

This article will discuss iSCSI options.

NFS is a very valid design choice with very flexible options for deployment. NFS for use with VMware is Fast and Flexible in other words a solid choice. But NFS is for another time and a different discussion.

Why IP Based storage??

In a single word: SPEED and Flexibility. Okay two words. Core networking speeds are no longer limited by 10baseT, 100baseT but 10 Gigabit Ethernet is more so the standard. You already have IP based network, why not try to leverage what is installed?

I have seen greater interest in deployment of iSCSI based storage for VMware lately. Not just 1 gb but 10 ‎Gigabit Ethernet is gaining more of a foothold for DataCenters, as customer upgrade their core networking from 1 Gb to 10Gb. The potential for performance gains is really a good thing, but fundamental to deployment is to consider a solid network design.

What I mean is end-to-end. What kind of switch are you using? Are you limited by a certain number of 10 gb ports? How many switches do you have. Do you have a single point of failure? This is more critical as you will be really leveraging your network for “double duty”. Instead of a discrete network designated for storage such as FC can provide you will now run storage information across your IP network. Ideally two separate physical switches are ideal. BUT at a minimum use VLANS for logical separation. And let me go ahead and say it… “VLAN 0 (zero) is a not really a scalable enterprise option!” That is a huge red flag you will require more network analysis and work to deploy a IP based storage solution.

There are many considerations for a successful iSCSI implementation.

1) Gather Customer Requirements 2) Design  3) Verify 4) Implement/ Deploy 5) TEST User Acceptance Testing

Ideally having two 10Gb switches for redundancy is a good thing! Be careful in the selection of a Enterprise grade switch. Have seen horrible experience when improper features are not enabled. i.e. flow control can be a good thing!

Software based iSCSI initiators. Don’t forget about Delayed ACK. See Sean’s excellent article here: Delayed ACK setting with VNX. Read more VMware details about how to implement Delayed ACK from KB1002598

“Modify the delayed ACK setting on a discovery address (recommended):

  1. On a discovery address, click the Dynamic Discovery tab.
  2. Click the Server Address tab.
  3. Click Settings > Advanced.”

Double, no TRIPLE check the manufacture recommended CABLE TYPE and LENGTH. For example: Does your 10GbE use fiber optic cable? Do you have the correct type? What about the SFP?  And if you are not using fiber optic, but choose to use TwinAX cabling. Do you have the correct cable as per manufacture requirements?

For example: Meraki, only makes and supports a 1 Meter 10g Passive Copper Cable for their switches.  If you look at any normal Cisco Business Class switch they support 1, 3, 5 meter passive and 7, 10 meter active cables on their switches. 

Active cables are generally more expensive, but could be a requirement depending on your datacenter and or colocation layout.

I try to approach the solution from both ends. Storage to the Host and the reverse Host to the storage.  Examine end-to-end dependancies. Even though your area of responsibility isn’t the network, you will be reliant on the network services and any misconfiguration will impact your ability to meet the design requirements stated. You may not have bought or had any input to the existing infrastructure but you will be impacted by what is there currently in use. How will you Keyword: Interoperability how each independent system will interact with another system. Upstream and downstream dependencies.

Other considerations:

For example: vmkernel port binding: The diagram below is from VMware KB 2038869 “Considerations for iSCSI Port Binding”

Port binding is used in iSCSI when multiple VMkernel ports for iSCSI reside in the same broadcast domain and IP subnet to allow multiple paths to an iSCSI array that broadcasts a single IP address. When using port binding, you must remember that:

  • Array Target iSCSI ports must reside in the same broadcast domain and IP subnet as the VMkernel port.
  • All VMkernel ports used for iSCSI connectivity must reside in the same broadcast domain and IP subnet.
  • All VMkernel ports used for iSCSI connectivity must reside in the same vSwitch.
  • Currently, port binding does not support network routing.”


While there isn’t FC zoning for IP based storage there will be a requirement for subletting and VLAN separation.

For VNX here are some design considerations for iSCSI design:

The following points are best practices for connecting iSCSI hosts to a CLARiiON or VNX:

  • iSCSI subnets must not overlap the management port or Service LAN (128.221.252.x).

  • For iSCSI, there is no zoning (unlike an FC SAN) so separate subnets are used to provide redundant paths to the iSCSI ports on the CLARiiON array. For iSCSI you should have mini-SANs (VLANs) with only one HBA per host in each VLAN with one port per storage processor (SP) (for example, A0 and  B0 in one VLAN, A1 and  B1 in another).  All connections from a single server to a single storage system must use the same interface type, either NIC or HBA, but not both.

  • It is a good practice to create a separate, isolated IP network/VLAN for the iSCSI subnet. This is because the iSCSI data is unencrypted and also having an iSCSI-only network makes troubleshooting easier.

  • If the host has only a single NIC/HBA, then it should connect to only one port per SP. If there are more NICs or HBAs in the host, then each NIC/HBA can connect to one port from SP A and one port from SP B. Connecting more SP ports to a single NIC can lead to discarded frames due to the NIC being overloaded.

  • In the iSCSI initiator, set a different “Source IP” value for each iSCSI connection to an SP.  In other words, make sure that each NIC IP address only appears twice in the host’s list of iSCSI Source IP addresses: once for a port on SP A and once for a port on SP B.

  • Make sure that the Storage Process Management ports do not use the same subnets as the iSCSI ports – see [Link Error:UrlName “emc235739-Changing-configuration-on-one-iSCSI-port-may-cause-I-O-interruption-to-all-iSCSI-ports-on-this-storage-processor-SP-if-using-IP-addresses-from-same-subnet” not found] for more information.

  • It is also a best practice to use a different IP switch for the second iSCSI port on each SP. This is to prevent the IP switch being a single point of failure. In this way, were one IP switch to completely fail, the host can failover (via PowerPath) to the paths on the other IP switch. In the same way, it would be advisable to use different switches for multiple IP connections in the host.

  • Gateways can be used, but the ideal configuration is for HBA to be on the same subnet as one SP A port and one SP B port, without using the gateway.

For example, a typical configuration for the iSCSI ports on a CLARiiON, with two iSCSI ports per SP would be:

A0: (Subnet mask
A1: (Subnet mask
B0: (Subnet mask
B1: (Subnet mask

A host with two NICs should have its connections configured similar to the following in the iSCSI initiator to allow for load balancing and failover:

NIC1 (for example, – SP A0 and SP B0 iSCSI connections
NIC2 (for example, – SP A1 and SP B1 iSCSI connections

Similarly, if there were four iSCSI ports per SP, four subnets would be used. Half of the hosts with two HBA would then use the first two subnets, and the rest would use the other two.

The management ports should also not overlap the iSCSI ports. As the iSCSI network is normally separated from the LAN used to manage the SP, this is rarely an issue, but to follow the example iSCSI addresses above, the other IP used by the array could be as following examples:

VNX Control Station 1:
VNX Control Station 2:
SP A management IP address:
SP B management IP address:

The High Availability Validation Tool will log an HAVT warning if it detects that a host is connected via a single iSCSI Initiator. Even if the initiator has a path to both SP’s it is still at HA risk from a host connectivity view. You will also see this if using unlicensed PowerPath.

Caution! Do not use the IP address range 192.168.1.x because this is used by the serial port PPP connection

Oh.. I haven’t even discussed VMware storage path policy, as that would really be dependent on your array. However VNX is ALUA 4 and RoundRobin works really well! If you don’t have or want PowerPath as an option!


VMware Storage Guide 5.5 (PDF)

VMware Storage Guide 6.0 (PDF)

“Best Practices for Running VMware vSphere on iSCSI” (TECHNICAL MARKETING DOCUMENTATION v 2.0A)

“Using VNX Storage with VMware vSphere” EMC TechBook


HLU ID.. aka Host LUN ID.. what?

Putting on the storage admin hat.. you have to remember the little details that a good storage admin knows instinctively. I was assisting a customer with allocating new LUNS to their file system (CIFS/ NFS)… and well I gave general high level instruction but after the customer had some errors.. here are the lessons learned.

Host LUN ID.
It is important.
For example.. a boot LUN needs to have HLU = 0
That is important when you boot from SAN.
For example each ESXI must have a separate storage group for the boot LUN, as there can only be on HLU = 0 per storage group.

Also: HLU IdS 6 to 15 are reserved for system use only

Now this is different than a LUN ID.

EMC tip:
When adding a LUN to a storage group; this is the opportunity (or only time) you have to change the HLU ID.

BUT if you forgot and didn’t pay attention when adding LUNS to your storage group..
(This is disruptive. You will lose all data on LUN as it has a new HLUID)

1. remove LUNS from storage group.
2. re-assign LUNS to storage group; remember to assign HLU ID at this time >16
3. I try to match ID with HLU ID if possible.

Possible errors is HLU ID is <16
SCAN failure message = Message ID 1360601492
SCAN from GUI (unisphere) or
from CLI: nas_diskmark -m -a (login to Control Station as NASADMIN and run cli)

Also.. when dealing with VNX file systems. Do not remove backend LUN without first removing dvols.
Without deleting dvol you should not remove backend LUN.. there is some binding that happens between the two.. and well, let's just say your storage pool for file really doesn't like it.

clarsas_archive what the heck??

What is it?
"This a system definded pool. If you create a RAID group at the backend with RAID 5 type and add the LUNs from that RG to ~fs group. It will automatically create a file pool by name clarsas_archive"

Well I don't want it around.. the remove dvols and then remove the supporting backend LUNS.

Just a few VNX storage tips. Hope you find it helpful.. be social, share.

EMC, Storage, vmware

Software Defined Storage: VMware VVOL

Closer to reality… Closer to GA

Yesterday VMware announced public beta2 for vsphere6 and VVOL.

Why separate Beta programs?

While vSphere 6 can be leveraged ontop of most existing hardware that supports vSphere 5.5; not all storage vendors will be ready for VVOL.

All major storage vendors will be supporting the new VVOL feature. And the impact of this is very big.

But let’s start with some background information….

What is VVOL

**please note the capital V 🙂

VVOL is a new paradigm — a standard, a model, template for storage. You won’t be doing storage like you have in the past; well not 100% the same. Sure you will have the traditional tasks a storage admin will have to do for fabric based storage arrays.. but what makes things really interesting is what I call the integration point.

Currently Storage arrays are not VM aware; VVOL changes that. I guess you can call it adding more “intelligence” to your storage.

The conversation is storage meet this VM/Application. It isn’t about just consuming a LUN, but more about making a VMDK a native object on the storage system.

This point/ intersection is where the conversation should grab your attention.

I won’t try to recap what the status quo is today for storage and vmware, but know this: it is complex. Time consuming at a high operational cost and requires specialized training to ensure availability, management of SLA: performance and capacity.

Very important reference reading: (much more detailed information)

Cormac Hogan’s blog:


Duncan Epping:


Chad Sakac



VMworld 2013 VVOL Session

VMworld 2012 VVOL Session


And when you are ready… go and DO THE BETA! (or at least join the beta community and learn all you can!!)

Rawlinson shows you all about the beta and examples of storage vendors using VVOL



EMC, ScaleIO, Storage, Uncategorized, Virtualization, vmware

Open Source tool: Vagrant OPENSOURCE. Learn it. Use it. Software tools are great and sharing tools and supporting the OpenSource community is a good thing. This is part of a multi-part post. I will share my experience setting up ScaleIO in my vmware Fusion Lab. First find a tool to make provisioning quicker, more consistent, and automated. Tool: Vagrant From the vagrant website:

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team. To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.”

more later..   vagrant and more about


Intro to ScaleIO


EMC ScaleIO at a Glance

EMC ScaleIO is a software-only solution that uses application hosts’ local disks to realize a virtual SAN that is comparable to or better than external SAN storage, but at a fraction of the cost and complexity. ScaleIO makes a convergence of the storage and application layers possible, ending up with a wall-to-wall single layer of hosts. The lightweight software components of ScaleIO are installed on the application hosts alongside applications like databases and hypervisors.

Breaking traditional barriers of storage scalability, ScaleIO scales out to hundreds and thousands of nodes. ScaleIO’s performance scales linearly with the number of application servers and disks. With ScaleIO, any administrator can add, move, or remove servers and capacity on demand during I/O operations. ScaleIO helps ensure the highest level of enterprise-grade resilience while maintaining maximum storage performance.

ScaleIO natively supports all the leading Linux distributions, Windows Server and hypervisors and works agnostically with any SSD, HDD, and network. The product includes encryption at rest and quality of service (QoS) of performance. ScaleIO can be managed from both a command-line interface (CLI) and an intuitive graphical user interface (GUI). Deploying ScaleIO in both greenfield and existing data center environments is a simple process and takes only a few minutes.”