Right here. Right now.

It has been a while since my last blog post, but there is a reason or in this case for me LOTS of reasons.

My intention to blog has always and still is to give back to the community. Take what you can find useful and share what you can to give back. I will post ideas and technical challenges I have come across in the field that I can share.

Well… My professional role has changed. I am less customer facing but still face the same challenges! I am putting together some cool – or what I think are cool relevant posts for the hard working techs out there doing the needful.

Stay tuned….

Change Control Station IP for VNX

The installation of a VNX2

The first day is the toughest and easiest. Like baking a cake you need to make sure all the ingredients are there and ready to go. You verify what  was ordered is installed and is in working order. Sounds simple enough?

As it turns out the initial VNX “Rack and Stack” was correct.

  • All cables were plugged in correctly
  • No amber lights on the system.
  • The DAE (Disk Assembly Enclosures) were recognized.
  • No faults in Unisphere

Ok. Good to go? No? Why. Mr. Customer provided a duplicate IP for one of the control stations. The correct IP is the same subnet and mask but needs to be changed.

Here are the steps

  1. Log onto Unisphere confirm IP information for Control Station
  2. Log onto Control station via ssh

Use root/nasadmin (default)

3. Verify status of Control station. You can make changes to the primary not standby.

In this case the primary Control station was IP’d correctly. I need to change the secondary Control station, the standby.

/nasmcd/getreason  <<– run this command as root

Here is what you will see:

output:
10 – slot_0 primary control station
11 – slot_1 secondary control station
 5 – slot_2 contacted
 5 – slot_3 contacted
4. Failover the Control station with this command:
    /nasmcd/sbin/cs_standby -failover
   
About 5 minutes later you will reboot the primary and failover to the secondary. 
5. Run this: /nasmcd/getreason
output:
11 – slot_0 secondary control station
10 – slot_1 primary control station
 5 – slot_2 contacted
 5 – slot_3 contacted

6. Log into Unisphere

Normally you login to the VNX with using the primary Control Station IP. However since it is in failover mode use the secondary IP. In this case I am using the duplicate IP and will change it to the correct IP by using the Unisphere GUI

Screen Shot 2016-02-19 at 5.10.39 PM

Log into the Unisphere GUI as root. Make sure you choose scope “local” not Global

Navigate to the “Control station Properties Tab”

Screen Shot 2016-02-19 at 5.11.43 PM

Change the IP of the Control Station

Screen Shot 2016-02-19 at 5.12.20 PM

 

Click Apply.

In Confirm Action, click OK.

WARNING:

Modifying the Control Station hostname, IP address, subnet mask, or gateway may disrupt the Unisphere software connection and any other client access to the Control Station. It may be necessary to reconnect to continue administrative activities. If you make a mistake changing the network, the Control Station may no longer be reachable remotely.

7. ssh to the primary Control station and failback. Verify that the primary control station is in slot 0

#  /nasmcd/getreason
10 – slot_0 primary control station
11 – slot_1 secondary control station
 5 – slot_2 contacted
 5 – slot_3 contacted

That is it.

 

Remember when you login to Unisphere you connect via the Primary Control Station IP, if you are failed over you will connect to the secondary Control Station.

 

Thanks for reading! If you have a more efficient way please share. Yes, you can ssh in and do all of this via cli. 😉

 

 

 

EMC Elect 2016.. by the numbers

The EMC Elect 2016 members have been announced.

PastedGraphic-5

Wow what a privileged honor to be a part of this list. This year will be filled with many new challenges as EMC will have a flurry of changes internal to help the customer make their goals a success! I refer to the Dell + EMC adventure. I look forward to helping my customers and fellow IT professionals in 2016. Don’t forget that is just the business side. The never stopping change of technology is still in full force! The information to be consumed is non-stop, and the challenges for the customer to take advantage of this is incredible.  I know I am excited for 2016. Sharing the information and empower each other!

EMC Elect 2016 — Official list

FACT:

EMC Elect Summary

  • 2015 – 102 members chosen out of 450 nominations
  • 2016 – 71 members chosen out of 700 nominations

In 2015 – There were 450 nominations leading to 200 finalists, the 102 official directory of its members for 2015
In 2016 – There were 700 nominations, leading to 150 finalists. Of those finalists, under the toughest selection process for the EMC Elect here are the 71 members for 2016

In my opinion it seems that for 2016.. the selection process was roughly 10% of the nominated persons were chosen. There was an increase in nominations, but a decrease in the number of selected. I can only imagine how much a difficult task this was. Thanks for all the hard work Mr. Mark Brown @DathBrun

Here are some interesting facts by the numbers:

 

Number crunching for 2015  Great job Mr. Henry!

My attempt inspired by Mr. Henry @davemhenry

PastedGraphic-3

PastedGraphic-4

 

I didn’t have a chance to calculate repeat EMC Elect members from former years. Metrics tell all sort of stories, but more over. Take a look at the content provided by the EMC Elect. There is A LOT!

Follow an EMC Elect member become informed and part of the discussion. Search for the tag #EMCElect.  Read the blogs (like this one) read up on the ECN (EMC Community Network). If you have a question reach out to a member via twitter or post on ECN. You will find very helpful individuals; who generally are very open to objective discussion on the technical merits of X, Y or Z!

Congrats to ALL the EMC ELECT 2016! And to all past EMC Elect… A famous quote to consider.

                   “If I have seen further than others, it is by standing upon the shoulders of giants.”                                                    -Sir Isaac Newton

@Digital_kungfu

 

 

HP C7000 iLo

What happened? Why doesn’t my iLo have an IP Address? The blade inserted has the correct profile but the iLo cannot obtain an IP address. I cannot start a remote session to the iLo….

What? What about configuration of the iLo IP Address?

HP iLo uses DHCP to obtain an IP address.

The Onboard Administrator configures this parameter.  The iLos have DHCP enabled and OA distributes addresses. Feature is called Enclosure Bay IP Addressing or EBIPA.

I was scratching my head, as it has been a while since I had to configure a new blade with the iLo.

I forget to run the full configuration for the iLo in the Onboard Administrator.

Verify the blade is functional powered on and recognized in the and the chassis isn’t throwing any errors.

Messages_Image_227066963_

Go into the OA.

Locate the Menu “First Time Setup Wizard >>> EBIPA section”

  1. Make sure the enable box is checked for the new blade
  2. Notice how the blade has not obtained an IP address yet, pending set of feature and reboot

C461BF12-14CA-4D11-87F7-F2FAAE29B20E

Once the blade has rebooted or ILo has rebooted it will have an IP address and you will be able to remotely console into the blade.

Remember to patch the latest firmware for the blade. Download the full ISO image of the SPP from HP’s site and boot from that ISO.

vmworld 2015 Highlights Day 3 VSAN 6.1

Well it is almost the end of vmworld2015. I have been incredibly busy attending so many sessions it is numbing both for the mind and my feet! The content this year has a different perspective for me. In years past it has been on the tip of everyone’s tongue a new release of x, y or z but this year it is more than that. For me it is Day 3 as I have been at vmworld 2015 since Sunday.

VSAN 6.1

This is the third major release for this product. VSAN has isn’t the new kid on the block. It is shown its ability to deliver Enterprise Class performance with the flexibility of the Software Defined Data Center. It is delivery flexibility and ease of operations as a foundation of SDS. Performance is just one of it’s attributes. The beta for a long awaited feature is announced. DEDUPLICATION

VSAN DE-DUPE BETA ”  SIGNUP FOR IT!!!

“VSAN 6.next beta – A glimpse of the future”

VSAN stretched clusters.

If you have a chance watch the session delivered by Rawlinson Rivera @Punchingclouds and Duncan Epping @duncanYB. It is a deep dive of the design of using VSAN as a stretched cluster option. Key points to remember are L2 L3 requirements and the newest feature the Witness VM. While VSAN requires L2 Multicast; the witness vm only requires L3 communication. The impact is a stretched cluster Site A and Site B while a lower latency site can host your witness VM with L3 requirements. As the Fault Domains are dependent on the use of a Witness VM and this re-design of the witness lends itself to use for VSAN as A ROBO solution.

Look for it STO5333 “Building a Stretched Cluster with Virtual SAN”

The impact of VSAN on design and deployment of a stretched / Metro cluster combined with the versatility and flexibility of vsphere 6 (Think PSC HA Think the flexibility of vmotion across VDS, VSS and different Datacenters!)

SC-VSAN

VSAN ROBO.

What is it? In nutshell VSAN can now be deployed at remote sites (Remote Office Branch Office) with only 2 nodes! Technically the third node is a witness vm.

robo

I am very excited to leverage this ROBO configuration for many customers. There is a lot of flexibility in how to implement this design. It decrease the complexity and cost of shared storage, but that is the crux of VSAN of vmware’s SDS.

Once last note about VSAN 6.1

I have had many conversations with customers and as well as the vmware VSAN team about the requirements and benefits of such architecture. This focus of this conversation was address directly in the session STO5336 “VMware Virtual SAN – Architecture Deep Dive” delivered by Rawlinson Rivera @Punchingclouds and Christos Karamanolis

Note the Hybrid Design vs the AFA VSAN are NOT the same. It that flame bait? No.

I will try to summarize it here. Both are hyper-converged SDS solutions employing the benefits of clustered storage and SSD. Both have simplified operations and lower TCO. Both deliver fantastic consistent predictable performance.

Cache is where it is at. Read Write

SSD is leverage in both the hybrid model and the AFA model but the key is the are leverage DIFFERENTLY.

70/30 Read Write for Hybrid moreover… this is the crux. The algorithm used to deliver performance for a AFA is different than the Hybrid approach.

  1.  LRU vs ARC.   That is why you cannot. I repeat CANNOT bolt on All SSD in a licensed Hybrid VSAN design and get the same performance of a  AFA solution. I have tried to explain this to a customer but one persons results skew their reasoning.  I would tend believe results that you and review that are from countless man-hours of R&D vetted against multiple manufacturers / vendors and various RA bench marks
  2.  Read Write characteristics of SSD / HDD   Not all SSD are created equal. SSD and HDD are different. vmware knows this. Hardware manufactures know this. Storage engineers know this. But Consumers sometimes know this. You have the flexibility of choosing your own hardware to fulfill your requirements for VSAN ref: vmware compatibility guide

“A candle that burns twice as bright burns half as long” 

VSAN 6.1 AFA design understands this at a much deeper level. It is designed and tested for not only performance but for longevity. VSAN 6.1 helps you sort out which SSD is appropriate.  HDD aka spinning magnetic disks can handle long term data and dense capacity at a price: IOPS. Conversely SSD can handle 10x increase in IOPS but at the cost of density and longevity.

Pick carefully your components. To borrow from the culinary world. The best ingredients make for the best food. You will consume something awesome if you cook something awesome. “You can’t make chicken salad out of chicken sh*t” lol

Reference Duncan’s explanation of the VSAN Ready Nodes http://www.yellow-bricks.com/2015/08/25/virtual-san-ready-nodes-taking-charge/

Well there is more to VSAN 6.1 and vmworld2015… more when I can find some time to share. Thanks for reading.

EMCWorld 2015 – Redefine Next … two big items to consider

emcworld

#EMCWorld

One “little” event, but this one event is in truth packed with so much activity;  you can actually feel the energy in the air.  For those who don’t know what it is about: EMC World is about: cloud computing, virtualization, Big Data storage and analytics, and IT security and protection. Each individual item alone is enough for a 4 day conference but all together under one roof. It is a lot to take in.

There is a wide audience in attendance: Vendors, Customers, Partners. So basically a little (or a lot) of something for everyone. There are multiple tracks in each topic: beginner, Intermediate and advanced. And there are different focus areas to learn about.

EMC isn’t just about storage, sure it is at its core DNA, but there is so much more to EMC than storage. If you don’t think so .. look at this:

EMC {code}

http://emccode.github.io

There are currently 33 Public Repos in development, not counting any forks.

Storage isn’t just about LUNS, RAID Groups and IOPS. No sir/ madam. DevOps is alive and well and pushing the envelop, really maximizing the technology under the hood. Making things happen you never thought possible, but more importantly redefining Next.. the next use case, the next different way to do things.

And if you are ready.. look at this current offering that well, is more mature than you might think. Don’t assume All Flash Arrays are equal. Today lots of different vendors are offering a flash array of some kind, hybrird or 100% All Flash array options.

I had a conversation recently and told my colleague I am storage agnostic, as I have worked with just about every storage vendor out there. BUT that doesn’t every storage is for everyone. You can’t force one-size fit all for storage.

There are requirements and understanding those requirements, well that is my job.

Enter…

xtremIO

xtremIO

Today I had the opportunity to meet some very incredible individuals. It may be over-used by “passionate” is what this guy is about. The details shared, I cannot discuss publicly at this time, but WOW. xtremIO 4.0 release isn’t a maintenance release it is HUGE.. to paraphrase “4.0 release is a big deal, if not as big a deal as the 1.0 release.”

Read more here:

http://www.xtremio.com

https://itzikr.wordpress.com/2015/04/10/emc-world-2015-the-xtremio-sessions/

Here is one thing to really dig deep into:

Think about how you do business today. What do you have in production? What is working? What could work better?

How could it work better?

Do you expect your Application development team to turn on a dime to adopt something new and shiny?

Agility may be a buzz word, but the cost to re-tool is a big deal. I mean, cap-ex is only one dimension. OpEX and adoption rate is not a small deal. If you have invested effort, evaluate how to make your TCO maximize the ROI. Buzz words or reality? You can’t expect to drop everything and jump to the latest and greatest just “because”. Nope. That is irresponsible. The luxury of a greenfield deployment isn’t something everyone has at their disposal.

xtremIO-ecosystem

Hint: dig deeper into “Snapshots” find about how to really leverage this for your organization.

More thoughts to consider later. Sure there are fun and games, but EMCWorld is a lot of game changing information to absorb. The theme this year is “Redefine Next” what is your next step? what is your next technology investment? Who will be on your side to put your organization in pro-active quadrant? How will you empower your team/company/self for the next 90, 180 days? How about the next fiscal year?

more later.

VMWARE Virtual SAN networking

VSAN networking can be a bit tricky to troubleshoot. Before I go deeper into the topic here is a very important concept to remember about VSAN clusters.

Given any VSAN cluster remember the following:

** “Introduction to Virtual SAN Networking

Before getting into network in detail, it is important to understand the roles that nodes/hosts can play in Virtual SAN. There are three roles in Virtual SAN: master, agent and backup. There is one master that is responsible for getting CMMDS (clustering service) updates from all nodes, and distributing these updates to agents. Roles are applied during cluster discovery, when all nodes participating in Virtual SAN elect a master. A vSphere administrator has no control over roles.”

** from Cormac’s troubleshooting guide

That is a lot to digest but if break it down you can see some key principles about a VSAN cluster to remember.

The roles in VSAN:
A master
B agent
C backup.

There is one master.
If you see more than one master there is something not quite right with you VSAN CLUSTER.

The VSAN admin does not control which node will be the master.

Example:
Log into each node of a three node VSAN. The normal pre-req for troubleshooting make sure ssh is enabled.

Run the following command on each node:
~ # esxcli vsan cluster get

Cluster Information will output below.
NODE 1

Cluster Information
Enabled: true

Current Local Time: 2015-03-30T22:38:38Z
Local Node UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Backup UUID:
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member UUIDs: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Membership UUID: a5ce1955-f5e5-5663-d338-a0369f58b8e4

Node 2
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-30T22:38:38Z
Local Node UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Backup UUID:
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member UUIDs: 55197cee-f530-4966-5ea6-a0369f58b8e4
Sub-Cluster Membership UUID: a5ce1955-f5e5-5663-d338-a0369f58b8e4

Node 3
~ # esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2015-03-30T22:56:46Z
Local Node UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Backup UUID: 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster UUID: 551374b5-03f9-7bd6-6257-a0369f58b8e8
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member UUIDs: 551374b5-03f9-7bd6-6257-a0369f58b8e8, 54f9dc6f-8674-f412-364d-a0369f58b5a8
Sub-Cluster Membership UUID: d6da1955-e2f8-38eb-d7f0-a0369f58b8e8

See the image below for the error seen in the web client.

From the output above can you see the problem?

IMG_2285.PNG

IMG_2285-0.PNG

EMC Elect 2015

347028-graphic-EMC+Elect+2015-hires.jpgI am truly honored to be chosen as a recipient of this recognition. There is always so much change happening in the world of storage. It doesn’t matter who you are a home consumer, SMB business, or large Enterprise Business. Understanding how that  change impacts you, your work, and your lifestyle.. that is the impact. That is the empowerment and that is the difference.

Old, new and not even on silicon.. Sharing about what technology does and doesn’t do … makes a difference in what technology can do and WILL do. That is a positive feedback loop. Contributing to advancement.

Here is the official announcement link:

“…

But finally the EMC Elect of 2015 were selected. Out of the 450 nominations leading to 200 finalists, the 102 official directory of its members for 2015 in alphabetical order are: …”

https://community.emc.com/community/connect/emc_elect/blog/2015/02/22/the-emc-elect-of-2015–official-list?et=blogs.comment.created#comment-36868

VMWARE PEX 2015

There is almost so much going on it is difficult to find enough time to attend every session that is of interest.

The challenge is if your interest falls in multiple areas.. then more often than not the in-depth sessions will have conflicts.

http://blogs.vmware.com/partner/tag/partner-exchange-2015

pex2015-300x153

VMware PEX (Partner Exchange) is a much different venue than VMWorld.

It isn’t that the topics are that much different. There are core areas: Virtualization, EUC, BCDR, SDDCU, Hybrid Clouds,

But hands on opportunity to talk to engineering staff and product management. You never know who you run into.

* Of course there are discounted exams, hands on training, Networking opportunities etc.

VFRC sizing considerations SSD quandry

Sure you know about it.. I am sure you have read about it.. but you even know how to enable it.
But are you sizing it correctly??
Recently a customer I was working with wanted to boot their ESX blades from RAID1 SSD drives!
Wow cool, but not cool. Let me explain…

ESXi is tiny relative to any other OS. Since this hypervisor is so small and loads entirely into memory when it boots, I explained that using the SSD for other options would be more beneficial. Using the SSD for boot would not really be taking advantage of what SSD can do for your Virtualized environment.

I didn’t go as far as to suggest using a stateless auto deploy boot system.. but something a bit easier to manage. Of course they could have used boot from SAN, but based on the reliability of their environment they wanted something a bit more independent.. flash devices fit their use case perfectly.

Now about those SSD drives.. what to do.. what to do..?

This particular customer has FC backed storage and blades so a hyper-converged solution like VSAN wasn’t a good fit. Yes 10G was available but the question was how to maximize the use of the SSD. Disclaimer I didn’t design the components of the compute nodes and local storage, but I wanted to make the most of it..
Queue in…. Enterprise Plus licensing and its additional licensed features.

“vSphere Flash Read Cache”

Some quick bullet items to summarize.

  • Hypervisor-based software defined flash storage tier solution
    • Cache is a high-speed memory that can be either a reserved section of main memory or a storage device
    • Supports Write-Through Cache Mode
    • Improve VMs performance by leveraging local flash devices
    • Ability to virtualize business critical applications
  • Aggregates local flash devices to provide a clustered flash resource for VM and vSphere host consumption
  • Leverages local flash devices as cache
  • Integrated with vCenter, HA, DRS and vMotion
    • VFFS Volume is created
    • Fine-grained control via per VMDK caching
    • Block Size configuration capabilities Min 4KB – Max 1024KB
    • Virtual Flash Host Swap Cache
      • Provides the ability to use virtual flash resource for memory swapping
      • Legacy support for Swap-to-SSD
    • Scale-out Storage Capability: 32 Nodes
    • Requirements
      • vSphere Server 5.5
      • vSphere Web Client
      • VM Hardware version 10
      • License – vSphere Enterprise Plus

In a nutshell 20% of VMDK size is used to determine size of VFRC

Interesting white paper analysis of VFRC
http://www.hds.com/assets/pdf/benefits-of-server-side-flash-with-vmware-vsphere-flash-read-cache-on-hitachi-compute-blade-500.pdf

There are many articles of how to use, how to implement but few articles of benchmarks

http://cormachogan.com/2014/02/14/a-closer-look-at-vsphere-flash-read-cache-vfrc/

http://www.yellow-bricks.com/2013/08/26/introduction-to-vsphere-flash-read-cache-aka-vflash/

http://www.vmware.com/files/pdf/techpaper/vfrc-perf-vsphere55.pdf

http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-Flash-Read-Cache-FAQ.pdf

Advanced professional services would go beyond recommending use of VFRC but actually creating a test bed for the customer to determine the best size of and which block size to use.
default block size is 8kb

Assessment tool would be
vscsiStats
http://nickmarshall.com.au/blog/2013/10/31/rightsizing-vflash-read-cache-vfrc

Notes to share..

-vp