The fundamentals cannot be over-emphasized. You need to ensure that the key components of your vSAN host is configured per recommendations.
Just a reminder of the HBA controller configuration.
- Make sure the device is on the Hardware Compatibility Guide (HCG)
- And verify the firmware is up-to-date.
I have seen first hand what impact different firmware can have on your environment.
Example: Dell Perc H310
Controller queue depth impacts the rebuild/resync times. A low controller queue depth may impact the availability of your production VMs during rebuild/resync. A minimum queue depth of 256 is required in vSAN. Some vSAN Ready Node profiles require minimum queue depth of 512, All Flash configs.
For more details see this: vSAN Hardware Quick Reference Guide
The availability of vSAN and VMFS can be vying for the same resource; the HBA.
Do NOT mix Disk Access modes to your Host Bus Adapter (HBA) also called an I/O Controller. Pass through configuration is preferred, but RAID-0 can work. vSAN prefers to have a more direct access to the device attached to the I/O Controller. So for example if the HBA is setup with some logic configuration the groups all the devices together before presenting to the ESXi host then you have some prep work to do. Several array controllers do not support pass through mode, to use this type of controller for vSAN, we need to create a single disk RAID-0 group for every SSD and HDD.
- RAID levels access for the devices attached.
- vSAN and VMFS devices on same HBA.
From the VMware KB:
- Do not mix the controller mode for vSAN and non-vSAN disks.
- If the vSAN disks are in pass-through/JBOD mode, the non-vSAN disks must also be in pass-through/JBOD mode.
- If the vSAN disks are in RAID mode, the non-vSAN disks must also be in RAID mode.
- Mixing the controller mode will mean that various disks will be handled in different ways by the storage controller. This introduces the possibility that issues affecting one configuration could also affect the other, with possible negative consequences for vSAN.
If you absolutely must use the same HBA:
- limit the use of the VMFS that is sharing the HBA with vSAN.
- AND DO NOT USE RDM for that shared device/HBA
- DO NOT have the boot device on the same vSAN controller
- If the non-vSAN disks are in use for VMFS, the VMFS datastore should be used only for scratch, logging and coredumps.
- Virtual machines should not be running from a disk or RAID group that shares its controller with vSAN disks or RAID groups.
- ESXi host installation is permitted on non-vSAN disks attached to same controller.
- Do not pass through non-vSAN disks to virtual machine guests as Raw Device Mappings (RDMs).
The number and type of drives plus their disk group configuration is not covered here but another topic of important discussion!