In some older releases of vSAN there is an issue where if applications are using AVX-512 CPU instructions data corruption can occur. This was first reported by customers using DB2 but can occur with other applications as well. Here is the KB describing the problem: https://knowledge.broadcom.com/external/article/367589/applications-using-avx512-instructions-i.html
Note that the issue has been already fixed in all ESX/vSAN branches so the best way to remediate is to patch your environment. However, if immediate patching is not an option, and you need a quick workaround before the patch can be applied then read on.
To quickly check if a particular VM can use AVX-512 instructions you can run this linux command:
cat /proc/cpuinfo |grep avx512
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xsaves arat pku md_clear flush_l1d arch_capabilities
For Windows VM you can use HWiNFO.
To reproduce the issue, the simplest test is to use ssh to transfer data and check for corruption. Ssh is using openssl that might leverage AVX-512 instructions.
Here I am using Photon OS 5.0.
$ dd if=/dev/zero bs=1M count=1024 | sha256sum - > sha256sums.zero # one time checksum generation
$ ( while true; rm -f /tmp/zero.$$.img ; do ssh root@localhost "dd if=/dev/zero iflag=fullblock bs=1M count=1024 status=progress" | dd iflag=fullblock bs=1M status=none of=/tmp/zero.$$.img && sha256sum - < /tmp/zero.$$.img | diff -u sha256sums.zero - || exit 1; done )After a while the checksum will fail:
...
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.11915 s, 507 MB/s
1013972992 bytes (1.0 GB, 967 MiB) copied, 2 s, 507 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.09118 s, 513 MB/s
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2 s, 532 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.01729 s, 532 MB/s
--- sha256sums.zero 2024-10-21 10:44:48.142238490 +0000
+++ - 2024-10-21 10:54:17.289072688 +0000
@@ -1 +1 @@
-49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14 -
+4e841c1d34a2a051fab082d10a8105cd34e10c95ee74ce20d280a7c022eaaa53 -
The workaround can be done either at the application level (see the DB2 specific instruction in the KB above), OS level where the CPU instructions flags can be masked in kernel (append clearcpuid=304 at the end of the grub commandline) or the best at the virtual hardware level which is obviously the most universal as it is neither application nor OS specific.
The following two Advanced Configuration Parameters will mask the problematic AVX-512 CPU instructions:
featMask.vm.cpuid.AVX512F = “Max:0”
featMask.vm.cpuid.AVX512FP16 = “Max:0“
They can be applied for each VM in vCenter, or alternatively in VMware Cloud Director environment we can create Compute Policy (in UI called VM Sizing Policy) that the end-users can apply to their VMs.
