The Mystery of Incompatibility Error when Upgrading vSphere to 6.7

Recently, one of our customers asked us to upgrade the current vSphere 6.0 and vSphere 6.7. Once all the pre-requisites works are done, we scheduled the upgrade process. The steps should be fairly simple and straight forward, import the vSphere ESXi iso file into update manager, create the baseline, attach to ESXi hosts and run the upgrade. Sounds easy?  yes!!! But we end up with the incompatible error and can't continue. Tried to search google, but most of search results are referring to VIB conflict, such as "The upgrade contains the following set of conflicting VIBs", not the missing VIB error in our case.

Alright, the following is our troubleshooting process to fix the issue:
1. Instead of using update manager, we started to use the manual upgrade process to see what exactly the error is. And then we found this:
 [DependencyError]
 VIB EMU_bootbank_scsi-be2iscsi_11.4.1210.0-1OEM.600.0.0.2494585 requires ima-version2, but the requirement cannot be satisfied within the ImageProfile.
 Please refer to the log file for more details.

2. Now we found this VIB EMU_bootbank_scsi-be2iscsi is causing the trouble, but why we need this VIB? The physical servers are all Cisco UCS and not use Emulux driver.

3. To make sure, run this command to query SCSI devices
esxcfg-scsidevs -a

and we get the following results, and none of them using Emulux driver.

vmhba0  ahci              link-n/a  sata.vmhba0                             (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba1  fnic              link-up   fc.20100025b5ff0009:20100025b50a0008    (0000:62:00.4) Cisco Systems Inc Cisco VIC FCoE HBA Driver
vmhba2  fnic              link-up   fc.20100025b5ff0009:20100025b50b0008    (0000:62:00.5) Cisco Systems Inc Cisco VIC FCoE HBA Driver
vmhba3  fnic              link-up   fc.20100025b5ff0009:20100025b50a0108    (0000:67:00.4) Cisco Systems Inc Cisco VIC FCoE HBA Driver
vmhba4  fnic              link-up   fc.20100025b5ff0009:20100025b50b0108    (0000:67:00.5) Cisco Systems Inc Cisco VIC FCoE HBA Driver
vmhba32 usb-storage       link-n/a  usb.vmhba32                             () USB
vmhba33 ahci              link-n/a  sata.vmhba33                            (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba34 ahci              link-n/a  sata.vmhba34                            (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba35 ahci              link-n/a  sata.vmhba35                            (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba36 ahci              link-n/a  sata.vmhba36                            (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller
vmhba37 ahci              link-n/a  sata.vmhba37                            (0000:00:11.5) Intel Corporation Lewisburg SATA AHCI Controller

4. All cleared, so we removed the Emulux drivers

esxcli software vib remove -n scsi-be2iscsi
esxcli software vib remove -n ima-be2iscsi

Now, the incompatibility issue is gone, and we continue to update the ESXi to 6.7. The question is why this is happening. Asked our customer how the ESXi 6.0 has been installed in the past. It turns out that the engineer installed the ESXi 6.0 in the past with the standard vSphere ISO, not the customized Cisco vSphere ISO, and then installed the Cisco driver VIB. When we use Cisco customized ISO to upgrade to 6.7, it doesn't include the Emulux VIB, which caused the incompatibility issue.