esxtop shows a bunch of stuff … what’s up with this!?

It’s been a while since I’ve had to run esxtop, but when I did recently, I got a bunch of stuff on my screen like this:

o termcap entry for a `xterm-256color’ terminal”(PDH-CSV 4.0) (UTC)(0)”,”\\\Memory\Memory Overcommit (1 Minute Avg)”,”\\\Memory\Memory Overcommit (5 Minute Avg)”,”\\\Memory\Memory Overcommit (15 Minute Avg)”,”\\\Physical Cpu Load\Cpu Load (1 Minute Avg)”,”\\\Physical Cpu Load\Cpu Load (5 Minute Avg)”,”\\\Physical Cpu Load\Cpu Load (15 Minute Avg)”,”\\\Physical Cpu(0)\% Processor Time”,”\\\Physical Cpu(1)\% Processor Time”,”\\\Physical Cpu(2)\% Processor Time”,”\\\Physical Cpu(3)\% Processor Time”,”\\\Physical Cpu(_Total)\% Processor Time”,”\\\Physical Cpu(0)\% Util Time”,”\\\Physical Cpu(1)\% Util Time”,”\\\Physical Cpu(2)\% Util Time”,”\\\Physical Cpu(3)\% Util Time”,”\\\Physical Cpu(_Total)\% Util Time”,”\\\Physical Cpu(0)\% Core Util Time”,”\\\Physical Cpu(1)\% Core Util Time”,”\\\Physical Cpu(2)\% Core Util Time”,”\\\Physical Cpu(3)\% Core Util Time”,”\\\Physical Cpu(_Total)\% Core Util Time”,”\\\Memory\Machine MBytes”,”\\\Memory\Kernel MBytes”,”\\\Memory\NonKernel MBytes”,”\\\Memory\Free MBytes”,”\\\Memory\Kernel Managed MBytes”,”\\\Memory\Kernel MinFree MBytes”,”\\\Memory\Kernel Reserved MBytes”,”\\\Memory\Kernel Unreserved MBytes”,”\\\Memory\Kernel State”,”\\\Memory\PShare Shared MBytes”,”\\\Memory\PShare Common MBytes”,”\\\Memory\PShare Savings MBytes”,”\\\Memory\Swap Used MBytes”,”\\\Memory\Swap Target MBytes”,”\\\Memory\Swap MBytes Read/sec”,”\\\Memory\Swap MBytes Write/sec”,”\\\Memory\Total Compressed MBytes”,”\\\Memory\Total Saved By Compression MBytes”,”\\\Memory\Memctl Current MBytes”,”\\\Memory\Memctl Target MBytes”,”\\\Memory\Memctl Max MBytes”,”\\\Power\Power Usage Now Watts”,”\\\Power\Power Usage Cap Watts”,”\\\Power(0)\P-State MHZ”,”\\\Power(1)\P-State MHZ”,”\\\Power(2)\P-State MHZ”,”\\\Power(3)\P-State MHZ”,”\\\Power(4)\P-State MHZ”,”\\\Power(5)\P-State MHZ”,”\\\Power(6)\P-State MHZ”,”\\\Power(7)\P-State MHZ”,”\\\Power(8)\P-State MHZ”,”\\\Power(9)\P-State MHZ”,”\\\Power(10)\P-State MHZ”,”\\\Power(11)\P-State MHZ”,”\\\Power(12)\P-State MHZ”,”\\\Power(13)\P-State MHZ”,”\\\VSAN\Enabled?”,”\\\Group Cpu(1:idle)\Members”,”\\\Group Cpu(1:idle)\% Used”,”\\\Group Cpu(1:idle)\% Run”,”\\\Group Cpu(1:idle)\% System”,”\\\Group Cpu(1:idle)\% Wait”,”\\\Group Cpu(1:idle)\% Ready”,”\\\Group Cpu(1:idle)\% Idle”,”\\\Group Cpu(1:idle)\% Overlap”,”\\\Group Cpu(1:idle)\% CoStop”,”\\\Group Cpu(1:idle)\% Max Limited”,”\\\Group Cpu(1:idle)\% Swap Wait”,”\\\Vcpu(1:idle:32769:idle1)\% Used”,”\\\Vcpu(1:idle:32769:idle1)\% Run”,”\\\Vcpu(1:idle:32769:idle1)\% System”,”\\\Vcpu(1:idle:32769:idle1)\% Wait”,”\\\Vcpu(1:idle:32769:idle1)\% VmWait”,”\\\Vcpu(1:idle:32769:idle1)\% Ready”,”\\\Vcpu(1:idle:32769:idle1)\% Idle”,”\\\Vcpu(1:idle:32769:idle1)\% Overlap”,”\\\Vcpu(1:idle:32769:idle1)\% CoStop”,”\\\Vcpu(1:idle:32769:idle1)\% Max Limited”,”\\\Vcpu(1:idle:32769:idle1)\% Swap Wait”,”\\\Vcpu(1:idle:32770:idle2)\% Used”,”\\\Vcpu(1:idle:32770:idle2)\% Run”,”\\\Vcpu(1:idle:32770:idle2)\% System”,”\\\Vcpu(1:idle:32770:idle2)\% Wait”,”\\\Vcpu(1:idle:32770:idle2)\% VmWait”,”\\\Vcpu(1:idle:32770:idle2)\% Ready”,”\\\Vcpu(1:idle:32770:idle2)\% Idle”,”\\\Vcpu(1:idle:32770:idle2)\% Overlap”,”\\\Vcpu(1:idle:32770:idle2)\% CoStop”,”\\\Vcpu(1:idle:32770:idle2)\% Max Limited”,”\\\Vcpu(1:idle:32770:idle2)\% Swap Wait”,”\\\Vcpu(1:idle:32771:idle3)\% Used”,”\\\Vcpu(1:idle:32771:idle3)\% Run”,”\\\Vcpu(1:idle:32771:idle3)\% System”,”\\\Vcpu(1:idle:32771:idle3)\% Wait”,”\\\Vcpu(1:idle:32771:idle3)\% VmWait”,”\\\Vcpu(1:idle:32771:idle3)\% Ready”,”\\\Vcpu(1:idle:32771:idle3)\% Idle”,”\\\Vcpu(1:idle:32771:idle3)\% Overlap”,”\\\Vcpu(1:idle:32771:idle3)\% CoStop”,”\\\Vcpu(1:idle:32771:idle3)\% Max Limited”,”\\\Vcpu(1:idle:32771:idle3)\% Swap Wait”,”\\\Vcpu(1:idle:32778:idle0)\% Used”,”\\\Vcpu(1:idle:32778:idle0)\% Run”,”\\\Vcpu(1:idle:32778:idle0)\% System”,”\\\Vcpu(1:idle:32778:idle0)\% Wait”,”\\\Vcpu(1:idle:32778:idle0)\% VmWait”,”\\\Vcpu(1:idle:32778:idle0)\% Ready”,”\\\Vcpu(1:idle:32778:idle0)\% Idle”,”\\\Vcpu(1:idle:32778:idle0)\% Overlap”,”\\\Vcpu(1:idle:32778:idle0)\% CoStop”,”\\\Vcpu(1:idle:32778:idle0)\% Max Limited”,”\\\Vcpu(1:idle:32778:idle0)\% Swap Wait”,”\\\Group Cpu(2:system)\Members”,”\\\Group Cpu(2:system)\% Used”,”\\\Group Cpu(2:system)\% Run”,”\\\Group Cpu(2:system)\% System”,”\\\Group Cpu(2:system)\% Wait”,”\\\Group Cpu(2:system)\% Ready”,”\\\Group Cpu(2:system)\% Idle”,”\\\Group Cpu(2:system)\% Overlap”,”\\\Group Cpu(2:system)\% CoStop”,”\\\Group Cpu(2:system)\% Max Limited”,”\\\Group Cpu(2:system)\% Swap Wait”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Used”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Run”,”\\\Vcpu(2:system:32772:SVGAConsole)\% System”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Wait”,”\\\Vcpu(2:system:32772:SVGAConsole)\% VmWait”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Ready”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Idle”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Overlap”,”\\\Vcpu(2:system:32772:SVGAConsole)\% CoStop”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Max Limited”,”\\\Vcpu(2:system:32772:SVGAConsole)\% Swap Wait”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Used”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Run”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% System”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Wait”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% VmWait”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Ready”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Idle”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Overlap”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% CoStop”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Max Limited”,”\\\Vcpu(2:system:32773:debugtermlivedump)\% Swap Wait”,”\\\Vcpu(2:system:32774:logSysAlert)\% Used”,”\\\Vcpu(2:system:32774:logSysAlert)\% Run”,”\\\Vcpu(2:system:32774:logSysAlert)\% System”,”\\\Vcpu(2:system:32774:logSysAlert)\% Wait”,”\\\Vcpu(2:system:32774:logSysAlert)\% VmWait”,”\\\Vcpu(2:system:32774:logSysAlert)\% Ready”,”\\\Vcpu(2:system:32774:logSysAlert)\% Idle”,”\\\Vcpu(2:system:32774:logSysAlert)\% Overlap”,”\\\Vcpu(2:system:32774:logSysAlert)\% CoStop”,”\\\Vcpu(2:system:32774:logSysAlert)\% Max Limited”,”\\\Vcpu(2:system:32774:logSysAlert)\% Swap Wait”,”\\\Vcpu(2:system:32775:serialLogger)\% Used”,”\\\Vcpu(2:system:32775:serialLogger)\% Run”,”\\\Vcpu(2:system:32775:serialLogger)\% System”,”\\\Vcpu(2:system:32775:serialLogger)\% Wait”,”\\\Vcpu(2:system:32775:serialLogger)\% VmWait”,”\\\Vcpu(2:system:32775:serialLogger)\% Ready”,”\\\Vcpu(2:system:32775:serialLogger)\% Idle”,”\\\Vcpu(2:system:32775:serialLogger)\% Overlap”,”\\\Vcpu(2:system:32775:serialLogger)\% CoStop”,”\\\Vcpu(2:system:32775:serialLogger)\% Max Limited”,”\\\Vcpu(2:system:32775:serialLogger)\% Swap Wait”,”\\\Vcpu(2:system:32776:tlbflushcount)\% Used”,”\\\Vcpu(2:system:32776:tlbflushcount)\% Run”,”\\\Vcpu(2:system:32776:tlbflushcount)\% System”,”\\\Vcpu(2:system:32776:tlbflushcount)\% Wait”,”\\\Vcpu(2:system:32776:tlbflushcount)\% VmWait”,”\\\Vcpu(2:system:32776:tlbflushcount)\% Ready

Continue reading

How to set the NTP server on all ESX hosts in vCenter at the same time with PowerCLI

I was frustrated with a number of my lab hosts have the wrong times/timezones. Rather than going to each ESXi host to update the time servers, I ran a simple script to do it with PowerCLI.

As before running any script, you must authenticate with the vCenter or the host by running

Connect-VIServer <hostname>

Then, we’ll cycle through each of the ESXi hosts and add an ntp server. Obviously, use the time server you want i.e. or 🙂

Get-VMHost | Add-VMHostNtpServer -ntpserver ""

With that set, you can then move on to punching the hole through the firewall to request the time from the servers.

get-vmhost | Get-VMHostFirewallException | where {$_.Name -eq "NTP client"} | Set-VMHostFirewallException -Enabled:$true

#Start NTP client service and set to automatic
get-vmhost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Start-VMHostService
get-vmhost | Get-VmHostService | Where-Object {$_.key -eq "ntpd"} | Set-VMHostService -policy "automatic"

How to use PowerCLI to register or unregister a bunch of templates at a time …

I recently resignatured a datastore that housed all of my templates and ran into an issue with a bunch of datastores that wouldn’t unmount and a slew orphaned templates. To get rid of them, I had to unregister them all and then register them, but had a ton, so I didn’t want to do them one at a time.

Unregistering the VM templates is easy. Just select them all, right click and click “remove from inventory”. The registering part is harder – need to browse the datastore and then right click and add to inventory and give it a name. So, I did it with PowerCLI.

The first thing we want to do is get a list of all of the vmtx files in the datastore that the templates reside in. I do it like this:

$vmtxs = gci vmstore:\$((Get-Datacenter -name "DCpod"))\templates -Recurse -Include *.vmtx | select -exp datastorefullpath

You can list it and you should see something like this:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> $vmtxs
[templates] Win8.1_Template/Win8.1_Template.vmtx
[templates] Windows 10 template/Windows 10 template.vmtx
[templates] Win2K8R2_Template_1/Win2K8R2_Template.vmtx
[templates] Ubuntu_Server_Template/Ubuntu_Server_Template.vmtx
[templates] Win2K12R2_Template/Win2K12R2_Template.vmtx
[templates] webservertemplate2/webservertemplate2.vmtx

With the list of files, you can loop through and register them all on a particular host like this:

PowerCLI C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> foreach($vmtx in $vmtxs)
$currentname = ($vmtx -split "/|\.")[1]
$vm = new-template -name $currentname -templatefilepath $vmtx -vmhost


ESX 6.0 – The operation is not allowed in the current connection state of the host.


The things in the posting did NOT work. According to IBM’s website, the fix is by removing a package:

esxcli software vib remove -n intel-cdmv2-provider

I got this message when trying to power on a vm, trying to deploy a template, and something else.

So I read this
and then I go to the host and try restarting hostd (/etc/init.d/hostd restart) and it will no longer restart.

Looking into /var/log/hostd.log, it appears that /tmp is full:

2016-02-09T06:01:01.648Z info hostd[79EC2B70] [[email protected] sub=Vimsvc.ha-eventmgr] Event 620 : The ramdisk ‘tmp’ is full.
As a result, the file /tmp/auto-backup.6942719/local.tgz could not be written.

So I look into the /tmp directory and find a file that’s 256MB – cimple_log_err_messages

cat cimple_log_err_messages
2016/02/09 23:28:57:LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var defined. Looking for CIMPLE_HOME
2016/02/09 23:28:57:LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var defined. Looking for CIMPLE_HOME

The first thing I do is to clear up the file – just run:
> cimple_log_err_messages

Then I’m able to start hostd, but how do I stop the file from filling up again?

I find this kb:

Not that it helps! It’s for 5.x. I go to the IBM url and it just leaves me more confused.

I look into disabling CIM. I find instructions here:

ummm it’s for 5.x again. 🙁
Run the instructions and the sfcb watchdog just restarts by itself!

Finally, I poke around in the vSphere client and notice in the profile that there is some CIM stuff there. For the heck of it, I try disallowing communication with IBMIMM.

Screen Shot 2016-02-10 at 11.45.09 AM

I believe the log messages continued after disabling it, so I continued to disable CIM as well. After disabling CIM, the log messages stopped.

Screen Shot 2016-02-29 at 12.14.11 PM


Getting VMware PowerCLI 6 to work on Windows 10

UPDATE: PowerCLI 6.3R1 has been released and fully supports  Windows 10, so there’s no need to do the hack anymore!


PowerCLI 6 is not supported on Windows 10 yet, but I just thought I would give it a try anyway. Upon installation of PowerCLI 6 on Windows 10, I was greeted with this unwelcome screen:


Luckily and very quickly, I found this link:

Just an environment variable. Need to add PSModulePath with “C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Modules” as the value and reboot.

Screen Shot 2016-02-18 at 12.31.24 PM

How to expand a LUN with SteelFusion

 Why would you want to do this?

No matter how well you plan, resources are almost never in abundance. There could be a few reasons why you want to expand a LUN. Perhaps you want to add more vmdks to the same VMFS volume. Another reason could be that you might want to increase the size of the partition of the guest VM. In either case, you might want some of the workarounds listed at the end of the article before continuing. If you’re convinced that expanding the LUN is what you’re going to do, here is how to do it.

Check feasibility. First check at the maximum size and expandability of the partitions involved.

  • Check the guest OS partition. If NTFS, the maximum partition size for NTFS is 256TiB according to You probably don’t need to go beyond that.
  • Check the partition size of VMFS. Consult VMware documentation to ensure that you do not exceed maximum size for your partition. For example, if your partition is VMFS-3 and formatted with a 1MB block size, your maximum is 2TB-512bytes. If you’re already at the maximum, see workarounds further in this document.

These are the steps in a nutshell:

  • Resize/expand the LUN on the backend array.
  • If the SteelFusion Core is a virtual machine and LUN is connected via fiber, run a Rescan/Refresh on the HBA on the ESXi server where the Core is hosted.
  • On SteelFusion Core, perform a LUN rescan
  • Following VMware documentation, resize the VMFS volume
  • Resize or map the device in Windows

Resizing the LUN on the backend array.

Consult your array’s documentation for resizing the LUN. Here’s an illustration on how it’s done with a Netapp array.

1) Locate the LUN you want to expand, right click and select Edit.


2) In the Edit LUN window, assign the new LUN size, and click save and close.


Seeing the new size on the SteelFusion Core

The SteelFusion Core may automatically detect the change in LUN size depending on how busy the LUN is. If there are read/write operations from the backup with a “LUN configuration changed” status, the Core will be forced to rescan and retry and it will automatically change. There’s also a periodic rescan every 5 minutes that would cover change it. If you need the change immediately, you can do a manual rescan and force the change. The size will highlight in orange when it changes.


Once the Core sees the change, the new size will be propagated to the Edge. If the LUN was pinned, the Edge will approve or deny the size increase based on Blockstore capacity.

Extending the VMFS volume size

Reference this VMware KB article for extending your VMFS volume. Here’s how it’s done with the vSphere Web Client if you’re using vCenter:


Find the Datastore, right click and select Increase Datastore Capacity.


Select Use Free space to expand the Datastore.


If you’re using the deprecated Windows client, you can effectively do the same in the Datastore view here:


Select the Datastore you are expanding, go to the Configuration tab and select the Properties of the Datastore. In the Properties box, Select Increase.


Follow the wizard and it will expand the volume.

Expanding the size of the virtual disk on the VMFS volume

If the UI doesn’t allow you perform the following steps, refer to this VMware KB:

Find the VM settings and change the size of the VMDK.


In the Windows client, it’s in the same area.


Expanding the partition in Windows

Lastly, go into the Windows guest and expand the volume size.

First, go to the Disk Management in the Computer Management application, right click and select Rescan Disks.


Upon rescanning, you should see the newly added space as unallocated.


Right click on the primary partition of the disk and select Extend Volume.


At completion, you now should have your expanded disk.


Consider inflating the VMDK file after doing the extension during non-peak hours. Because the added capacity that you just added to the VMDK file is sparse, you may want to fill it up with zeroes. The reasoning behind this is so that you are not struck with first write or double-write penalty (in all copy-on-write disks) when your guest writes to the disk. You can read a little about the penalty here:

You can do this in the Datastore browser. Just find the VMDK file of your VM, right click and select Inflate.


Functionality also is available in the Windows client.



Here are some workarounds to consider LUN expansion is not feasible:

  • Consider moving swap files to another VMFS volume, preferably on a Local LUN. You can do this in the Options tab of a VM’s settings. This will require your VM to be shutdown, but the benefit of having this on a Local LUN is that it does not need to be replicated.
  • Consider moving guest page file, swap files, and print spools to another drive. Like workaround #1, these things do not need to be replicated, so moving them to a Local LUN can help with your WAN traffic.

Not-recommended workaround

A workaround that may sound logical, but is not recommended is to Storage vMotion the disk to another LUN. This should be your last resort. This may be something that vSphere administrators do commonly, but because the Blockstore on the Edge is a cache and not ordinary storage, this will result in a lot of WAN traffic and can be potentially dangerous.

Converting a VMware Fusion VM to ESXi

One of the easiest ways to get a VMware Fusion VM up to ESXi is to use the Upload to server option
Screen Shot 2015-12-14 at 12.17.08 PM

If your VM was installed in more seamless mode though, after getting your VM up and running in ESXi, your local folders will be missing.
Screen Shot 2015-12-14 at 12.21.19 PM

I love running my desktop in more seamless mode in Fusion, but obviously, this won’t work in ESXi. To fix this, go into the VM and make the following changes.

Navigate to these two locations:

– HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders
– HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders

Make sure that you see these values (among others).  Update these entries, if they have a different data value.

  1. Navigate to these two locations:

    • HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders
    • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders

  2. Make sure that you see these values (among others). Update these entries, if they have a different data value:

    Windows XP / Server 2003 

    Name Data value
    Desktop %USERPROFILE%\Desktop
    My Music %USERPROFILE%\My Documents\My Music
    My Pictures %USERPROFILE%\My Documents\My Pictures
    Personal %USERPROFILE%\My Documents

    Windows Vista / 7 / Server 2008 / 8 / 10

    Name Data value
    Desktop %USERPROFILE%\Desktop
    My Music %USERPROFILE%\Music
    My Pictures %USERPROFILE%\Pictures
    Personal %USERPROFILE%\Documents