Monitor snapshot consolidation in VMware ESXi

This article provides information on how to monitor snapshot consolidation in VMware ESXi hosts. We will be making use of a watch command and also will dig deeper as well.

Let me give you a background on why I decided to write this article. In my current job as an Infrastructure Engineer at VMware, I deal with issues related to snapshot and consolidation on a regular basis.

I get to talk to customers and the major concerns that they have is to know what is the progress with the consolidation.

Although you can see the progress in the vSphere Client or the Web Client, it just gives you a percentage on the task completed.

If you would like to dig a little to deeper to understand on which disk is being consolidated at the time and the size of the consolidation left, you would have to get to the console of the ESXi host.

This method works withESXi 4.x and above.

Note: This method does not work if the base disks are virtual-mode RDM. You see the read and the delta files touch time, but the time stamps of the RDM pointer file are not updated.

First login as root to the ESXi host using SSH. Navigate to the Virtual Machine directory containing the virtual disks associated with the Virtual Machine that is being consolidated.

cd /vmfs/volumes/Datastore_name/Virtual_Machine_name/

Run the below command to list the content of the directory.

ls -lah

If you have any VM_NAME-00000#.vmdk or VM_NAME-00000#-delta.vmdk files within the directory, that means that the Virtual Machine has snapshots associated with it at this time.

To monitor the VMDK snapshot and base disks which are currently being updated, run this watch command:

watch -d 'ls -luth | grep -E "delta|flat|sesparse"'

Note: sesparse disks will be present from ESXi 5.5 if the size of the Virtual Disk is more than 2 TB.

where:

-d highlights the differences between successive updates.
t sorts by modification time.
l shows a long listing which displays additional file information.
u sorts by and shows access time.
h prints size in a readable format.

By default, the watch command is updated every 2 seconds.

Now, the above watch command that we saw is the most commonly used method. There is also another method that we could use which is using the vim-cmd utility.

For more detailed monitoring, type these commands to confirm that the consolidation / deletion are active.

In an SSH shell session on the ESXI host that is performing the consolidation, snapshot deletion task, run the following Command:

vim-cmd vimsvc/task_list

You see a task similar to:

# vim-cmd vimsvc/task_list
(ManagedObjectReference) [
'vim.Task:haTask-7-vim.vm.Snapshot.remove-920'
]

Now we have the details of the Snapshot which is running, we can now request for more information.

Type from within the SSH Shell :

The following command will provide you with more information on the snapshot.

vim-cmd vimsvc/task_info <followed by the task listed from the first command>

vim-cmd vimsvc/task_info haTask-7-vim.vm.Snapshot.remove-920

The output is going to look like something similar to below:

(vim.TaskInfo) {
dynamicType = <unset>,
key = "haTask-7-vim.vm.Snapshot.remove-920",
task = 'vim.Task:haTask-7-vim.vm.Snapshot.remove-920',
description = (vmodl.LocalizableMessage) null,
name = "vim.vm.Snapshot.remove",
descriptionId = "vm.Snapshot.remove", - Runnning Process
entity = 'vim.VirtualMachine:7',
entityName = "Tiny", - Virtual Machine Name
state = "running", - The status should be in running, not in a error state
cancelled = false,
cancelable = false,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 99, - progress of task
reason = (vim.TaskReasonUser) {
dynamicType = <unset>,
userName = "vpxuser",
},
queueTime = "2016-06-03T07:28:43.607887Z",
startTime = "2016-06-03T07:28:43.608304Z",
completeTime = <unset>,
eventChainId = 920,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
}

I have marked the important lines to look for in RED colour so that it is easy to read through the content.

Well, that is all I have for today. I hope it was informative and thank you for reading!