XenDesktop 4 : The delivery controller cannot find any available virtual desktops Error 1301

In a new mission, I had to learn a new environment based on Citrix XenDesktop 4, Provisioning Services 5.6 and vmware vSphere 4.1. This week, I had a weird issue, I didn’t change anything, I just didn’t understood why suddenly VMs stopped to be available, in fact VMs were available but for some reason, it was impossible for everyone to access it through the Web Interface.

VMs were working well
XenDesktop brokers were fine
Web Interface was ok
Citrix License Server was up and running with correct license
vmware vSphere was ok as well, VMs were running without any problem on it

On the Web Interface, the following message was display while trying to launch a XenDesktop virtual desktop : “xxxxx is currently unavailable. try reconnecting and, if the problem persists, contact your administrator.”

On the DDC, XenDesktop Desktop Delivery Controller, I found event logs with ID 1301, source : Citrix Desktop Delivery Controller, with the following description : “The delivery controller failed to broker a connection for user xxxxx to desktop group yyyyy. The delivery controller cannot find any available virtual desktops.

Please add more virtual desktops to the desktops group. If the problem is due to existing virtual desktops not becoming available, refer to Citrix Knowledge Base article CTX117248 for further information.”

It look like a communication problem between XenDesktop DDCs and vmware Virtual Center, so I checked every component, DDCs, Virtual Center…. I found nothing really relevant. So the next step was to enable extended logs on the DDC side, after a short search on Citrix website I found how to do do with CTX117452. I got a lot of logs, but after one day scratching my head to try to understand why without changing anything I had such behavior, I just had bunch of logs but I was missing something…

Here is a short part of the pool_log.log file :

pool_log.log

13/07/11 15:40:41.8971 : ActionOnDisconnect=DoNothing
13/07/11 15:40:41.8971 : ActionOnLogOff=DoNothing
13/07/11 15:40:41.8971 : DisconnectActionDelay=300
13/07/11 15:40:41.8971 : LogOffActionDelay=420
13/07/11 15:40:41.8971 : MaximumStateAgeForDecisions=5
13/07/11 15:40:41.8971 : MaximumStateAgeWithNotify=600
13/07/11 15:40:41.8971 : MaximumStateAgeWithoutNotify=30
13/07/11 15:40:41.8971 : MinimumOperationInfoAge=7200
13/07/11 15:40:41.8971 : TaintAction=DoNothing
13/07/11 15:40:41.8971 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1720816 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.8971 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1720816 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.8971 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722195 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.8971 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722195 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.8971 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722235 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.8971 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722235 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.8971 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722229 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.8971 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722229 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.9128 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722237 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.9128 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722237 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.9128 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722238 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.9128 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722238 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.9128 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722239 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.9128 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722239 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.9128 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722240 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.9128 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722240 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.9128 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722241 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.9128 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722241 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareSvcUtil.GetVmInfo(VmwareVmConnection connection, String configKey, String configValue)
at Citrix.PoolManagement.VMManager.VmmImplementation.Vmware.VmwareVmManager.GetMachineInfoByGuestOSId(String[] guestOSIds)
13/07/11 15:40:41.9128 : VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722242 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.
13/07/11 15:40:41.9128 : Citrix.PoolManagement.MachineManager.MachineManagementApi.NoSuchGuestOSIdException: VM info for VM with guestOsId S-1-5-21-839522115-287218729-725345543-1722242 is not loaded in the cache. Please verify the guestOsId is correct and the read-only privilege is set one level above this VM. Otherwise it could be a caching error.

I also took a log in the virtual desktops pool properties to check if everything was alright… It wasn’t :

All the VMs within the Citrix Delivery Service Console were disassociated with Active Directory and a message confirm I had communication problem between vmware Virtual Center and Citrix XenDesktop : “Virtual machines could not be retrieved from the hosting infrastructure” and then a pop-up saying : “Error occurred whilst validating the list of virtual desktops. For more information about each error, hover the mouse over the error icon for a virtual desktop in the list”

I made a quick tweet about my issue, Ryan Gallier asked me to jump on CitrixIrc.com channel, I went in there and we were trying to look around what was wrong… Jarian Gibson arrived in the chat and after watching logs asked me if my GuestOsIds were fine on the Virtual Center side… He was right, all the GuestOsIds were missing…

Jarian found the issue, GuestOsId were missing, but what is this value ? Jarian send me a very interesting blog about that : XenDesktop on vmware infrastructure @ WWT Datacenter Services Team Blog and I found a Citrix Knowladge Base article after-while The Virtual Machine, within the Access Management Console, becomes Disassociated with Active Directory Object.

Now why these value disappeared ?vmware guy where I had this issue just deleted these entry because he didn’t knew it was used by Citrix XenDesktop to identify virtual machines on the vmware Virtual Center.

Now I knew why I had issue, I needed to know how to but everything back online with correct values. One way was to delete all the virtual machines from the pool properties and put it back, so the value will be recreated in the Virtual Center. And there is the scripted automated way but that would have supposed I had export a correct workings list of VMs for each Virtual Desktop Pool properties before… Instead I made a CSV file with Virtual Machines for each Virtual Desktop Pool and I import in each pool the list of available desktops.