Better Understanding the VMware App Volumes Agent

This blog post is not intended to explain the basic of the App Volumes solution. If you’re looking for basic information on App Volumes, you should look at the VMware EUC Channel here.

Getting Started

The App Volumes agent is installed on a non-persistent guest OS. The guest could be a physical or virtual desktop or server.

Attaching AppStacks
AppStacks are assigned to users or computers. For user-based assignments, AppStacks are attached to the guest at logon. For computer-based assignments, AppStacks are attached to the guest at startup.

In each case, the AppStack attachment process has two distinct operations:
1. Disk Mount
The AppStack VMDK is attached to the guest.
2. Content Virtualization
The files and registry entries in the AppStack are virtualized or merged with the guest OS.

How it Works
The App Volumes agent is a minifilter driver. Having that minifilter driver in place is what enables App Volumes to virtualize the content of the “C: Drive” and the Windows Registry. Microsoft uses a concept of altitudes for filter drivers. The concept is the larger the number the “higher” the altitude. Minifilter drivers can only see other filter drivers that are at a higher altitude.  The actions done at a lower altitude are not able to be seen by filter drivers operating at a higher level. 

The lower number minifilter drivers (#2 in image above) are the first to interact with a request from the OS or other applications. Generally speaking the requests are then given to the next minifilter driver in the stack (highest number) after the first is finished processing the request. However, this is not always the case as some minifilter drivers may not release the request and “close” it out to the OS or application. Anti-Virus and other security software are famous for doing these types of actions. The Security software will sometime see the App Volumes Agent task as attaching external drives, so, by default, it tries to scan or “hold” the task.

In this case the other minifilter driver would never have seen the request at all. If this happens with an application running at a lower altitude than App Volumes, then the App Volumes minifilter driver would never get a chance to process the request, and thus would not be able to virtualize the I/O as expected.
 Legacy file system filter drivers (#3 in the image above) add additional complexity in that they can attach only to the top of an existing file system driver stack and cannot attach in the middle of a stack. As a result, interaction with App Volumes can get interrupted because the legacy drivers will be loaded at a higher level in the stack.
This is the primary reason certain applications that use a minifilter driver should be disabled or removed from the OS while installing applications with App Volumes and vice versa. There might be additional scenarios where App Volumes should be disabled allowing other applications to install correctly in the base OS.
 In addition, there will potentially be entire applications that will not be able to work alongside App Volumes on the same OS image. 

Troubleshooting tips
One of the scenarios to troubleshoot with the App Volumes Agent is to find out if there are external filter drivers conflicting with that agent. In most cases, security applications will have built-in safeguard that will prevent you from stopping their services, as a security measure. What you can potentially do, is to go to a command prompt and look at the currently loaded Filter Drivers.

You can do so by opening a Command Prompt, with Administrative Privilege, and typing: “FLTMC” and hitting enter.

On a Windows 10 system, with the App Volumes Agent installed, your output will look something like this (the list depends on the drivers used on your system):

You can see from the image that the VMware App Volumes filter Driver (svdriver) has an altitude of 190,000. If you want more information on attribution of the Altitude level, you can consult the Microsoft link, altitude assigned by Microsoft.

From the image above, you can also see that the driver called “wdfilter”, which stands for Windows Defender, has an altitude of 328,010. So, this driver could potentially impact the App Volumes driver, since it is at a higher altitude.

In most cases, one of two things can happen when these type of conflicts arise. Either the App Volumes AppStack does not attach at all or there are long delays because the other filter driver is scanning every file/registry entries on attachment.

Here is another example where the Security Software driver is also at an altitude of 320,xxx (specific altitude removed to avoid naming the vendor)

In the above image, the security software is actually preventing the App Volumes agent from working properly, end result was that the AppStack did not mount. It sees the App Volumes Agent has an external drive and is scanning every file before releasing the hold on the process.

The perceived negative result of that action is that the user logon is delayed by multiples seconds, even sometimes minutes. In extremely rare cases, we even saw that the security software would prevent the AppStack from being seen by the Virtual Desktop. So, even though, the AppStack is successfully assigned to the user, the user when they login to their desktop, do not see any applications installed.

One quick test to do when this happens, is to unload the conflicting filter driver from the troubled system. You can do this by opening an administrative level Command Prompt and typing “FLTMC unload ”

In most cases, by executing that action, you will see the AppStack complete its virtualization process and the applications will appear as available in the virtual desktop.

Best Practices to achieve successful Application AppStacks 
The following is a brief description of situations and application types where App Volumes may need special attention to work properly or should be avoided.
Deep OS integrated Applications:
Applications that integrate deeply with the OS (like IE) should not be virtualized in an AppStack. The reasons for this when these apps are real time removed from the OS they can cause issues with the OS. This is again if the application should be present when the user is logged out it should be in the base and not an AppStack. Applications that start at boot time or need to perform and action before a user is completely logged on.  Applications such as Firewalls, Antivirus and Internet Explorer fall into this category. 
Poorly written large or legacy applications:
Apps with heavy registry enumerations can potentially have a performance impact when used with App Volumes. This is highly dependent on the application however and the overall performance impact will vary greatly depending on the environment and application itself.
Know thy app:
In the rare event that an issue with an application does present itself, it is important to have a thorough understanding of how the application functions. Understanding processes that are spawned, file and registry interactions and where files are created and stored become useful information in the troubleshooting process. 
Junk in Junk out:
App Volumes is a delivery mechanism for applications. It is important to understand that App Volumes does an intelligent recording of an installation during the provisioning process and then delivers that installation. If the installation is not accurate or configured correctly, then the delivery of that installation will also be incorrect. It is important to verify and test the installation process to insure a consistent and reliable App Volumes delivery

If you want to dig deeper in understanding filter drivers, this MSDN blog post, even though a bit old, is still a very good tutorial on the matter:

Windows uses a dedicated set of load order groups for file system filter drivers and minifilter drivers that are loaded at system startup.
Legacy file system filter drivers can attach only to the top of an existing file system driver stack and cannot attach in the middle of a stack. As a result, the start type for a driver and load order group are important to legacy file system filter drivers, because the earlier a filter driver loads, the lower it can attach on the file system driver stack.
Drivers are loaded first based on the start type for the driver, which represents phases of booting a system. For more information about start types, see “Driver Start Types” in What Determines When a Driver Is Loaded. All file system filter drivers and minifilter drivers that specify a start type of SERVICE_BOOT_START will be loaded before drivers with a start type of SERVICE_SYSTEM_START or SERVICE_AUTO_START. The start type is specified by the StartTypeentry in the ServiceInstall Section of an INF file that is used to install the minifilter driver. Within each start type category, the load order group determines when file system filter drivers and minifilter drivers will be loaded.

Take the time to understand how the stack is built, how it interacts and how external components will influence the behavior of filter drivers, your deployment will be better because of it!

2 thoughts on “Better Understanding the VMware App Volumes Agent”

  1. Tristan Todd says:

    Nice article Stephane!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.