Process Virtualization Technology

Process Virtualization Technology

IncrediBuild accelerates time-consuming products or executions by distributing their processes in parallel across idle cores over the local network machines on which IncrediBuild is installed. For example, IncrediBuild enables an execution running on a four-core local machine, to run on 400 cores in the network, thus significantly accelerating the overall execution time.

IncrediBuild uses a unique technology to accelerate these processes and applications: Process Virtualization. This technology virtualizes processes on a remote machine as if they are running on the local machine.

Note:

Process Virtualization technology ensures that a Task is executed exactly as if it is being executed on the computer that initiated the Job (Initiating Machine), regardless of the remote Agent's file system, installation base, and environment.

Process Virtualization does not require any installations on remote agents, and does not require files to be copied from Initiating Machine. The only requirement is that the small and light IncrediBuild agent is installed on all machines. Process Virtualization ensures that files (including DLLs and others) required by the remote process are transferred automatically and transparently from Initiating Machine to Helper Machines.

Process Virtualization provides an additional feature – interception, which takes place on the local machine. If a parent process has to run many child processes, and wants to run these child processes on a remote machine, IncrediBuild knows to listen to all process executions of the parent process. With the help of a small XML file that defines the processes that should be run in remote, IncrediBuild knows to monitor all the process creations of the parent, and for each process that it creates, looks at a predefined XML file to see whether the process has been defined by the user to run remotely.

Process Virtualization technology is appropriate to all IncrediBuild solutions (VS, Make and Build Tools, and Dev Tools), with only small differences between them:

Remote Virtualized Environment

Helper agents are client programs on various machines in the network that run parts (Tasks) of an executed Job in a virtual environment so that the results are the same as if the Tasks would be run on the local machine. Helper agents communicate with the Coordinator, reporting the availability of their resources (cores). As a result, the Coordinator is always up to date as to how much processing power each Helper agent can provide.

Note

When a Task is assigned to execute on a Helper agent via IncrediBuild, a process representing that Task runs on the Helper machine in a special virtualized environment. Any child processes that are run by this process are also virtualized processes.

IncrediBuild fully emulates the initiating machine's environment for the Helper agent, including: file system, registry, process DLLs, standard output, and directory information. All remotely-performed Tasks run in this encapsulated environment. There is no need to copy files from the original machine or install applications on remote machines. IncrediBuild supports interaction among all Windows-based agents on a network, even when they are running different versions of Windows operating systems.

Virtualized Processing Data Flow

The Helper executes the process and surrounds it with the virtualization environment, simulating the environment of the originating machine. In the Helper machine, each call that the process makes to the operating system is intercepted by the virtualization environment. If a call is related to the file system, the virtualization environment synchronizes the file from the initiating machine to the remote machine, and caches it there in a dedicated folder. The virtualization environment then replaces the original path (for example c:\a.txt) with the path to the cache (for example c:\IncrediBuild\cache\a.txt). The same logic applies to registry access, DLL loading, etc.

At the end of the process, the Initiator machine gets the agent back as available, and decides whether it wants to run another Task on it, or wants to free it to the Coordinator.

For the host software running on the Initiating machine (for example, Visual Studio), this entire process is completely transparent (i.e., it appears as if the process was handled by the local operating system). In the Helper machine, the Task is handled by IncrediBuild on the OS as a virtualized process.

If the Helper machine is also being used by a conventional user, the additional (IB-initiated) processing is not apparent to the user. Most of the time, computers use very little of their CPU power. IncrediBuild makes use of much of this available CPU time that isn't in ongoing use by computers in the organization. The remote (virtualized) processes executed on the Helper machine only use the idle CPU cycles of the machine. In a medium-sized organization, this means that at any given moment, IncrediBuild can use hundreds of unused cores to accelerate processes in the organization. Each workstation in your organization that can do initiation of processes can serve as a workstation of 200 cores instead of only 8 or 16 cores!

As the Helper machine does not have any of the related resources that are relevant to completing the Task, such as files or DLLs, the IncrediBuild Agent runs the process on the Helper and surrounds it with a virtual environment, requesting the required resources from the Initiator as it processes the task. These resources are stored in the Helper’s IncrediBuild file cache, so that the next time the process runs in the virtualized environment, the Helper can use the resources previously stored in its cache (assuming that they have not been modified since the last execution).

IncrediBuild has a simple set of interfaces that enables you to either directly ask to run a command on a remote Helper, or to define in advance names of processes, which upon detection, IncrediBuild runs automatically on a remote computer, provided one is available. For example, when IncrediBuild accelerates builds in Visual Studio, it does so by adding the cl.exe process, the compiler of Visual Studio, and C++, to the list of predefined process names to be virtualized. This way, each time Visual Studio runs the cl.exe process, IncrediBuild automatically runs this process on remote computers. See Automatic Interception Interface.

If you run a batch file, which in turn runs many testing processes, you can specify that all processes running in the batch file are to run in parallel and on remote machines. See Submission Interface.

Process Virtualization in VS Compilation of C++ Projects

This section describes in detail how the IncrediBuild architecture is used for Process Virtualization, for the case that VS does compilation for C++ projects.

Note:

The following describes the sequence of events for the case that VS does compilation for C++ projects:

  1. In Visual Studio builds, IncrediBuild executes a VS command (such as DevEnv or MSBuild command), and requests VS to build a solution. When the VS command runs and the VS process is ready to build the solution that IncrediBuild passed to it, IncrediBuild injects DevEnv or MSBuild into the VS process, and acts as a mediator between the calls that the VS process makes and the OS. It monitors all the calls to the OS and decides how to process them.



  2. As part of the build command, the build runs compilation processes (cl.exe). When VS runs a cl.exe process (a compilation process), VS runs a Create Process command, which is intercepted by IncrediBuild before it reaches the OS. IncrediBuild detects cl.exe as a process predefined to run on a remote machine, and does not allow this process to reach the OS; instead, it tells VS that a dummy process has been created.



  3. Instead of letting the cl.exe process run on the OS, IncrediBuild asks the Coordinator component for a Helper Agent (on a single core), on which it can run the compilation process.



  4. The Coordinator, which has many Helpers connected to it, brings an available Helper (core) that is also currently the strongest in the network.



  5. Coordinator connects Helper to the Initiating machine.



  6. Initiating Machine and Helper Machine communicate directly with each other, without the help of Coordinator. Note that the IncrediBuild Agent is installed on both Helper Machine and Initiating Machine.



  7. The compiler is transferred from Initiating Machine to Helper Machine.



  8. The IncrediBuild agent on Initiating Machine sends a request to the IncrediBuild agent on Helper Machine to run cl.exe.



  9. When the IncrediBuild agent on Helper Machine runs cl.exe, the IncrediBuild Virtualization layer is once again injected into the process that monitors all the calls that cl.exe sends to the OS.



  10. Processing now takes place on Helper Machine. cl.exe sends many calls and requests to the OS. However, each file-system-related request is sent to IncrediBuild Virtualization, as it cannot be performed directly by OS, as the files that cl.exe requests are not present on the remote machine. For example, when cl.exe sends an Open File request, the request goes to IncrediBuild Virtualization, which knows that the file that needs to be opened is not on the remote machine, but is on Initiating Machine that initiated this process execution.




  11. The IncrediBuild agent on Helper Machine requests from the IncrediBuild agent on Initiating Machine to bring the file.


  12. The requested file is copied from the file system of Initiating Machine to a cache of IncrediBuild in a dedicated remote directory.




  13. IncrediBuild Virtualization does redirection to the parameter that contains the path to the file that cl.exe asked to open.




  14. Instead of giving the OS the original path address of the file that cl.exe requested to open, IncrediBuild Virtualization gives the OS the path to the file in the cache.
    The same virtualization techniques are applied to other file-system events such as Registry calls, DLL loading, and the running of executables. The virtualization environment handles StdOut, StdErr, and StdIn for the process in a similar manner.



  15. IncrediBuild Virtualization makes use of many Helpers to significantly accelerate processes and to supply Initiating Machine with hundreds of cores, which it can use to accelerate its processes instead of only using the cores on its local machine.