Xen 3.3 Feature : Memory Overcommit

From Dan Magenheimer at Oracle:

Memory overcommit provides the ability for the sum of the physical memory allocated to all active domains to exceed the total physical memory on the system. For example, if your machine has 4GB of RAM and you want to run as many 1GB domains as possible, you can run at most three — because Xen and domain 0 require some physical memory also. With the new memory overcommit feature in Xen 3.3, in some environments, you can run six or ten or even more.

To be clear, there is no magic: Memory overcommit may have some performance impact and may be unusable in some environments. Memory for new domains is obtained by taking it away from currently running domains so environments where all domains heavily utilize memory are not a candidate for memory overcommit. And to maximize benefit, all domains must be properly configured. But for environments which require a ratio of high virtual-domains-to- physical-machines and that are willing to make some tradeoffs, memory overcommit can substantially increase “VM density” and save cost.

Memory is taken from one domain and given to another using the existing Xen “ballooning” mechanism, which has recently been improved to be more robust. For example, a domain that is idle (or nearly so) is probably not using much memory; this memory can be made available to use in another domain, or for a newly created domain. The tricky part is to determine how MUCH memory can be taken away from domains without causing problems for them; and, even more importantly, how to give the memory back if a domain suddenly needs it again.

This careful memory balancing ideally should be done in a management tool that can monitor memory needs of all domains and add or subtract memory from each domain as needed. A very simple management tool supplied with Xen 3.3 provides “self-ballooning” and, while more sophisticated tools may be needed in the future, self-ballooning is sufficient for many environments.

To best implement memory overcommit, all domains should be configured with a properly sized and configured virtual swap disk and all HVM domains must have a working balloon driver and runnable Xenstore tools. Next self-ballooning scripts are installed in each domain and enabled as a service.

The scripts, along with a comprehensive README, are found in xen.hg/tools/xenballoond in the open source Xen distribution. Once all domains are rebooted, automatic memory balancing will occur and idle memory is freed up to run additional domains, thus resulting in memory overcommit!