29 Apr

Nested ESXi in vCloud Air-Routed Edition


Over the past 3 years I have spent a lot of time building and refining nested ESXi as part of my day job.  I worked for a large service provider at the time, and we developed our own portal on top of both the vSphere API as well as the vCloud Director API.  Nesting VMware products to enable multiple developers to build against the different API’s was a perfect use case for nested virtualization.  We really didn’t care about the lack of performance, we really only cared about having an API to build stuff (mostly just empty shell VMs, but sometimes small linux boxes to test networking).

I have recently been working on building a nested lab in vCloud Air. There was one well documented limitation of using vCloud Air, the fact that Promiscuous Mode & Forge Transmits was not enabled. Just recently there was an excellent article posted that described how to setup nested ESXi on vCloud Air, which included some great diagrams explaining why these settings are needed.  The article explains how to use a 1:1 mapping of nested VM MAC address to virtual ESXi MAC addresses. This is a great option for running one VM on a nested host, but he also hit on something I hadn’t thought of…running a router to give multiple VMs outbound network access. This was something I had to explore.

The Setup

vCloud Air SetupI wanted to make this as simple of an example as I could while trying to build out the above diagram from the VMware blog article. To do this I am just using an ESXi host and not a full vCenter setup.  This will come in another blog article.

One thing that was interesting was the use of a second port group on the vSwitch, I can see using this to keep things organized but the second port group isn’t really necessary.

I started in vCloud Air by build two VMs, an ESXi host and a windows box. I will use the windows VM to configure the network and test connectivity. Nested inside of my ESXi host I will create three VMs: a router and two Linux boxes.

From my Windows VM I download and setup the vSphere Client so I can configure the VMs I need directly on the ESXi host.

VyOS Setup (http://vyos.net/)

Build a shell Linux box:

  • 1 CPU
  • 512 MB RM
  • 2 GB Disk
  • Guest OS –  Other Linux 32Bit

Install VyOS:
Screen Shot 2015-04-29 at 5.30.44 PM
Boot from the disk (I downloaded the Virtual 32-bit version), it is a live CD, but we will install from here.

Default Login:

  • U: vyos
  • P: vyos

To Install:

From here I just accepted the defaults, finished the install and shutdown the VM.

Linux Test VMs

Screen Shot 2015-04-29 at 10.05.29 PMInstead of using another version of Linux to build my other two test VMs I ended up just making two more copies of the VyOS VM.  Now is a great time to point out that after the install VyOS was less then 225MB on disk. If you don’t turn on any of the networking features, its a nice small Linux box that I may use more often for a nested guest OS.

All three of these VMs are on the same default “VM Network” port group. We will add a second NIC to the VyOS-Router VM to act as the backend or inside network. To really show the example of how the routing will work I decided to use for my inside network.  LinBox01 and LinBox02 have been configured with IP addresses on this network.


Configure VyOS-Router VM and Network

First we need to find the MAC address of our virtual ESXi host.


Next we edit the MAC of the VyOS-Router VM to be the same MAC as the virtual ESXi host as well as add a second NIC on the same port group.


Once the VM has been powered back on it is time to configure the network:

Login into the VM and execute:

Now we need to identify the interfaces:

In my case the interface that I set the MAC on came up with eth2 so this will be my outside interface and eth1 will be my inside interface.

To configure the outside interface you execute:

And for my inside:

Now we commit our changes and save:

To check our work we run “show interfaces” again:

show interfaces

 Diagram Break

Now that we have completed the initial setup lets look at where we are:


It is important to note that all the NICs that are running on the nested VMs are connected to the same port group.

Who can talk to Who?

  • Inside of the nested ESXi host: LinBox01, LinBox02 and the VyOS-Router on the network can ping.
  • Outside of the host: ESXi VM and WinBox can ping on the network.
  • The WinBox can also ping the VyOS-Router box on because we setup the correct MAC.

What is left is getting LinBox01 and LinBox02 to be able to pin WinBox. To do this we need to create a source NAT rule on the VyOS-Router box to allow pings to leave the network.

From the VyOS-Router terminal:

Once this is in place LinBox01( and LinBox02( can ping the WinBox(


This is a very basic example, but gives quite a bit to build off.  I used this as a building block to build a full vSphere environment, and much more. Stay tuned for more info.


One thought on “Nested ESXi in vCloud Air-Routed Edition

  1. Pingback: Nested ESXi on vCloud Air - Links - DavidHill.co

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.