The HYDRA multiheaded virtual computer

Thought I'd try and jam as many GPU's in 1 box as possible, out of what I have laying around.

The hard part was finding 10 working GPU's that are passthrough capable. I had to move some fans from Geforce cards to Radeons that had stuck fans.

I am using the PLX PEX 8508 8-lane, 5-port PCI Express Switch RDK (PCIe 1x to 4 PCIe 16x) to "fan out" the slots to accommodate all the cards . I bought 2 of these for $25 on ebay (haven't seen that price again) and am using the working one. The other has a fried cap (at least). I might try to fix it....

Also, when you pass PCI cards (not PCIe), you have to pass ALL PCI devices to one VM. So I had to pass 2 FX5200's to one XP VM. (no win7 driver from Nvidia for these, and I'm not putting vista on it). For some reason, although both work individually, I can't get a display on both of them. Maybe it's because the cards are identical, and the Nvidia driver gets confused? I could try an HP lights out board with VGA, it's really a computer by itself.....

So I added some lines to the XP's vmx to hide it's "virtualness", and installed VMWare Workstation. With an xp VM this makes 10 VM's .

Also running Mac OSx & Android Kit Kat, plus 6 Win7's.

I guess it's a case mod rather than "scratch built" since I started with this:

HP DC7700 SFF

All the parts

Cannibalized Dell monitor stand, sheet metal from Lowes & Harbor Freight brake

Case fits together

Ports facing up for better airflow & wire management

Modded Supermicro RM PSU

This connects.....

To this for the monitor power

Plexiglas back with grouped HDMI/USB & Ethernet

Directed airflow to GPU's

ALL wiring out the back

Of course, I would do it differently the next time....

The goal of the Hydra project is to bring the benefits of Datacenter virtualization to the desktop. Without the Datacenter, and without the personnel and associated expertise required to manage it. VMWare's Vsphere Client of course allows for almost complete configuration (Vcenter is required for some things as of 6.0 ), but it's a little too complete to expect all but a few end-users to use. A little curiosity would likely result in a tech support incident. It also requires another system, running Win7 or later, to get an empty Hydra system up and running some useful VM's. So we need an alternate management system.

I tried a few different approaches, a serial connected Arduino device (too much programming) , a Python menu system running at the console (no ability to view a VM screen), and an Android App, The Android App worked pretty good, but I was writing it in MIT App Inventor since I'm lazy and it was easy. The limitations of App Inventor and Android ver 2.0 on the crappy tablet I was using ended development here.

Early last year the Guys at Androidx86 posted an ISO tuned to use Nvidia GPU's. I found it to work on PCIe GPU's as well as the FX5200 I use as the boot GPU. So the Hydra boots VMWare ESXi and the screen switches to Android with a touchscreen(with the ability to plug in a keyboard & mouse) I also took the time to learn a more comprehensive Android development platform - Basic4Android. Which is nearly as powerful as Eclipse, but very easy and fast to learn.

As a result I believe I have hit upon the best method of management.

The otherwise useless console is "stolen" from ESXi, which means no extra hardware.

Android is free, ESXi is free, and can be distributed with the free license (I checked)

B4A allowed me to cover all the management bases, edit VM's, assign PCIe devices, connect to other hosts, and build a new VM from scratch from an inserted CD. This is all from the touchscreen console, or with the app installed on a phone or tablet. Over IP, or serial.

A new Hydra would presumably come with Free ESXi, and an Android VM running the app. The end user inserts an OS cd, and builds a VM with whichever GPU is selected.

Discovering VM's (Flash animation with Hydra playing catch with the Droid)

Found the VM's, their OS by icon & power state by label color. A new Hydra would prompt for host IP, and it can be changed later. The IP button performs operations on the host, such as power etc. DCUI launches the ESXi console we stole, so it's not gone.

Single click to power on a VM (or power off according to vmware tools state) and it launches the boot screen of the VM. Long click allows power on to bios, launch screen independently, and other power operations.

Build a new VM - this is the current screen. I'm working on some more such as cores & sockets. Here I'm building Win8, 2GB, 20G HDD, and assigning an HD6450 and USB 1a1.

VMWare ESXi being a Datacenter Hypervisor, the management available from VMWare is the VSphere Client, which needs to run from another PC. Or from a VM on your host, but you still need another PC to get that up and running. You can do it from the console, but probably not without another PC to Google yourself through it. And the VSphere Client has many options an end-user does not need, and some an end-user should NOT have access to.

If this is ever going to be a end-user friendly system, I needed some other interface. A VM with the VSphere Client requires Windows, and a license.....So I can't "include" a mgmt VM without the whole OEM M$ etc....

So the VSphere Client is of course available from VMWare for those that are comfortable with it. But I needed a simpler management interface that would be always available without extra hardware, needs to communicate via SSH or Serial to send the vim-cmd commands to the Host, and needs to be able to VNC into a VM screen (for building or troubleshooting VM's).

First I was using an HP Lights Out card as the primary adapter. You could remote in and mount ISO's, and power the system on/off. The colors on the DCUI were all messed up, but it worked.

Then I wrote an Android App that would run on a cheapo tablet that I would include. It needed WiFi, so I ran a VM running a soft-AP using a Ralink USB. They don't last very long, not surprising considering the source & price ($4). And when the radio dies, you're stuck with VSphere. But it would VNC into a booted VM, so you could get out of "startup repair" if needed.

Then I wrote a Python Menu, as an installable VIB. This always worked for enumerating existing VM's and controlling their power & snapshots. No VNC from the console, even tried some frame-buffer VNC without success. This code is probably useful somewhere though....

Then I found I could "steal" the primary GPU from the Host, and give it to a VM. It would not work with the ILO card, so it took me a while to discover this. The ESXi screen freezes, and the VM using the GPU appears at the console when booted. No text mode, so you can't troubleshoot a non booting VM from here. (so whatever I run here needs to be RELIABLE) But, this is a PCI (not PCIe) card, and all PCI slots in the Host can only be assigned to 1 VM (so these slots are kind of useless anyway). I have these populated with a Geforce FX5200 & a USB card. I connected a touch screen monitor and am running Android x86 ICS with these 2 PCI cards. Now I can run the Android App from an autostarting VM on every boot. A nice touch screen interface requires no keyboard or mouse. The VM with the App can be "frozen" at a working state and return to the snapshot on every boot. The same App could also be on a phone or tablet the EU already has - or not. I could potentially modify the install ISO to include my App and run it live. I could also put the cheapo Ralink in the free USB and run an Android Tether soft AP in case there is no existing AP. The DCUI can be accessed over SSH, so that's not gone. No M$ license needed either. This seems to be the best EU friendly management interface and I am finally satisfied with it.

Now I'm re-writing the App.....

So, I got an Oculus Rift DK2 cheap off CL. Tried for a solid day trying to get it running on the Hydra Game console, but no go due to the Quadro GPU. The six headed Hydra here uses HD6450's, and it lit right up. Just another peripheral, but 5 Oculus Rifts connected to 5 VM's seems to suggest some very cool 5 way gaming setups. Already have the Leap Motion working (in Ubuntu, with Compiz extensions you can spin the globe with your hand) and will be checking out some Leap + Rift games/content. Any suggestions?

I'll add more later, here's the youtube vid:

https://www.youtube.com/watch?v=7cov4LHAduo&feature=em-upload_owner#action=share

First of all, before the Hydra was eliminated from the space trip contest, my attention was unfortunately entirely focused on income. This has been back-burnered since late July, with not much forward motion since. I didn't really expect to place or show, but entered for the publicity & collaborative interaction it promised. But at this point it's still just a hobby....

I downloaded 6.0 beta with the best of intentions, and it just sat in a folder, until Susie Felt from VMware called me and I asked her about PCI-Passthrough in 6.0, but she could not comment on it as it is still un-released. So I had to try for myself.....

I have been installing Digital Menu Boards for McDonald's lately, and the Hydra certainly could drive those 5 HDTV's. Maybe not for McD's, but there are MANY others who have not converted from cardboard menus yet. This would be a new market for VMware, and their interest would help to accelerate development.

I have been stuck on ESXi 5.0 build 768111 (this is 5.0 with some patches to fix VM auto starting) since ALL later builds break the ability to pass through onboard PCI devices. This made all the USB's unavailable, as well as any disk controllers (if I was inclined to try). 5.0 works, but how can I build a system that requires anything but the latest build?

In 5.1 they would revert back to not enabled for passthrough after the required reboot.

In 5.5 they would persist, and I could assign them to a VM, but the VM would not boot with these devices.

In 6.0 build 2159203 initial testing I have an XP VM using the AMD 4650 add in card, as well as 1 onboard USB. I have no reason to expect anything less than the functionality I had in 5.0 with the patches. VMware has a "bug hunt" contest, so I expect basic features like autostart to work as well.

I will be trying the PCIe switch next.....

Mike makes it look so easy...

I can make ESXi do tricks but videos still seem to be a challenge....

This diagram shows the basic components of the Hydra system design. There are many details which need to be addressed as these parts are assembled, but this overview shows all of the components apart from the obvious power supply, case, and RAM to meet your requirements.