VMware: Selecting Among Benchmarks

The necessary first step to effective and useful benchmarking is to decide what you specifically want to test. Now this is a very subjective exercise that can be answered differently by different people, or even differently by the same people at different times. There really is no single, definitive benchmark to run under all circumstances it all depends. Nonetheless, here are some useful guidelines:

• For OLTP database workloads, consider industry standard TPC-C or newer TPC-E

• For Data Warehouse workloads, consider industry standard TPC-H (requires many disks)

• For narrow/specific application workloads, replay associated Oracle trace files

• For broad/overall application workloads, replay transactions using Oracle 11gs new Real Application Testing (RAT)

However, you should not feel compelled to strictly adhere to the original benchmarking specification or guidelines entirely, as there will obviously be mitigating or special circumstances in many real world scenarios.

One question that quite often arises when discussing benchmarking on virtualized machines is what about using VMwares new VMark benchmark? On the following page in Figure 10 is an architectural diagram about the VMark test from VMwares web site.

Figure 10:  VMark Test Diagram

The problem with this benchmark is that it is far too broad or generic  and therefore, not database nor Oracle specific. By testing along all of these six categories - email server, java server, standby server, web server, database server and file server - the benchmark is really meant to test the relative efficiency of the underlying virtualization architecture and specific platforms execution. Thus, this book will stick with using the simple TPC-C OLTP test which is quite familiar to many DBAs.

Relative Results

So far there has been quite a substantial build-up to what is actually a very short and concise set of benchmark observations as shown in the chart below (Figure 11). I basically tested the same hardware under four different configurations and ran the TPC-C benchmark for 50 to 500 users, watching that the response time remained two seconds or less.

Figure 11:  Comparison Chart of Various Benchmark Configurations

Remember, the goal was to show that virtualization is a reasonable choice. Furthermore, that the various virtualization options offer different performance characteristics, one of which should suffice for your database needs. So let us examine the results now and summarize what the findings tell us.

Of course, the native operating system serves as your baseline. Two clear observations surface immediately and both which make sense. First, the native operating system wins since it has the least amount of overhead. Second, the hypervisor based solution is very close to that of the native operating system, i.e. the hypervisor really does add a very low overhead. Another finding (although a little harder to see), is that the DExtreme I/O accelerator for VMware GSX improves with increasing user load. That makes sense since it basically serves as a much smarter I/O cache than what is on the native OS file system, and it scales much better. But the biggest finding is that virtualization of the database did not cause you to exceed your user requirements of two second or less response time and that all the configurations were pretty close, i.e. no shocking surprises nor major disappointments.

Furthermore, additional tests show that as you increase the number of databases and thus, the number of virtual machines, a very similar pattern emerges. That confirms our objective:  virtualization is, in fact, a reasonable alternative for database deployments.

Conclusion

In this chapter, we wanted to observe the performance characteristics of a virtualized database solution to see if it made sense to use this technology. First, we made sure to define what and how one would perform a successful benchmark, and then how one would best interpret those results. Then we applied these techniques to our range of obvious alternatives: native OS vs. VMware GSX vs. VMware GSX plus a smart cache vs. VMware ESX. Regardless of what performance pattern or characteristics the results implied, the major finding was that we could virtualize our database and still accomplish our response time requirements and all within an acceptable proximity of results that is offset by the added benefit of flexibility that virtualization offers.

This is an excerpt from Oracle on VMWare: Expert tips for database virtualization by Rampant TechPress.