Benchmarks are a critical part of the IT industry. They allow us to measure technological progress, enable buyers to compare offerings from multiple vendors and help vendors improve their products. In the world of database servers, the Transaction Processing Performance Council Benchmark C (TPC-C) has had a starring role in the online transaction processing (OLTP) space for more than 15 years. Few vendors publish data server specifications without TPC-C numbers. Now, TPC offers a new, rich and carefully designed benchmark -- TPC-E -- intended to become, like TPC-C, a standard measure of data server performance. TPC-E is designed to be broadly representative of contemporary OLTP systems, challenging today's fast processors with a rich data model and demanding calculations.
The TPC-E model
TPC-E is modeled on a brokerage with its database of customers, accounts and securities. In a
typical brokerage setting, customers generate data research requests, account inquiries and trade
requests. The brokerage house receives trade requests, sends them to the market and returns the
results to the customer. When developing TPC-E, TPC consulted with well-known brokerage houses to
develop a model that is as realistic as possible. TPC-E uses multiple trade types -- market orders,
limit orders, stop orders and more. There are several tiers of simulated customers, each with
distinct trading behaviors. Simulated companies issue both preferred and common stock.
Requires Free Membership to View
When
customers make research requests, the system responds with large text documents, which include
line-of-business (LOB) data as well as historic financial and market data.
The resulting model comprises 33 tables, 22 check constraints and referential integrity constraints, including 33 primary and 50 foreign keys. TPC-E has three dimensions of data, posing realistic partitioning challenges. The model implements business logic using 10 transactions, many with various modes of execution. The rich schema and diverse transactions of TPC-E provide new opportunities for vendors to optimize system performance.
Modern database features
TPC-E is designed to match several additional features of contemporary databases:
- Some transactions in TPC-E cannot be implemented as a single stored procedure, enforcing multiple round trips for certain data.
- Strict database isolation requirements in TPC-E require improved database locking performance.
- Constraint checking, now a standard feature of all database management systems, is enforced in TPC-E.
- TPC-E provides a business intelligence-style transaction (broker-volume) that requires the processing of various-sized data sets, including volatile data.
- TPC-E requires database disks to be RAID-protected, reflecting typical real-world configurations. In addition, sponsors must report the time interval between catastrophic disk failure as achieving 95% of pre-failure throughput. Buyers will be able to better judge the tradeoffs between performance and reliability, and vendors will be rewarded for maximizing both together.
Low barriers to TCP benchmark adoption
Benchmark sponsors will find that running TPC-E is relatively easy and affordable. TPC-E has been
carefully designed and tested to balance CPU demands with physical I/O, minimizing the number of
disk drives required to run it. In addition to shrinking the cost of running TPC-E, lower I/O
requirements will result in a closer resemblance between benchmark configurations and those used in
real data centers. Also, TPC-E is focused on benchmarking the database server and requires fewer
front-end systems to emulate user actions. Finally, TPC has provided code that generates test data
for the initial database as well as implementations of key parts of the workload logic. This lets
test sponsors focus on optimizing their products rather than developing and optimizing the
benchmark application. All of these design choices reduce cost and complexity, putting TPC-E within
reach of a broader spectrum of hardware vendors. The more broadly a benchmark is used, the more
helpful it is for both buyers and manufacturers.
Transitioning to TPC-E
In many industries that use OLTP, TPC-E will deliver highly relevant performance measurements and
drive practical system improvements. While lower cost and simplified infrastructure will allow
TPC-E to be adopted rapidly, TPC-C remains a crucial part of the technology industry. Vendors can
continue maximizing the value of their investment in TPC-C while preparing to transition to or add
TPC-E to their environments. Customers can gain expertise in interpreting TPC-E results and shift
their decision frameworks gradually to include the new benchmark. Just as TPC-C has done, TPC-E
will help the transaction processing industry achieve new innovations and new heights of
performance.
ABOUT THE AUTHOR: Matthew Emmerton is the chairperson of the TPC-E Maintenance Subcommittee. He has been involved in the development and maintenance of the TPC-E benchmark standard since 2005. He is also a member of the TPC's Technical Advisory Board.