NEAL NELSON BENCHMARK LABORATORY
Neal Nelson Transaction Benchmark(TM) - Frequently Asked Questions
Benchmark Laboratory
Home Page 		Click Here to
Close This Window
Why was this benchmark created?
We could not find a benchmark that was oriented toward small business transaction processing.
What about the results from Transaction Processing Council(TPC)?
When I have reviewed test results from the TPC I have generally seen very large and very expensive machine configurations running very exotic and very expensive software packages.

For example one set of TPC-C results used a $15,000.000.00 computer cluster that processed 3,000,000 transactions per minute. I don't know of any neighborhood gas station or restaurant that needs to process 3,000,000 charges per minute.

We felt that there was a need for a transaction benchmark oriented toward the small business environment.

Why doesn't this benchmark use a database like Oracle?
I understand that Oracle prohibits use of their software for benchmarks of this kind plus many small business aplications have their own access methods and do not use commercial databases like Oracle.
How is this benchmark similar to/different from the TPC benchmarks?
I have created a separate page where I list some similarities and differences with the TPC-C benchmark. Please click here to read those comments.
Why are there 'Standard Configuration' and 'Non-Standard Configuration' tests?
Auto racing provides the simplest answer to this question. In the case of IROC racing the cars are all identical. In NASCAR racing each team can make some custom changes but there are strict limits to what can be done. For Funny Car racing there are essentially no limits.

My 'Standard Configuration' is like IROC. I provide the disk drives, limit the memory size and provide/configure/tune both the application and operating system software.

My 'Non-Standard Configuration' is like NASCAR. I allow any configuration for disks and memory but I still provide/configure/tune the application and operating system software.

I do not presently offer a transaction benchmark that corresponds to Funny Car racing.

How do you calculate your "system cost"?
If the machine arrives as a complete system we take the manufacturer's stated price and add the cost of D-ISAM and SUSE Linux to calculate the system cost.
If the machine is assembled from components, we check www.pricewatch.com to find the lowest price for each of the components and total these with the costs of D-ISAM and SUSE to arrive at the system cost.
If the machine is assembled from components, and there are components that can not be found on pricewatch we search ebay and/or the web to try and find a cost for the components. We then add these to the costs of D-ISAM and SUSE to get the system cost.
Why do we need another transaction benchmark? We already have TPC.
That's like saying we shouldn't have NASCAR because the Indianapolis 500 was started in 1911. This benchmark is very different from any TPC benchmark. If it collects useful information it will find a niche. If it does not provide value, it will fade away.
Why do you limit "system tuning"?
Small businesses tend to run their systems with little or no system tuning.
What is D-ISAM? Why is it used?
ISAM stands for Indexed Sequential Access Method. D-ISAM is a software package from Byte Designs that implements a B-tree version of ISAM.

ISAMs have been used since the 1970's for small business transaction processing. They have proven themselves to be both fast and reliable for these tasks. D-ISAM is available as source code and is compiled along with the application code for this benchmark. This improves repeatability and provides enhanced software version control.

If a benchmark of this type is desirable, shouldn't it come from one of the industry standard groups like SPEC or TPC?
Virtually all of the "industry standard" benchmarks were created by individuals and, if they proved their usefulness, were then submitted to, and adopted by, an industry standard group. If this benchmark proves to have value, it will almost certainly be offered to one or more of the industry standard bodies for acceptance.
Is this benchmark flawed?
Possibly. TPC-C has been in process since 1992 and is currently in release 5.6. This is a new transaction benchmark and it is very likely that as it is run, and results are examined, weaknesses will be identified. We expect to correct those weaknesses by making revisions to the benchmark just as TPC-C has has been revised and improved over the years.
Your tests all seem to be for single machines. Why is that?
It is rare for a small business to install a multi-tier architecture. Small businesses will generally configure a server to support a single activity area with data storage, application processing and server functionality all on the same box. We have followed this model with the small business transaction benchmark.
Copyright © 2006-2007
Neal Nelson & Associates 		Trademarks which may be mentioned on this site are the property of their owners.