Usually, benchmarks are measured in transactions per second, but the TPC-C and TPROC-C, benchmarks are measured in new orders per minute (NOPM). What is a new order? It’s simply a predefined operation on the database, that even fails 1% of the time. This is a better metric than transactions because you can have transactions on the database that are not part of the benchmark. Autoanalyze uses a transaction, for example.
I do TPROC-C benchmarking, which is the unofficial version of TPC-C that forgoes things like think-time. HammerDB can do both types.
Benchmarking with NOPM also allows me to use monitoring tools that cause transactions, but don’t (obviously) create new orders. If we were measuring transactions, we could just send a solitary semicolon to the database in a tight loop and get a whole bunch for free. By measuring new orders, we are testing how fast the database can do actual work. A side effect of this is that the NOPM are much lower than the TPM. That doesn’t matter. What matters is that the NOPM stay relatively constant compared to each other.
PostgreSQL Development with Daily Users
Every day, I do a HammerDB benchmark run for two hours using 500 users on the first commit of the day UTC. I do this to detect if a new patch has inadvertently caused a drop in performance.
Here is the graph for January 2021:
This is the raw data:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(There were no commits all on the 3rd and the 10th.)
Because it’s only a single run as opposed to an average of five runs or so, there is some variation every day, but no major impact. This is good news.
Making Sense of the Data
If you would like to study all of the data that I collect before, during, and after a run; I have set up a public repository. My goal for the near future is to write some scripts to extract data and make pretty graphs, facilitating the analysis of these runs.
Tuning PostgreSQL is very important, but it is equally important to resist tuning for the benchmark. This is why I am not concerned with the exact number of NOPM I get, only that it remains relatively stable.
In a future blog post, I will share the scripts that I use to run these benchmarks so that you can run them yourselves, so stay tuned!
Relevant Blogs
Release Radar: New Postgres Extension Support Announced in Fully Managed BigAnimal Postgres-as-a-Service
EDB is one of the major contributors to Postgres, and we are dedicated to enabling our customers maximize their Postgres database environments by leveraging both EDB commercial and open source...
Unlock Possibilities with Free Postgres Training from EDB
Postgres is the database of the future. In recent years it’s skyrocketed in popularity across industries and established itself as the go-to database for innovators. It’s considered the most loved...
Solving Your DBA Talent Gap: Upskill vs. Outsource
For businesses looking to leverage the full potential of Postgres database, there are few roles as important as database administrators (DBAs). DBAs provide vital management and monitoring for your organization’s...More Blogs
Is Your Postgres Support Failing You? Here Are 6 Signs It Might Be
Postgres has established itself as the database for modern businesses—the most well-loved and advanced open source database on the market. But even the best databases experience issues. Business-critical systems need...