Back in the early 80s, Pepsi ran a marketing campaign called the “Pepsi Challenge.” The idea was to get people to do blind taste tests against Coke, the top cola brand at the time. This became a cultural phenomenon with celebrities and regular people alike trying this test for themselves. This conversation exploded in an era that was well before the days of social media and wide-scale viral marketing campaigns.
Fast forward to 2019 and the open source DBMS market. For enterprise class applications, many are now making the choice between Postgres and MongoDB (or maybe MySQL et. al). Open source DBMS’s (OSDBMS) are an increasingly popular database of choice for today’s next-generation applications, especially in the cloud.
The Benchmark Heard Around the World
With this challenge in mind, we recently chose to sponsor a benchmark of Postgres vs. MongoDB. We hired Ongres to compare the performance results of these two products across a series of benchmarks. The report showed that on all three performance benchmarks, Postgres came out on top. And in addition to performance, Dr. Michael Stonebraker recently reminded us all why the relational model provides superior data independence, as well.
The benchmark exercise was designed to be reproducible and run on a public cloud, so anyone who wanted to compare Postgres and MongoDB could easily do so. And in the spirit of open source, Ongres also made all the code used in the testing open and available on GitLab.
About a week after releasing this benchmark report, the team at MongoDB offered a response to the results of this exercise. However, they did not share the technical details of what they did, and their approach has been widely panned by many as lacking professionalism and less than transparent. In light of these events detailed below, we completely agree with this assessment.
There is No There There
First, MongoDB declared the results of the Ongres benchmark report to be “null and void”, releasing their own test results refuting the findings. Then, they proceeded to openly mock Ongres as a bunch of amateurs who were wasting the precious innovative energy of MongoDB developers. However, when pressed, MongoDB declined to publish their testing methodology and source data, which seems to miss the “open” part of being open source. This made it so no one could validate or recreate their results. In our opinion, why even bother responding?
To pile on, MongoDB even sent out one of their communications people to bully people on Twitter who were asking legitimate questions about how MongoDB ran their follow-up tests.
A Fact-Based Conversation
In contrast to MongoDB, EDB is happy to have a fact-based conversation about the benchmark results data and its approach will all interested parties. Ongres even professionally and thoroughly responded to MongoDB’s criticism, elaborating on their benchmark tactics, thought processes, and also invited further challenges.
Take the Postgres Challenge!
MongoDB’s covert response certainly makes us wonder how they got to their results and we’d guess we are not alone in that thinking. However, by contrast, we have nothing to hide. Here is where you can download the Ongres benchmark report and source code. Just check out the transparent approach utilized by Ongres and test the results for yourself.
At this point, our question to DBAs and developers is to take a closer look at the Ongres benchmark data. Look at the facts and performance numbers, not the MongoDB buzz and stock price. Like many today, we hope you’ll discover that Postgres is the right choice as a full featured and high performance DBMS.