The Cloud and Database Benchmarking Blog

Today in DataScaleFail, we revisit the benchANT Database Ranking and discuss its benefits forreal-world use cases. Further, we present TimechoDB as the new leader of the time-series ranking. Finally, we introduce a case study where we benchmarked databases under failure conditions. Topics: (i) Reintroducing the benchANT Database Ranking (ii) TimechoDB Takes the Lead in the benchANT Time-series Ranking (iii) Benchmarking Beyond Performance: A Resilience Case Study with Aerosp

Choosing the right database for your application is a critical decision that can significantly impact performance, scalability, and cost-efficiency. With the rapid evolution of database technologies and the proliferation of cloud services, this choice has become increasingly complex. With more than 1,000 products to choose from, the landscape is vast and varied. The benchANT database ranking is a public source of information that aims to simplify this decision-making process

TimechoDB, an enterprise-grade time-series database system based on Apache IoTDB, has taken the lead in the benchANT Time-Series Database Performance Ranking. Evaluated using the Time-Series Benchmark Suite (TSBS) with the DevOps workload, TimechoDB (built on Apache IoTDB v1.3.4) delivered standout results in the small-scale benchmark category running on a single AWS EC2 node. All benchmarks followed benchANT’s reproducible TSBS methodology, ensuring transparency and compara

Traditional database benchmarks often stop at measuring throughput and latency. While those metrics quantify raw speed, they fail to capture how systems behave when the environment changes, e.g., when nodes fail, workloads spike, or capacity must grow. In production, these behaviour in these moments is just as important as steady-state performance. This article discusses how the benchANT benchmarking framework handles such cases, and presents results of a case study which ev

PostgreSQL is available as a managed service from nearly every cloud provider, but performance and cost can differ significantly under the hood. In this independent study, we benchmark OVHcloud’s PostgreSQL DBaaS against AWS RDS PostgreSQL, Azure Flexible PostgreSQL, and DigitalOcean PostgreSQL using OLTP workloads and a strict price-performance methodology. We compared throughput, latency, and efficiency on equivalent instance types to provide an objective view of how these

DataScaleFail is back! After a busy year filled with in-depth benchmarking projects, the DataScaleFail newsletter is returning! In this comback edition we share the results of two of our benchmarking projects and introduce a new database to our Database Ranking. Newsletter topics: (i) YugabyteDB enters the Database Ranking (ii) Vector Benchmarking for SingleStore (iii) Benchmarking OVHcloud's PostgreSQL against Hyperscalers

YugabyteDB enters the benchANT Database Performance Ranking for CRUD workloads. Tested via its PostgreSQL interface on medium, large, and xlarge clusters, it delivers over 100,000 ops/s with stable read/write latencies. Throughput scales well, and price-performance remains steady, highlighting predictable behavior across cluster sizes. This makes YugabyteDB a balanced alternative with PostgreSQL compatibility and hence, an attractive option for enterprise customers.

Curious about how general-purpose databases stack up against native vector databases in search performance? Our latest benchmark study pits SingleStore Helios against Pinecone and Zilliz using the established VectorDBBench suite. With 10 million 768-dimensional vectors as the battleground, we provide technical insights into how a general-purpose database like SingleStore performans against this specialized database domainn.

The capabilities to store and search for vectors, also called embeddings, is a crucial capability in many fields of artificial intelligence (AI). When selecting a database technology for vector workloads, many selection criteria are similar to other application domains for databases. Yet, some new ones show up as well. This article revisits what vector databases are and what makes them different to other use cases.

In this edition of DataScaleFail, we share our findings about the price updates of the IaaS and DBaaS flavours of the benchANT database ranking. Also, we continue our Vector benchmarking journey where we first look into what is special when benchmarking this type of databases. Finally, we complete our analysis of benchmarking suites for Vector databases. Newsletter topics: (i) Database Ranking - Cloud & DBaaS Pricing Update (ii) Benchmarking Vector Databases (iii) Vector DB

The popular database ranking consists of performance and cost metrics for the most important databases and selected DBaaS products. We have compared the current cloud and DBaaS prices with the prices for 2022 and are quite surprised at the different trends.

Due to many VectorDB benchmarking requests popping up at benchANT, we start with a discussing on the situation of the DeWitt clause with Vector DBaaS's. Second, we continue our analysis of benchmarking suites and look into the time-series domain today. Finally, we introduce Telegraf-TS, a benchmark tailored to backends of the Telegraf monitoring agent. Topics: (i) VectorDB Benchmarking (ii) Time-Series Benchmarking Suites (iii) The Telegraf-TS Benchmark

Many benchmarking suites make use of synthetic data what makes them very flexible. Yet, it introduces some fuzziness when evaluating very specific use cases such as the storage, retrieval, and analytics of IT monitoring data. Telegraf is a popular tool for collecting that data and storing it into a data backend. For choosing the right data backend for your analytics queries, Telegraf-TS has been introduced. It is an extension to YCSB, specifically tailored to evaluate the pe

Vector databases are the latest trend in the database market, addressing the needs of AI applications through dedicated vector search features. And with every new type of database there always comes the challenge for the fastest database. But what about performance benchmarking of vector databases under the shadow of the DeWitt clause? We have analyzed the current vector database market with respect to their DeWitt clause and which provider might sue you for running and pub

We continue our analysis of benchmarking suites in the NoSQL area and have analysed more than 12 NoSQL benchmarks. In addition, new benchmark measurements on MongoDB DBaaS alternatives as well as a technical deep-dive on how to customise the YCSB Benchmarking Suite almost limitlessly. Newsletter contents: (i) 12 NoSQL Benchmarking Suites (ii) MongoDB DBaaS Alternatives - Performance/Cost Comparison (iii) YCSB Customising Guide

Learn the basics of cloud database benchmarking from scratch. Learn everything worth knowing from classic database benchmarking to cloud database benchmarking. Start here!

In the dynamic database management systems market, performance is a key differentiator. Database benchmarking suites provide the tool to assess performance, guide us through the vast landscape of options, and help developers, engineers, and decision-makers make informed choices. This article dives into the various benchmarking suites for different workload domains and provides an overview of their origin, implementations, and unique features.

The Yahoo Cloud Serving Benchmark (YCSB) is a well-known tool to measure and evaluate the performance of database management systems (DBMS). One of the major advantages of YCSB is the very simple workload it uses as this allows support for many different DBMS. On the down-side, YCSB standard workloads do not make use of any even marginally sophisticated DBMS features. In this article, we sketch how to exploit YCSB in order to build different workloads and also make use of o

DBaaS Navigator - holistic comparison of the most popular Database-as-a-Service products. More than 60 attributes on technical features, management tools, support contracts, performance measurements and price-performance assessment. - Step-by-Step Guide - Roadmap - Disclaimer

Database-as-a-Service (DBaaS) has the potential to reduce required manpower, lower time-to-delivery, and has the potential to reduce the total cost of ownership (TCO) of database services. As with most other cloud services, the DBaaS market worldwide is dominated by the American companies, in particular the hyperscalers AWS, Microsoft Azure, and Google Cloud Platform (GCP). Unfortunately, relying solely on U.S. companies is not without risk for European customers, for instanc