On StackOverflow, the question "Why Spring is faster than Vert. x?" in its various variations is asked once a month on average. After all, Spring is still by far the most popular JVM framework, and many businesses rely on it. However, the Spring Framework is not well-known for its performance. Vert. x, on the other hand, is regarded as one of the best JVM frameworks. As a result, Vert.x is expected to outperform Spring in any benchmark. That, however, is not the case.
In this blog, I'd like to discuss the various causes of those counterintuitive results, as well as offer some suggestions for how to improve your benchmarking strategy.
To begin, what do we mean when we say a framework or language is "fast"? When it comes to web services, we don't talk about response time, also known as request latency. What we usually mean is a different metric known as throughput. Latency refers to the amount of time it takes to respond to a single request. Throughput refers to how many requests a server can handle in a given amount of time. Typically, within a second.
Let's look at where developers get the idea that Vert. x should be faster than Spring. A popular benchmark for web frameworks powered by TechEmpowered attempts to measure the throughput of various languages, runtimes, and frameworks using a few scenarios. Typically, the Vert.x framework performs admirably in these tests.
In the 20th round, for example, Vert.x is ranked 10th with 572K requests per second, while Spring is ranked 219th with 102K requests per second. This is truly impressive.
However, replicating those impressive results can be difficult at times, hence the title's question.
Let's try to figure out what the main flaws are with the benchmarking strategy.
When I say Spring, I mean the Spring Framework, not Spring WebFlux / Project Reactor, which works in a different way. In addition, I'll assume that the Spring application is running in a Tomcat container.
Vert.x is I/O focused
The Vert. x framework's ingenuity recognized early on that the bottleneck of most real-world applications is waiting for I/O. That is, it makes no difference how well your application is written, how clever the JIT optimizations are, or how cutting-edge the JVM GC is. The majority of the time, your application will be waiting for a response from the database or from a service written in Python or PHP 10 years ago.
Vert. x addresses this issue by placing all I/O work in a queue. Because adding a new task to a queue is not a particularly time-consuming operation, Vert. x can handle hundreds of thousands of them per second.
Of course, this is a very simplified explanation. There are multiple queues, context switches, reactive drivers, and a slew of other interesting features that I won't go into detail about. What I do want you to keep in mind is that Vert. x is designed for I/O.
Let's take a look at how Vert.x performance is typically measured:
app.get("/json").handler(ctx -> {
ctx.response().end("Hello, World!");
});
Let's compare the preceding example to the code from the Vert.x benchmark, which still performs quite well, with a throughput of 4M requests per second, but not as well as some other languages and frameworks:
app.get("/json").handler(ctx -> {
ctx.response()
.putHeader(HttpHeaders.SERVER, SERVER)
.putHeader(HttpHeaders.DATE, date)
.putHeader(HttpHeaders.CONTENT_TYPE, "application/json")
.end(Json.encodeToBuffer(new Message("Hello, World!")));
}
);
Can you spot the distinction? There is almost no I/O in the benchmark that most developers run. There are some because receiving a request and writing a response is still an I/O operation, but not much when compared to interacting with a database or a filesystem.
As a result, the benefit of using a reactive framework like Vert. x is diminished by that test.
Write a benchmark application that does some I/O work, such as writing to a database or reading from a remote service, if you want to see real benefits from a reactive framework like Vert. x.
Benchmarking with Low Concurrency
Concurrency is handled by Spring Framework by allocating a thread pool dedicated to serve incoming requests. This is also referred to as the "thread per request" model. When you run out of threads, your Spring application's throughput begins to suffer.
ab -c 100 http://localhost:8080/
To bombard our service with requests, we use a tool called Apache HTTP Benchmark. The -c flag instructs the server to run 100 concurrent requests at the same time.
You run this test on two services, one written in Spring and one in Vert.x, and there is no difference in performance. Why is this the case?
Unlike Vert.x, Spring Framework does not directly control the number of threads it employs. Instead, the container, in our case, Tomcat, determines the number of threads. Tomcat's default setting for the maximum number of threads is 200. This means that there shouldn't be much of a difference between Spring and Vert. x applications until you have at least 200 concurrent requests. Simply put, you're not emphasizing your application enough.
Set the number of concurrent requests higher than the maximum size of your thread pool if you want to stress your Spring application.
Benchmarking on the Same Machine
Let us return to how Vert. x works. I've already mentioned that Vert. x improves performance by queuing all incoming requests. When a response is received, it is also added to the same queue. There are only a few threads, known as EventLoop threads, that are busy processing that queue. The greater the number of requests, the busier the EventLoop threads become and the more CPU they consume.
What now happens when you run a benchmark on your computer? As an example:
ab -c 1000 http://localhost:8080/
The following is what will happen next. The benchmark tool will attempt to generate as many requests as possible while utilizing all of your machine's CPU resources. The Vert. x service will attempt to serve all of those requests while also attempting to use all of the available resources.
To maximize the performance of the Vert. x application during the benchmark, run it on a separate machine that does not share CPU with the benchmark machines.
This brings us to the following point.
- The Spring Framework's Performance Is Excellent
- I've been a huge fan of Vert. x for at least the last 5 years. But first, consider the throughput of the Spring application in the earlier-mentioned benchmarks.
- Plaintext: 28K
- JSON serialization: 20K
- Single query: 14K
- Fortunes: 6K
- Multiple querie s: 1,8K
- Data updates: 0,8K
Conclusion
As software engineers, we enjoy comparing the performance of our favorite programming language or framework to that of others.
It's also critical to use objective metrics when doing so. Measuring service throughput with a benchmark is a good place to start, but it must be done correctly.
Check to see if the test you're running is CPU or I/O bound, or if it has another bottleneck.
Also, run your benchmarks on a separate machine than the one that runs your application code. Otherwise, you might be disappointed with the results.
Finally, I've witnessed companies encountering throughput bottlenecks in their language or framework, and I've even assisted in the resolution of some of them. However, there are many successful businesses out there that may not require all of that throughput, and you may be working for one of them. Creating a good benchmark is difficult and time-consuming. Consider whether that is the most pressing issue you should be addressing. If you have any doubt about the above topic. Don’t hesitate to contact us. Airo Global Software will be your digital partner.
E-mail id: [email protected]
Author - Johnson Augustine
Chief Technical Director and Programmer
Founder: Airo Global Software Inc
LinkedIn Profile: www.linkedin.com/in/johnsontaugustine/
Author - Johnson Augustine
Author - Johnson Augustine
