If i'm understanding the repository correctly, it looks like each language reads from a file, does some I/O printing to console, then computes the value, then some more console printing and exits.
In my opinion, the comparisons could be better if the file I/O and console printing were removed.
I'm not too familiar with the JVM so perhaps I'm missing something here: how would that help? The file is tiny, just a few bytes, so I'd expect the main slowdown to come from system call overhead. With non-mmap file I/O you've got the open/read/close trio, and only one read(2) should be needed, so that's three expensive trips into kernel space. With mmap, you've got open/stat/mmap/munmap/close.
Memory-mapped I/O can be great in some circumstances, but a one-time read of a small file is one of the canonical examples for when it isn't worth the hassle and setup/teardown overhead.
I'm not sure why the contents of rounds.txt isn't just provided as some kind of argument instead of read in from a file. Given all the other boilerplate involved, I would have expected it to be trivial to add relevant templating.
After seeing Swift's result I had to look into the source to confirm that yes, it was not written by someone who knows the language.
But this is a good benchmark results that demonstrate what performance level can you expect from every language when someone not versed in it does the code porting. Fair play
- There's a stark jump of times going from ~200ms to ~900ms. (Rust v1.92.0 being an in-between outlier.)
- C# gets massive boost (990->225ms) when using SIMD.
- But C++ somehow gets slower when using SIMD.
- Zig very fast*!
- Rust got big boost (630ms->230ms) upgrading v1.92.0->1.94.0.
- Nim (that compiles to C then native via GCC) somehow faster than GCC-compiled C.
- Julia keeps proving high-level languages can be fast too**.
- Swift gets faster when using SIMD but loses much accuracy.
- Go fastest language with own compiler (ie not dependent to GCC/LLVM).
- V (also compiles to C) expected it (appearing similar) be close to Nim.
- Odin (LLVM) & Ada (GCC) surprisingly slow. (Was expecting them to be close to Zig/Fortran.)
- Crystal slowest LLVM-based language.
- Pure CPython unsurpassable turtle.
Curious how D's reference compiler (DMD) compares to the LLVM/GCC front-ends, how LFortran to gfortran, and QBE to GCC/LLVM. Also would like to see Scala Native (Scala currently being inside the 900~1000ms bunch).
* Note that uses `@setFloatMode(.Optimized)` which according to docs is equivalent to `--fast-math` but only D/Fortran use this flag (C/C++ do not).
** Uses `@fastmath` AND `@simd`. The comparison supposedly is for performance on idiomatic code and for Julia SIMD is a simple annotation applied to the loop (and Julia may even auto do it) but should still be noted because (as seen in C# example) it can be big.
Looking closer at the benchmarks, it seems that C# benchmark is not using AOT, so Go and even Java GraalVM get here an unfair advantage (when looking at the non SIMD versions). There is a non trivial startup time for JIT.
Sorry, I can't seem to edit my answer anymore, but I was mistaken, C# version is using AOT. But the are other significant differences here:
> var rounds = int.Parse(File.ReadAllText("rounds.txt"));
> var pi = 1.0D;
> var x = 1.0D;
> for (var i = 2; i < rounds + 2; i++) {
> x = -x;
> pi += x / (2 \* i - 1);
> }
> pi \*= 4;
> Console.WriteLine(pi);
For example, if we change the type of 'rounds' variable here from int to double (like it is also in Go version), the code runs significantly faster on my machine.
Try that on ARM64 and the result will be the opposite :)
On M4 Max, Go takes 0.982s to run while C# (non-SIMD) and F# are ~0.51s. Changing it to be closer to Go makes the performance worse in a similar manner.
Startup time doesn’t seem to be factored in correctly, so any language that uses a bytecode (e.g. Java) or is compiling from source (e.g. Ruby, Python, etc.) will look poor on this. If the kids of applications that you write are ones that exit after a fraction of a second, then sure, this will tell you something. But if you’re writing server apps that run for days/weeks/months, then this is useless.
Python took 86 seconds, if I'm reading it correctly. I can see your point holding for a language like Java, but most of Python's time spent cannot have been startup time, but actual execution time.
This sort of thing is pretty meaningless unless the code is all written by people who know how to get performance out of their languages (and they're allowed to do so). Did you use the right representation of the data? Did you use the right library? Did you use the right optimization options? Did you choose the fast compiler or the slow one? Did you know there was a faster or slower one? If you're using fancy stuff, did you use it right?
I did the same sort of thing with the Seive of Eratosthenes once, on a smaller scale. My Haskell and Python implementations varied by almost a factor of 4 (although you could argue that I changed the algorithm too much on the fastest Python one). OK, yes, all the Haskell ones were faster than the fastest Python one, and the C one was another 4 times faster than the fastest Haskell one... but they were still over the place.
It's true this is a microbenchmark and not super informative about "Big Problems" (because nothing is). But it absolutely shows up code generation and interpretation performance in an interesting way.
Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it.
The delta there is because the Rust 1.92 version uses the straightforward iterative code and the 1.94-nightly version explicitly uses std::simd vectorization. Compare the source code:
> Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it.
The benchmark also includes startup time, file I/O, and console printing. There could have been a one-time startup cost somewhere that got removed.
The benchmark is not really testing the Leibniz loop performance for the very fast languages, it's testing startup, I/O, console printing, etc.
Reading the fine print, the benchmark is not just the Leibniz formula like it says in the chart title. It also includes file I/O, startup time, and console printing:
> Why do you also count reading a file and printing the output?
> Because I think this is a more realistic scenario to compare speeds.
Which is fine, but should be noted more prominently. The startup time and console printing obviously aren't relevant for something like the Python run, but at the top of the chart where runs are a fraction of a second it probably accounts for a lot of the differences.
Running the inner loop 100 times over would have made the other effects negligible. As written, trying to measure millisecond differences between entire programs isn't really useful unless someone has a highly specific use case where they're re-running a program for fractions of a second instead of using a long-running process.
The Clojure version is not AOT'd, so it's measuring startup + compiler time. When properly compiled it should be comparable to the Java implementation.
That first big jump in the graph, I thought that it must be the divide between auto-gc'd and non auto-gc'd languages. But then I noticed that Rust is on the wrong side of the divide.
I think I get why C++ thru C are all similar (all compile to similar assembly?), but I don't get why Go thru maybe Racket are all in what looks like a pretty narrow clump. Is there a common element there?
The common element is that they're written with the most obvious version of the code, while the ones in the faster bucket are either explicitly vectorized or written in non-obvious ways to help the compiler auto-vectorize. For example, consider the Objective C version of the loop in leibniz.m:
for (long i = 2; i <= rounds + 2; i++) {
x *= -1.0;
pi += x / (2.0 * i - 1.0);
}
With my older version of Clang, the resulting assembly at -O3 isn't vectorized. Now look at the C version in leibniz.c:
rounds += 2u; // do this outside the loop
for (unsigned i=2u; i < rounds; ++i) // use ++i instead of i++
{
double x = -1.0 + 2.0 * (i & 0x1); // allows vectorization
pi += (x / (2u * i - 1u)); // double / unsigned = double
}
This produces vectorized code when I compile it. When I replace the Objective C loop with that code, the compiler also produces vectorized code.
You see something similar in the other kings-of-speed languages. Zig? It's the C code ported directly to a different syntax. D? Exact same. Fortran 90? Slightly different, but still obviously written with compiler vectorization in mind.
(For what it's worth, the trunk version of Clang is able to auto-vectorize either version of the loop without help.)
They’re measuring program execution time, including program startup and tear down. Languages with a more complex runtime take longer for the startup, and all seem to have roughly equally optimized that.
I think it's SIMD generation. Managed runtimes have a much harder time autovectorizing, because you can't do any static analysis about things like array sizes. Note that the true low-level tools are all clustered around 2-300ms, and that the next level up are all the "managed" runtimes around 1-2s.
The one exception is sort of an exception that proves the rule: it's marked "C# (SIMD)", and looks like a native compiler and not a managed one.
This is meaningless. The benchmarks are (1) run in github actions, (2) include file and console IO, and (3) are compiled with different compiler flags...
The code looks 100% identical except for the namespace prefixes. Must be something particular about github setup, because on mine (gcc15.2.1/clang20.1.8/Ryzen5600X) the run time is indistinguishably close. Interestingly, with default flags but -O3 clang is 30% slower, with flags from the script (-s -static -flto $MARCH_FLAG -mtune=native -fomit-frame-pointer -fno-signed-zeros -fno-trapping-math -fassociative-math) clang is a bit faster.
A nitpick is that benchmarking C/C++ with $MARCH_FLAG -mtune=native and math magic is kinda unfair for Zig/Julia (Nim seem to support those) - unless you are running Gentoo it's unlikely to be used for real applications.
I suspect this is it. Any benchmark that takes less than a second to run should have its iteration count increased such that it takes at least a second, and preferably 5+ seconds, to run. Otherwise CPU scheduling, network processing, etc. is perturbing everything.
BenchExec "uses the cgroups feature of the Linux kernel to correctly handle groups of processes and uses Linux user namespaces to create a container that restricts interference of [each program] with the benchmarking host."
(I have spent a good amount of time hacking the llvm pass pipeline for my personal project so if there was a significant difference I probably would have seen it by now)
As with all the ahead-of-time compiled languages that I checked, the answer is that it generates non-SIMD code for the hot loop. The assembly code I see in godbolt.org isn't bad at all; the compiler just didn't do anything super clever.
Some implementations seem vectorization-friendly like the C one that uses a bit-twiddling trick to avoid the `x = -x` line that the Odin implementation and others have.
When you put these programs into Godbolt to see what's going on with them, so much of the code is just the I/O part that it's annoying to analyze
In my opinion, the comparisons could be better if the file I/O and console printing were removed.
Memory-mapped I/O can be great in some circumstances, but a one-time read of a small file is one of the canonical examples for when it isn't worth the hassle and setup/teardown overhead.
But this is a good benchmark results that demonstrate what performance level can you expect from every language when someone not versed in it does the code porting. Fair play
Makes the benchmarks game 100 lines seem like major apps.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- C++ unsurpassable king.
- There's a stark jump of times going from ~200ms to ~900ms. (Rust v1.92.0 being an in-between outlier.)
- C# gets massive boost (990->225ms) when using SIMD.
- But C++ somehow gets slower when using SIMD.
- Zig very fast*!
- Rust got big boost (630ms->230ms) upgrading v1.92.0->1.94.0.
- Nim (that compiles to C then native via GCC) somehow faster than GCC-compiled C.
- Julia keeps proving high-level languages can be fast too**.
- Swift gets faster when using SIMD but loses much accuracy.
- Go fastest language with own compiler (ie not dependent to GCC/LLVM).
- V (also compiles to C) expected it (appearing similar) be close to Nim.
- Odin (LLVM) & Ada (GCC) surprisingly slow. (Was expecting them to be close to Zig/Fortran.)
- Crystal slowest LLVM-based language.
- Pure CPython unsurpassable turtle.
Curious how D's reference compiler (DMD) compares to the LLVM/GCC front-ends, how LFortran to gfortran, and QBE to GCC/LLVM. Also would like to see Scala Native (Scala currently being inside the 900~1000ms bunch).
* Note that uses `@setFloatMode(.Optimized)` which according to docs is equivalent to `--fast-math` but only D/Fortran use this flag (C/C++ do not).
** Uses `@fastmath` AND `@simd`. The comparison supposedly is for performance on idiomatic code and for Julia SIMD is a simple annotation applied to the loop (and Julia may even auto do it) but should still be noted because (as seen in C# example) it can be big.
For the sub-second compiled languages, it's basically a benchmark of startup times, not performance in the hot loop.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
On M4 Max, Go takes 0.982s to run while C# (non-SIMD) and F# are ~0.51s. Changing it to be closer to Go makes the performance worse in a similar manner.
C# is using CoreCLR/NativeAOT. Which does not use GCC or LLVM also. Its compiler is more capable than that of Go.
I did the same sort of thing with the Seive of Eratosthenes once, on a smaller scale. My Haskell and Python implementations varied by almost a factor of 4 (although you could argue that I changed the algorithm too much on the fastest Python one). OK, yes, all the Haskell ones were faster than the fastest Python one, and the C one was another 4 times faster than the fastest Haskell one... but they were still over the place.
It's true this is a microbenchmark and not super informative about "Big Problems" (because nothing is). But it absolutely shows up code generation and interpretation performance in an interesting way.
Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it.
https://github.com/niklas-heer/speed-comparison/blob/master/...
https://github.com/niklas-heer/speed-comparison/blob/master/...
The benchmark also includes startup time, file I/O, and console printing. There could have been a one-time startup cost somewhere that got removed.
The benchmark is not really testing the Leibniz loop performance for the very fast languages, it's testing startup, I/O, console printing, etc.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
How significant are the differences for such tiny tiny programs?
There is very little superfluous or that cannot be inferred by the compiler here: https://github.com/niklas-heer/speed-comparison/blob/master/...
> Why do you also count reading a file and printing the output?
> Because I think this is a more realistic scenario to compare speeds.
Which is fine, but should be noted more prominently. The startup time and console printing obviously aren't relevant for something like the Python run, but at the top of the chart where runs are a fraction of a second it probably accounts for a lot of the differences.
Running the inner loop 100 times over would have made the other effects negligible. As written, trying to measure millisecond differences between entire programs isn't really useful unless someone has a highly specific use case where they're re-running a program for fractions of a second instead of using a long-running process.
You see something similar in the other kings-of-speed languages. Zig? It's the C code ported directly to a different syntax. D? Exact same. Fortran 90? Slightly different, but still obviously written with compiler vectorization in mind.
(For what it's worth, the trunk version of Clang is able to auto-vectorize either version of the loop without help.)
- run bytecode - very high level - GC memory
But not all have these traits. Not sure.
The one exception is sort of an exception that proves the rule: it's marked "C# (SIMD)", and looks like a native compiler and not a managed one.
Also, winners don’t make excuses.
(Not even being snarky. You have to spiritually accept that as a fact if you are in the PL perf game.)
Swift: 3.7
Python: that's incorrect!
Swift: yeah, but it's fast!
A nitpick is that benchmarking C/C++ with $MARCH_FLAG -mtune=native and math magic is kinda unfair for Zig/Julia (Nim seem to support those) - unless you are running Gentoo it's unlikely to be used for real applications.
BenchExec "uses the cgroups feature of the Linux kernel to correctly handle groups of processes and uses Linux user namespaces to create a container that restricts interference of [each program] with the benchmarking host."
https://github.com/sosy-lab/benchexec
(I have spent a good amount of time hacking the llvm pass pipeline for my personal project so if there was a significant difference I probably would have seen it by now)
What do you think they could have done better assuming that the IO is a necessary part of the benchmark?
Also good job to the Rust devs for making the benchmark so much faster in nightly. I wonder what they did.
The differences among the really fast languages are probably in different startup times if I had to guess.
Startup times matter a great deal.
When you put these programs into Godbolt to see what's going on with them, so much of the code is just the I/O part that it's annoying to analyze