Benchmarks differences in keynote-2

[brandonpollack23]

I just was taking a look and found some things im curious about, would you mind explaining why some of these differences were done (different clients and protocols mainly, as well as a lack of warmup on other clients).

https://news.ycombinator.com/item?id=47148920

[Centril]

Hey Brandon,

First, a small note about warmup. Ripping the warmup period out makes no difference for SpacetimeDB's TPS numbers, so that can simply be removed from he benchmark without affecting the results, so we'll do that.

I cannot speak to the benchmarks for the other databases, but I did write the Rust benchmark client, so I can give some insights into that client. I wrote this client to measure the maximum TPS throughput of the host alone, without care for the client's performance. While optimizing the datastore of SpacetimeDB and looking at flamegraphs,
using the typescript client and our higher level APIs, the host was getting bottlenecked on the client using too many resources and not sending requests fast enough.
So I wrote a raw client to do the bare minimum work necessary to talk to the host so that the host's performance could be measured and improved.

Beyond my notes here, we're going to write a proper technical writeup addressing the concerns you've highlighted and those discussed elsewhere as well.

[brandonpollack23]

Thanks so much for your prompt reply. I realized after some things I said weren't quite right so I'm going to add an addendum.

First off setting sqlite with an invalid sync param will likely turn it off according to my genie, and also one thing worth noting that wasn't immediately clear to me is even though sqlite and the others have this extra rpc hop, that is exactly the thing spacetime doesn't have since everything is a stored procedure in the db with wasm. What isn't quite fair is that you use HTTP for those with JSON etc and tons of overhead but Spacetime gets to use a socket in your benchmark.

I also am curious if you compare to other direct backends with the extra server hop (since that's fair) but using a socket connection to the server (which is what a game would do) as well as using NodeJS with SpacetimeDB. Looks like it still uses websockets.

Basically we're comparing two different use cases RPC vs REST which are different so it'd be cool to see apples to apples comparisons :)

[xthexder]

Quoting from the README:

When running the benchmark for SpacetimeDB on higher-end hardware we found out that we were actually bottlnecked on our test TypeScript client. To get the absolute most out of the performance of SpacetimeDB we wrote a custom Rust client that allows us to send a much larger number of requests then we could otherwise. We didn't do this for the other backends/databases as they maxed out before the client.

This claim that all the other clients are bottlenecked on the database, and not Node.js and JSON serialization is a little hard to believe. How exactly did you determine that the other backends were "maxed out"? Using more efficient communication methods than REST reduces load on both the clients and the server.

[buggymcbugfix]

Here is a good writeup by @tanayvk that picks out the SQLite case and shows it can be brought up to the same ballpark without much hassle.

[cloutiertyler]

Also worth noting that I responded to @tanayvk's write up here: https://spacetimedb.com/blog/benchmarking#claim-12-sqlite-can-achieve-the-same-performance-as-spacetimedb-with-batching