I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.
These seem like some fairly standard approaches for reducing memory usage. I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
For the curious, this is technically possible in Rust using a dynamically sized type [1], but in practice is difficult and doesn't really play nice with the rest of the language. The nomicon entry concludes with "Yes, custom DSTs are a largely half-baked feature for now." [2]
One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
Maybe I'm misunderstanding, but this powers 1.1.1.1, it doesn't front an internal dataset. A cache miss hits a nameserver. Which is to say, the dataset is "every DNS record in the world"
You have to cache, cloudflare doesn't know all the records ahead of time, they have to do recursive lookups to the authoritative servers that own the records and that is only good for the period of the TTL of the record. There is no "global" DNS record database or something like that.
With my own MaraDNS, I aggressively optimized the memory usage of blacklist entries by having a single really big malloc() to allocate the memory for the entries, then traversing that memory block for potentially blacklisted entries.
When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
It is often not worth optimising in the early days. You don't know how popular it will become, you might not know how many DNS records you will hold, it was possibly written in an earlier language and ported as-is.
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
It only looks super obvious in hindsight and the well explained blog post. when a team of 5 is tasked with getting a completely new DNS up at the scale and integrate well with cloudflare.
if you spend cycles on nitty gritty opinions like this time to market goes out further and further out. some napkin math, 130 gen13 servers cost "only" ~$2.6M. relative to the importance of the 1.1.1.1 and the market at the time. that is nothing to cloudflare.
this is not to say good system design does not matter. it very much does, but making that call at that time would've butchered the prodcut very much similar to google+, youtube etc.
Premature optimization argument fits right in. Now that memory is up to 10x more expensive it is worth considering optimizing programs with large memory footprint.
One of the "evils" of premature optimization is how much time you spend on the optimization vs. the benefit you get from it. If your goal is correctness and shipping fast and you're not memory constrained then spending time using the least amount of memory is a waste of time specifically because you want to ship fast.
Another interesting thing that happens is you don't necessarily know what form your actual optimizations will need to take. Later when your systems grow you discover the suboptimal parts you hadn't optimized for.
Very early on at Cloudflare I worked on part of the DNS infrastructure that took DNS records from the UI and got them in a state for actual authoritative serving. The system had been constructed anticipating Cloudflare having millions of customers with unique domains, but it had not been constructed for a single customer with a single domain with millions of records. This caused a periodic slow down in DNS record updating while the system churned on that one customer.
In a different job I worked on a piece of optimization software that needed to keep track of "node" A is reachable from node "B". This had been implemented as a matrix (literally a malloced NxN matrix of ints storing 0 or 1) which worked really well for small systems. But you'd be out of memory really fast on a large project. I replaced the matrix with a hash table and all was good because the matrix was actually really sparse.
Absolutely true, but I will say that LLMs have changed the equation somewhat.
With a rather short prompt, claude/codex will take your code, write a harness, profile it, build experiments, profile those, and give some pretty solid advice which one to pick. Then integrate the changes. It's the kind of goal-directed, bite-sized job that LLMs excel at. Extremely low-commitment.
Except for the whole "making changes in production at scale" problem, of course.
Engineers are expensive, especially good system engineers who are trained in your code base. Very possible that this just hadn't gotten to the top of the priority list.
I don't understand why you need training on your code base to design a cache format for read only vs rw workloads, but anyway yours is a comment about neglect, not the "evil" that would happen if you did that design
I see your point but disagree. Engineering is about constraints. Time, materials, labor, scope.
The “evil” of premature optimization is that it’s a misapplication of priority. If I have an acute medical problem that needs attention, it’s not the right time to talk about chloresterol and statins, get my broken leg set.
There’s always a tension between engineering management who needs to deliver a solution to the business and engineers who want to deliver a beautiful object.
> I don't understand why you need training on your code base to design a cache format
Because anyone willing to come in just to design your cache format is going to expect payment that is many multiples more than the engineers you already cannot afford? Long-term employees cost less, which brings them closer to being affordable, but you have to be able to keep them busy for long periods of time to realize that reduction in cost. A engineer who doesn't understand your codebase isn't going to be useful for very long.
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
[1] https://doc.rust-lang.org/reference/dynamically-sized-types....
[2] https://doc.rust-lang.org/nomicon/exotic-sizes.html
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
It's an interesting problem.
When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry...
Interestingly this is exactly how netlink works-ish: https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.ht...
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
Were there no design discussions/reviews when the system was setup to catch trivial things like this?
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
So how would you decide which path to take in situations like this?
if you spend cycles on nitty gritty opinions like this time to market goes out further and further out. some napkin math, 130 gen13 servers cost "only" ~$2.6M. relative to the importance of the 1.1.1.1 and the market at the time. that is nothing to cloudflare.
this is not to say good system design does not matter. it very much does, but making that call at that time would've butchered the prodcut very much similar to google+, youtube etc.
Another interesting thing that happens is you don't necessarily know what form your actual optimizations will need to take. Later when your systems grow you discover the suboptimal parts you hadn't optimized for.
Very early on at Cloudflare I worked on part of the DNS infrastructure that took DNS records from the UI and got them in a state for actual authoritative serving. The system had been constructed anticipating Cloudflare having millions of customers with unique domains, but it had not been constructed for a single customer with a single domain with millions of records. This caused a periodic slow down in DNS record updating while the system churned on that one customer.
In a different job I worked on a piece of optimization software that needed to keep track of "node" A is reachable from node "B". This had been implemented as a matrix (literally a malloced NxN matrix of ints storing 0 or 1) which worked really well for small systems. But you'd be out of memory really fast on a large project. I replaced the matrix with a hash table and all was good because the matrix was actually really sparse.
With a rather short prompt, claude/codex will take your code, write a harness, profile it, build experiments, profile those, and give some pretty solid advice which one to pick. Then integrate the changes. It's the kind of goal-directed, bite-sized job that LLMs excel at. Extremely low-commitment.
Except for the whole "making changes in production at scale" problem, of course.
The “evil” of premature optimization is that it’s a misapplication of priority. If I have an acute medical problem that needs attention, it’s not the right time to talk about chloresterol and statins, get my broken leg set.
There’s always a tension between engineering management who needs to deliver a solution to the business and engineers who want to deliver a beautiful object.
Because anyone willing to come in just to design your cache format is going to expect payment that is many multiples more than the engineers you already cannot afford? Long-term employees cost less, which brings them closer to being affordable, but you have to be able to keep them busy for long periods of time to realize that reduction in cost. A engineer who doesn't understand your codebase isn't going to be useful for very long.