Cooling Infrastructure with Ellie Ford

Welcome to Fork Around and Find Out, the podcast about building, running, and maintaining
software and systems.
Hello, and welcome back to Fork Around and Find Out.
I am your host, Justin Garrison, and with me today is Autumn Nash, as always.
How's it going, Autumn?
Surprising?
Like, I don't know.
The silence, I know.
I know.
There's a lot going on.
Your daily distraction, or your monthly distraction from all the other bad news.
Let's talk about outages and software.
It seems so much better right now.
It's just a lot, okay?
Joining us on the podcast today, we have Ellie Ford, staff software engineer at DevZero.
Thanks for coming on, Ellie.
Of course.
Thank you for inviting me on.
I absolutely invited you on because your blue sky feed was for a time phenomenal for giving
me continued insights into what's going on in data centers and hardware, because I really
kind of miss it.
Like, there's a lot of things that were just like, if you stay away from it for one or
two cycles, like, you just, it's gone.
Like, you just, you don't know anything that's going on anymore.
You go to the cloud for four or five years and you're like, well, this must be state
of the art, right?
No.
Like this is like the amount of performance and price per watt and everything else you
can get by just buying some hardware is really changed over over time.
Yeah, I think like one of the things that has been the most interesting is watching
like the blue sky, you know, the company behind blue sky, like build their own infra and build
it for kind of, you know, a tremendous scale.
They are one of the biggest proponents of like hosting it your own because they too have
realized and done the math and they're like, you know, to run this on AWS and with all
of the egress pricing and all of, like, like all of the miscellaneous Bezos tax that you
pay for, like it is so tremendously cheap to actually just go out and buy hardware and
depreciate it over like seven years and just like stick it in a colo, right?
It doesn't matter if you have like remote hands and like you have to pay for like people
to staff it.
It's just, it is so tremendously cheaper and you get more performance and you get more
control and kind of at the, you know, you trade that off with like you have to do more
capacity planning.
You have to think more through like, okay, you know, we can only handle so much load
on this hardware, but then you kind of get into things like performance tuning and things
like that, which is so much fun.
I mean, I'm a big proponent of people should budget there, like plan to some capacity and
just make a budget and say, I'm not going over this.
And that's like the cloud in general with infinite, infinite scaling credit card bills
is more along the lines of not infinite scaling infrastructure, but you can spend as much
money as you want on anything and never have a plan for it.
There was, so we've been doing a lot of Kubernetes optimization work at my work lately.
And there was a study by DataDog, I think, that showed that I think 80% of containerized
workloads are over provisioned, right?
So like, and it's, it has been crazy, like doing metrics and doing like a detailed rigorous
analysis of like how much waste is actually happening inside of Kubernetes environments.
And it's insane.
And you're paying for all of that too.
Is that new though?
Not really.
Like we used to buy servers and they'd sit there, you know, 10% utilization maybe.
And then we'd like, oh, we need to utilize this more or less use VMs.
And then we had a bunch of VMs that were 10% utilized and then we put it in containers
and those are all, you know.
In every case, I feel like we're just like, like try to densely pack these things more
and more.
And at some point you hide it, right?
We're like, VMware had a memory sharing, right?
You're like, I got 20 VMs on this, but I'm only using the RAM from one of them because
it's all doing the same thing.
Is the waste, is the waste a bad thing though?
Like, I mean, so my background is, is a SRE, right?
So I typically look at waste as capacity rather than actual waste, which very much flies contrary
with like what the bean counters like to think and everything like that.
Especially once you're talking about, you know, like reducing tail latency and like
meeting strict SLAs, it is so much harder to do that when you are on a system that is
like fully loaded.
And one of the things that's been really interesting is like the Linux kernel isn't really well
designed to handle large amounts of load, right?
You think of things like the out of memory killer, the out of memory killer wants to
preserve kernel functionality rather than user space functionality.
So you will have the out of memory.
I have a four by four that says OOM on it for my, it's my, it's my prop for out of memory
killers.
Yeah, it's just so funny.
Like you will have to go out and you will like to make any system functional like Facebook
has found this out the hard way.
You have to have an early OOM daemon that kills processes before it even starts to trigger
the kernel out of memory reaper because it will just like lock up that whole system and
you'll have to power cycle it.
Yeah, the work Facebook and Meta has been doing on a user space OOM killer is fascinating
to me because it just shows that even though the kernel has all the power, it doesn't have
all the context and information for like what's actually important.
And we try to do that on the operating system I work on where it's like, how do we protect
the things that we want to protect?
It's like, we can't just do it in kernel.
Like we have to have something that knows what's actually going on.
Yeah, like you have like niceness values, which are supposed to do that.
But if you've ever looked on like any regular Linux system, you'll see that like some processes
are set as negative 20, which are like never kill and then others are like zero and like
no one actually uses these kernel tunables.
So you get, you quite often get stuck in this situation where you just run into out
of memory and you can't do anything about it, right?
So what sort of performance tuning have you been doing on Kubernetes?
Fine.
You know, you're just trying to eliminate the over provisioning or is it a latency or
what is it?
Not necessarily latency.
So before we started doing the cost optimization work, you know, startups, we pivot all the
time.
We used to do cloud development environments before that.
And a lot of the people at Dev Zero have really like they have worked on developer environments
at Uber.
We have people from Facebook.
We have people from Carta.
A lot of people who have done these developer environments.
And one of the things that we were trying to do is like trying to make it snappy and
responsive and actually functional, right?
We wanted to do everything that an EC2 virtual machine could do, but then better and on your
own infrastructure, which is not particularly easy.
And it was in kind of like a different vein from like what Coder and everything does because
Coder, like to their credit, it's more or less just the Terraform wrapper.
It just provisions that environment for you and that does some like initial setup.
It was, we were a lot more controlled.
There were, I introduced micro VMs into the mix, which made things chaotic and so fun.
I had to debug multiple hypervisor bugs.
It was so gory.
And you know, developers, we were trying to also solve like for developer efficiency
at the same time and respond like responsiveness and everything like that, which basically
meant that like we took it like Ubuntu distribution.
We pinned the kernel at a specific version.
We made our own kernel config with our own options to like slim it down as much as possible
to reduce startup time.
And then we had a custom PID one, like PID one that would run before anything else in
the container that would copy all of the contents over to a persistent block device.
It was, it was batshit crazy.
I think that's kind of the struggle though.
People want to create Linux distributions that have the least amount of what you need,
you know, to do it because for one, it makes it more secure, two, it makes it faster.
But try to figure out exactly what you need to be usable as the hard part balancing that
and actually being usable.
No, absolutely.
It's, it is so much back and forth.
It is, it is tuning.
It is trial and error.
It is, it is like trying 15 different kernel configs to find out which one works, which
one doesn't.
It is, it is an egregious amount of time and I have so, so much newfound respect for people
who maintain distributions after that because I was like, this is, you could probably spend
the rest of your life just tuning the Linux kernel.
And not just that, but like Linux users are very specific about what they want.
You know what I mean?
Like it's, it's its own religion.
Like people.
You calling me out?
What is this?
Yes.
Yes, I'm calling you out, sir.
What do you mean?
I got a new fun Linux distro this week was it Rocknix?
Justin, every other week, he's like, I found a new distro and this is how I'm going to
hurt my own feelings this week.
Like, why are you like this?
It's like you like paint or something for you and all our listeners.
If you haven't compiled the Linux kernel the last couple of weeks, maybe you should.
Maybe you should try.
You see this?
He's like, call these Linux isn't enough.
I have to go with her.
It's okay.
I'm going to in high school.
So just started out like, yeah, choosing violence, just choosing violence.
Like, like I was, I was sitting in like AP human geography and I was compiling Firefox
from source and like, laptop would go to a hundred percent, fans would kick on.
It was, it was a sight to behold.
I feel like you were doing the right thing because I was so bored in high school.
At least she found something that was like fun to do.
Like I was just sitting there doodling.
I don't know.
I think it's like, it's also hard too because there's no like, I think a lot of the other
open source projects that everybody use have this like one united overall project and they
might kind of branch off, but Linux distributions are different.
Like there are so many sources of truth and there's so many like, there's so many different
ways where people will pick the packages they want in their distribution and a lot of open
source and the beauty of open source is maintaining things with a community, right?
So you're not maintaining it by yourself and kind of like having that continuity over the
packages that are built.
And then trying to figure out what you want to continue to have is hard.
It's even like, like when you're building an end user application for a Linux system,
it's like, what versions of glibc do you need to support?
How do you want to package it?
Do you want to use RPM?
Do you want to use dead?
I feel like I have PTSD this week, good lord.
I have learned the correct way is to statically link everything to Musil Libc and ship people's
static binaries because everyone will just figure it out if you give them a static binary
that just works on their system.
But Linux distribution, like okay, when you're, Linux users are a special like, you have to
know your customer, right?
Like when you're building anything, you have to know your customer and like something about
developers, especially Linux developers are like, I want to have all like, it's like people
that like, there's people that use Macs in real life or some sort of like operating system
in real life, but they're Linux developers, right?
And then you've got the people who have to have Linux on everything, Justin.
And then like, like, you know what I mean, like they're different flavors of like they
want all the knobs, or I want it to be bare metal, like, and just do only what I want
it to do and have nothing else, you know what I mean?
Like, so it's like, what is your distribution for?
Is it for the people that want just peer secure?
There's so many different levels and you're building for like, who are, who's your audience?
You know, like, if, yeah, absolutely.
And that's like, like, that's why if I was to tell anyone, like, like any advice
from building cloud developer environments for like the past, like our company's been
doing it for the past three years, right?
If there's anything we've learned, it's like, don't do it, don't do it.
It's so bad, so awful.
It's kind of fun too, though.
Like when you were talking earlier about how you like, and I put this like cloud
vm and I did this and I was just like, it's because you like chaotic things.
Like I think we all have a little bit of like, you just kind of like the fire a little bit.
And then you have to figure out like how much you're like risk tolerances for, you know,
like, but another thing that you mentioned that I thought was interesting is you
mentioned Colos and I've talked to people at Blue Sky who do that.
And it's just, I don't think people realize that that's the intermediate, right?
Like you have a cloud where you completely don't know what's going on.
You have nothing to do with it.
You've got Colos where you buy all the hardware, you can depreciate it, you ship
it, and it's cheaper, right?
But you still have some kind of like, they're your VMs, but somebody else
may be maintaining it.
You might have to pay for hands, whatever.
And then you have completely on-prem, you are writing, running a data center.
And I feel like we talk about on-prem, we talk about cloud, but nobody talks
about Colos and how, like, because we're talking about on-prem, like everyone's
going back to on-prem.
People are not going back to truly writing your own data centers.
Don't lie.
Like, you know what I mean?
Like people keep writing these articles for spicy, like takes and I'm like,
bullshit, like that's not what we're doing right now.
Like people are going to the middle option of like, not today Satan.
I don't want to go back to holding a pager myself, you know, like, but buying
my own hardware is cheaper.
So like, I love that you brought that up because I feel like we, like.
People are writing a lot of articles to be spicy, right?
And like there is value in it being cheaper, but don't lie.
Nobody's doing that kind of work.
And the fact that we like, and that's the wild thing, because all of a sudden we
now need DBAs, we need like people that run data centers.
And not everybody has that talent.
And most of the people that do are these huge companies that do nothing, but, you
know, run people's Colos or the people that are, you know, have these big data
centers.
I find that really fascinating where the, the people that are making on-prem and
data centers sound so cool are the cloud providers.
It's like literally Amazon's like, we're making the best, biggest, coolest
data center ever existed.
And everyone's like, I want one of those, right?
And like, well, let's just use ours.
You can just, you can just pay for it.
It's funny because it goes back to exactly what we're talking about Linux.
It's the different levels of abstraction.
How many knobs do you want to press?
How many, like how much responsibility do you want here?
It's like Macedon versus blue sky.
People were like, yeah, and we're going to have our own stuff and we're going to
do it.
And then we got halfway through.
We're like, this is terrible.
I don't want this much responsibility.
I just wanted it to be fun.
And then we were like, but we don't want to go back to the evil place and the
bad place.
So then they were like, okay, the middle blue sky is the Colos of like social,
of like social media, right?
Like you got like, we don't want to completely be responsible because we do
that all day.
And nobody wants to run their own PDS.
Did it?
Okay.
But did you remember earlier when I said that Justin chooses violence and he
likes to be like, am I wrong?
Fatal masochism is a disease.
I'm telling you, you just got to pick what level of pain that you're just like
excited about.
I have to know how it works and I have to do it.
So there's also this, he made it where he couldn't even like direct message me
and talk to me on this guy for like a month.
Okay.
And then I was just like, Justin, every time I tried to at you and he's like,
no, it's fine.
What are you talking about?
It was a feature.
It was not fine.
Ali, he didn't even like figure it out for a month.
Okay.
Yeah.
So whenever someone talks about on-premise, I show them the picture of the OVH data
center fire.
The big one that just lit on fire.
Right.
I'm like, you want to deal with that?
Not just that, but now people are like, I'm running a server in my grandma's
basement.
Please don't let your grandma's house on fire.
Like grandma did not ask, are you hoping grandma paid this electricity bill?
Like goodness.
And that's, that is one of the things that I find super fascinating, especially
about hardware, which I do want to talk about is how much cheaper,
and quieter, and low power these, especially like the home lab systems,
like the N100 chips are like, they just trickle power and can run a bunch of
stuff.
It's just super fascinating.
We met Matthew works for them.
Oxide, yeah.
Oh my God, their servers are wild.
I have never geeked out about a server so hard.
I mean, usually like, ew, I don't want to touch those things.
But that one, I was like, what?
Like, I'm not buying one of these like, like, and then the tests that they do,
they'll like go with their little mics and you're just like,
I'm like, I have been begging the oxide people to like give me a rack for free.
And I will like, I will do horrendous, awful things to it.
I want to know how you break it.
I want to know how you put it together.
I wouldn't like, you know what I mean?
I'm just like, you can do what?
Like, do you know how many hood rat things you could run on that?
And just so much fun.
You know, I, I've been, we've been talking about, um, like the, the poly
keel, poly keel oxide rack, one of, one of the, uh, you know, understated
things about polyamory.
I know like three people that would be on this like server.
Like, oh, I can't wait to this episode goes live.
I'm just going to be like, I talked about you at Charlotte.
I'm going to be like, I love you so much.
I'm always the friend that like, I told them I want a T-shirt friend of the
poly keel, like poly keel.
I'm just like, you guys keep adding like more people and I just love everybody.
Yeah.
Like the, so the thing is, like hardware is getting more, more like efficient,
more powerful, but then at the same time, the AI craze is happening too.
And that's like, I don't know if you've ever seen an A 100 while it's running.
I don't know if you've ever had to work with an A 100 while it's running, but
it is a passively cooled 400 watt card.
So like, that's a lot of electrons.
Yeah.
That like it, it gets real hot, real toasty.
And that's kind of been one of the funny things about like, I've been working
with NVIDIA hardware for like the past year.
God knows how long before that, um, and like a data center capacity.
And like, no one doesn't like, no one tells you NVIDIA cards.
They get so hot.
They get so hot, so quick, even when they're idle, right?
I have had an A 100 shut down on me because it was passively cooled
and I didn't have any airflow going over it.
And so there are like 3D printed brackets that you will have to get and you
will have to point a whole 120 millimeter fan just straight at the fins to
cool it down because otherwise it will shut down.
Right.
So it's been funny.
Like CPUs are so efficient now, right?
Um, especially once you look at like the arm space, what Apple's doing with
like their M4 chips now, I think they're on.
Yeah.
Um, I got a new M4 laptop and I swear to God, I've never, it was the most
beautiful thing I've ever opened in my, like the things that you can do on
that laptop are just should be like, it's great.
I when the, when the first like ends, like when they were first talking
about switching over to arm, I was so skeptical charcoal ones.
Oh God, they're so, when they were first talking about switching over to arm,
I was like, so skeptical.
I was like, no way in hell.
They pulled it off.
They pulled it off.
I'm sorry.
Like even, even working at a like, like language and runtime and seeing all
the extra stuff you had to do for those M1 M4 trips, worth it.
Well, I'm like, I, I swear to God, like I run exclusively Max.
I had one laptop, right?
It was a terabyte of RAM.
It was like, oh, it's going to be my new gaming computer.
And I would put one here and then I had an M4 laptop.
This fan's like kicking on.
It's dying.
You'd run all the same stuff on an M4 and it's like, oh, like, it's
you're just like, how like there, it's a year difference between these two computers.
It's insane.
Yeah, like, like things are so efficient now.
Well, they have to be because it's the only way we're going to be able
to continue to this whole AI race because there's no way they can continue
to like use this much electricity and water, you know what I mean?
So they're going to have to figure it out and they want to put it on everything.
One of the things that nobody really talks about, it isn't really about the power.
I mean, like the power is obviously one of the big constraints in like growing
from the grid that much power is a huge concern.
And like what, what I think XAI is doing is they have gas turbines at their plant
now, which is like spitting, like spitting emissions into the air.
Polluting, like a small under.
Privilege community that they just keep pretending like they're not in line.
Oh, yeah.
Yeah, like, like the power is one thing, but no one talks about cooling, right?
Like, well, that's why some of them are using water, but it's crazy.
Like, did you see the study they did in Alaska?
They were like, well, just go to the cold.
I'm not Alaska and good drinker.
They're like, we'll build them there and then we'll open the windows.
And I was like, what are you doing to the environment by changing the
temperatures? And then they were like, we'll build one under water.
And it's not like we're already killing fish or anything.
Matt has been doing the open the windows for a long time now, because they have
the, the. Well, they're trying to build them in colder places.
Yeah, well, I think one of the things like they built these data centers
in what used to be colder places that are now getting warmer.
Hey, we need to move closer to the poles because it's close to it's colder there.
Yeah, there's going to be a lot of money for the first person to make a
colo at Antarctica, right?
So they're already trying, but I'm just saying, like, it's really working out
from my home value because Washington's going to be California.
Other one day, well, everything else is hot as balls.
So it's going to be like, great, like.
Now, question, though, if they build a data center or region at the South Pole,
is that, is that a West or an East eliminated?
Can we name it Santa or like, yeah, can I be like, can North Pole be the
like, instead of like East one?
Can it be like North Pole one?
Cause like with like a little like candy, like a candy cane, like emoji.
Emoji regions, that would be like, like East, like West, like, what is it?
West. Let's test all your Unicode.
West should be like a little, like, West should be like a little beach, you know?
And like, like New York East one or two or whatever.
Send us a blue.
What should the emojis be for each region for regions?
Dude, we're going to have, we're going to have a West coast,
like East coast works if you're going to be great.
I love it.
I don't know, but I mean, it's, it's fascinating.
Like you mentioned water cooling, though, right?
It's actually going further than water cooling.
It's like full immersion, like mineral, oil, immersion, cooling.
But they're talking about under the water.
We're right under the sea.
Yeah, there was that.
I think when I was at Facebook, when I was at Disney, Pixar had
a immersion data center, which was fascinating because you walk in there,
it's just quiet because there's no fans.
There's no, it's just like, what's going on?
Like it's hot in there too, because they run at a higher ambient temperature.
So you're in the, in the data center and it's like 85 Fahrenheit or something like that.
You're like, it's, it is muggy in here right now.
What is going on?
It's like, but it was, it was super cool because it was just like, yeah,
we can run these, you know, harder, longer and we have less breakage on things.
It wasn't all their data centers, but it was super fascinating just seeing
how that was playing out in like a real environment.
Yeah, I think, I think in California, I think I remember reading some of the
data centers are there because like they need to prioritize capacity so much.
And you, like the only way to get higher capacity inside of a data
center like that is to go immersion, right?
And so I think like some of the data centers in San Jose, at least last time
I was reading are like fully immersion as well, because it's come so far
and like a couple of years because they want to be able to push the constraints.
And it's amazing.
You, like, do you remember when it was just hobbyists, like the fish tank PC
on like Linus tech tips back in the day?
Do you remember, I remember going to PAX a couple of years ago and people were
walking in with their like PCs underwater, but it was to be cool.
Like it wasn't like, you know what I mean?
Like, no, like you need it to like run.
The bubbles coming off of this also the dedication to pull, to like drag a whole
ass PC down to like Seattle, like props.
It's, it's kind of funny.
It's like, there's like, it's very much a push and pull factor.
I think we're going to keep with that push and pull.
It's always, it's like, oh, cloud.
And then it's like data centers and like, there's always a limit, right?
You go, you find the next limit and you work around it.
Not even just the limit though, it's whatever people want to worship that day.
You know, religion, like, and then you're just like, there, there's a real tools
that you could just kind of weigh your options, but then it's just like, what,
like, what, what direction is the religion going on today?
And people are going to find their version of social media to argue to the
depths, even though there could be like a middle argument and there's like a good
use case for both, but you know, keep going.
Cause it wouldn't be fun if you weren't arguing.
No, I, well, yeah.
And you know, like so many push and pull factors.
I mean, like, because the internet is growing and continuing to grow, like we
are getting faster and faster networks, right?
I think they are now testing like 1.6 terabit optics.
Just wild.
Do you remember how long it took to like, download a movie back in the day?
Do you remember being on like.
I mean, I think one of the things that we're trying to talk about is like the
history of computing, like back in 2013, 2014, like it was rare to see a 40 gigabit
network even.
I think like at the high end, people would have 10 gig now, but like, you can
go talk to an ISP and get a dedicated 10 gig fiber line for about a thousand
dollars a month right now.
I know.
Cause I just talked to like Zeo and, and astound they're a local ISP in
the Seattle area.
Um, and it's like, it is, it is so cheap to get 10 gig now.
Um, and even on the hobbyist end, you can get a hundred gig equipment for under
$2,000.
Right.
I used Melanox switch will only run you about $1,500.
Don't tell me that.
No, no, no, poor Beth.
She's just going to have to put up with this now.
She'd be like, Justin, what are you doing in the garage?
I'll send you the eBay list because, because we just built a lab off of like a
hundred don't help this on eBay.
Literally yesterday.
Have you seen his garage?
Look behind him.
He doesn't need any help.
So the secret is, and before the tariffs, it was like the Chinese are
incredibly good at recycling data center equipment and, and turning it into
hobbyist grade equipment so you can get, um, Bluefield twos.
They're, they're insanely cool, um, network cards.
I can get into them a little bit more.
They're a hundred gig capable, actually 200 gig capable.
And like, you can get them for like $300 each, which is the same.
Are those the ones you were running?
They have like, you can run a full operating system on them.
Oh yeah.
Yeah, yeah, yeah.
I run, I ran a blue sky PBS on one of them.
Don't give it on a network card.
I remember, I remember you posted that.
And I spent the entire night watching YouTube videos about these cards because
I didn't know anything about all these different hacks that people were doing.
I'm not going to be able to add him again for like another month.
Cause he's going to break it.
So, so that's funny.
This has been a whole research article or like whole research point of research for
me.
There's a blog post about this, uh, at my work blog.
Um, I was going to say, if you're not on blue sky, are you talking about this
somewhere else?
Cause like then we'll lose all the information.
Come back to us and just like, don't, just ignore everybody.
Like, don't actually talk to people.
Just post it and then just, just post.
Come back, just drop the mic after.
Yeah.
Um, it's like, NVIDIA is doing some insanely cool stuff with these DPUs.
The blue field, two cards, I mean, $300 each.
They have a full arm VA core on them and they're like tied to a
Kinect X seven network adapter.
And just like it's in a really weird way where that arm chip has like full
control over that network adapter and create like pseudo interfaces that show
up on the host that are actually routed through that arm core out to the
public internet through the ethernet port on the back or technically it's an
SMV port.
It's going to get to a point where we have so many of these fast ships that
they will like just be everywhere.
Cause it seems like, you know what I mean?
Like they just keep building them faster and faster.
So people are going to want the new ones and it's just like, we're just
going to end up with this crazy surplus of chips.
Well, not only just a crazy surplus, but just so many stacked on layers of
like abstraction and complexity.
I mean, I really don't envy someone who has to learn all of like the basics of
PC hardware from scratch.
And then now you're throwing on like DPUs.
You're throwing on all of the extra complexity of like, oh, you know, how
do you handle a hundred gig, gigabit line rates, right?
Accelerator cards.
My little brother was learning how to build his own PC like two years ago and
he was trying to add more RAM and like get a new chip because he was running
video games.
And I remember like, you couldn't find like the crappiest card anywhere.
Like it was wild.
And I'm like, what are we going to do when they've got like, just at some point
there's going to be too many chips because they keep making new ones, you
know, what's going to be like, I wonder how that affects.
One of the things I was thinking of just as I speak about hardware data
centers is just that recycle factor of like companies handing out old hardware
to employees that are interested in the amount of things that I learned over
the years by having free access to random compute things that I didn't know
what to do with or not.
Like I've, I've taken my fair share of rackmount servers home, not knowing
that I could, if I could power or cool it, but it was still interesting to be
able to say that you didn't burn your out.
So in the accident, this was in a dorm room, right?
Like this is just taking out the whole door.
There's someone burning popcorn.
Don't worry about it.
Yeah.
Like this is, I feel like we, a lot of people miss out on that, right?
Cause so many of these like college students and early career people are only
working in cloud providers and they don't actually get access to know how
this stuff works at a lower level, be able to freely break something and have
constraints around like, I'm not going to, it's not going to accidentally charge
me $10,000 if I, you know, like for loop this Lambda function, right?
Like at some point, there's different trade-offs though, like a lot of times
you get extra credits and you can experiment, but it'd be, give people
access to both, you know what I mean?
But I mean, is, is there anything like the constraints of a, here's a box
that runs something like here's a, here's a DPU or even a GPU or an old desktop
or something like the amount of home labs I've built on old desktops is just
because I was like, I always had free access to them and it was fascinating
how much I could learn within that constraints of like, oh, what can this do?
So it's funny that you mentioned this.
This is kind of one of the things that I've always wanted to do.
Um, there's this project called cloud lab.
It's run by, I think like the university of Georgia or, or like some, maybe Clemson,
Clemson, yeah, Clemson, the university of Utah participate.
And they basically built this like software defined data center where you
can rent some of this hardware for like academic research use.
And like, I just think like that's the type of things that needs to be in the
world more, right?
You need to give like so many people who don't have access, who can't
afford this hardware, because a lot of it is so expensive.
Like you need to give them access.
Otherwise, like, how do you expect anyone to really get knowledgeable?
I was going to say, cause like, I mean, I do think that having a box is great,
but sometimes you may not have the room, you may not have one stable place to
live, like there's so many different, like, I know people who were living in
their car and using libraries, like to do development work.
So like, I think there's a place for everything.
I do think something that has no limits that you can break and fix is definitely
different than the cloud.
Yeah, no, absolutely.
Like I, like me personally, like I've moved so much, right?
I, I just finished up a move from North Carolina to Seattle and I, it is, I had
to pack up all of the servers in my house that were just sitting there and
shipped them across the country.
And a lot of them got broken along the way.
And it's just like, like, it is such a privilege to have so much space and so
much, like all of that power available in order to be able to power this equipment.
But there are a lot of people who don't, right?
And I definitely do think that, like, there will always be a market in making
these things accessible.
Yes.
Because like, there's just surpluses, right?
Like, because of the way that this hardware is depreciated and like the way that
it's written off, like to a company is basically useless, probably after like
maybe three or four years, right?
And just the advances they're going to get new.
Yeah, and it's, it's been so cool because you will see, like a lot of Chinese
companies that are solely like built around recycled or like secondhand, you
know, enterprise hardware, which is like, it's no longer in fashion mainstream and
like current data centers, but it's like still valuable, right?
That's been, that's been like the fascinating thing to watch about deep
seek, right?
They basically, you know, their principles are all correct.
They're using the right stack.
They're doing the right things.
I, I am so bullish on deep seek.
If anyone is getting to AGI, which, you know, I don't know if that's even a possibility.
I, I remain skeptical, but if anyone is going to do it, I definitely think it's
going to be the Chinese, right?
I think the fact that they were willing to put that clearly opens up.
Yeah, yeah.
That's ballsy.
That's ballsy, you know, saying I know that we're really, really good because
I'll let you have the code and see all.
That's how, that's how you disrupt markets anyway, right?
Like you make the thing that someone else finds valuable and just run it to zero.
Right.
Like that's what Kubernetes exists.
As Google said, we wanted everyone to rethink how to deploy applications and we
thought containers were better than VMs.
And so we wanted to ruin Amazon's business.
So Kubernetes exists.
I don't think it's that, but I also think it's the Kubernetes community because
that's probably the coolest community I've ever, well, sure.
Yeah.
Like, like the people.
I think that they had to do things differently because they had other open
source containerization things that exist.
Me, so, but even Google, like Google had, let me container that containerize that
for you and like these, there was, there, there was Mesa's, there was no mad where
the two that were, were around at the time, Kubernetes, um, yeah, and it's funny
too, if people contributed so much that that is the most diverse, like people
with pink hair, people that are brown trans with just all the things.
And I like, like you walk into a Kubernetes conference and I'm like, I
want to run this stuff, even if I don't need to use it because I want to
hang out with all these people.
Like, yeah, like, like people like Ian Coldwater are my, like, are my role.
Dude, like you actually, like, there's so many open source projects where
they're just like old, grouchy people and you're like, oh, I like your stuff.
But like, y'all, like Kubernetes, you're like, oh, it's painful, but the people are
rad.
It's, it's kind of like the untold secret that trans people tend to make up a
lot of the backbone of most distributed systems, which is why I think it's
funny when they're like, oh, we don't care about them.
And I'm like, yeah, we tell your infrastructure to like breaks.
Don't call anybody.
They're not coming to help you.
It's, it's like an unspoken fact that trans people and furries tend to make up
most of this nation's critical infrastructure.
Dude, it's, it's like when people are like, oh, we don't like immigration.
I'm like, you shouldn't eat the most like unseasoned, like horrible food for
the rest of your life.
You're not allowed to have a taco.
Okay.
Like I just, nothing unless it's like the, just.
I hope you eat the most bland food for the rest of your life and also no
infrastructure for you.
Like.
So, so I grew up half Japanese in Salt Lake City, Utah.
So.
Oh, girl.
So, um, Mormon too, Mormon too, should, should mention Mormon.
Right.
So I, I, I was an assumption made Salt Lake City that you got it.
You have to clarify assumptions.
Yeah.
Yeah.
You got it.
Yeah.
It was Mormon.
It is, it is, it is shocking.
Um, and then I moved to North Carolina right after that.
And I was like, wow, food can taste good.
It's like, it's like the, like, I don't know.
It's like the great, I don't know, like it gets like, it's like the universe's
way of being like, you don't deserve good food.
The equalizing factor.
You know, like I grew up a jove witness and I was like, all my friends who came
from very, very, very, very religious backgrounds are all like so much fun.
And I was like, you can tell we didn't get to do anything when we were little.
Yeah.
But it's, it's funny that you mentioned Kubernetes too, because if you really
start looking at the background on like who's pushing a lot of the needle forward
on a lot of this Kubernetes research too, isn't really the big cloud providers.
It isn't really like, like the giants in the U S of a lot of the stuff that I've
been looking at it, like overlay BD, um, is, it's, it's a fascinating project.
If you've not looked into overlay BD, they basically made like a block device,
which remotely streams in its content from a, a Docker registry.
Like that's, that's I think a 10 cent project.
Right.
Um, GCP has that, right?
GCP has a way to, to stream.
Yeah, yeah, yeah.
Like E-Star GZ.
Yeah.
It's I, I like there.
I think one of the main, main tanners behind container D is also like these
Japanese Akiharo Suda, right?
That is this handle, I think.
Like there are people across the ocean who are kind of leading the way on a lot
of this Kubernetes research and it's kind of fascinating.
If you look at it all, like just all the way, that's why like I'm just like,
when people say that like diversity work is like some sort of like you're doing
it to be nice and help people.
I'm like, um, have you ever heard of the bottom line and building things?
Like, cause like this is not charity work out here.
Like you are not doing anything for people.
Like you, to get the best people that are like really into whatever they're
building and they're just doing it, you have to go everywhere and allow everyone
there.
You literally get better products and more of a return on investment.
Some of the smartest people I've ever worked with were high school dropouts.
And that's, that's like a bizarre factor.
I'm a college dropout myself, right?
I didn't get to finish.
Some of the best people, college dropouts, theater degrees, absolutely no
real computer site.
Like, and it's, it makes me sad because I've ran into so many times where we're
like, you don't have real computer science to create or like, like, and I think
that we've got this elitist vocab, you know, that if you don't sound exactly
the same as other people that people think that you like, or if you didn't go
to a fancy school and I'm like, some of the best technologists I know have
invented half of these as vocab words and like, don't have that degree.
Like that's ridiculous.
Like, yeah, I think that's been one of the most interesting factors.
Like, because, because of like open source technology is it is so, so like, it
is so easy for people from underserved communities to like break into tech by
just contributing to open source projects.
That's how I got my break.
It also helps you to be able to play with technology because it's all
there in the open and there's usually good documentation or a doctor will
have a walkthrough.
Yeah, it's, it's like, it's like, I really admire tail scale.
I really admire oxide, how they all have their core basically open in public
for people to look at.
Right.
A lot of people don't have the ability to do internships.
Internships are especially like unpaid ones.
That's for rich kids.
Like, not everybody can do, not everybody can move to Seattle or New York
or Silicon Valley and how are you supposed to pay the extra high rent for
three months, even if you are making some like, you know what I mean?
Like, people don't realize how much of the traditional process of getting into
tech is gatekeeper by like your social economic class about where you're able
to get like your education, like there's so much of it.
And I think open source is the great equalizer.
And 70% of infrastructure comes from open source.
And that's the other thing I was saying with like the trickle down of free
hardware, right?
Like, is that goes, it ends up going far and wide just beyond the person that
gets it, like the amount of computers that I've, I've got from companies and
then also given away to other people that I'm like, Oh, you should get into this.
I play hockey with a, with someone and they work a service job and they seem
really kind of apps to technology.
And I'm like, Hey, I have a spare laptop.
Do you want it so that we can do some programming classes?
And they're like, I've never had a laptop before.
And I'm like, I always give my laptops away because I just definitely
the kid that didn't have one.
Yeah.
And it totally has a second order effective.
Like if you have, you have more time than ability at some point, you're like,
just give me access to something and I will bang my head against the desk until
I figure something out in, in for systems work, for programming, for all sorts of
things, we can't replace the desire to learn with more training, right?
Like the, the amount of curiosity is one thing you can't train.
Yeah.
And like the, the amount like, Oh, we're just going to train up a new group
of people to do this thing.
I'm like, no, you're not.
Like if they're not interested, they're not going to learn it.
You just can't.
But the rabbit hole and the research that you will go and you will learn when
you are just like, want to solve a problem.
Like that's why the best projects come from like somebody wanting to solve
an issue that thing.
Or prove someone wrong.
Oh, oh my God, my life is spite driven development.
I swear to God.
Yeah.
No.
So like this has kind of been one of the most frustrating things about watching
modern computer science education.
Like I, I am.
The interviewing.
Don't forget the interviewing.
The, the interviewing process is frustrating, but like watching people
who are so reliant on chat, GBT and like don't want to learn anything.
It's so frustrating because I'm really scared for future developers.
Like I think we're going to make people like dumber, but I think that AI could
be useful in certain ways, but I just, it's so weird to me that they're
shopping it as the black box of box of answers instead of this new way to learn.
Like I, it's so weird to me that that would be so like also just the way that we,
the fact that you can have AI do process on Apple phones and everything is
processed in chip.
That's fricking amazing.
You know what I mean?
Like there's like, there's so many cool ways that we could use this to help
people and we're just, let's make them dumber is a really weird flex.
Like I have, I have such a nuanced opinion about AI because on the one hand,
I think it's so cool.
Like this is the next frontier of natural language processing.
Like you can talk to a computer and like literally just plain text.
This can be your rubber ducky.
It can help you like when you're a neuro-spicy person and you're trying
to figure out how to interact with ready, but like, does this sound dumb?
How do I rewrite this?
Like, you know what I mean?
Like here's all my neuro-spicy ideas that I'm too excited to make into words.
Can you help me make it into work?
Like, you know, like a structured like paragraph from my bullet list of ideas.
Like, yeah, but like at the same time, like I spent so much time around my
friends who are still in college who are just graduating and like, like chat
GBT has completely changed how the kids don't want to write paragraphs.
Yeah, they don't, they don't actually try.
They just feed everything into chat GBT, use the output, like output, turn it in,
maybe change some words here and there.
So to like avoid that.
Yeah, I played all the M dashes, which is funny because I feel like we did.
Remember when they told people that you can't use an ID, you have to use Vim.
And then they were like, oh, well, like you can't copy and paste.
You got to make your projects for like, people have been doing this shit for forever.
Right.
But like, I think this is just like, it gives you such a finished product that's
so tailored to what you want before you could go copy some code, but you had to
make it, you had to change enough to do what you want it to.
You know, the one, the one radicalizing thing about learning about higher
institutions and how education is actually conducted there is it isn't like,
not only is the curriculum more advanced, the cheating is more advanced, right?
They get so creative on cheating.
There are a whole cheating rings form and like disbanded on a daily basis.
There are discord servers created.
So every now and then I want to give them a little award like that dude who did
the Trojan horse and the like, and he like bullied the maintainer into
making him a maintainer, like, should you do that?
No, was it creative?
You get an A plus for creativity.
And I want to teach this in a computer science cyber class, but that was mean.
Yeah, it's, you know, that is one of the things that I, it just, I, I think we
don't do a good job of making people curious.
And that's one of the things, but in a way that is curiosity and problem
solving, but they're using it for bad.
It's like when you have kids, okay?
My kids use their intelligence to do little evil things.
And I'm like, you could have done this the right way.
And it would have took less brain power.
I might have been talking crap about you earlier, but that was so rude.
Oh, would you work with people that are your friends and they know you for real?
It's so hard.
Also, my kids worship him.
I know I was like, damn, and he's not wrong.
So I don't even have a rebuttal to the heart.
What do you think is interesting in hardware in the next five years?
I was like, all these accelerators right now are like, everyone, I feel
like everyone's just like throwing more electrons at AI right now, but it's
not super interesting to me.
Yeah.
I think, I think one of the most fascinating things is like, people are
now kind of revisiting the basics when it comes to the data center technology.
Bye, Autumn.
Um, it, and like, there's so much talk now about disaggregation, right?
Because now you have GPUs, which are basically their own hole.
Like they consume the whole unit.
They are basically the majority of the power unit of a lot of these rack units.
There's a lot more talk about making them into their own standalone rack units
and connecting over CXL.
That's the standard that's coming out, I think, and it's going GA this year.
And people, you know, putting in more memory than what like these servers
are physically capable of, um, like dynamically allocating and swapping
around memory between servers without ever turning them off.
It's, there's, there's so much cool research there.
I had a call with both Memverge and Liquid.
Those are two startups that are, um, very hot and playing in the disaggregation space.
And it is insane what's going on there.
It is, I, so what they're able to do is they basically realize
okay, PCIe is a switched, um, switch interconnect, right?
It is, it is, if you squint, it's like a network protocol, right?
And so they basically made it so they break out the PCIe slot from inside
of the computer into a copper like interconnect that can go 15 meters, 10 to 15 meters.
And so you're able to have these individual thin, thin, um, you know, thin racks,
thin, thin rack units, and then you're able to have all of these copper
interconnects into, um, like a GPU box where you have like eight GPUs all in
like a four U enclosure.
And that's all it is, right?
There's, you know, power, there's a power distribution system that's more
equipped to handle all of that load for the, those GPUs, you know, or, or like even
you could do it like from like another rack unit over, right?
That's been one of the fascinating things.
When I, when I see like that and NVME over fabric and these things that are
just like, let's just put everything as a network protocol and, and connect it
over, over more generic cabling, right?
This is basically what we're doing.
And then I, when I envision it, like initially I think that like a rack, an
entire rack is basically going to look like what a desktop PC looks like today.
Like if I open my desktop PC, I got the CPU on this side.
I got the storage over here.
I got the GPU down here, whatever else.
And like, we're just doing that at the rack scale to say, Oh, it's fast enough
to connect these things together with the network or some fabric connectivity.
It's close enough.
So now we can just like, I always envision like pulling off the side of a
rack, like a big door and it's like a big PC where I'm looking at, okay, I'm
going to change the CPU cooler here or something like that.
But that, that seems like where we're going, where we want these, these
specific units that do one thing.
And we say, Oh, we need to swap the, you know, compute Bay two or something.
Right.
We got like, pull it out, hot swap a CPU out.
Okay.
We fix that, put it back in.
It's back into.
Yeah.
That's, that's where we're going.
I like, like, you know, that saying it's turtles all the way down, it's computers
all the way down, right?
But yeah.
And that's the thing I have a question about is at that point, every one of
those components has an arm chip attached to it or something that is its
operating system.
Mm hmm.
What is the operating system for the rack?
Right.
And I don't think Kubernetes is the right abstraction for that.
I, I think like there has been some thought there.
I think open stack was a very interesting player in that space.
I would probably say they were the closest to get to get right, but it was just
too complicated.
Um, I, I been looking at other projects, open Nebula.
They're, you know, it's like no one has really figured that out.
Right.
I, I think the enterprise is kind of standardizing around Kubernetes as is
right now, but it'll kind of be very interesting to see what comes out of this.
Um, because I don't think Kubernetes provides the lower level abstractions
that people want to manage that hardware, uh, especially if it's a, a per
nick operating system, right?
Yeah.
So, so that's actually, it's actually funny that you mentioned this.
Um, one of my research topics at work has been the doka platform.
Um, basically NVIDIA built a bunch of CRDs, uh, custom resource definitions on
top of the Kubernetes API to like declaratively manage the DPU networking
cards that we're talking about earlier.
And it's so weird.
It's like, it's like, yeah, you could do this, but should you?
Yeah.
Um, and the answer for me was kind of like, no, this, like, and, and like the
way that they want you to set it up and, and like the way that you want to maintain
it, it's like you're running three Kubernetes clusters at the same time there,
which is insane.
Yeah.
Right.
And there are like projects like Kamadi, which is what I use to manage, um, those
three Kubernetes clusters, which is so cool because it lets you like, it lets you
declaratively create Kubernetes clusters inside of Kubernetes clusters, which
is, which is, you know, uh, batshit and saying, but it's like, you know, this
doesn't feel like what we should be doing with Kubernetes because it wasn't
really designed for this is it like CRDs were really meant to be kind of, if you
need something more than just a deployment or replica set, something like that.
It still should be at the application layer.
Right.
It still should be.
If I think of the like OSI model, right?
That's still at the top of the, like CRDs are not down here in like the physical
sort of like TCP area.
This is like, no, there needs to be something else that's managing everything
below that.
And, and to some degree, that's just the operating system has traditionally done
that.
I don't know that general purpose operating systems are the right thing for that either.
Right.
That's where we just start getting into like, what is firmware?
It's like, well, it's like a little operating system that does one thing.
Basically.
Yeah.
I mean, and it's like, you could probably cobble together like all of the individual
parts and individual software to like make like a whole rack scale, you know,
computer software.
But it's just like, you know, I spent so much time trying to get even like
net box working correctly.
And it's just, it's a headache.
Like there, no one has really thought of it that way as like a rack scale computer.
I think the closest people would probably be Oxide, honestly.
Because, you know, if there was anyone that was going to figure it out,
I would probably assume it's them.
Just because they're, you know, they're building out all of the key building
blocks and the way that you would expect, right?
Well, they're doing, I think they're doing it in such a different way.
Right.
Like they're, they're just saying, like, you know what, we don't need a BIOS.
Like it just doesn't exist.
Like we don't, we're not building that kind of machine.
We know everything about this hardware and how to initialize it when we plug it in.
So we don't need that general purpose BIOS UEFI stuff.
And it's just more complicated than it's worth to have that interface in the system.
So we'll get rid of it.
Yeah.
And in the more of the more of those layers, we can remove the better in a lot
of these cases where it's just like, actually, what is, what is valuable in this stack?
Yeah.
Well, I mean, disaggregation is cool, right?
It is, it is so cool.
It is so, so wild.
But at the same time, like there are limitations due to the speed of light.
A lot of, a lot of times, right?
There is an unavoidable amount of latency involved in like converting optical signals
to electricity, like two copper signals, right?
And it's like, it's just, it's going to be really fascinating to see, like, to see
all of this operationalize and see, like maybe like 10, 15 years from now, like
were we correct about disaggregation, right?
Were we not correct?
And like the answer is kind of like, you have to try and find out, right?
Yeah.
But like from an SRE standpoint, disaggregation, it makes everything a black box.
Means you can't trust your storage devices.
It can't, you can't trust your network.
And one of the first things you learn about distributed systems is that the
network is not reliable, right?
Like there is a reason why TCP is basically the de facto norm.
And it is because packets are dropped pretty frequently, right?
And I mean, if like, it has been a long, long time since I haven't trusted
a front side bus between the CPU and a PCI, like that has always worked for me.
Or most, most of my life that has worked.
And I've never had to think about maybe I need to debug this layer.
But when you go to a distributed system or disaggregation, you're like,
everything is network.
And now I don't trust any of those buses.
Yes.
Yeah.
It is, it is going to be so, so wild to see like all of this productionized.
Like, like, what does the SRE do 10, 15 years from now?
Like, are we teaching people how to like attach an oscilloscope and like
see the actual signal coming over the wire, like trace it?
Like, it kind of seems like where we're headed because, you know, one of the
things that I, I really pride myself on is like, reverse engineering and
knowing every single stack and every single layer of like, involved in like
an OS call or a sys call or something like that.
But then what's the, like, through disaggregation in the mix now?
Like, you can't even trust like an M map, right?
And M maps can fail.
And then what does that do?
Like software, right?
Like, it's, it's funny because like all of our hardware is advancing at such
a fast pace, but it doesn't feel like the software is keeping up.
Hmm.
And that, that has kind of been the, the, like killer, killer, sorry.
What's like the term here?
It's like nailing the coffin for a lot of things.
Like you think of itanium, you think of Zion, fuck me.
Zion fi, I think, like Intel's experiments to try and make really cool hardware.
Right.
The software just never capped up for either to, for either itanium or Zion
fi, which led to both of them failing.
Right.
And so, you know, that's, that's kind of going to be really interesting to see
with Memvirgin liquid, because they're both betting on Kubernetes as being kind
of like the management plan for these things.
Interesting.
So like in some ways, like Kubernetes is, is not the right fit for any of
these, but it is what the enterprise is kind of standardizing on, because the
alternative is like proprietary vendor locked software.
Yeah.
Um, you know, like ESXi, but everyone was running on ESXi for a while until
Broadcom bought VMware and then was like, Hey, we're going to screw up all of
your licensing, by the way.
Uh, you mentioned provisioning there for a minute and I'm kind of curious, it's
always been like this passion of mine throughout my entire career.
I was just like, I, I love the, I love pixie booting things.
I love provisioning automated.
Like I can install a hard, like an operating system.
What's the, do you know what state of art, like state of the art, like for
provisioning systems these days?
Like when I did it, like I literally was like cobbler back in the day and a
satellite foreman, um, doing that sort of like red hat stack for quite a while.
And then I was like switching over to like when I did a core OS, like their whole
like install script was like a, it was just a bash script that just like
did, did an image to this kind of like, that's fascinating.
That's all you need.
I was like, yeah, that's it.
And, and I've kind of, I've, I've moved away from a lot of stuff.
I'm kind of curious what is, oh, net box.
That's what you mentioned.
Well, so net box is just like the network source of truth.
It is.
Okay.
That's just the, not IMDB, the CMDB, right?
So it's the database of like what's, what's there in the network.
So Nix is really interesting.
Nix is really interesting, but so far I haven't found a single person who's
gotten Nix to work as like that sort of like provisioning, you know,
Nix as the OS or Nix.
Yeah.
Nix OS, right?
I haven't seen a single person who has managed to make Nix work.
I think the state of the art there is still like puppet chef, Ansible.
Um, but I genuinely don't think that we've evolved beyond puppet or chef.
I mean, if you go to like the big hyperscalers, yeah, they, they totally
have a replacement for puppet and chef and everything like that.
And they definitely have like integrated solutions for all of these things.
But, um, yeah, you know, I think the people that I know who have made
advancements here are like packet before they got bought out by Equinex and
turned it to Equinex metal.
And now it gets shut down.
Yeah.
You know, like there hasn't been a lot of like, and this is something that
OpenStack was actually trying to do as well.
Right.
Ironic.
Yeah.
Yeah.
It's, it's like, it was, it was just a couple of years too ahead of its time.
And maybe now if someone tried to do OpenStack again, we'd be like, okay, you
know, and that's going to be really interesting to see also from like,
um, what was it?
Cidero too.
Right.
Like how is that Talos Linux stuff going to work?
Yeah.
And, and it's been interesting on like both sides of how things get provisioned
because I look at people that went to the cloud, just ignored the whole
how to provision things and it's just like boot the generic Ubuntu and, and
can, and cloud init the whole thing.
Right.
And like cloud init is the config management system, which is just such
a, a slow, difficult path to like actually make something work.
And then coming from like data centers where they're like, oh, we're running
puppet agent or chef or something.
And that's, uh, it is, is a, is a lot more powerful than cloud in its, but
also can, can quickly become more complex and harder to maintain.
Yes.
Um, and I, I keep telling friends at work, like people at work.
I'm like, I'm so spoiled by Talos now because like, I don't, I don't worry
about either of those things where it's like, I call an API and my OS gets
installed and that's weird.
I had to do a, I told you before we started recording, I'm doing performance
testing on a bunch of different platforms and operating systems.
And like every, I have a KVM and I'm mounting like two gig of Ubuntu
ISOs to it, to like install Ubuntu.
I'm like, this is so painful.
Like I can't believe it's no wonder that people wanted to get away from this.
Uh, because my, you know, my Talos Linux is like 80 megs in it.
It just like one API was okay.
It's installed, I'm ready to go.
And I'm like, oh, I'm still, it's still booting.
Like Ubuntu is still kind of going through that.
And I can't believe that nothing has changed that ecosystem in drastic ways.
I'm, I'm using things like, um, Silverblue, Bluefin for my desktop, which is, you
know, imaged base sort of installs, it's still large because it's still a
general purpose sort of Fedora installation, but the maintenance side of it has improved.
And so there's all these like weird areas where we're, we're kind of making
changes in some places and not a lot of people are rethinking the possibilities.
Yeah.
No, so many, so many of these like second order cloud providers, they're
still using cloud in it, right?
It is all, it is basically Libvert with cloud in it and they are using like the CD
data store that is it that, you know, and like you look at the cloud providers,
they're doing things like MMDS, right?
With the EC2 instances where you can call, I think 196.254.196.254 and it's like
this API where you get all sorts of metadata about your, um, like things
like that are probably the future and where we need to move towards.
And, you know, I just think like no one has really tackled that space.
And it's understandable why, but it's like, at the same time, like, you know,
you have a feeling like someone could do this better with like Nix or like
Talos or something like that.
It's just, you know, I just, I just don't have the time to do it.
Well, thank you so much for taking the time for this podcast.
This has been a lot of fun.
Uh, uh, if, if, if no one has seen it, I'm going to put it in show notes.
Uh, le.fm at your website is something that everyone needs to go to.
At least once experience.
I'm not going to throw out any spoilers yet.
Just, just click the link.
Uh, I hope that's not mobile friendly because it's not, it's not, right.
This is great.
Yeah.
Not optimized for mobile whatsoever.
If there's not horizontal scrolling, I don't want it.
So I know you're, uh, I don't know if people can reach out to you
somewhere, or at least, uh, where you could see where people are talking about,
uh, you know, I lurk, I lurk, I will, I will, I will see, I will see the
discussions, maybe, maybe I'll pop in on whatever new blue sky account I have at
the time.
Sounds great.
And I look forward to learning more about, uh, DPUs and, and going down, uh,
rabbit holes of, of YouTube videos of, I didn't know that was possible from
you in the future.
Have a good one.
Yeah.
Thanks.
And everyone, thank you for listening and we will talk to you again soon.
Thank you for listening to this episode of fork around and find out.
If you like this show, please consider sharing it with a friend, a coworker, a
family member, or even an enemy.
However we get the word out about this show helps it to become sustainable for
the longterm.
If you want to sponsor this show, please go to FAFO.fm slash sponsor and reach
out to us there about what you're interested in sponsoring and how we can
help.
We hope your system stay available and your pagers stay quiet.
We'll see you again next time.