Laying the Foundation with Joe Beda

I think I may have lost it.
Not only have I, in the past three weeks,
rebuilt my entire network at my house.
I bought all new gear.
We're going 2.5 gig, 10 gig for the backbone, all this.
We went all in on new networking gear.
I also reinstalled both my laptop and my desktop
with NixOS.
Not because I'm a fan of it,
but because I wanted to try it
and I maybe like some pain.
And also I've shifted my entire home lab
over to Kubernetes,
a technology that wouldn't exist
with the guests on today's show.
If you don't know Joe Beda, look him up.
We're gonna go through this history,
a little bit brief of his history in technology
in this episode.
And it's just a fantastic look back
at the last 20-ish years of what people have been doing
that isn't necessarily what the state we're in today.
And I love it as a reminder that technology in general
is a lot bigger than a single technology you work on.
And being able to shift your focus and try different things
actually works out really well
because you get a lot of difference in experiences
and just understanding how the pieces might fit together
in different ways.
And don't forget to subscribe to the newsletter.
I have a list of outages that I'm putting
in each one of the newsletters.
And I'm gonna tell you that this was a really hard one
to write because there were so many people
that were finding out this month.
I couldn't put all of them in there.
I'm just putting some highlights,
ones that I like, postmortems that I read
and learned something from.
So if you like reading postmortems
and wanna gather some of that,
go ahead and sign up for the newsletter
at sub.fafo.fm, S-U-B.F-A-F-O-F-M.
It's in the links in the show notes, just click on it.
Enjoy the episode.
Welcome to Fork Around and Find Out,
the podcast about building, running,
and maintaining software and systems.
Hello and welcome to Fork Around and Find Out,
the podcast with more pods than your Kubernetes cluster.
I'm your host, Justin Garrison.
Oh, wow.
And we're here with the amazing Justin Garrison.
Coming in hot.
You know, I had to bring out another pun.
You know, I forgot to do them for a little while
and this is the perfect show for it
because on the show today as a guest,
we have Jo Beta.
Jo, welcome to the show.
Thank you so much for having me, guys.
I feel like your name sounds so cool.
Like it's like one of like,
it could be in like a Marvel superhero-like name.
Well, that does seem like a-
What I love about my last name
is that it's actually hexadecimal.
Oh, that is sweet.
And so if you turn it into a color
and you do 00Beta or Beta00,
it's actually this nice chartreuse
or sort of like a lime green.
And when I did Spiffy,
I actually used that as the colors for the Spiffy logo.
So the Easter egg is that I used my last name
as sort of the color codes in the Spiffy logo.
You were meant for this life.
I tried to do the same thing with SideArrow
where it's like I just converted as much as possible
like leet-speak hex
and it turns into like this bright green.
Then I was like, that's kind of like console green.
This is pretty good, yeah.
Sweet.
Joe, the first time I remember interacting with you,
I was working at Disney Animation.
I don't remember how we jumped on a call,
but you offered to jump on a call with me
while I was trying to understand Kubernetes in general
and how things would be laid out.
And I had this idea about hierarchies and things,
and you were so nice to not only jump a call
and just explain labels to me, just in general.
I still remember this because at the time
I was still trying to secretly bring Kubernetes
into the space.
And so I rented one of the conference rooms
that were down the hall that no one would see me in
that I could go sit in and do my,
I gotta do some Kubernetes research over here.
And we sat there and you gave me this great overview
about why Kubernetes went with labels,
why you were doing it this way.
And ever since then, I've always known you
as being one of the most approachable, nicest people
in the cloud native community in general
and just like always open to talk to someone
and help them better themselves in any way they wanted to.
And so, first of all, I just appreciate that
because I think that does set like
what a lot of people know of as today,
as like cloud native is being welcoming
is a lot thanks to folks like you and Sarah Novotny
and people that were doing this super early
that were like, we just need to make sure
that everyone has a place
and it doesn't matter their skillset
or their knowledge level or any of their misconceptions
about what this is.
We want them to be welcome to be able to come
and just try to learn and do things.
I mean, we're all newbies at one point, right?
Even the creator of Kubernetes.
Oh yeah.
I mean, this is one of the,
I love that like you have to be crappy at something
before you can get good at something, right?
And that's just, you know, there's no way to get good
without going through the crappy stage
and some of us stay at the crappy stage
depending on the activity, right?
Like I got into like, you know, modular synthesizers
cause they're really cool and I suck, I'm so bad
but it doesn't matter as long as you're having fun,
you know.
That's so cool though.
Like those are my favorite people
that are just absolutely brilliant
and you've meet them at a conference somewhere
and you're like, not only are you talking to me
but you're willing to teach me.
Like, I just feel like that's like the best humans
and you get like the best karma for that.
And speaking of being crappy,
Joe, can you tell us about your career,
about the things you started doing?
Because I think that a lot of people realize
that they can dislike you for more than just the YAML.
Oh yeah, no, you can definitely blame me
for some of the YAML.
I like how he complimented you first
and then he was like, and then the YAML's your fault.
Well, I know the IE6 as well
and so that's what I wanted to kind of go into.
Yeah, we'll talk about IE.
But like I did give this talk, it was at a software circus
and it was during COVID and they did it all online
and it was a Halloween themed talk
and it was sorry for the YAML
where I talked about sort of like the ins and outs
of why YAML and the problems there.
But I gave it in a blow up Pickle Rick costume
that I just gave away as part of this latest house move.
Oh my God, you're my favorite.
And then I'm like, how am I gonna mic myself up
in the Pickle Rick costume?
So like Justin, I just got more gear
and so I have like a wireless lavalier mic
that I got just so that I can mic myself up
inside the inflatable Pickle Rick costume.
But that's, you know, that was the YAML story.
Yeah, totally worth it.
You know, it's committing to the bit.
Okay, so my career.
So I went to, so first of all,
I'm a second generation programmer.
My dad was an entrepreneur.
He started a company when I was growing up
that back when the postal service
was adding zip codes to mailing lists,
if you were sending out a lot of mail
and you were, you know, sending out catalogs or whatever
and you had zip codes, you'd get a discount on your postage.
And so he wrote a company
that would essentially attach zip codes to,
you know, address lists.
And it was, ended up selling that company Pitney Bowes
that does a lot of mailing stuff.
But it was one of those sort of like very clear,
it's like, you pay us so much,
you save that much on postage,
you'll earn it out in a year type of thing.
And so he, I remember growing up,
he would bring home punch cards
we would use as a shopping lists paper
and he would disappear at night to program an IBM 360s
cause that's when he could get computer time and all that.
So a lot of my like early life was programming
and just introduction to computers.
And then went to college at Harvey Mudd.
We were just talking about that
before we jumped on the call here down in LA.
Met my wife there, did a computer science degree there
and did a couple of internships at Microsoft
while I was there.
And then joined Microsoft out of college
and I had a choice between the NT Kernel team
and Internet Explorer.
Now this was in 97 and I was like,
I don't know if this internet thing is going anywhere,
but like there's a lot more people doing stuff,
moving faster, shipping faster.
And so like at that point I chose IE
because I'm like, when there's a lot of action,
when there's a lot of stuff going on,
that's gonna be good for your career.
And I think that, you know, I think when you see a downturn
it can be a bit of a double-edged sword,
but I think when things are moving,
when people are active, when you're shipping
and like there's that excitement,
that's always something that I've gravitated to a little bit
in terms of what I pick.
So I joined Internet Explorer right around IE 4.
So that through IE 5, IE 5.5 and then IE 6
and then stayed on in Microsoft.
The eternal browser, IE 6 lasted forever.
Well, this is my thing is that like,
and I will defend this to the death.
And like, I think a lot of the work that we did
between IE 4 and IE 6 established
essentially the modern web.
It was done sort of contemporaneously with the W3C stuff.
But the idea of the modern DOM
where it's like you have this object model,
you tweak something, it shows up on screen.
That was really pioneered by the IE team
back in the IE 4 days.
The competition at that point was Netscape Navigator
and their sort of dynamic solution
was to have essentially these layers is what they called it
where you could have multiple web pages
that kind of layered and composited on top of each other,
not nearly as interactive.
Now, I'm not gonna claim credit for that
because I was just a newbie right out of college,
had no idea what was going on.
But a lot of the work that we did in that time
was essentially modernizing the rendering engine
so that it could deal with more and more dynacism
and really led to the web that we have today.
I think the sin was Microsoft stopping development
and essentially letting IE 6 rot for God knows how long
and declare victory when in reality there was more to do.
And so, yeah, so I was at Microsoft,
I did a client framework.
So this Windows Presentation Foundation
was codenamed to Avalon.
That's a lot of the IE team's transition to do that,
which was really about sort of bringing
some of the early,
some of the sort of dynacism that you get on the web
to Windows development and some way success.
Did that first land in, that was pre Vista, right?
Yeah, so this was happening during Vista
and then landed with Vista.
And I ended up leaving Microsoft for Google
right before Vista shipped.
But it was one of those like, you know,
you saw the writing on the wall,
it was a project that wasn't converging.
And you're like, you know, career wise,
you know, at Microsoft at that time,
the only, you know, you would ship a version of Windows
and then there would be a game of musical chairs
as different people left.
And then there'd be opportunity for advancement
to take on more responsibility.
And so if things aren't shipping,
then that game of musical chairs doesn't run
and there's a lot less, you know,
room for you to actually step up
and take on more responsibility.
So that was one of the things that led to me
leaving Microsoft at that time.
The code name was Longhorn, right?
For Vista?
So it was like, that was, it took a while.
I was running the Longhorn betas like back in the day.
And like before Vista finally came out,
it was just like, okay, this is a little rough
because it was a big change from XP.
Yeah, and Longhorn was really sort of three big efforts
all built on, you know, CLR, Managed Code, C-Sharp stuff.
There was Windows Presentation Foundation.
There was Windows File System,
which was essentially re-envisioning the file system
as a database.
Some really cool ideas there,
but like, I think like a lot of stuff,
it was probably ahead of its time.
And then there was a, what was,
there was a whole sort of networking subsystem
and communication subsystem that I even forget about.
But I think out of those things,
I think Windows Presentation Foundation
probably saw the most sort of uptake through time.
But even then it's been seen twists and turns
and so many different interesting aspects out of that
and so many lessons to be learned.
What was it like working at Microsoft
in those like early days and working on,
you know, Internet Explorer?
It was, it was an exciting place to be because,
you know, there's been times in my career
when you felt like everything was happening all at once.
And I think the early internet working on the browser,
it was that, it was Netscape, eventually Firefox,
a lot of that stuff,
those sort of early browser wars was incredibly exciting.
In some ways that reminds me of like early Kubernetes days
when we were, you know, and cloud when you're like,
I remember it's like, oh, what's the latest announcement?
How does this change our plans?
Oh my God, they're ahead of us.
It's like, there's this sort of like this,
this sort of energy in the air
when some of those things are happening.
I think you're definitely seeing that with AI now,
whether you, you know, like it or not,
it's definitely a moment with a ton of excitement,
a ton of action.
Nobody quite knows where things are going
and the next announcement from whomever
can change the landscape pretty dramatically.
And so I think there were definitely parallels
between early browser wars, cloud native cloud stuff with,
you know, when I was working on GCE and then Kubernetes
and then I think what we're seeing happening today with AI.
It's so fascinating you see those parallels happen.
What do you, when you look at AI and the parallels,
I know everyone's probably asking you this question,
but like, what do you think is gonna stick?
Like, do you think it's really gonna revolutionize
technology the same way like the cloud did
and like internet or-
Oh, to move your mic a little closer.
Oh, sorry.
Yeah.
I think we're still figuring,
there's definitely a hype cycle around AI.
And I think there's certain domains
where it's been overplayed
and there's other domains where it's clearly effective.
I think that when it comes to coding
and sort of the stuff that a lot of us do day to day,
it's almost perfectly well-suited
for a lot of the AI stuff that we see.
It's like, there's a ton of examples
out there with GitHub, right?
I mean, it's like, I name a domain
where there's as much prior art,
except for maybe writing, right?
Whereas we're seeing that.
But I think part of it also is that the tools to validate
whether something is, you know, holds water or not
are also automatable and work in sort of computer time
versus human time, right?
And I think this is one of those things
that when certain things happen in human time, right?
Like think about, you know,
racking and stacking servers pre-cloud, right?
You get a certain level of output.
And then you see these phase shifts where with cloud,
all of a sudden things that used to take, you know,
weeks or months now can happen in minutes
or hours on the outside
that fundamentally changes how you actually view technology.
And I think we see similar stuff with AI
where there are certain tasks
that were happening at human time
that can now happen at computer time.
And it changes how you view that.
And even if it's not exactly the same thing,
the fact that you can do, you know,
10 of things, a hundred of things all at once,
and it can happen, you know, within 10 minutes
or, you know, an hour on the outside
versus taking weeks or days,
even if the quality is not there,
it fundamentally changes it.
And I think the coding aspect,
the computer side of this stuff,
because compilers and, you know,
and unit tests and all that stuff is out there,
that means that you can iterate at computer time
instead of iterating at human time.
Whereas if you think about like, hey, like medicine or law
or so many other domains where people want to apply AI,
there's some utility there, I'm sure,
but you can't turn that crank
and actually get results on computer time
because like at the end of the day,
is this legal brief good or not?
I don't know, you really kind of need at some point
a reality check there that is harder to do
with just a purely computerized system.
But programming is actually, I think,
the sweet spot right now,
and I don't think we're going to see it go away.
But I think there's, you know, I've been, you know,
I like to try and come up with sort of pithy ways
to think about these things.
And nothing I've seen has dissuaded me of the fact
that AI can enhance expertise,
but it cannot replace expertise.
And so even, you know, in the domain where we're seeing
so much AI have impact on programmers,
you have to understand what the technology can do.
You have to understand what's possible.
You have to have taste in terms of what good looks like.
If you're applying it to like replace SAS,
like, oh, I want to like write AI that does a tax thing.
Like you need to understand the tax code well enough
to be able to guide the AI.
I'm not, you know, you can't trust it to do medicine
or tax or lawyering all on its own.
But if you know what good looks like,
if you know when it's going off the rails,
it can be a huge accelerator.
And I think that that is something
that we're seeing again and again.
But we're all over the place on topics.
I don't know where you want to take this,
Justin, I'm just ranting at you.
That was a really good,
a really level-headed assessment of AI, right?
Like I think people were like, oh, it's magic.
And then like now I feel like we're finally getting
to the point of people seeing where it's good
and where it's bad and what you can do with it.
And I'm like, yes, like heads are,
like level-headedness is prevailing.
How do you think we teach people the taste though?
How do we teach young engineers taste?
How do you learn that expertise in general?
Where do you get the smell test?
You know what I mean?
And how do you like still get those like,
the iteration to like grow, you know, and to get better?
I think that's a really good question.
I think one of the things that I think computer people
excel at is being systems thinkers,
looking at just not something,
but looking at how it relates to other things, right?
Whether it's like, you know, there's this whole idea
of sort of systems thinking.
There's the thinking in systems book, right?
Or, you know.
I just read that.
Yeah, which is fascinating
because it has nothing to do with computers.
So what's the author's name?
Danielle, I want to say.
Don't quiz me on it.
Yeah.
Was it a good book?
Yes.
Yeah, I'm trying to think.
Yeah.
Donnell, Donnella?
Donnell Meadows, yeah.
Donnell Meadows.
So what's interesting is that it's,
and this was one of the courses I took in college
was called systems engineering,
where you're looking at whether it's water systems
or electrical systems,
but also when you look at supply chain systems,
when you look at flow of data through a bunch of cues
in a distributed system,
all of these things have a lot of sort of commonality
between them.
And I think as, you know, technical people
thinking about computers,
you start to get an intuitive understanding
of how these things connect
and how something over here can impact something over here
and that nothing really stands alone.
And I think, you know, through my career,
I've seen that apply to people in organizations too, right?
So when you think about the organization of like,
how does a company work?
That's a system, right?
And how does an industry work?
That's a system also.
And there's sort of human systems,
there's commercial systems,
there's technical systems.
But when you look at sort of the programming
as a discipline, as sort of a labor market,
that's a system also.
And so the question that, you know,
you're asking of like, how do you build that taste?
How do you build that experience?
Well, like, you know, a lot of places,
there's an apprenticeship, right?
When I joined, I did a lot of scut work
on the IE team, right?
You know, and some of it was like real engineering
or like redesigning the core data structures
of the rendering engine.
A lot of it was just, you know, BS busy work, right?
And, you know, sweeping the floors,
but, you know, with respect to the code base.
And I think that's something that a lot of AI
can take over for, which is good.
But also it means that,
how do you actually get started in the industry?
How do people actually get familiar with things
to sort of build that taste,
to build that idea of what good looks like?
And I think that that's something
that we're gonna have to face as an industry
is that the sort of the opportunities for apprenticeship
are less and less as people just focus on output
instead of actually thinking about
how do I actually grow the market?
I think, you know, I mentioned
that I started off at Microsoft.
I did a couple of internships there while I was in college.
Those things were incredibly, you know, formative.
And I think Microsoft at the time
saw it as their duty to actually, you know,
build the next generation of programmers
both for Microsoft and for others
and did stuff like the internship program.
And I worry that those things are gonna disappear
and that we're not gonna see as much of that going forward
because of AI and LLM.
I think they just wanna get senior engineers
and I'm like, but how do you grow that?
Like that's-
At some point there won't be any senior engineers
on the market, right?
Not just that, but like, I think that,
I don't think that LeetCode
was ever really a good interview process,
but it's going to even more.
Like, I think that like from what I've heard
from a lot of really smart people like yourself
is that like creativity, imagination,
the systems thinking, none of that is LeetCode.
You know what I mean?
None of that shows you the curiosity
or like, I think the almost kind of relentlessness
of what you're gonna need to like stay relevant.
And like, so how do we even look for that in people?
Yeah, I found that I, you know,
the way I interview people has changed over time.
I think, you know, having some level of like,
can you take a problem?
Can you break it down?
Can you develop an algorithm that maybe is a little novel
and can you translate that to code?
I think being able to show that in an interview
is reasonable, but the LeetCode stuff
where it's like, people are just like-
Memorizing.
You know, memorizing all these different things.
It's just like, I don't think that's not the job, right?
So I think, you know, through Heptio,
the startup that I did and VMware
and sort of, you know,
as I help people out with interviewing now,
I focus more on like, can you describe a problem well?
Can you actually write a document
that actually like, you know, represents your thinking?
Can you work with others?
So I think communication, technical reasoning
and sort of like intuition,
those are the types of things that I look for now
when I interview more than just sort of like,
hey, LeetCode type of problems.
I was never a huge fan of those and even less so over time.
That like literally makes my heart happy though,
because I think a lot of people that,
especially like women and people that don't come from
like very fancy backgrounds or even more like kind of,
what is the word I'm looking for?
I just like the statistics of women
retrying technical like interviews,
like a lot of them will just not finish, especially girls.
And I think that like a lot of this
is just going to make it worse to keep diversity in tech.
And I think that the way that you're actually
describing that, that's actually interviewing for the job,
you know, especially like if you're not neurotypical,
like those interviews are rough, you know?
So I think that that makes me like happy
that people that have that kind of power,
but also that our voices that we listen to
are saying these things out loud.
Yeah, totally agree.
I think that, you know, there are,
we have this tendency to try and over-focus
on the things we can measure
and the LeetCode type of interview stuff
is something that's pretty measurable.
But I think, you know, anytime you're building a product,
anytime you're hiring, there's like, you know,
12 different axes that you're evaluating stuff on,
but you tend to over-focus on the things you can measure
instead of like, hey, how do you measure, you know,
technical sort of, you know, reasoning,
systems thinking, communication, teamwork,
like those things are much squishier.
And so I think, you know, there's definitely
an over-focus on other things,
but we see it in technical things also.
When I was working on GCE, for example,
you know, we had certain metrics that we would measure,
like how fast could we boot up a VM, right?
And like, I know at that point, Amazon was five minutes,
GCE was like a minute.
And I'm like, there's other things we should work on,
but then we had other people that are like,
well, can we get that down to 15 seconds?
I'm like, that's not a bad thing,
but also what about this other stuff, early Kubernetes?
It's like we had a scaling limit of 100 nodes
and I'm like, there's nothing, you know,
that's just work to get to a thousand or whatever,
but other people like, you know,
we're knocking on Kubernetes because of that.
I'm like, we have other fish to fry,
but like oftentimes if there's a number,
people over-focus on that.
And I think you see the same thing with hiring
and things like leak code.
I generally describe it as like being data-blinded, right?
When you're data-driven, you want the numbers
and then all of a sudden the number's the only thing
and you're like, actually it doesn't matter anymore.
Not just that, but this industry very much incentivizes
the data-driven for like promotion and for like-
It's a badge of honor for companies that are like,
we want to be like Google, so we're data-driven.
And I'm like, but data's not the whole story, right?
Like data's gonna give you pieces.
It's the causation versus like actually like being,
you're proving something, you know?
Like, are you actually proving something?
Or like, cause think about it.
If you're, if just like you said,
at five minutes versus a minute,
is that not good enough to now go to the next thing?
You know what I mean?
Like, what's good?
Like-
So Vista's coming out.
It's gonna be a failure.
Let's go back to this timeline.
And now you're going over to Google.
What, why, why'd you switch?
What would you switch doing?
So a bunch of the folks that I worked with early on in IE
that I enjoyed working with had went over and joined Google
and we're starting up the office in the Seattle area,
starting in Kirkland, which is on the East side.
And this is actually one of the things in my career
is that like, when I'm evaluating an opportunity,
there's all sorts of things that you can be like,
what's important to me as I'm doing this?
And I think for me, I think, you know,
I've tried to stay consistent over time of like,
who I'm working with is number one.
Who I'm working for is actually important.
Like, I think, you know, in your career,
you can choose your boss.
And I think that applies both to your immediate boss,
but like the organization that you're working for,
and that has a huge impact on your success.
What I'm working on then comes in, right?
And then I think like things like comp
are further down on the list.
I kind of let that, you know,
once you're at a certain level,
I let that take care of itself.
Now, that being said, I'm not, you know,
I'm not super human, right?
At some point when people are, you know,
waving big checks in front of you,
it does shift your thinking, right?
So none of these things are pure,
but I think who you're working with day to day,
I think has such an impact on, you know,
whether you like going to work.
And so a lot of the folks that I had enjoyed working with
early on in IE, and then at some point,
like things shatter and people start going
their own, their different ways.
A lot of those folks were starting at Google.
And so I joined Google and at that time,
Google viewed engineers as fungible.
So that you would come in and be like,
oh, smart people could work on anything.
And so I had no idea what I was going to be working on
when I joined Google.
And-
And this would have been 2006?
2004.
So here's my, I have this here.
Your timeline?
So when 10 years, so like, I was at Microsoft
for like seven or eight years.
And then my tenure thing at Google was a little thing.
I still have that sitting around.
But so 2004, right after the IPO,
in fact, I was interviewing pre-IPO, joint post-IPO,
which was definitely a very exciting time to be at Google.
But also at that time, like it was before
the sort of rivalry between Google and Microsoft
was heating up.
But like, as that happened,
as we opened the Kirkland office, we, you know,
Google started hiring a ton of people from Microsoft
and things got pretty nasty.
You know, Steve Ballmer throwing chairs,
that type of thing.
But I think, you know, people who aren't in the Seattle area
don't understand sort of how, back at that time,
how sort of like dominant and stable Microsoft
was in this area.
I think, you know, I have a friend who's been at Microsoft
for about 30 years, right?
He's, you know, a couple of years older than me,
joined right out of colleges and been there the whole time.
That level of retention is almost unheard of
in our industry.
But I think Microsoft has that sort of that level
of old school Microsoft people who've been there forever,
who live and bleed Microsoft.
It's changing now, but at that time,
having somebody like Google come in
and start hiring people really,
really ruffled a bunch of feathers
because it was shaking things up in a way
that made a lot of people very uncomfortable.
So yeah, so I joined Google.
I ended up working on Google Talk,
the chat system, but along the way we built out,
not me personally, but the people on the team
built out the peer-to-peer stack
for doing voice video
and then eventually things like file transfer.
That got open sourced and released
as this thing called LibJingle,
which ends up being the sort of underpinnings
for WebRTC in the browser.
So it's one of these things where we did this stuff
for Google Talk and the code lived on in new ways.
It's same protocols, different code base,
underpins things like tail scale
for doing sort of peer-to-peer connectivity,
that type of thing.
So that was fun to work on.
I led a team that hooked us, Google,
up to the telephone system.
And so learned a lot about SIP
and did a signaling protocol for XMPP,
all sorts of fun stuff.
That wasn't Google Voice, was it?
It was Google Voice, yeah.
So it was like Google Talk and Google Voice.
So the way that it worked is we did Google Talk.
At PM at that point, went out
and Spear led the Grand Central acquisition.
And I think sometimes you have people
who just do an acquisition
because they want to do an acquisition
because they want to feel like they're important.
And so there was a little bit of that going on,
maybe with Grand Central.
That turned into Google Voice.
These things kind of got blended together.
But, and it was one of those things where the Google,
the Grand Central folks came in and said,
you guys don't know what you're doing.
But honestly, when I look back at the stuff
that we did early on with Google Talk
and hook into the telephone system,
it has a lot of echoes of like Twilio.
And so like, I think if we had kept that stuff
and moved it forward and offered it as part of cloud,
it would have been a really great competitor to Twilio,
but it kind of got lost in the shuffle
with the Grand Central Google Voice stuff.
But then all that stuff turned into Hangouts
and like, you know, a lot of the technology
that we're using right now to do this podcast
is built on some of that stuff.
And then I transitioned, I'm like, I got to pay the bills.
And so I did some advertisement stuff,
this thing called, it was essentially ads optimization.
So when people are in the ads console,
how do we actually give them suggestions
so that they can actually advertise
and, you know, really do win-win for them.
And Google at the time was like,
we help you find ads that are going to pay off
and that helps everybody and, you know,
and all that type of thing.
But it was early sort of machine learning,
pre sort of LLM sort of AI stuff
that was interesting to be involved and see.
I mean, Google has always been a machine learning AI company
at heart, even before sort of the latest sort of,
you know, explosion.
But it was interesting to sort of see that early on.
And then we had a kid and came back
and I could not make myself give a crap about ads anymore.
And so there was a director in Seattle,
the Seattle Fremont office of Google
who was focused on system stuff.
And so we were like, hey, let's,
at the time there were these underpowered computers
called netbooks.
And we're like, let's build an operating system
for netbooks and that,
and there were some folks down in Mountain View
that were having the same idea
that eventually became Chrome OS and Chromebooks.
But because of sort of politics at Google,
there wasn't a lot of room for leadership from Seattle
when it was the Chrome folks.
And we wanted to build on top of Android,
but at that point, like Android and Chrome
hated each other, right?
So Chrome OS for, now it's all sort of like,
they looked at it and eventually sanity prevails
and you're like, really coming full circle.
Having like two Linux sort of like operating systems, right?
But like, you know, I'm like,
can we reuse the compositor for Android
and use that on the Chromebooks and stuff like that,
ARM, all that stuff.
But like, you know, so we kind of got squeezed
out of the Chrome OS stuff early on.
And so plan B was doing infrastructure cloud.
And in fact, we called it plan B,
the original codename for Google Cloud was plan B.
Now at that point, Google had-
I feel like you have to have the best stories.
You're talking about Microsoft versus Google
and then like Android and like Chrome, like,
do you have like a crazy story that like nobody else
do you think knows outside of the Seattle scene?
Yeah, this stuff just happens.
But I do think it's fascinating that like,
like Google, Microsoft and Amazon,
all the cloud stuff is actually, you know,
started out of Seattle.
Now, a lot of the cloud stuff has moved down
to Mountain View and, you know,
and Amazon's spread all over, but the center is there.
Now, just to be fair, like there was already Google App
Engine that had been around for a while
that was being run out of the San Francisco office,
but doing infrastructure level cloud.
I mean, like I looked at it and I'm like,
look, we're not really in this game
unless we're doing VMs and storage, you know,
the sort of the, you know, the core parts of cloud.
And so, yeah, so there were a group of us
that started the infrastructure cloud.
We eventually called that Nebula
and the three starting projects,
it was the internal codename, were-
I was gonna say, who comes up with these names?
I feel like they're working at Amazon and Microsoft.
The names that they name things are just,
it's so interesting.
Well, you come up with great names internally
and none of those things see the light of day.
Oh my God, that is so true.
The internal codenames are so much better
because I'm just about to tell you
the three sort of initial efforts
that we did as part of Nebula was BigStore,
which turned into Google Cloud Storage,
BigQuery, which is still BigQuery, right?
Because, you know, I looked around and I'm like,
what are internal Google services
that are almost ready to be packaged up and externalized?
And Dremel, which turned into, you know,
which is the basis for BigQuery
was like probably 80% there already.
And so, and it was a unique thing
that took advantage of the switching architecture
in Google's data centers.
And so that was the first thing.
And then BigCluster was, ended up being GCE, right?
Horrible name, right?
They were like, we're making everything big.
Google BigCluster would be the best.
And I still have early, the original logo.
I have some early artwork that I stole off the walls
and like an old sweatshirt and stuff like that.
So I keep some of that stuff around.
But at some point, but like Google,
like the Google leadership at the time
was not a fan of infrastructure cloud at all.
Because they couldn't build data centers fast enough
to satisfy internal demand for things like Gmail.
And so a lot of the thinking there was like,
look, if we're gonna put a machine in a data center,
we're better off doing something high value,
high margin, like inventing the next Gmail
or the next ad, whatever,
instead of just running it out for parts.
Now, eventually they changed their mind on that stuff.
And I think a couple of drivers there,
number one is that the margins are not that low.
I think there's a lot of room
if you can run this stuff efficiently to make real money,
especially when it comes to like the,
one of the things that's obviously super high margin
is network egress for clouds, right?
Because you're charging per bytes,
but you buy per bandwidth.
And so there's all sorts of arbitrage to be had there.
And nobody's seen those prices go down over time.
And you know that the cost of goods for doing like,
bandwidth and fiber has just dropped dramatically
over time for doing egress.
Which is also funny to think that they,
like Gmail was high value,
like Gmail as the free service that everyone uses
has a lot of value to Google.
Right, because I mean, it's a place to show ads
and ads are just that lucrative.
But I think the other thing there is that
when I joined Google,
they love to brag,
Urs who was head of a technical infrastructure
at Google forever and is now,
I think still officially at Google,
but working out of New Zealand.
He was always against people working remotely
until COVID hit and he moved to New Zealand
and made an exception for him.
So who knew, right?
Everybody else is coming back to the office,
but he's still in New Zealand.
Oh, you keep it real and it's my favorite.
Yes.
But anyway, so like, like at the end of the day,
people like Urs, like, you know,
they love to brag that, you know,
Google was probably the second largest computer manufacturer
back in the early 2000s, right?
Like behind Dell or HP.
I forget which one is probably Dell,
but it was only for Google data centers.
Google was building like, you know,
stuff for their own use, not selling it externally.
And with that, they had enormous pricing power,
both when it came to data centers, dark fiber,
but also negotiations with like Intel and AMD.
And so there's lots of situations where
Google would be like, hey, like, you know,
Carolinas, we want to put a data center in the Southwest
who can give us the best tax deal.
And then North Carolina, South Carolina would compete
to see who could give the better deal.
And they'd be like, okay, we're taking both of them.
Right, so they would do that type of thing a lot.
Buy up dark fiber, Chris Saka,
who, you know, was an early lawyer at Google,
helped sponsor some of their early Google talk stuff,
went on to become a VC and like was on TV
doing Shark Tank and stuff.
Did a whole bunch of early stuff around like, you know,
buying data center, you know,
this is before Google was building their own data centers.
They would sell you a data center per square foot,
but Google would actually, you know,
be able to actually use more power
than anybody else based on density.
Now you buy data center space based on kilowatts,
not square foot, but like that was back in the day.
So all sorts of arbitrage there.
But with that, Google had a lot of purchasing power
and negotiating power with vendors,
things like Intel and AMD.
And with the rise of cloud, all of a sudden,
Google was no longer the big kid on the block.
And so to some degree, to maintain those advantages,
cloud was one of those ways to just make sure
that Google was still operating at the scale
that actually kept them competitive
when it came to actually acquiring compute resources.
So I think that was one of the drivers over time
to make cloud real also.
But this was your first like infrastructure job, right?
Yeah, I mean, before that, I mean, like, you know,
I had done, I'd never worked on Borg.
I never worked on the data center stuff.
I had always done sort of like ads and stuff like that.
But, you know, it's like,
I spent an enormous amount of time just studying Amazon,
using it, talking to users, understanding what cloud was,
comparing and contrasting that to Google's internal stuff.
And, you know, read a lot about OpenStack,
you know, at the time.
And all that went into designing GCE
and influencing things like GCS.
I was a little bit involved in GCS,
but not nearly as much as with GCE.
And so GCE was a really fascinating project
to be part of from the get-go,
because our goal was to build on
as much Google infrastructure as possible.
So we wanted to run on top of Borg.
Google did not believe in virtualization at that point,
especially not hardware-assisted virtualization.
So it was all containerization.
And there were a lot of folks that were,
I didn't know them personally,
but my understanding was that they had been at Transmeta,
which was this sort of like startup that did like chips,
but the whole idea is that they would do live translation
of code before it executed type of thing.
So it's a different approach to sort of doing virtualization
versus what we ended up doing with GCE
was building on top of KVM that used,
processor features to be able to do virtualization.
And that ended up being,
allowed us to run VMs inside of containers on top of Borg,
when Borg was essentially the internal container orchestrator
that predated Kubernetes.
And so, but we didn't have any virtual networks.
So we had to create a virtual network system
from the get-go.
Google didn't have a virtual block device
because everything was GFS and Colossus,
which if you squint at it,
it looks a little bit like S3 or GCS.
And so we had to build a block device,
distributed, fault-tolerant,
snapshot-able block device from the ground up.
Multi-zone too, which was a big feature.
Yeah, so like a lot of the stuff
that we were thinking about from the get-go
was like, how do we use,
like because Google had so much dark fiber
between regions and stuff,
like by default Google's network
is actually connected between regions.
So you don't have to set up sort of like hacky VPNs
to actually talk between regions.
You can just talk from one region to another region
with Google and it's all private.
So stuff like that,
we had a lot more capabilities and bandwidth
for moving stuff between zones and even between regions.
And so we were able to take advantage of that.
So in a lot of places we were able to differentiate
versus sort of where, you know, Amazon and AWS
was at the time.
But the truth of the matter is that this stuff
was all at the edges.
And so we launched GCE.
It was a ton of work.
The company started seeing value of like,
hey, we really want to be in the cloud game,
but it really didn't move the needle, right?
At the end of the day,
it was working around the edges
and playing catch up in a lot of ways.
And so that's what really led to,
number one, it was getting,
that led to Kubernetes.
And so me and Brendan and Craig
were rolling off of other projects.
It was getting harder to get stuff done in cloud
because as it got more important,
as more people came on board,
there was more process.
It felt like you were sort of
swimming through molasses to get anything done.
And we knew that we needed to shake things up
to actually make Google competitive in cloud.
And so that's what led to Kubernetes.
Before we jump into the Kubernetes,
you mentioned the,
just the politics and dynamics
between the Chrome OS team
and the Android team originally.
How was that with GCE and App Engine?
Oh, it was awful.
All right, that's what I thought, all right.
No, and it was just,
it's a fundamentally different way of viewing the world.
Right?
I think the App Engine folks,
I don't want to misrepresent
because I think there are some good people.
And it's like, it's,
you're always these cases
where you like somebody personally.
Like Kevin Gibbs was the guy
who led App Engine at the time.
Great guy.
I just like didn't agree with him
on a lot of technical stuff.
That's the thing.
That's totally it.
As an outsider of the whole thing,
it always seemed that App Engine said,
you form your app to our infrastructure.
And GCE was exactly the opposite.
It was like, we will form our infrastructure to your app.
And that's like the fundamental differences of like,
hey, if you want to do it the Google way
or the way that we recommend, use App Engine.
And if we want to make money from customers,
if we want actual customers,
we got to go adjust.
I think there's,
I mean, you see this story play out
so many different times
when it's like platform as a service types of things
where, and we saw it in App Engine,
but you'll see it with Heroku.
You'll see it with Cloud Foundry.
You'll see it with things like Vercel
where these things are,
they're essentially fixed architecture.
If your app fits into a specific architecture,
these things can work very well.
And they're more opinionated about how things work.
But as apps or applications see success,
they'll hit limits of those platforms, right?
And so the question is,
is that there's this thinking
from the App Engine team at the time of like,
well, if we just add this feature or that feature,
then that'll get us over the hump
and then we'll solve everybody's problem.
But there's an endless list of small things
that need to be done.
So for example, like App Engine at the time,
every request coming in,
it would be captured and forwarded.
There was no streaming.
And so there was a limit on the size
of the request coming in, right?
And so I think that limit was like, I don't know.
And then they kept increasing the limit over time.
But at the end of the day,
there's gonna be no limit that's gonna be big enough.
So then there was a way to sort of like transfer it over
and have it actually write into an object.
But if you're doing video transcoding or streaming stuff,
there's always gonna be some case there
where you actually need a more raw network
versus sort of just doing HTTP type stuff.
And this is the fundamental problem I have
with platform engineering as like a thing
that companies are trying to do
where they're trying to build their own Heroku.
And I'm like, do all your apps look the same?
Are you trying to build one platform
that exists for everything?
Because what happens when they outgrow it
or there's the stateful stuff over here
that doesn't fit into that thing?
It's great for maybe 60, 70%
if you actually can just move that direction.
That was Disney+.
We had a great platform that we forced everyone,
you have to meet our architecture.
But then yeah, everything else that outgrew it
or had different requirements,
like sorry, you gotta do everything yourself.
And we can't even help you with a little bit.
It's like, no, no, no, you go get a different AWS account
and go have fun.
I really think that there could be a middle ground,
but it's just those things.
I mean, if I had it in me to do another startup,
I would do one around platform engineering,
but I'll explain in a second what that would be.
I do have two questions before we get back to Kubernetes.
One thing that I thought was really,
like that stood out about what you were saying
is that you went and used AWS
and you really thought about,
and why don't we promote the strategy
and the customer obsession truly as an engineer?
I feel like it's so disjointed from actual
being technical in business, but that's why they pay us.
Like, you know what I mean?
To me, that's so interesting,
the way that business and technology come together
and how you can really be competitive in the market.
And I feel like we make such bonehead mistakes
ignoring, like really looking at our competitors
when we're building things
and looking at what's already out there
that you just end up reinventing the wheel
over and over and over again.
Like, why is that not a bigger part?
I have theories.
You wanna hear my theories?
I do. Yes.
So I think a lot of business leaders,
so it comes back to like, you know,
remember when, oh God, what's his face
who led Pepsi took over Apple, right?
So there is this management theory
that like good management actually transcends domains.
And baked into that is this idea of management and labor.
And if you're gonna have labor,
if you're gonna view people like,
and the question is like, is the work that we do,
is it management?
Is it labor?
Is it knowledge work?
Is it something in between?
And how does that change with AI?
I don't know, right?
This is-
I always wonder if it's gonna make people,
if it's gonna make the soft skills and the business
and being like, almost like if engineers
will become engineers and PMs at the same time
or just kind of take a little bit of that middle work out
because I really wonder if,
and honestly, I'm just excited
because I think women are great at that.
Right.
Well, I think-
This is actually what I'm getting to
is that I think part of that mindset
is that you have job descriptions
and those job descriptions are useful
in terms of hiring, in terms of leveling,
in terms of pay scales, all these different things,
but the map is not the territory.
People are not their job description.
Oh my God, that's so true.
And so I think one of the things that you find
is that like, I've always viewed myself
as a business person who specializes in technology, right?
And so through my career, I'm like,
I wanna understand the business context,
I wanna understand the customers,
and then like the programming, the architecture,
that stuff is all downstream
of actually sort of being based in the business,
but there's a lot of places in large companies,
and I saw this at Microsoft, I saw this at Google,
whereas if you're an engineer and you're like,
well, I have opinions on the business model,
they're like, stay in your lane, that's not your job.
Right?
Oh, I felt like I deal with that all the time.
So how do you like navigate that,
asking for a friend friends me and everybody else?
Like how did you build this wonderful career,
but still get to do that when the, like,
when the job description didn't fit you
and you were trying to make it fit you
or trying to show that you had these skills?
I mean, the answer is burnout.
I mean.
Again, you keep it real.
For me, what happens is that
if you're one of those people that is cross-disciplined
where you don't fit neatly into a job description,
what you'll find is that you look around
and you're like, everything is so dumb,
why don't we just fix it?
Right?
And for me, the driver for burnout
is when you feel ownership over something,
but you don't have the tools to impact it.
When you are pushing a rock that's not gonna move,
you're gonna burn out.
And I think if you're at a big company
and you feel ownership over sort of like,
you know, cross-functional sort of like type of things,
and I think good people feel a deep level
of ownership over things.
They want it to be successful.
They wanna be proud of their work.
That's a combination for burnout.
And there's another theory I have.
So I'm not gonna name-
You didn't have to attack us like that, Joe.
But I think that talking to you is so rad.
But number one, this is one of the reasons
why people go to startups
because startups, one of their superpowers-
I've been thinking about that
and I was like, I'll only do big tech.
I'm never going to startup.
And now I'm like, hmm,
Oxide and some other places look kind of cool.
One of the superpowers of startups is that like,
you know, they can look at autumn
and they can say like,
we have an autumn-shaped hole over here
that we can actually make work, right?
And it's very difficult to actually
sort of write your own ticket
and find a job that is like really suited to you
when you don't fit cleanly into one of the disciplines.
Can you write a book for engineers
trying to navigate the crazy world that we're in?
Because you have such a good way of being realistic,
but also like, I don't know.
The problem is, is that if I were to write a book,
I think most business books,
like, do you listen to this podcast,
If Books Could Kill?
I love it.
This is Michael Hobbs and-
No, but that sounds very interesting.
Oh, so they take airport books and self-help books
and they just tear them apart and it's just hilarious.
But they have this one book theory,
which is like, all these books are actually the same book.
It's just sort of like set in different ways, right?
And I think some of this happens-
I think a lot of processes are like that.
I think it's a lot of patterns that are just recycled.
Yeah, so I think most business books could be like,
you know, boiled down to like a single eight and a half
by 11, you know, piece of paper, write a handout, right?
The best ones, you need to use both sides.
The crappier ones, you could do it all on one side, right?
And so I hesitate to write a book
because I think it would probably be like,
it would be a pamphlet.
I would try like-
That's fine, you can make a pamphlet.
But that would still be cool though.
Maybe I should do a zine, I don't know, right?
But one other question before I forget.
Do you think that Google's former partnerships with AMD
and how they were building their data centers
and kind of the fact that they were
an early machine learning shop gave them an edge
when they were building Gemini?
Because it was behind for a while and then it came back.
So, wrong part.
The T in GPT stands for Google.
So, that's it.
So, if you wanna go deep on this stuff,
there's another podcast I'll recommend,
The Acquired Podcast.
I'm sure you guys have seen this one.
They did a, you know, one on Google and Google AI.
And I think, you know, number one,
Google Cloud is actually, it's one of those things
where nobody knew that AI was gonna show up
the way that it did and Google Cloud being there
and having that infrastructure, that customer base,
the ability to sell enterprise.
Obviously, all of that stuff is super convenient
to have now as we're entering the AI era.
And also, if you look at, like,
Google started making early investments
in building their own silicon
for doing this stuff with TPUs.
And they're on, like, I don't know what,
like generation seven of TPUs internally.
They're not selling these things externally.
You can get access to them through cloud.
But I think that did give them the advantage, right?
I think Google saw early on that hardware acceleration,
you know, through all the machinations.
Again, listen to the podcast
if you wanna hear all the ins and outs.
I'm not an expert there.
But like, I think, you know, Gemini is probably
one of the only sort of frontier models run in a way
that is not losing outrageous amounts of money
because Google owns their infrastructure
and actually the TPUs are so much more power efficient
than GPUs and just they're building for a real business
versus, you know, a lot of the funny money
that's happening out in the industry right now
is my take on it.
And that's going to be such a big determiner
because I think a lot of this is not sustainable.
You know what I mean?
We keep saying that it's cheaper than engineers,
but is it though?
You know what I mean?
I mean, like, you know,
already in Uber rides used to be cheaper than taxis.
Oh my God, I was writing a blog about this,
like this weekend and I was thinking about that
because so many people are, now I use AI.
I'm not saying that it's like completely,
I was very like anti and now I think
I'm somewhere in the middle.
But it's like, I want to keep collecting knowledge
and using it to learn
because I think at some point
we could very likely be the people that are,
like, you don't want to be so reliant on it
and then they raise the price
and they raise the ability to have it
and then you no longer can use your brain, right?
So it's like, I want to almost extract
all I can from it, you know?
For sure, for sure.
Joe, where were we?
I don't know.
There are like 12 things on the stack.
Tell me what you're talking about.
We have so much ADHD here, like.
Well, we at minimum have to get
to the Kubernetes piece here.
At some point we're gonna have to have you on
for a second round of everything else.
What would you do differently
if you were building Kubernetes today?
Okay, the things I thought we did right,
so number one is we open sourced it.
There were some people who wanted to actually keep it
as a GCP service
or like take Borg and find a way to externalize that.
Borg was this hairy C++ code base
that was so entwined with the rest of Google.
There was no way.
So I think starting something new, writing it in Go,
because I think that was sort of the right level
of approachability and systems level.
Open sourcing it, you know, trying to sort of utilize
as much as sort of the Docker stuff at the time.
You know, I think all that stuff was right.
The way we built community and approached community,
I think, you know, paid off in spades.
I think you should be really proud of yourself there
because it's still like the cool,
one of the coolest open source communities out there.
Cloud Native, like I'm over there.
I will learn how to do Kubernetes
just to hang out with you guys.
Oh, that's awesome.
I mean, but it was everybody, you know,
and I think it was a lot of relatively low ego
senior engineers that are like,
hey, we just got so much shit to do.
Let's just get it done, right?
That's hard to find.
And I think, and then you have folks like, you know,
Justin, you mentioned Sarah, I'll mention Tim Hawkins,
right, like these people are just the absolute backbone
of this stuff, right, more than anybody else.
But even starting out, I think, you know,
when we first announced Kubernetes,
we're like, I remember I was trading, you know,
tweets with Deepak,
who now heads up all compute stuff at AWS.
And I'm like, hey, you guys are welcome to join.
And like, I'm like, seriously, like, you know,
if you want to get involved, get involved.
And, you know, he's very dismissive at the time.
Like, well, you know, well, we'll do Kubernetes
when customers start asking for it,
which is essentially like, bless your heart in, you know,
in Twitter-y.
I want that on a t-shirt.
An Amazon-ese, yeah.
But like, you know, the, but then I do remember
the re-invent where they announced EKS.
And that definitely felt like the end of the beginning
in terms of this stuff, right?
Like, okay, now everybody was on board.
And it was fascinating to see that stuff come full circle.
But like, even from the get-go,
we knew that we wanted more people involved.
We knew that for the to be able to do that,
to be successful as open source,
it couldn't just be a Google thing.
It had to be a community thing.
And both in terms of technology and partners and people,
we wanted that to be.
Did you feel a little bit of petty, like I told you so,
when they released EKS?
I would've been, like, I would've screenshot it,
the old pro stuff.
Not that, because honestly, Deepak is such a nice guy.
He really is.
You know, I ran into him at Costco years later.
At Costco, I love that so much.
Seattle's pretty small.
That is such some Seattle stuff, though.
Like, two people that have been in tech forever
just randomly talking at Costco.
But there are people that I do feel,
you know, there are places where I do have
some pettiness in my heart,
and I try not to talk about it.
I try and let it go.
But, you know, AWS and those guys, not as much.
I mean, they are what they are, right?
It's like, you know, I mean,
there was some underhanded shit early on in cloud.
Like, early on, like, we were undercutting.
Give us the tea.
Yeah, we were undercutting S3 prices with GCS.
And we had some early customers
that wanted to move a lot of data from S3 to GCS.
And I know that Amazon was screwing with BGP routes
to actually send it through low-bandwidth links.
Shut up.
So that you'd really slow and crappy
to move stuff over, right?
Because they didn't have a lot of traffic
going that way otherwise.
And so they're like, might as well screw stuff
going over to Google infrastructure.
Wow, that is petty.
There was some petty bullshit like that early on.
But, you know, but Amazon is Amazon.
It is what it is, right?
It's like early on in Kirkland,
I was working with Steve Yaggy
and to hear him talk about sort of like Jeff Bezos
and working in Amazon, you know what you're getting there.
It's kind of like, it says it on the tin, right?
What was it like?
You mentioned, I know Justin,
you have complex feelings there.
And they all have complex feelings.
Autumn was there too.
It was a lot of like, I mean,
I worked directly under Deepak for quite a while.
So, yeah.
You mentioned early on, like doing,
going Kubernetes was a shift from GCE of like,
you were kind of having a hard time making changes
and you wanted to change the industry.
Do you think that the things that you set out to do
were successful?
Like not just the technology limitations, but like.
Yeah, no, I think Kubernetes was very successful that way.
Oh, you were asking what would I have done differently?
Oh, we dropped that thread.
I think eventually, you know,
Brendan started playing around with third-party resources,
which eventually became CRDs.
And I think I understood sort of in an intuitive way,
the power of having a clean API.
And so, you know, the original API definitions
was stuff that I did based off of knowledge
from doing the GCE API, which I spent a lot of time on,
which is based off of knowledge of looking at AWS
and OpenStack and stuff like that.
And so understanding the power of a clean API,
I think was something we got right in Kubernetes.
I wish we had understood the power of CRDs,
what eventually, third-party resources
eventually became CRDs.
And this idea of having sort of a, you know,
domain independent API core, that was, you know,
I think, you know, if we could go through
and extract all of this sort of the container stuff
out of Kubernetes core and have the API server be pure,
kind of like the, what is it?
The KCP project type of stuff.
I think like that would have been a better way to go,
but I think we had to learn to get there, you know?
Yeah, Tim often talks about like,
I wish pods weren't a first-party thing in Kubernetes,
right?
Like if we could just have,
here's the API, implement whatever you want,
and that makes it a little more flexible.
Right now, it feels like we're shoehorning a lot of things
like KCP and cross-plane and those sorts of things.
We're like, well, we like the API.
We like the predictability of this API
and being able to do CRUD actions off of,
here, store my stuff for me.
Yeah, the controller pattern and the spec versus status,
all those patterns I think have served us well, but yeah.
What are you most proud of?
I mean, the community, honestly, right?
I think it's like, I think,
so there was a tricky point in Kubernetes history
where early on, I mentioned it was essentially
a bunch of low-ego engineers that are like,
okay, we just got a lot of stuff to get done.
Let's get the work done, right?
And there was enough excitement
that everybody and their brother came in
and wanted to add stuff to Kubernetes, add features,
and figuring out what to say no to, what to say yes to,
how to create rooms for people to extend,
all that stuff was really difficult.
And we recognized at some point
that we didn't have a governance structure
that was well-defined.
And it was really just sort of by osmosis
and sort of influence.
And through that, we wanted to actually get more formal.
And so eventually that process led to things
like the steering committee and SIGARCH
and like some of the other sort of like processes
that are still being used today.
But at the time, we were in charge
because we were kind of the ones in charge,
but we wanted to actually sort of slip in
this governance structure without alienating the community.
And so we had a set of meetings
that we called this sort of the steering bootstrap committee
or something like that, where governance bootstrap,
where there were a set of us that are like,
okay, what does good governance look like?
How do we actually do this?
How do we actually...
And so I think being able to define that
and get people on board without people freaking out
and abandoning the project,
I think all that has led
to a lot of Kubernetes longevity over time.
And I'm proud of the fact that like,
hey, we've handed that off to new people
and they've taken it on.
And there's been ups and downs
and things that have worked and not worked.
But I think the fact that the set of people
leading the project now weren't the set of people
leading the project at the beginning,
I think it's a healthy thing.
I think the BDFL, the benevolent dictator for life thing
is almost an anti-pattern at best,
very difficult to do right, at worst an anti-pattern.
And I think we avoided that and I'm really proud of that.
How core was CNCF formation to...
Like CNCF was before the governance pieces, right?
This was just a neutral legal body essentially.
The relationship between Kubernetes and the CNCF
has always been complicated.
The LF at the end of the day is a marketing organization,
it's a trade association.
And their goal is to boost their members,
which are the people who are paying dues
to the organization.
And the goal of the Kubernetes project
is the Kubernetes project.
And so there are times where these things are at odds.
And I wish that there was more transparency
from the CNCF and the LF in general around budget
and who's like where the money's going
and how it's being spent.
And it's just not there.
That being said, I think having a neutral place
for this stuff to exist from a legal point of view,
from a trademark.
And I have issues with the way in the unevenness
they've enforced some of the trademark policies too.
But when you look at this stuff,
I think that having places where companies
can come together and collaborate
and having some sort of neutral ground
is incredibly important.
I think that was a big part of Kubernetes success also.
But I think that the governance sort of predated anything
with like the CNCF and the TOC actually deciding
sort of, okay, at this point that the CNCF is like,
if you wanna be graduated,
you have to have some governance, right?
But it's like, I think we were really sort of making up
as we went along, I think a little bit.
And I think it's worked out well.
I'm proud of that work.
I think other stuff that I'm proud of that I did
is I started off the whole CAP process.
At some point I'm like, okay,
a lot of stuff is changing the project mature enough.
Kubernetes enhancement proposals.
Yeah, yeah.
And so you look at things like,
JEPs in Java and PEPs in Python.
And I'm like, we need something similar
where it's time to have a little bit more process
around how do we add new features and make changes.
There were a lot of people that were resistant to that
because they were used to just making stuff up
and checking it in and going.
But I think in terms of looking at it
from the point of view of the user community,
having those signals of what's happening
and having some of that sort of documentation over time,
I think has played out pretty well.
I think processes are one of the underrated parts
of engineering.
Like writing code is cool,
but having a structured way to like work together
is really important.
And making that process not only have longevity,
but also have like a larger scale
and just who can be involved.
It's like a huge thing for a lot of the stuff
where whoever had commit rights,
decided it was a good idea is not the way to go.
No, not for a sustainable community.
Yeah, and there's no one size fits all, right?
For what works for a small project
or for one project may not work for another project.
And so I think there's so many of these problems
where as engineers, we want to be like,
here's the one true solution to this
and there is no one true solution to so many problems.
Yeah, it's why the words, it depends is so popular.
Exactly, exactly.
Joe, this has been great.
Thank you so much for giving us the time.
We're gonna have to have you back so we can-
Please do, seriously, please come back.
Day two Kubernetes stuff.
We got Heptio stuff.
I wanna hear all your thoughts on platform engineering
and AI.
Do you have any like one cool Kubernetes story
or something crazy that happened with the original,
doesn't have to be bad, crazy,
but you know, like a cool memory you have
from building Kubernetes or working with-
I remember we knew we were,
okay, so two things I'm gonna finish with.
So I ended up leaving Google because I don't think,
I didn't see a career ahead there for me, right?
So I launched Kubernetes and I got a 3.0 on my next review,
which was essentially telling me that like, I'm not-
What?
No way.
I'm not on track to getting,
I was a senior staff engineer,
I'm not on track to getting to principal.
And so that's pretty much, you know, that started-
What exactly did they want?
You're like firstborn child, a kidney?
Like what did they want for principal?
There were a lot of people that like,
when we were launching and there's one marketing guys,
like, you know, it's like,
I know there's a unicorn in there somewhere
because I see a lot of unicorn shit,
but I don't know what it is.
Like, they just did not understand it
because they didn't understand
the relationship to the business.
Because again, systems thinker, right?
I'm like, look, if Google can actually go through
and do something where for every new user onto technology,
X Google gets 80% and Amazon gets 20,
that 20% of Amazon, you know,
like it may still be larger in numbers,
but like, you know, in terms of Amazon doing stuff, right?
But like Google's,
if they're grabbing more than their fair share
of new users based on technology,
that's a win over time for Google.
And that's the way you think about
eating away at markets over time.
Unless you're Azure
and you just try to give it away for free.
Yeah, I mean, well,
there's different ways that you can do this, right?
Where you can essentially,
so open-source Kubernetes was a way that sort of-
But I love that you're such a big picture thinker.
I love that you're such a big picture thinker.
Like that's so underrated.
And I feel like-
It's not always appreciated.
It's not always appreciated.
No, but I think it gets squished out of a lot of people,
right?
Because we put so much premise on being technical
and fitting these modes.
Like, I think that's really rad.
Yeah, so like, yeah,
that is kind of like,
so I think, you know,
there wasn't a lot of appreciation
for Kubernetes early on at Google.
And I think even over time,
when you look at sort of the governance
and the sort of the drama around donating things
like Knative or Istio to the CNCF,
Google wanted to hold on control
because I think there were a lot of people
that felt like they gave away too much with Kubernetes.
Not recognizing that Kubernetes
would not have been Kubernetes if not for the community,
if not for everything that went into it.
Especially when you get that many people
working together on one thing,
like it makes you more successful.
Yeah, and so I think it, you know,
Google definitely didn't know what they had
right after we launched it.
And even over time,
I think that, you know,
it's different leaderships rotating through
so in different ways.
But the thing I'll leave you with
is that the time that I knew that we were onto something
is that I think it was Brendan and I
were down in a bar in San Francisco
and we overheard, because, you know, San Francisco,
we overheard somebody else in the bar
talking about Kubernetes.
And this was pretty early on.
You're like, okay, maybe we're onto something
when you overhear somebody else talking about your thing.
And like, you know, yeah, that was pretty crazy.
I hope the guy that, or whoever that was in leadership
who gave you that review and that didn't listen to you,
I hope it like lives in their nightmares forever.
I don't know.
Like Google lost such a cool thing in you.
Words is doing fine in New Zealand.
Words is fine, yeah, this is, wow.
Well, thank you so much, Joe.
This was just fantastic.
And again, we'll schedule another time
because I want to hear the rest of these stories.
All right, yeah, we'll keep talking.
It's so good to hang out with you guys.
And I always enjoy doing these things.
Thank you so much for coming.
I can't wait till part two.
All right.
And thank you everyone for listening.
And we'll talk to you all next week.
Thank you for listening to this episode
of Fork Around and Find Out.
If you liked this show, please consider sharing it
with a friend, a coworker, a family member,
or even an enemy.
However we get the word out about this show
helps it to become sustainable for the longterm.
If you want to sponsor this show,
please go to fafo.fm slash sponsor
and reach out to us on Facebook, Twitter,
about what you're interested in sponsoring
and how we can help.
We hope your systems stay available
and your pagers stay quiet.
We'll see you again next time.