Integrating with Kubernetes with Matthew Sanabria

Welcome to Fork Around and Find Out, the podcast about building, running, and maintaining software
and systems.
Bienvenido a Fork Around and Find Out, soy Justin, Justine Gerrison, and with me as always
is Autumn Nash.
Welcome, Autumn.
This, this is almost as bad as your dad jokes.
I just, I just saw Matt's video saying that he has been learning Spanish or relearning
Spanish.
I was wondering, yeah, does he always introduce himself in Spanish?
Yeah.
No, it's usually like a really bad dad joke, but like.
That was just really bad Spanish.
So it's.
Bienvenido a Fafo en Español.
Fafo.
Fafo.
Fafo.
Fafo.
I like that better.
Yeah.
I feel like, I feel like you can't see Matt's like oh so pleasant face right now and it's
like the, like you're just, the smile is just so big, like it's so pleasant.
But you can if you subscribe to our social channels.
We have clips.
So you would see Matt's beautiful face.
I'll post the face on socials if you need it.
You just let me know.
Okay.
I will post a big ass picture of my face.
Welcome to the show, Matt.
Matt, you are a solutions architect or leads, I forget the actual title.
So it's funny because like we have both at Oxide.
We have solutions architects and we have solutions software engineers.
And I'm, I was the first hire for solutions software engineering, which is like a team
in engineering that focuses on customer facing problems and integrations, but not necessarily
like solutions architecture, which is different.
That's under GTM and like sales that partners with sales.
So trying to split the difference because we, Brian and I agreed that companies tend
to not prioritize the customer facing work.
And like you can see things like rot over time, like their tariff and providers or Kubernetes
integrations.
They rot over time because nobody really owns them.
So like our team is meant to like find those and own them and do that sort of work.
So you're strictly, you're under engineering then at that point and you just, you own the
customer facing things.
So solutions architect is like, Hey, we need this problem.
It may not go directly in the product, but we want some integration here that a customer
needs.
Exactly.
So like we'll be building things for ourselves, for our customers, for our solutions architects
to go use on those customer calls and like unblock things and then receive that feedback
from them and say, Hey, like I met with seven customers this week and they all want Kubernetes
plus plus plus.
Like, what do you think?
And then we have that conversation and make it a potential project.
I feel like that's always a gap, right?
Like because it's solutions architects trying to go and convince product and engineering
like to go build something, which is really hard when you've got a roadmap of that product
that's stacked and you already don't have time for.
So like either solutions architects are trying to hack something together or it never gets
done.
So I think that's like a really cool solution to like a problem that everybody faces.
Yeah.
I mean, I have strong thoughts on this topic, honestly, because this is one of the reasons
why I joined Dockside, because I've seen this breakdown at other other companies and I wanted
to like own this and make sure that it doesn't.
It's hard to get, you're right.
It's hard to get solution architects to like convince engineering to make changes, right?
So that's why we sit in engineering so we can like make those changes.
And sometimes there are things that are out of our scope where we go partner with the
rest of the engineering teams and like say, hey, this project's bigger than our team.
Let's partner and get this going.
But I've seen like many companies do solutions architecture.
And when I if you send me a link to your personal GitHub for a company call, I'm losing my mind
because then I then I know your solutions architecture has failed, like, or your company
has failed you.
And that's like my sort of hot take with solutions architecture, like sure, making a personal
GitHub repo to unblock the customer works for you and your commission check, but you leave.
Now what? Yeah, well, the thing that I find hilarious is even at like these large companies
like Amazon to get a repo created on the AWS Labs GitHub was right.
Like that was so hard.
It was just it was multiple forms.
You had to have a meeting about it.
You had to have this ongoing thing.
And then like part of the open source team, like the open source organization for that
company, because every company has their own like GitHub and then they have the open
source organization and you have to do the training to get through it.
And like, you're like the barrier to entry to solve someone's problem is so high that,
yeah, they just they end up on a personal GitHub and then I'm not interested.
Yeah. Yeah. We've we've did this as an industry and we made we failed.
We required all this red tape to get a repo created to do the right thing.
And then even same thing for docs and other like sort of typo fixes.
God, people really just like I know that docs were like ignored before, but now they're
just adamant that we don't need them anymore because yeah.
And I was like, like that is the opposite.
Like if anything, you want to give more context.
But also like think of the contribution flow for docs.
I want to just drive by fix a typo.
That's so obviously wrong.
Right. So now I have to go to your repo, fork it, make a change, push it, sign the CLA,
get some frickin CI to review it, get someone to review it just to fix a typo.
I just want to say it was in East like it was in US East one, not US East three.
Yeah. It's like three days.
And it's like no wonder why your docs rot.
And then now people are like, oh, just throw LLMs at the problem.
But you're still requiring pull requests for these things.
Like you realize that the process is the same.
Yeah. Yeah. And you didn't change the process.
So it still sucks.
Yeah. And yes, we we had our own docs as like the product team.
Like we're just going to own the docs because the AWS docs around it was just like it was such a hard process to get anything written.
That's another thing.
They're never it's never which I understand sometimes you can't have a one size fits all.
But like there's never continuity about what to do with docs.
Like Coretto had different docs than when we had at Keyspaces.
We had a docs person.
You know what I mean?
Like and so it's there's no continuity, even though like I still think that technically, like when it comes to operational excellence, Amazon still one of
used to be one of the leaders.
But it's just like there's so many things to try to get you to do the right thing.
And I think developers are lazy, which is not always a bad thing.
That's why we automate the shit out of everything.
But like, yeah, you know what I mean?
Like it gets to the point where like if you put so like if everyone's trying to do a million things and you make it so hard to do the right thing, you're just setting yourself up for failure.
Yeah, I think hiring writers is even if they're not writing the docs today is important just for keeping the consistency of like frame of mind of like how I organize things, like even if it's not the way I do at their jobs.
Yeah, because like I think a lot of engineers are not always the best at teach.
Like for one, they forget where the person started.
Right. Like so just because you built something, you don't remember where like you can't look at it from a frame of like, I've never touched this before.
This is completely new.
Like they're good at getting the beginning, the middle and the end and then giving you feedback and the information that they pull from those docs with just like how many times one person is like, like a ton of people are hitting one doc or this troubleshooting guide.
Like, I feel like they're a very they don't get enough credit.
They're very much appreciated.
I mean, that's this is where like different organizations have different concepts of being like sales led, product led, engineering led.
And if you go down either one of those or any one of those paths too far and solely, you end up in a problem.
You have to kind of be a mix of all of them and get that feedback going.
And then there's there's a lot of organizations out there that are very heavily engineering led to the point where it's like, dude, you forgot who you're building for.
You're not even using your product.
I don't think they even like like.
I don't know what I don't I don't want to talk like in a generalized statement, but I think a lot of times engineers get stuck in the I want to build something cool and not who you're building it for.
Or how that person's going to use it or how you make it accessible for them.
And I think that's why I enjoy like public facing.
But I want to be like an engineer.
I think that's like always a struggle when I'm looking for like a role is like I want to do engineering, but I want to do it with a customer and like hit their needs.
But I don't always want to be part of sales all the time, you know, like so I feel like it's the feedback culture to like autumn.
Like this is where I talk to my wife.
She's in she's in the culinary world.
Right. Where she makes she has a chocolate shop.
She makes chocolate feedback in how awesome your life is like you had a cute kid and you get chocolate and your wife.
Listen, I can't eat all the chocolate.
Okay, I'm trying to get back in shape.
Trying to get back in shape.
She is cute.
I love that little little girl.
But but like feedback culture is center to those fields, artists, musicians, like culinary chefs and all that stuff.
Feedback is there.
You want someone to eat your food, to look at your painting, to review your your song, whatever, to listen to it and give you feedback.
And like because you want to make it better and know who you're kind of building for.
Right. If you tell me it's also an excitement to it.
You know what I mean?
Yeah, there's an excitement to it.
You're like, oh, I don't want your like opinions.
You don't know what I did.
Exactly.
So weird, like coming from art, like how you said, because it's like, do you remember just when you make something for your family and they taste it?
And you're like, oh, my God, how is it?
How is it like?
Yeah, yeah, yeah.
You'd crave that feedback.
And in this like engineering world, we can build like tools just for ourselves.
And we can claim like, ah, this is just for me.
Don't worry about it. But but thousands of people are using this.
Don't you want feedback?
And then when you have like programming languages or other open source tools that are really for the people and then you invite people to give their feedback and then you tell them like, actually, now your feedback isn't valid.
But they're users of the tool.
Imagine me telling someone that I cooked a meal from that, like if they told me, oh, Matt, this meal sucked, then here's why I need to understand that to make it better.
And when you just dismiss it, it's like, what are we?
Do you even remember who you're building for?
If you can't take that feedback, why even build?
Don't call yourself a builder.
I also think that we've like we've we've separated them so much.
Like, I think when you go and you look at like what companies think being a good engineer about, it's never customers or making something that like it's always like.
Like what you think about when you're doing their code review and what people like what gets you a promotion is never like the feedback or like the pain point that we experienced last point last time, even when it's your pain point that is going to affect you.
Like, I think that we've just gotten like it almost becomes like a pissing contest about like elitistness, about like, but this is like the best way to write a for loop.
Like, who cares?
Like, is this going to work for your customer?
Is this like are you listening to like what they need?
Did you really think about the requirements before you ran in to build it?
You know what I mean?
Right.
Honestly, I'm tired of like just doing stuff that like.
Engineers think are cool with other engineers, like I want to go solve a physical or like metaphysical, but like a problem for somebody, like it's so exciting to see, like when you solve an actual customer problem and you unblock them.
Speaking of this, I know you were.
Yeah, no, I mean, I was just gonna say that building is so much easier when you don't have to think about money or people.
Right.
Like if you just eliminate and that's where a lot of people like that's the easy route to go.
So if you just do the easy thing, you're just going to build in a bubble and you're like, I don't care.
I think that's why the personal software stuff with AI is taking off, because you don't have to worry about others.
You're just building for yourself.
And that's that's that's your goal, which is perfectly fine.
And the money, it's cheaper.
Yes.
I'm putting that in air quotes.
It's cheaper.
You have to scale it and maintain it.
Yeah, it's cheaper.
Time investment.
Yeah, like the time cost is so much less.
And that is where, like, I have so many projects that I wanted to do.
I'm just like, I'm never gonna do this because I'm never going to spend that many nights in interested in the problem.
And so now it actually makes me excited.
Like, I will say that, like the second I realized I was fun and played, I was like, I have so many ideas I wanted to do.
Yeah, I mean, I've been using AI to do that, and even on my walks in the park, I'll like talk to AI and say, hey, let's move this forward or let's understand, let's talk about the design.
Yeah, I want to get to that in just a minute, but I want to start first.
We're like, the reason we kind of brought you on here was solving customer problems and building things.
And you had a blog post on the Oxide blog about the like three integrations you had with Kubernetes, because Oxide as a rack doesn't have any native Kubernetes pieces.
So you're using external tools to kind of do that.
And I'm kind of curious what that experience was like building those three tools and integrations for different customers, but similar use cases, which is like, I want a different tool for it.
It's like, I want Ansible instead of Puppet.
It's just like, oh, that's
Can you explain what like Matthew does at Oxide and just Oxide in general, for people that don't know?
No, everyone on this podcast should know who Oxide is.
I can give you the one-liner, yeah.
So Oxide, the company, Oxide Computer Company, right, Oxide.computer.
So we make a big rack scale computer that we build the hardware, we write the software, it's all integrated.
And the whole purpose of it is you put it in your data center somewhere, and you have essentially a cloud provider like API to provision infrastructure.
And like, we have a certain amount of primitives that we give you, we give you the virtual machine, the virtual disk, and like VPC, and you build on top of that.
So my job is to help customers integrate with Oxide and build integrations that they want or need to make their journey coming to Oxide and staying on Oxide easier and better.
So that's why I was hired.
I was hired as the first solution software engineer, which is a new role that Brian and I kind of came up with to focus on these problems that we saw being neglected at other companies that we've both worked with and for, right.
So that's the spiel.
Now, to answer your question, Justin, like, it's, it was simultaneously fun, and also difficult, because the problem that I have, especially when I was a team of one, now, now we have four people on the team, right, myself and three others, which is amazing.
And my co workers are fantastic.
They are so much better than me in many ways, which is great, which I love, because they get to dive deep on these problems.
And like, the communities, the state of communities on Oxide wouldn't be where it is today without them.
I helped get it started, but they definitely are taking it over the finish line.
The problem was, the fun part was, love talking to customers, like, it's so nice to be able to say, customer, I'm literally in a Slack with you, like, or on a meeting with you, let's talk, what do you need?
Oh, I'm running Rancher and da da da, and here's where I'm at.
And we have this, or, oh, I need the ability to create a load balancer.
And like, there's no way to do that on Oxide.
And I know that you don't have a native load balancer yet, blah, blah, blah, and just get all the information and then determine what you can build.
Flip side of that is, there's so many things you can choose to build.
And there's context switching.
When like, I wrote the blog post, and I published it, and then you get people on Hacker News that are like, oh, why don't you just have a primitive in Oxide to do containers?
Because we decided at Oxide that the primitive was the virtual machine and to create a container primitive would require going into the host OS, which we can do.
But we'd have to decide that as a direction for the company, and it requires more work than like, just my single team, right?
We'd have to do hypervisor work, host OS work, there's more to do, do I want to maybe, but like, I also need to solve the immediate customer problem.
Because that that's, you're, you're talking about like a six to 12 month roadmap, rather than or even more than like a quarter roadmap.
So I have to balance the urgency of there too.
To speak to the, the, hey, shiny object problem that engineers tend to have, because there's so many cool things to be working on, how do I pick which one to work on?
And how do I like, context switch when I need to work on multiple things?
That was that was the hard part, and which led to us, yeah, which led to us scaling the team up.
Sorry, I hope I answered your maybe I tangented off a little bit there.
But no, that was good.
Like, that's a good starting point.
So like, like, walk us through like, what was so someone came and said, like, hey, we want the rancher node driver, and we just want to work with oxide, right?
So like, you already have a machine, it has an API, it has authentication has like all of the things that they might need for projects and all this stuff.
And you say, Okay, I'm gonna go put this in the the Matthew GitHub, org and and run it from definitely not.
What's like, at that point, it's just like, oh, we need to go solve this problem.
But I'm sure there's other customer problems.
There's other yes demands.
I was fortunate enough to join oxide where our customers were like, few, right?
Because we had just I joined in late 2024.
The rack was for first available in I think, late 2023.
So the rack wasn't even on the market for like, a year yet.
So we obviously had a few customers, which meant that I was able to focus on the customers.
And when I joined, they were like, hey, this customer already submitted a pull request for a rancher node driver, we have no one to really review
and maintain it.
Like, this is kind of your first, like, this is your first thing, go do it.
So okay, so I had never used rancher before.
Never, never use rancher never used SUSE products in that capacity.
So I was like, let me go figure out what this even is and how it works.
So there's all that discovery to go do and I did and I reviewed it and I got it all merged.
And then I'm like, okay, the problem here is that customers are going to want to provision Kubernetes on oxide.
And I don't mean like, in their bespoke, I have a script that does it way.
I mean, they're gonna want to be able to provision it in using real native integrations for their respective distribution of Kubernetes, right?
Rancher or Omni or cluster API, they're gonna want to use the standards there rather than some bespoke thing that I give them.
So that's when that's what led me to like, there's more here than just creating this one integration for this one customer.
Let's solve the problem overall and use customer information to guide it.
Because while this customer wanted rancher, the next customer either doesn't or didn't want rancher, right?
They wanted like the cluster API, for example.
And then just repeating that and saying, hey, get the feedback in, look at your pile of stuff that you can build and pick the ones that would impact the most customers in a decent amount of time.
And that was it on like the provisioning side, honestly.
And, you know, you and I spoke because I ended up doing an Omni plugin with Oxide, right?
An Omni infrastructure provider.
And that was more like, it was customer driven, but more prospect customer driven, right?
Not like we didn't have an existing customers that wanted that.
Yeah, yeah.
And it was just prospects that are saying, hey, I'm Talos plus Omni plus, you know, whatever other other places, how would that work in Oxide?
And then we got enough people asking that.
And also, I was technically interested in that stack.
So like, let me go see how this works.
And so that was more of a build.
That was more of a plugin that I built to explore the customer problem space and get more information about when you build something, when you build it like two or three times, then you really understand like where the edges are.
And that kind of helped me understand.
And that's really what I wanted to, like, you've now built from fresh eyes against three different Kubernetes, you know, self-hosted Kubernetes sort of infrastructure, like, and they have trade-offs on how they were designed and what they're doing.
And in taking my like Zidero Labs hat off, like, you don't have to say like the Talos side is good.
I know we had plenty of bugs.
I'm kind of curious what like your, what you learned through that of like, wow, these are, this is why these are different or what the pros and cons of this type of architecture might be.
For this use case.
So good question.
The Rancher node driver, I don't like, because it's just Docker machine.
It's Docker machine was great for the day.
Docker machine was great for the day.
Yeah.
But it, it's not something that's good for like an item potent operation.
It didn't give me the ability to like, put logs anywhere.
I just felt blind building that thing because Rancher would expect you to give you a Docker machine binary essentially, right?
Because it actually has to be called Docker machine driver oxide or whatever, or else Rancher won't run it.
So like kind of the plugin system.
Yeah.
Yep.
So then I have to build that binary, give it to Rancher, Rancher executes it, but then I don't get my logs because it's executing it in like a pod.
It's wrapped inside of the thing.
And so, yeah, exactly.
So when I was building that thing, I was like, I can't stand developing this because I have no visibility.
I, at one point I like integrated open telemetry and sent logs to like honeycomb or axiom or whatever it was.
And I was just like, I can't get logs out of this thing easily.
I'm sending it up.
Like I'm just sending it somewhere else so I can get observability on this thing.
But even so, the way it's called, like it creates a new SSH key per instance.
And like, that means if you have a seven node cluster, you have seven SSH keys because you can't reuse SSH key because it's stored on disk over there.
And good luck trying to figure out what's on disk and the JSON serialization format for how it describes the state of the cluster.
You know, this goes in this VPC and this subnet, that serialization format is actually like a breaking change if you change it.
Right.
Like, and it's not a versionable thing.
Yeah.
So now you have to, you wouldn't know that from reading the plugin.
Different releases of your plugin to say this one supports this other schema.
Correct.
Yep.
I mean, you can like, you could handle versioning yourself if you wanted to, but then you'd have, you had to know that the JSON serialization was load bearing.
Reading, from reading how to build that plugin, it's basically just like, yeah, implement Docker machine driver, have fun.
And it doesn't tell you any of the edge cases of like, here's what this does or why it matters.
And that's been true of like most of the plugins that I've been building.
You go read on how to build a plugin and they're just like, implement interface by, you know, like, I don't give you go find it and then do have fun.
I don't know.
And like, I do feel that way about a lot of the, like Rancher was one of the earlier sort of Kubernetes, like self hosted management things.
And like everything being SSH based is just like, it just feels, I feel bad about it in a lot of it.
Yeah, I don't love it either.
I don't even, I don't even think the customers truly love it.
It's just that that's where they had to start because they're using Rancher.
And I basically like the Rancher integration that we have, I'm letting customers kind of drive the development of that from via feature requests, features, tell me what's wrong.
It's, I don't want to say it's in maintenance mode, because maintenance mode has a bad connotation.
Like it is, it is maintained by us.
It's actively putting like new features in it, unless the customer wants their needs.
Yeah, and that's our customers, it's feature complete.
And that's totally fine.
And primarily because Rancher itself is going to be moving towards the cluster API.
Like, eventually, right, just added tech preview support or whatever.
So they're going to be moving in that direction.
So why even like waste more time developing their node driver?
Like, let's focus on cluster API.
And cluster API was the third thing you implemented, right?
Yeah, that was after Omni, actually.
So I guess we can talk about Omni next if you want.
We'll go to cluster API, because I'm actually kind of curious on that one of like,
So cluster API was one of those things that I was, it was just me at the time.
And I wrote down that we want to do this.
But instead of me going to build cluster API, I took a tangent to build the cloud controller manager instead, because I realized like, oh, all of these Kubernetes clusters are going to want to have the CCM installed.
So that way, they can have access to the load balancer controller to node help like lifecycle controllers.
For anyone that's not familiar with that piece of it, right?
Like that ties your Kubernetes cluster to where your environment where you're running it to give more information about like availability zones and in any information that Kubernetes wants to like schedule pods against, right?
Like the CCM is what like pulls that in from metadata in the environment, and like shoves it into nodes.
Yep.
So if you were if you were running a Kubernetes cluster on a cloud provider, there was a CCM running under the hood, powering the load balancer controller and other things.
And same thing like for other providers that are not cloud native.
If you want to integrate with those cloud providers, the CCM was the mechanism to do that.
And it had like three controller types that you would implement one for nodes, one for services and one for routes.
And those are the primary three controllers.
And you can add more controllers, but those are the primary three.
So for oxide, I was like, hey, people need load balancers, right?
People need to know their node health, because what happens if you accidentally put a network partition in your Kubernetes nodes in oxide?
What's gonna happen?
Well, without that, without the CCM querying oxide for the real instance state, you can, you can end up in a way where Kubernetes is like, I'm gonna get that node out of here.
Right?
So I figured, let me go solve that problem.
And cluster API was a large project.
And I didn't really have the bandwidth to take on that large project, while maintaining the go SDK and the terraform provider and building a packer plug and doing other things.
So I was just like, deferred, we need this, but deferred until we get more bandwidth.
And then more bandwidth came, we hired people right on my team.
So from there, my colleagues, and Brandon are the ones that built the cluster API, they're the ones actually got off the ground.
Which I love.
Yeah.
Shout out to Brandon, we work together.
Yeah.
Josh, Brandon, you guys are great.
Love having you on the team.
And they both come with extensive Kubernetes experience.
And they were able to see that forward and get it going.
And we were talking about how to build it and whatnot.
But the thing I don't like about cluster API is, there's just a lot to it.
Right?
Not only, it's not as simple as the rancher node driver, where it's like, hey, implement this interface, and you're done.
It's make these custom resource definitions, right?
These CRDs.
Also, write the controllers for it.
Also, make sure you get the ability to generate the manifest.
And now when you deploy this thing, you put like the CRDs and the controllers on the management cluster.
And then you have it with provision, like a workload downstream cluster.
There's a lot of moving parts, right?
Like, and I found that to be annoying.
But I guess if you are a Kubernetes native shop, and you already think in CRDs and controllers, guess it makes sense.
I mean, that was like the architecture of cluster API was, hey, what if everyone just had to write their controller piece to provision machines for their environments, and then everything else would just get the quote unquote benefits of Kubernetes, right?
It was just like, hey, we get a database, and we get these like, magic objects that we can just store and retrieve and stuff like that.
But then you're dependent on that magic database of Kubernetes and upgrading three times a year and making sure that CRDs are compatible and all of that stuff becomes a lot of overhead.
If you're like a heavy platform engineering team that's working across clouds, or like many, many clusters that you're spinning up and tearing down, I can see where it makes sense because you can just add it to like your GitOps pipeline or whatever.
It unifies the interface for all the environments.
Yeah, it unifies the interface.
You're just running Kubernetes manifest, like, cool, that makes sense.
But if the providers themselves like implement breaking changes in the CRDs, that's annoying to deal with, you know, how do you deal with this?
That's part of the downside of cluster API.
I don't mind it.
I think it's fine.
Overall, like, I don't, you know, my colleagues and I are using it to create internal clusters.
And it's, it works.
And like, the nice part about it is you don't need you don't need a control plane.
And I'm air quoting that you don't need a control plane.
But you do need like, step one is create a Kubernetes cluster.
You know what I mean?
So you do need something.
Just it basically treats the Kubernetes cluster and etcd as it's like control plane storage.
So it's like step one, create Kubernetes cluster, step two, deploy the CRDs for cluster API, step three, create cluster.
It's like, dude, step one is a lot.
Yeah.
Like, when we were bootstrapping EKS anywhere, like that was you run a kind cluster locally.
It's like, and every time I told him, like, I don't want my laptop to be like, production should not be dependent on my laptop.
Like, I know we're gonna pivot things over to the final cluster or the management cluster eventually.
But then like, when I need to upgrade that, like, we were literally like, backing up etcd, restoring it to my laptop, and then sending it back in.
But no, no, keep my laptop out of the loop of this.
Yeah, I do not want it.
So the guide that we have on oxide that tells you how to use the cluster API uses kind locally as a manager cluster, and then deployed to cluster oxide.
And then my colleague, Brandon wrote some logic to create clusters internally, that will bootstrap a cluster, do the CRD stuff for cluster API, create the actual management cluster, move the CRDs in there using like cluster CTL move.
And then from there, create work like workload clusters.
And that's fine.
That works.
It does everything.
But again, think about the complexity you're asking people to do, right?
It's it's terraform, a plan in terraform apply sounds great right about now.
Yeah, like create this cluster that's going to actually die, to then create the real cluster.
So it's managed, and then create downstream clusters.
And I get why you do that.
I think that is the correct approach.
But it's just at what point do we say that cluster API is wild?
Yeah, at what point did we just admit to ourselves like there's a lot going on?
Yeah, and I usually tell people like the, the benefit of that complexity usually comes when you're running in in roughly like three, maybe four environments, right?
Once you were like, I need to have all these different environments.
And each one is going to have its own opinions about its own operating system and its own networking and all that stuff.
It's like, okay, now, this might make more sense.
Because like, you can get by easily in three different clouds with terraform, like just do that for a while, right?
Like terraform is going to be okay.
But if you're like, I'm in five clouds, and we have eight different operating systems, you're like, okay, yeah, you need to figure out.
I totally agree.
And like, since we maintain the cluster API provider, we also need to dog food it so that we can keep continuing to test it and improve it.
And that's unlike the rancher node driver, where we're kind of in maintenance mode there in a way, and accepting customer bugs and feature requests, the cluster API, we're actively using to improve ourselves.
We don't have to wait for a customer feature request or bug to come in.
We're going to go fix that, because we're using it and trying to, we're trying to improve it to the point where we can get a stable interface before declaring it.
Cluster API is still going through a lot of changes, right?
Like they, they, I think this, this episode is going to come out in September, probably.
And like in September, there's like a big release for like new features in cluster API, which is like my work was like, hey, we can't spend the time to keep investing in this because that's not where our customers are looking.
They're, they don't want, they're not actively looking to use this in business context.
And so like, we can't justify spending the time.
And, and on the like, the other side of it is like, it is very opinionated about how, how it gets used in some ways where it always assumes that it can do like rolling upgrades for nodes.
And you always have additional capacity and you have an API that you call for these types of machines, which were all like problems that we ran into in the past.
Yeah, especially if you're like strictly bare metal, what's your API there?
Like that's where it gets even more difficult.
Like unless you have proper APIs for IPMI and controlling power and all that stuff, that gets rough.
But Oxide is a pleasure to build on because it has those APIs for the, for the VMs.
And it's like, this makes my life easier to build because I'm just calling APIs.
I don't have to like implement new features from scratch per se.
But yeah, I mean, the cluster API is a weird one.
I don't hate it.
I don't love it.
We wanted to support it because we did want that vendor agnostic approach of Kubernetes on Oxide.
Like, hey, all you need is to give us some image to be your Kubernetes nodes, right?
Some Ubuntu image.
From there, no vendors, just cluster API and Kubernetes and have fun, which is nice.
And not that vendors are bad, because obviously, like we have the Cider Labs Omni infrastructure provider and we have the Rancher stuff.
But like some customers totally want vendor neutrality.
Yeah.
Yeah.
Yeah.
Even when they're spending a bunch of money for a rack, right?
It's like, even when they're spending a bunch of money for a racket.
Really?
Guess what?
We're always gonna be partners with Oxide, but like nothing else.
Like we don't want to pay anyone else.
But like, when you compare that to what they're doing with, say, commodity hardware plus VMware, you're paying for the hardware.
And then now you have to pay for software to run on top.
And then that software is like per core license or something.
So the more you use it, the more you pay.
I don't want our customers to deal with that.
Like just buy our hardware, the software is free.
And then from there, you want to run Kubernetes, here's your no cost cluster API method.
You want to use Omni because like you're standardizing around Omni, go for it.
We have an integration for that too.
And like, but you obviously have to pay for Omni, right?
So now you have to do that.
But I think that's worth it, depending on where you're at as an organization.
Because I use, my personal cluster running on Oxide is running through Omni and running Talos.
And I like that flow.
To me, it makes sense.
I like it.
The integration is simple to build.
It's easy to reason about.
And obviously, once I learned more about how state and everything worked, I get it now, right?
So comparing the node driver to the cluster API to then the Omni infrastructure provider, you just implement two methods.
It's provision steps and deprovision.
Like provision steps get called in order.
And they're meant to be idempotent, right?
Because the contract that Omni says is, A, we're going to repeatedly call these steps until the step passes.
Then we're going to go, we're never going to call that step again.
And we're going to go to the next step and so on and so forth.
And then you use states that you store in Omni to like kind of have your little retry points in a way.
And I like that.
I wanted that in the Rancher node driver because there's times where like things break.
And now if you recall the whole method from the beginning, now you get conflicts or something.
And it just makes it very hard to reason about.
So from that perspective, the Omni infrastructure provider felt nicer to develop.
And you all were very helpful on the discussions that I started.
It was a big lesson we learned because we built the cluster API stuff first and then realized that like, wow, A, we don't want to make Kubernetes a dependency for Kubernetes, right?
It's like that we just want to, even though Omni is still like etcd on the back end and all that sort of stuff, but like it's a simpler thing to run as an interface.
It's like, here's a single container that has embedded etcd that's just like, you can just do this piece of it and that does more for you.
But then the like the in-place upgrades of like, hey, guess what?
We're going to remove all of this complexity of like different operating systems because we only work with Talos and Talos has an API.
And it's just like, guess what?
We can do all this stuff we need after the fact of like Talos boots.
We can call the API.
Now we do everything we want and it can be a generic piece of that and not have to like pre-build those, bake those images, get cloud in it and all that other stuff.
But that's another thing.
So I was just talking to my colleague about, hey, we're going to start creating like standardized oxide Kubernetes clusters that have these sort of shapes.
And what is, what do the disks look like, right?
You have the OS disk, you have an attached oxide disk for maybe Longhorn, you have an attached oxide local disk for maybe the local path provisioner.
How does the cluster API configure those disks and also from an oxide perspective, it's easy, right?
Just create the instance with the disks.
But then you have to do more than just create the disk.
You have to, you have to actually like format them, partition them and set them up for use.
And now you're kind of have, you have to kind of fall back to cloud in it for that, right?
Because cloud in it's too late, right?
But cloud in it's too late and cloud in it is not guaranteed to, to retry or pass, you know what I mean?
Like, unless you write logic to do something there, it's, it's not even guaranteed to run actually.
What if, what if cloud in it started up and had an error and immediately exited, you'll never get it.
It's bashing a Python's wrapper.
Yeah, it's a bad hook to hook into.
But then what's your alternative?
Like, how do you, how do you tell cluster API to spin up a node and then configure disks outside of cloud in it?
You give it an SH key and you SH in after the fact.
Well, now we're back to the Rancho node driver.
So like, where's the hook?
And this is where I think Talos makes sense because you have the API there.
And in the Talos world, I just drop a YAML manifest that says, hey, user volume.
Boom. Right.
Like whatever the CRD is for that.
Which is like the way that everyone else solves that is right.
Like they have that shim that a NITRAMFS like piece of like butane or ignition sort of thing of like, hey, we have to run a bunch of stuff before the machine boots.
I'm like, that sucks.
Like, what are you doing here?
And like, you have like two layers of cloud in it to like format drives, drop system D units before we actually hand over to system D.
That's essentially Red Hat OpenShift in a nutshell, right?
They rely heavily on ignition to bootstrap the cluster.
And then now the cloud providers that are running RHEL have to use ignition.
And you could use the normal cloud and user data path and put ignition config in there and it would respect that.
But oftentimes you need more ignition config that can fit into the 2K that cloud init.
It's limited. Yeah, yeah.
And then now it's like, what are we doing?
Like, this is where I think Talos makes sense because it's API driven.
And I wish more people would see that, I guess, is what I'm trying to say.
So maybe that's good.
We're talking about that in this podcast.
Like, that's the thing that I've been telling people for years, like whatever happens with SideArrow and its products, like there needs to be a API for Linux distros.
Like that is the core initiative that I just wish it existed more in the world.
I want more distros that have an API interface that I can call as like declaratively.
And also, you can't rely on it as like a side way to get in.
Right. Like doing this stuff with like Webman, right?
Like Webman existed for decades and like it was like, cool, we can get a web interface to like the user interface.
It's like, no, no, no. The server needs to only be the API.
And if you can't do it through the API, then you have to extend the API some way because there shouldn't be a back door to this.
Otherwise, everyone's going to just use the back door. That's easier.
This is where, like, people will be like, well, just use NixOS.
Well, but NixOS requires you to drop a config somewhere and then rerun it.
Like it's not serving up some sort of externally accessible API.
You know, I've been running. It's like it's it's there after this point.
You know what I mean? Yeah. So I really do.
I've been running it on on three of my machines for since scale this year.
So it's like for four months now or so.
And it's like it is the steepest learning curve of any technology I've used for a long, long time.
And I'm just like, wow, you have to get over a lot of stuff to like understand even like I don't even write in the config.
Right. I just want to like conceptually understand, like, what is this thing doing?
And it puts outputs. Yeah, that's basically what it boils down to.
And then the outputs are special.
If you if you put an output that that happens to be named this, then some part of Nix will interpret that as a special thing.
Yeah. And like that's what it bothers me a little bit about Nix.
It's it's a Linux sister with baked in config management.
Right. Like we've been always putting that stuff on top.
And again, Nick says, no, you have to go through the config management piece.
Right. Just like tells you have to go through the API.
But no, this is requiring their own cult at this point.
Absolutely. But but but that's because it's the frame of mind of like, if you understand Nix, you don't want to know anything else.
And you're like, I love how this works because I took all of that effort to understand it.
I've been running for like four months now and I got a new laptop.
It won't boot now. Actually, I tried to do an update last night and the system won't boot.
And I can fail. I can fall back.
But I can't get it into the next stage.
I want to upgrade my kernel and it doesn't start, you know, a graphical user target.
And I'm just like, this sucks.
Right. Like this is still a bad situation for me.
I was just talking to complicated.
It is. That seems like you are getting in between me being productive.
And that's just too much for me.
Yeah. But that's that's the Linux way.
OK. That's why I said people like look at me like I'm crazy when I say that Linux is like its own.
Like it's like different denominations of like a religion and people get so passionate about it.
And like people will fight you like tooth and nail over like their distro is better.
And why? For me, like.
To close the NixOS bit or the Nix and NixOS bit, I was just chatting with the Phlox folks yesterday.
We're chatting all about Phlox because Phlox is built essentially on Nix and makes Nix easier to use.
My biggest gripe with Nix was essentially what you said.
The config learning curve is steep.
And then like you're pinned to shaws and revisions, not to versions.
So like if I want to like pin my stuff to a specific version of application, I now have to like go a layer deeper and understand what shaw that that refers to.
And Nix purists will say like, that's how you should do it.
And you're right. But also, I kind of don't always care for perfect reproducibility.
I just want to know that it's always this version.
And that's where you get a little bit of tension.
And also, if I want to spin up a fleet of NixOS machines, then I have to kind of pre-know my config up front of how I'm going to do that.
Or at least enough to get a user and SSH key on it.
Exactly. Enough to get a user and SSH key and then dump the real config.
There's really no good bootstrap option.
You're still back to something like a cloud init or whatever.
You're still going back to that.
Yeah, starting each of my my my laptops and my desktop when I'm configuring them, I have to like literally copy my SSH key to a USB drive and plug it in and like copy it and then like, you know, SCP files over and then I can start to bootstrap this and then I can get to the next wave of like now I can do the rest of it.
And like, that's not that's not OK.
This isn't what I'm trying to do.
I don't like it. And I run Talos Linux at home and it makes my life easier.
I just Talos ETL my way through it.
It's nice.
But it is such a slim use case, right?
Like, that's the other piece of like Nix OS.
Like, I'm able to do a whole lot with it, which I love.
Like, I'm having so much fun with Niri is like the manager on Wayland.
I want to try Niri. I'm waiting for my framework 13 pro to come in.
Niri in Noctalia 5 is like I'm having more fun.
Like, I haven't had this much fun since I like Compass Barrel days of like spinning the cube.
And like, I used to sit in those settings in my college, in my dorm room.
And like, I was like, what is this setting doing?
Like the windows would catch on fire and all this cool stuff.
How often does Autumn call you a nerd?
Every day, all the time?
Well, it depends.
Because sometimes I'm in there with him.
And sometimes I'm just like, when he says fun, I'm like,
like, you're conflating pain with fun.
Sometimes like you like when he says, Oh, this is fun, I'm like, this is fun?
Like, like my kind of fun or your kind of fun?
They're two different like he does like what's that type two fun?
Like, you know, when people say they like type two fun, yeah, and type two fun is like,
usually like, making their life as horrible as possible on purpose.
I mean, yeah, like, it's all pain in other ways, right?
Like, I'm loving playing hockey, but I took a puck to the leg this last week, and I got a big old bruise.
And it sucks.
It's like wobbling around.
No, but it was fun, right?
Like, this is the same same vibe.
I like using Linux a lot.
And I could I could understand where people are like, Hey, Matt, if you just use Mac OS, like it just works.
And you don't have to worry about certain things like joining Microsoft Teams calls.
They have an app for that.
And no, but you're right problems.
Yeah, exactly.
Also, if I'm joining Microsoft Teams calls, I think there's Yeah, there's other problems in my life that led me there.
It's all died here.
So you can't solve that with a Mac.
Shade.
Shade.
Sorry, Autumn.
But like, what gets me is that I don't feel happy or productive using Mac OS.
Actually, I hate like, the absolutely, like moving around the windows for me takes eternity.
I'm always using my mouse and my keyboard because it is slow.
All of the animations, everything in it.
Yeah, I was chatting about this with my colleague because they bought, they bought a MacBook and they ran Mac OS on it, obviously have to.
And they were like, creating do they call them virtual desktops or workspaces, whatever, when you click the green button, it makes a full screen in another desktop, I guess.
And they were switching between those desktops with the Mac OS key bind, which is I think is control arrow.
Commander, like, why is this so slow?
I actually know, I think it is control for the window switching.
But yeah, haters.
OK, like my like, this is why Matt's on the show.
Yeah, I know.
I can get baby pictures.
What do you mean that has nothing to do with this?
Like, I like his spicy opinions and like, also look at that smile.
That's what that's the smile he makes when he talks about his wife and his baby.
It's so adorable.
But anyway, like, that's not why he's here.
OK, but I am like, like my solutions architect roots are like, I want the tool to do the job when I'm trying to mom.
I don't want to play with Linux configurations.
I don't want to, like, worry about that stuff when I want something to work.
I want to use Mac OS.
When I want to, like, develop an environment where I can work, I want Linux.
I don't like to struggle for free.
Like, I just I'm not doing it for no reason.
Like, this is why your computer crashes in the middle of our episodes, because you just like.
A lot of reasons for that.
A new Linux thing.
I'm with you on that.
Like, I that's why I bought a MacBook Air.
That's why I have the iPhone Air.
Functionality.
OK, like I need it to do its job in that moment.
I want it to sync all of my family's like stuff together and be usable.
And then when I want to play with something, I buy a server in a warehouse.
And how's that server going for you?
Sweet Jesus.
Why? Why did I do this to myself?
Oh, my God.
I've just been fighting like.
So I've landed in the middle where Mac OS is going to become my strictly personal stuff, and I mean personal.
I don't mean I don't mean personal plus a side project.
I mean, like, no, I'm ordering food to go and I'm, you know, sharing photos with my wife or something.
And the baby photos.
Yeah, yeah, yeah.
That's what it is.
That's what it's turning into.
Because I don't get real work done on Mac OS because I'm too.
I'm thinking constantly about the animations that are slowing me down.
Yeah.
And I'm just like, I just feel I feel out of whack on Mac OS.
It's not what it's for, though.
You know what I mean?
Yeah, it's not for getting work done.
No, but and honestly, right now, like you're they're looking at making your life where everything comes to one place.
Everything is seen and it's easier to do things.
And when I like when you have a baby in one hand, are you trying to like, you know what I mean?
Like you're trying to get things done.
That's what I did with a baby in one hand.
Yeah, dude.
I got went to AWS and graduated a college degree with a literal baby.
But that's why I didn't need to argue with Linux in the middle of it.
OK, I needed it to work.
I just find like for my work at Oxide, Linux is better for me.
But also that was that's like straightforward, right?
Like if you're working on hardware and you are doing integrations, you don't want all the extra stuff in the background because that's not what they're like.
You want it as close to what your customer like is going to have.
Right. So you need something that's going to work and you can have more control over the environment.
I very much wish more companies would allow people to have that flexibility to decide which like because it is always the like you get a Windows machine, you get a Mac, whatever.
And in AWS was the first place that actually had a supported laptop build of Linux that I could run.
And every every job I had before, like I just ripped off the OS and it was like, I'm just installing Linux.
Like I can't do it.
Oxide is mostly Linux people like using Linux on as their primary workstation.
Now it's probably more of a split because some people are just like, I'll just remote into Oxide VMs and do it.
Yeah. Do the work remotely and disconnect.
Yeah. I haven't found the remote development experience to be useful for me.
Like what I mean by that is, OK, if I treat my Mac as a thin client and I say, hey, my Mac's going to be for browsing and like joining the MS team calls and then everything else, I open up a terminal and I SH somewhere else to do the real work.
That's how I do it.
Like, I don't love that flow because I'm still bound by the window manager on my machine.
Unless it's the remote machine has like Tmux installed or whatever, that's what you got to move.
You also can use IDEs like there's so many different ways to get it, like to get into like a different VM or SSH or like there's so many different options.
I guess where I've settled is I rather my window management experience be on my machine and good on my machine.
I don't I don't actually care about Tmux.
I don't want I want window management just for my terminal.
I want window management for everything.
I want to organize.
Well, and that's like I mean, running Linux on the Neri on my laptop.
But all of my development is done on my desktop, which is like tail scale plus Mosh plus Tmux is fantastic, right?
Like, it's just I open it up and it's already connected.
It's where I was when I literally just walk away from my desk and like go over my laptop.
I'm like, oh, I'm still working on the same thing.
So that's where I'm at.
Once the framework 13 pro comes in, that's going to be my oxide work laptop.
This MacBook Air is going to be a personal browser.
Plus, maybe I'll edit the occasional on the go video if DaVinci Resolve is giving me a problem and like I'll be able to take FaceTime calls from it with my family.
Whatever. It's a purpose built tool for that.
Exactly. Purpose built tools.
I just don't have that that need.
That's fine. Yeah.
All that I have is that all my family and everyone uses Apple devices and like sharing the photos and FaceTime with them is miserable if I don't have an Apple device.
So I'm basically I'm just I'm bifurcating work and personal and Mac OS is going to be the personal stuff.
And anytime I'm doing work that's happening on a Linux machine, that's essentially how I'm doing it.
And all my kids have Apple like.
Yeah, the ecosystem has a lot of gravity, which is like I'm switching Android and it's going to be a fun like let's see who complains about the green bubbles.
Yeah, my mom was like, how do I FaceTime you?
I'm like, I don't want to really teach you how to do that because you can you can actually create a link and share with me.
But then I can do it in the browser.
And it's like, but but it's those.
Also, one reason why I don't like to use Apple products is because Apple makes it very Apple's like that one kid that'll like brag about everything they have, but never share anything with others.
Yeah. And that's what they do.
It's it's the business model gets in the way of the user experience.
Right. Of like of like making this universally work.
Right. Like I'm moving to Android.
The way or does it do what's intentional of keeping you in the ecosystem?
It keeps you in the ecosystem, which is the business model and the business model ecosystem.
So nice and cushy.
It's not that good if you're it's honestly not that good.
Yeah. Like if you're Apple Docs, right.
Compared to Google Docs, go use like anything.
Like, why can't I use Google Docs in a browser?
Like, it's not that big of a deal.
Why can't I use Apple pages in a browser?
I mean, I guess you can now with iCloud.com, right?
I don't know.
That's what I'm saying.
Like, you're like limited.
You're like, I want to go to the depths of the earth and fight with Nix.
But God forbid I have to do it like Google in a browser.
That doesn't make any sense.
You're just picking your heart.
I don't appreciate being yelled at right now.
OK.
Don't appreciate.
Listen, do you see this Apple logo on my phone?
OK, this is this.
Yeah. Like y'all are just ridiculous.
All right. Switching gears a little bit, because it's like we're we're pretty far into this.
And I think we have at least another half hour on this.
Matt, you seem like I don't want to say I pilled, but you you definitely use a lot of
A.I. in in a lot of the stuff I do.
Like he's very baseline normal.
And I feel like we know some A.I.
pilled people and we definitely like I don't feel like you're you're out there.
Like you are sharing what you're doing to some things.
And like you got me started on AMP for a little while and I was using it.
And I liked the harness.
But like it was just too expensive.
I'm like, I'm not doing this.
Yeah. But like I get the benefits.
And but like you've been using it for for longer than I think I was.
And you integrate it in more places than I think some people do.
And. What are you seeing as like the benefits or like the next the next step of like is
everyone software factories?
Are we all like is is the personal A.I.
pins or whatever, like the thing that you're waiting for?
What do you what are you working on?
Yeah, good question.
So my primary harness of choice is AMP still to this day.
And much like my personal work separation for the computer layer, I'm also doing a personal
work separation of the A.I.
layer. So like I log into AMP and chat, GBT and Claude, how do we actually don't even use
Claude? I logged out of that.
I don't care about Claude personally.
I just don't find their interfaces to be pleasant.
It feels it feels like nobody designed their product that they just like, ha ha ha, Claude
make my thing. That's how it feels.
And Codex or Chachapiti app now feel more intentionally designed, and I like that.
So I sign into my work stuff on AMP and Chachapiti.
I recently signed back up for subscriptions for AMP and Chachapiti personally because my
work Oxide has A.P.I.
billing. And there are certain things that you can't do an A.P.I.
billing that you can do on subscription based like voice mode.
So I've been using the voice mode on my personal account and like kind of just moving
this forward. Right. And saying.
Go to my GitHub issues and see what this is doing.
Let's get a review started on this pull request or let's talk about how to integrate Oxide
and XYZ and like go through that architecture and talk to it.
Lately, I'm not so.
I'm not as AMP pilled as the AMP team is
on social media. They're wild.
Like they're just like orbs for days.
But I get it. Your business model was wrapping A.P.I.
prices as a middleman.
That's not a tenable long term business model.
You need to charge something else.
And orbs it is. And I get it.
And the user experience. I don't know what orbs is.
What is this? Orbs is like they're remote runners.
It's like, you know, exe.dev where it's like a VM that you get automatically.
Yeah, it was just on the show a couple months ago.
Yeah. It's basically that.
But meant to run AMP AI threads and those threads can talk to one another.
So you can like say, hey, start a thread on the Terraform project and work issue
one, two, three, and then start another thread and say, hey, start a thread on the
Packer project and work issue four, five, six.
Actually, tell thread one, two, three over there to like do something different
because I've thought about it and give it some and the threads can talk to one
another. So it's nice.
So if you if you were someone that likes this sort of like cross threading talking
AMP makes sense there. But to bring it all back to answering your question.
The things that I'm focused on are one.
Using the voice modes of AI to get away from my computer.
Right. I'm not like I want to go for walks with my daughter in the park,
but I found myself at my desk thinking about a problem and just saying, oh,
just give me 10 more minutes to think about this problem and then we'll go for the
walk and then we'll do the thing and then I'll cut the grass instead of that.
Instead of being tied to my desk for another 10, 20, 30 minutes, I'm like, you
know what? I'm out.
Going to go walk, spin up the AMP thread or spin up the chat GPT voice or whatever
and just say, hey, like, let's continue chatting about X, Y, Z.
I was thinking that we can go up and just start chatting with the thing.
And then at the end, the best ideas or solutions come when you walk away for a
little bit. Yeah. And you're already thinking about the problem.
Yeah. I just want to get some exercise while I do it.
You know what I mean? Not just that.
I feel like when you're away from your keyboard,
you really put thought and you really go down the rabbit holes and, you know,
really think through your requirements because you're not tempted to just go
right into like solving it.
Yeah.
And I'm not like some max person that always has to have agents running and like
I'm always chatting. Like if I'm away from my computer,
then I'm always on a AI threaded with my voice or I'm not that, that person.
Like today I went for a walk in the park, didn't have anything playing.
I had my headphones in to track my heart rate, nothing playing in them at all.
No AI, nothing. So that's kind of,
that's kind of where I landed on the voice stuff.
Like I want to use the voice stuff to get me away from my desk.
And so is the, is the voice specifically.
So you're not looking at a screen.
The voice is just so that I exercise and that I,
like you could type it on your phone. Like, I don't know the like,
I want my hands free. Maybe I'm like, you know,
pushing my daughter around or maybe I'm like, yeah, yeah.
Or maybe I'm like playing with the dogs or maybe I'm cleaning my grill.
Like I'm just doing other things that are not.
I actually think that's like what AI should be used for,
to be able to give you life back.
Like for you to be able to like have time back. Like that to me, like,
I'm like, okay. Like.
Yeah. Cause like if I'm washing the dishes and I'm just chatting, like,
I just thought about this, the integration that we have oxide here, there,
that has a problem with this.
Can you create an issue for it and like summarize what we're talking about here?
Boom. Now that's out of my head.
I have the best ideas that way and then I don't forget stuff.
Yeah. And I found myself either.
Forgetting those things when I'm like out and about and I'm thinking about
something or I found myself being tied to my desk to just like finish the
thought to find a breaking point before I can like go for my walk.
And I think the AI voice mode bridges the gap for me. Right.
So that's what I use it for. And then general work stuff.
I make heavy use of like purpose-built threads for whatever I'm working on.
And I kind of treat AMP as my worker and AMP will be like, you know,
start thread in this project, work on this issue,
talk to this other thread and orchestrate threads that way.
And I see ChachiBT as more of like my researcher daily questioner.
When you use those threads and those like, I guess, agents in a way,
like how do you get the context that you need?
I'm usually telling it to pull from like some linear issue or GitHub issue or
whatever.
And you have AMP on the back end connected to those things.
So yeah. So I say like Puck,
Puck is their meta agent that's always available.
Like it's meant to be like the orchestrator that you talk to that then do work.
So I'll say, Hey, Puck, linear issue one, two, three,
let's get that started on the Terraform project.
Go spin up a thread there and it'll do that.
And then it'll move the project forward. And then it'll say, and I'll say like,
Hey, like monitor, monitor CI for failures and like report back.
And then I'll report back. And then that allows me to balance multiple units of
work. So now instead of just saying, Hey, just go start unit one, two, three,
or linear one, two, three, I can say,
go start work for linear issues one, two, three, four, five, six, seven, eight,
and spawn them in separate threads.
And that's what I did the other day for working on a Terraform thing. I was like,
Hey, we have these eight Terraform issues,
spawn sub-agents to work on all eight of them. Let me know when they're,
when the threads are blocked and I'll like tell you how to proceed.
And then now I had a pull request at the end of that. Like,
I didn't have to write them, right?
Because Terraform is a lot of boilerplate generation.
I'm okay with letting AI take those forward.
And then my job turns into reviewing them.
And then while those eight sub-agents were running, I was talking with AI agent,
like, Hey, I want to integrate Oxide with Buildkite to do some CICD stuff.
Cause we're working on blah, blah, blah. And what sort of APIs does Buildkite have?
Oh, it has this model. Oh, okay. But how does that,
how does that relate to the actions way while you're talking?
No, I don't use this voice specifically, but.
How do you get the context though of like what's in those pull requests?
Cause I've been like working on my like whole workflow of using AI and I've been
trying to tweak it.
So the ones that I say start from a linear issue have pretty detailed linear
issues. So for those, I said,
we need to add a new Terraform resource called this.
The resource should look like this. And I gave it Terraform syntax,
like the config. And I say, cause Terraform is basically a CRUD wrapper,
create, read, update, delete. And I say, here are the CRUD APIs.
You should be calling from Oxide. And that, that's what I give it.
And linear you're using as internal bug tracking or issues, right? So.
I am on my team, but like Oxide as a whole uses GitHub and I have linear sync to
GitHub.
So that's where I was kind of confused. Cause like linear can sync to GitHub and
then AMP could call GitHub, but like, I don't know where you're like,
that seems like a lot of places to store.
I find GitHub issue tracking to be miserable personally,
especially when you're working across projects. And that's what I do.
I work across multiple projects or when it's down.
Shade. Shade.
Yeah. Even their GitHub projects feature that just,
I find it to be a little annoying. So,
and then linear has this concept of like attachments that you can put on the
issue itself and there you don't have to do a comment for them and they're just
separate. So I'll like, I'll talk to the AI agent.
And so to answer your question, Autumn,
the context comes from usually the issue, right? Say, Hey, if it's a,
if it's a well-defined issue, I can just spawn an agent to go do the thing.
And then I'll be left with a pull request that I can review or whatever.
If it's something open-ended where I'm trying to build the context, like for
example, that, that oxide build kite stuff that I was thinking through,
the whole point of that conversation was to flesh out how would this look like?
And I would tell the agent via voice,
I already have a GitHub actions, ephemeral runner. It works this way.
It uses these APIs. What's the equivalent in build kite? Oh,
the equivalent does this. Well,
actually wouldn't have a problem with oxide here because it doesn't have XYZ.
Oh yeah, it would. What are my options? Oh, your options are to do XYZ.
And then at the end of that conversation,
we're left with all of those sort of end that those tangents kind of discussed
and like, we have a decision for them. And then I tell it,
please summarize all of this and put it on linear issue, you know,
four or five, six as an attachment. Or if we don't have a linear issue,
I'll say create linear issue four or five, you know,
create a new linear issue in my SSE team and put this as an attachment on it.
I've been doing it where like, I have like a works, a workspace,
and then I have everything broken down to different projects.
And then I have like an investigation.
So while we're talking through it or like put like going through it and doing
the trade-offs, it's like, it writes the notes. And then when we get through,
like, Hey, we got stuck here or we got, you know,
into this issue or we learned this new thing and had to like re like structure.
I have it taking notes. So I like can remember like what I learned, what like,
you know, what we did, what was the solution? Where did we get blocked?
Just trying to make sure that like,
I'm always getting the context from that situation.
That's effectively what it turns into.
It turns into some markdown document attached to the linear issue.
And then now when I'm ready to go work that issue,
I can just go read that markdown or I can go have the LLM say like, Hey,
linear issue four, five, six, there's an attachment on it.
Where did we end up here?
Exactly.
Sometimes it doesn't always have context with the context for you and the agent
sometimes, you know.
And I find that to be helpful. Like I don't do any, I don't.
And I chose that because I don't want to keep any context on my local machine.
I don't want to use beads. I don't want to use markdown files locally.
I don't want to use obsidian. I don't use any of that.
I want it all stored where myself and others can get to it.
Which is really smart. I like that idea.
I really like that Oxide is like wrote a whole white paper on using
AI and then they did such a good job getting like public comment and thinking
through, and then you guys did so many iterations and then you have Oxide and
friends. Like I thought it was so cool to see you all.
Like I feel like a lot of times,
especially at these big companies they're investing so heavily in AI that it's
just like reckless and they don't,
they're not telling you what they want from you with AI, but just use it.
But then don't use it this way, but like use it in everything,
which is like not very productive.
And Oxide had such a really cool way of, I think,
kind of giving like guardrails and kind of like real
expectations and how we think you should use it.
And just kind of taking real feedback and just seeing that you worked through
that issue. I thought that was so impressive.
Like do you feel like that helped you to like figure out like how to experiment
with AI and like to do it,
like enjoy like being a part of the feedback and writing that paper?
Yeah. Yeah. I mean,
I commented on that RFD and we still actually were talking about that RFD just
today, even before I hopped on.
It's good guidance. Cause I,
I use AI to help me write issues, to help me write code,
but I don't use it to like speak for me. Right. Like I use it.
Yeah. And that's basically what we were discussing today internally of it's not
okay to use AI to write for you, especially on public blog posts and all that
stuff. Did I use AI to edit my blog post of the Kubernetes blog post? Yeah,
I did.
How do you get it to not edit with em dashes in it? Cause then people think that
that's, that's exactly what we were talking about.
And we were talking about that today and yesterday. Cause we,
we have this like tool that we use to detect AI written responses.
And I put my Kubernetes blog posts in it.
And like the sections that I had AI helped me edit flagged as AI.
See, that's what I'm worried about because I feel like I'm so like.
Same here. And then they're like same here,
but then there were sections that I wrote that it flagged as AI,
but I didn't use AI for that.
And then there were sections that it wrote or like helped me edit that it didn't
flag as AI. So it's not perfect. Yeah. Yeah. It's not perfect.
But like I'm petrified of having bad grammar or like, you know,
like not sounding polished,
but then I feel like if you have AI help you with that,
then it sounds like AI wrote it when you spent all this time putting in this
effort, you know?
I make heavy use of like commas at the start of the sentence or at the trailing
end of the sentence. So I'll say, you know,
Matthew went on a walk today comma leading to increased weight loss or
something. Right. And I'll,
I'll make use of commas like that or I'll say like actually comma or otherwise
comma. I'll use that in my writing.
Are we going to change the way that people write so that I think so,
you know what I mean? Like, so,
which is funny because I've been every,
every professional blog I've written on for the last decade,
I've tried to either put in an em dash or a semi-colon and that was long before
AI, like this was just me as like, I want to,
I want to feel like I'm writing smart things and I have like punctuation that
people don't often use. And then when it was like, Hey, I use em dash,
I'm like, screw you. I'm still doing it. Like guys, this is still just how I
write.
I generally write with grammatically correct sentences with punctuation and
casing and all that.
The only reason why I don't use dashes in my writing as often as I probably
should is that I don't know how to type them quickly on the keyboard.
Control you 214, two zero one four. Like I remember, yeah, I, I don't,
I don't have that memorized in my muscle memory. So like I either,
I like Google the Unicode real quick and copy it or whatever. Yeah.
Otherwise I would use em dashes more because there's times where it makes sense
in the writing. I don't really like colons though.
I usually pull those out of my writing, but I'm turning around to them.
But it's so weird though,
because like even Grammarly now uses so much AI that like,
how are you supposed to distinguish it when you're not even technically using AI?
It's we've been using Grammarly for forever. You know what I mean? Like,
no, I agree. It's to me, it's,
it's definitely affecting the way I think about writing.
Yeah. I'm not going to let it change my writing.
Maybe not worse,
but like I'm almost trying to make sure that I'm like showing that it's human.
You know, like I was looking at like an app.
I'm sorry. No, go finish your thought.
Like I was looking at an application the other day I was like putting in and I
was like, Oh God, is this going to sound like AI?
Like should I make like some of the punctuate punctuations like worse?
Add five exclamation points.
I got,
I got feedback on the blog post internally about like places where it can be
improved or whatever.
The last thing you want to do is rewrite the whole thing.
I don't want to rewrite the whole blog post.
Cause I already have a nice solid flow here,
but I got feedback on the opening and I was like, okay,
I agree with this feedback.
And I was like working with an LLM to try to edit the opening without making that
spread through the document. Cause it's very easy for them to be like, Hey,
I edited your opening. And now that I edited your opening,
this time together. Yep. So I spent, I kid you not.
I spent about two plus hours with the LLM trying to like get that opening
edited without introducing it's crappy speak.
And I got to a place where I'm pretty happy with and the,
the tool that flags AI writing kind of flagged like half of the opening as AI
written and the other half as human written.
And like I thought that was a pretty good compromise because I spent a lot of
time like iterating with, with LLM to be like, okay.
I almost feel like I'm spending more time trying to sound human.
You know what I mean? And then like also like there's so much context,
like you have to give more context now because I feel like for one,
I'm either going to forget because we're working on so many different threads or
like I want the LLM to have context,
but then you feel like you're overly giving context.
And then a human has to read that too. It's like such a weird.
One thing for me especially is that I don't have any, I don't have any skills.
I don't use any skills like any AI skills or whatever. I don't use them.
I don't have,
I don't have a single one that I've written primarily because I find that the
models are changing too frequently for those skills to matter.
Plus the things that I'm working on can vary.
Like if I want to have a writing skill for the Kubernetes blog posts,
well that's actually different than reviewing technical documentation.
So the skill wouldn't actually even matter.
I kind of want to tell it on demand what I'm thinking of.
And this is where I use dictation. So not,
not the voice mode where it talks back to me,
but just where I talk to an input field. This is where I use dictation.
I'm just like, Hey, this opening,
the problem that I'm having with it is that this is wrong. That's wrong.
Let's let's fix it for this sort of flow and just dump all my thoughts in there.
And that allows me to get more into the context window than I would when typing.
Because honestly when I'm typing, I'm not typing paragraphs at the AI.
Like I'm really not, I'm typing two, three, four sentences.
I'm not typing seven paragraphs and then hitting enter.
But with dictation you can kind of type seven.
You throw a lot at it. Yeah.
Yeah. And I think that helps.
One of the things we landed on or at least I personally landed on,
we had internal conversations like how do we make things sound less AI written?
And I was like, it's not about what's written.
It's about the process of writing it. And I find that if you,
if you did the full first draft yourself and then had AI edit it,
it'll sound a lot less like AI than if you start with AI,
give me an outline and then I'm going to write it.
Like those will sound differently. And,
and going first of you doing the hard work of staring at the blank page and
writing the thing down.
It's even more upsetting though when it comes out,
like flagged as AI after you did do all the hard work and all you did was polish
it.
Personally for me, like I don't care if things get flagged or not.
It's like, cause I know my process.
And I'm like my process is if I'm writing long form or something that someone
else is going to read,
I'm going to write the whole thing first.
And I'm going to be as concise as possible that I think you should have all this
context.
And I'm going to write as few words as possible to save your time as a reader.
And then I'm going to have AI,
like help me figure out the things that like I understand that I should add in
there. Like, Oh yeah, this context may be missing or whatever.
And like maybe move around some paragraphs, but like, I,
I try to use it as a editor, not a writer,
which I think is roughly the like oxide RFD of like, yeah, it should be.
That's essentially what it comes down to. Yeah. Use it as an editor,
not as a writer, respect your, your readers, right?
Respect the audience that you're, that you are writing for. Um, for,
and that's,
that's really what it comes down to because we were having a chat internally
about writing like AI writing on pull requests and like reviews.
Do we do even,
do we ask a human to review a pull request if they're just going to like AI
review it and post the content?
Why not just ask AI to review it in general?
Yeah. For me, it depends.
I'm perfectly fine having AI agents spin up and do adversarial adversarial
review and all that stuff. That's good.
But if I'm working on something that needs to be designed and thought about,
there are multiple options. There's a difference for me. Yeah. So for me,
the approach I've been taking was, okay,
I want to make a change to the oxide control plane API,
but I know that there's one of like three different ways that this change can be
made and they are,
they all have slightly different designs and slightly different constraints.
I'm going to use AI to prototype one of them.
And then from there I'm going to talk to my human colleagues and say, Hey,
how does this feel? Like if you were using this API, like how,
how does this feel to you?
And if you want to use AI to critique it for certain things, go for it.
But for me,
it's more important to get the initial design out there and get it critiqued
because I'm not going to waste my time polishing something that we're just
going to say, Hey, drop it. Yeah.
See to me, I don't mind it summarizing a pull request or like, you know,
doing a review because I love to have two different adversarial reviews,
but like when it comes to design and architecture, those are sacred to me.
Yeah. So it's like, but me as a human,
am I going to think through the design and architecture in totality?
Not exactly, because that'll take a long time.
I'd rather get a working prototype, test how it feels, and then be like,
actually, this sucks. We're going to go this other way.
And AI lets me move faster in that regard. And then once we did,
once we finalize that design and we say, Hey, yeah, I like the shape of this,
then I'm going to give that PR a nice final pass and polish it up and remove all
the AI generated garbage. Right. But until then,
if we haven't finalized on the design, I'm using AI to iterate on this. Sorry.
I also think it depends on how much time you have and how many things you're
working on too. Yeah. Yeah, exactly. That's another thing.
I have other things to do. I cannot spend all day on this one issue.
Yeah. Not when the design hasn't been finalized yet.
So I use it as a tool for that regard. So that's kind of where I'm at.
Use it as a tool to free up my time for other things,
especially like personal life things.
And that's where the Spanish learning came into play. I was like, wait,
this thing could just talk to me in whatever language I want. Let's,
let's talk Spanish, man.
And that's really cool though,
because you're learning something else and it's not just like offloading,
you know? And I also think it's great rubber ducky when you're like,
kick the tires on this, like find all the edge cancels.
I'm more excited. Like, I don't want to sit at a screen clicking. Like,
what is this image? La bicicleta, click. Like, I don't,
that's too slow for me. I want to, I want to talk to a, a human sounding,
natural sounding thing that I can move faster.
And then I have my daughter in front of me pushing her on the stroller.
And I'm like, I'm walking in the park with my daughter.
What am I seeing in the park? Give me some words. Arboles,
that's trees like Pasto, that's grass. And it's just,
that's how I can look at,
I can look at her and I can just be like, Mira los arboles. And she's like,
you know, just laughing at me. You know what I mean?
And like it's, it's not exactly daddy daughter bonding time,
but it is, you know, like it actually is.
Life is a lot like trying to be a good professional and trying to be like a good,
like dad and all the things. So I honestly,
like if you can make it work where you're still being productive, but like,
I'm actually really proud of you for like making that time to like do the
outside. Cause like, it's really easy to lose, you know,
like focus and just be like,
I have to work really hard and I have to do all these things.
Cause look at the pressure that we're all going through, you know?
And that's what I was doing. I was working at my desk all day, every day for,
till late, late in the night. And I'm like, what am I doing here?
I need to go get exercise. I can't, I can't do that. I need to be good.
Let me go take my daughter out. Like instead of holding her here.
Professionally though, like, because like when you're not,
your body's not getting exercise and you don't feel good and you like,
you know what I mean? Like,
I feel like I used to think like when my kids were like older or needed less
help, I would be a better engineer.
And then when I got my first long stretch with no kids,
I was so bad at it because I wasn't taking breaks.
So I was just like stuck constantly.
Exactly.
You need to give your body some grace and like exercise it in other ways than
just work. And this is where like at work,
the nature of the work that I do at Oxide is very,
it can be context switching heavy and it's across multiple different repos that
are in different domains. I could be working on Terraform,
Packer, Kubernetes, OpenTelemetry all in one day.
And those are all vastly different things in different domains.
So being able to context switch between them efficiently and using AI to help me
pick up the pieces is really helpful for me. And like it's not,
and then you can say, Hey Matt, well,
why don't you hire more people just to focus on those things?
Because then you risk the silo creation again,
where if you're just focused on one aspect, you're not talking to one another.
Someone has to look at the holistic picture as well and see how that maps.
And like, that's, that's where I'm at.
Like when I put this blog post out there,
I got some people that reach out to me like,
why don't you just run Kubernetes on bare metal? But why don't you just,
why don't you just like,
I thought about everything that you all have said so far,
but I also have other things to do. You know what I mean?
Like we have to be thinking incremental building here.
We can't build everything perfect right off the, right off the bat.
We have to be able to incrementally build.
And it's hard to convey that in a blog post where it's like,
why did you stop here, Matt? You know, floating IPs don't work for that.
I know they don't work for this, but we had nothing. You know what I mean?
We had nothing before. Now we have something. So.
I would say one of the things that I've changed in my writing over the years is
I've removed just so many times in so many, especially written contexts,
because like people I've had feedback in the past that like,
I sound harsh, like typing in Slack when I'm not talking to someone.
And I'm like, yeah, I was like, why is that?
And I found myself like saying just in literally just like belittling their
thought process of like, well, why don't you just do this? I was like, Oh wow.
You've spent a lot of time probably thinking about this.
Like there's no just in here.
Like this is a hard problem and I need to respect that and not like
belittle the effort you've put into this.
I think I've gotten,
I've gotten similar feedback and I've also I'm trying to not let it get to me
either. Cause it's so, it's so easy.
When you are like in a generalist mindset,
cause you're working across various different things and you're trying to move
them forward.
And then you're talking to like someone that's only had a specialist mindset.
And then they're like, how dare you not solve this in totality? It's like, dude,
I solve this and five other things. Like, yes, come on bro. You know,
like really we're going to, we're going to do this.
I think those are two different mentalities and they need,
they both are valued, but I think it's hard to make them.
It's you got to find like a kind of middle ground on how they can speak to each
other.
Yeah. And it goes both ways, right? It's not like,
it's not like a generalist sort of thinker. Can't talk to a special,
they do it to specialist thinkers too, where it's like, Hey man,
why don't you think about this other far reaching thing?
It's because they've been so focused on a specific problem.
They actually weren't looking around. So this is why you need a mix of both.
You're like, I didn't know our customers used it in space. I'm sorry.
Yeah. Yeah.
But also I think that like a lot of it goes back to just being able to
experiment and share ideas and like have that psychological safety.
Because like,
if you don't have like a covenant of being able to like share ideas and get
feedback and not be like, Oh God, why didn't you do it this way?
Why didn't you do it that way? Like it breaks down your, like the, like,
what is the word I'm looking for?
Like being able to contribute and like get the feedback because you're like,
instead you're trying to like, make sure you've thought about it.
You're trying to survive.
Yeah. I think, yes.
And like, if you're trying to survive,
you can never be open and communicative and like share ideas freely.
And I think the more that everything gets so competitive and like hunger games
with like engineering and all of this stuff. Like, I think like, just like,
that's wild. Like you put it like time into a blog post and somebody decided
that they were going to come back and say, why didn't you do all these things?
Like, I feel like we're like,
everybody's trying to have the pissing contest of I know everything. And I'm
like, that's, but that's not why we're here.
Like we're learning out loud to get feedback and to share the process with each
other. Why can't we just do that?
It's it's diversity of thought is really important.
Being able to be vulnerable and that feedback culture that I was selling,
talking about earlier in the show, we don't have that in engineering.
I think we've really lost that.
We've lost it.
I think I want to go back to being a solutions architect cause I'm just tired of
having the pissing contest.
Like cause I don't mind if you tell me my work sucks. I actually don't care.
I actually prefer you to come up to me and be like, Matt, this sucks.
Like what's going on here? Let's talk about it.
It's all about why it sucks though,
because I feel like that's a growth moment if you can say it right. You know,
if you can come at it the right way,
but if you're not allowed to grow and be vulnerable,
you can't get the opportunity to grow or share any of that stuff.
That's where like much like talking out loud in the park to an AI agent feels
weird. And people are going to look at me like I'm weird. I don't care.
You know, cause you won't,
you don't know why because they're not the ones trying to learn Spanish.
They're not the one that has the linear ticket assigned to them. I do.
So like I'm going to get this going and I don't really care because your
perception of me in the park of talking to myself doesn't matter.
Also you shouldn't care because you're winning at life.
Cause you got the cute kid.
You're spending time with them getting that linear ticket done and you're
getting some spent. Like,
I mean most people are not doing a good job at doing that balance and finding a
way to fight for their life, you know,
like the actual life part and do good at professional.
So I think like people are always like, well,
you shouldn't do that or you shouldn't want this remote position or you
shouldn't do that. And I'm just like,
I don't really care if you feel that way because I know what I'm trying to
achieve in life.
I have no problem like quote unquote looking stupid in front of people by
asking a question. I don't care. I'll go on a random GitHub issue and be like,
wait a minute, how does this actually work or what's going on here?
I'm happy to do that.
And now with AI I can do it a little less cause I can just ask AI to help me
with that stuff.
But I find not asking the question or not doing those things to be far worse
than being afraid of some feedback.
I think we're really going to lose that though.
Like I really think that we're losing so much like knowledge sharing and with
all the tribal knowledge,
these companies are losing and everybody's too scared to be vulnerable and have
that knowledge sharing and to like collaborate.
I really feel like we're losing a lot of,
I guess invention and just kind of like things.
All of the problems are going into individual contexts of AI conversations with
people.
And we're not putting it on Stack Overflow anymore.
Think about it. People are not going to forums.
I cannot stand that everything's going hyper towards the individual.
This is why I like to post like my AI,
like this is why I want to move the AI context out of anything that's on my
local machine, put it in the linear issue. I don't want it.
I don't want the context cause it's not mine. It's out of my hands.
It's the team's context.
And like I don't think many people do a good job of this and maybe I'm not even
doing a job of it. Cause you know,
if my team is not looking at the linear context and whatever.
Well, and they can't read the linear in Spanish. So it's, yeah, exactly.
It's just, I don't want it to, I don't want it.
I don't want to pull things towards me. I want to like put it in the open.
And the other thing I wanted to say is that at some point you have to call
certain things done for now and move forward.
Perfection's the enemy of good sometimes.
And that's,
that's another thing of feedback around the Kubernetes blog posts and whatever
else. They're like, Hey, you know, you can do more here. I'm like, yep.
But I have to ship something.
Well that's what it used to be like at AWS.
Like there were so many rounds of review that it took a year to get a blog post
out, you know? And you're just like, bro, it's, it's old already.
I had, I had blogs that shipped at Disney when I didn't work there anymore.
And I was like, this is not fun with that.
I am not someone who's like,
I want to sacrifice rigor wherever possible for speed. I like,
that's not how I live. But also this is technology.
If I ship you a bug right now, I can fix it in two seconds.
But so why am I so worried?
We're so, yeah, we're losing that. Like, dude,
like by the time you sit here and like, and like agonize over this,
like you could have already fixed the situation and we could have iterated it to
make it more optimum like later.
Balancing that has been one of the harder parts of my career of like how much
upfront discussion and planning do we do versus like getting something out there
and getting feedback and iterating quickly.
Cause both have their pros and cons.
And I tend to fall on the side of like, let me get you something out there.
If it doesn't work perfectly, let's talk about it.
And fix it where possible. Cause I can cut, I can cut your release right now.
I can't like, why is this hard?
That gets customers moving though. You know what I mean?
It gets people unblocked and then you make it better and you get that feedback
later. And I think like,
one of our customers had a problem with a tariff provider where,
where their plans were taking 40 minutes because of the data structure for like
how the VPC firewall rules worked. And I was like, yo, this sucks.
And what am I going to tell them? Sorry, get to it in the next release.
See you in Kubernetes 1.37 bro. For real?
You're telling us that we have ultimate software power at our hands.
We can't cut a release like that.
Like what are we saying? And so I talked to this customer and I was like, Hey,
I just need a minute to think about the design of this.
And then once I settled on the design, I'm like, here, patch, here's your patch,
man. Have fun. Like, let me know how this works.
I didn't have to wait to the next release cycle for this. That's, that's a,
that's like a slap in the face to the customer. Say, Hey, sorry,
I know this problem is critical to you,
but see you next quarter when we convene for our release cycle.
Like, but I thought you were a software engineer.
Putting your process over the customer is.
That's that being customer-oriented and like, and that makes money.
You know what I mean? And I'm not saying to be reckless. That's like, I want,
I want to make sure that that's being heard by the listeners.
Like I'm not saying be reckless. I'm saying,
if a customer has a hard pain point,
you don't need to wait for your next release cycle to fix it.
But I'm not on call for their production. It's fine. Be reckless. Yeah. Well,
yeah, there's that. You are in control.
Do you think we're losing like because of the lack of like talking to each other
because we're talking to AI more and the lack of like stack overflow and ask
questions and all these different like websites that people used to use.
Do you think that's going to affect engineering as we go forward?
Not as much learning out loud and solving problems and getting stuck with other
people?
I think it's going to affect it. I mean,
we're already seeing it to some regard or to some effect where, Hey,
I built this software. No, you didn't. Claude did actually. You didn't want to,
you didn't say that though. You said I built this software.
And really what you did was you prompted, Hey, make me a so-and-so clone.
And then now instead of talking about how you designed and built this thing,
you're just like sticking it on the wall and say, Hey, look, here you go.
Problem solved. And then it's on the reader. I guess,
I guess what I'm trying to say is we're putting too much burden on the reader
now rather than sharing things from the writer's perspective.
I think that affects engineering negatively because now what do I get as a
reader?
I open up social media and I get bombarded with all of the projects people
vibe coded or built. And now if I find interest in any one of them,
there's nothing really written down about it. I have to,
I have to use my LLM to scan it.
So I think that nature of like relying on community to share information with
one another changes. And now it's,
it's becoming where people are expecting you to use an LLM to get up to speed
instead of sharing their thoughts and talking about it.
And I think that's a little sad.
Well, and they're not being like thinking of their reader. They're not,
they're not making it easier for the reader to understand.
So when it is all generated from the writer's perspective and they're like,
I don't care about your time,
then you have to now spend your tokens to go figure out what all this means.
And, and that's very.
So I think we'll see, I think conferences and meetups will be the,
the, what's the word I'm looking for? The middle ground here.
I think those will make some of what of a resurgence in a way,
because I think people still yearn for that.
It's exciting to be around other people that are excited about the same things
you are, right? Like going to a hockey game yourself.
It could be fun if you're into hockey,
but going to a hockey game with other people that are into hockey is like,
yo, this is amazing. I love this. Yeah. You know?
And it's the same thing with our, with our jobs. Like sure.
I can post on socials and say, Hey, I'm building this really cool project.
Like come talk to me about it and I'll get some people.
But like being in the same room as people that are excited about these things as
you are and just iterating with one another and like riffing off one another,
you can't replicate that with LLMs or over remote connections.
You have to be with people.
So I think that's where we'll start to see these ideas and these,
the community be focused because nobody's incentivized to write a stack overflow
answer anymore. Nobody's incentivized to write a Reddit post anymore. Heck,
nobody's incentivized to write a blog post anymore unless they're trying to grow
their personal brand. Yeah. So like,
and even then they're not getting people to read it.
They're getting LLMs to summarize it.
They're just writing for themselves and posting it.
Which is always how you should write. In my opinion,
you should always be writing for yourself.
Totally. So I think that's just a natural consequence of where we're headed.
And I don't love it.
And that's why I've been trying to work on like my personal professional
relationships and making sure I'm still talking to people because yeah,
it's sad that this sort of communal knowledge is being moved essentially.
And not to end on a downer, but you know,
where can people find you online to talk to you? I know you have fall through FM.
Oh yeah. Podcast.
We have your blue sky in the blue sky starter starter pack for FAFO FM.
So if people join the starter pack or look through it,
they can subscribe to everyone in there or else.
Yeah. Matthew Sanabria.com is the easiest place cause everything's listed on
there. Look, find me there. And my email's there.
You can send me an email if you want to chat.
I'm actually really good with email so it will not be lost.
I'm an inbox zero type of person.
Matthew. Thank you. Thank you so much for coming on.
And for everyone listening, thank you. We will talk to you again soon.
Bye.
Thank you for listening to this episode of fork around and find out if you like
this show, please consider sharing it with a friend, a coworker,
a family member, or even an enemy.
However we get the word out about this show helps it to become sustainable for
the longterm. If you want to sponsor this show,
please go to F A F O dot F M slash sponsor and reach out to us there about what
you're interested in sponsoring and how we can help.
We hope your system stay available and your pagers stay quiet.
We'll see you again next time.
Okay.