Laughing Through The Pain with David Crawshaw

1
00:00:05,520 --> 00:00:10,400
Welcome to Fork Around and Find Out, the podcast about building, running, and

2
00:00:10,400 --> 00:00:12,220
maintaining software and systems.

3
00:00:20,320 --> 00:00:22,960
Hello, and welcome to Fork Around and Find Out.

4
00:00:23,300 --> 00:00:26,840
I am your host, Justin Garrison, and with me as always is Autumn Nash.

5
00:00:26,860 --> 00:00:27,640
How's it going, Autumn?

6
00:00:28,240 --> 00:00:29,400
This is delightful.

7
00:00:29,400 --> 00:00:30,360
That's how it's going.

8
00:00:31,340 --> 00:00:34,640
We've been in this like this room for 10 minutes now.

9
00:00:34,900 --> 00:00:36,720
And the guest on the show is David Croshaw.

10
00:00:36,780 --> 00:00:38,380
Thank you so much, David, for coming on the show.

11
00:00:38,740 --> 00:00:39,520
Thanks for having me.

12
00:00:40,060 --> 00:00:41,400
So this has been amazing.

13
00:00:41,580 --> 00:00:42,720
All of us have had a rodeo trip.

14
00:00:43,060 --> 00:00:46,020
And so I had to hit record just to make sure that we don't miss out on anything

15
00:00:46,020 --> 00:00:47,200
else, because this is too good.

16
00:00:47,580 --> 00:00:48,980
Like my eyeliner is smearing.

17
00:00:51,100 --> 00:00:54,840
Yeah, go, go subscribe on, on YouTube, or we actually have an

18
00:00:54,840 --> 00:00:56,580
Instagram and we put the clips on.

19
00:00:56,580 --> 00:01:00,700
There is a dedicated Fork Around and Find Out on LinkedIn, which I just love

20
00:01:00,700 --> 00:01:04,880
that, like, it's like a business LinkedIn account, which we post clips on there.

21
00:01:04,960 --> 00:01:07,860
So if you're not subscribed to like the social channels, we post clips of all

22
00:01:07,860 --> 00:01:10,900
these shows and they're super fun because it's all the little fun things about,

23
00:01:11,040 --> 00:01:13,540
you know, seeing Autumn's mascara smearing or something like that.

24
00:01:14,280 --> 00:01:16,020
Or the faces Justin makes.

25
00:01:16,920 --> 00:01:20,960
Right before he says something really shady and then he like smiles really big.

26
00:01:21,000 --> 00:01:21,500
And it's great.

27
00:01:21,600 --> 00:01:23,240
I'm glad that the world can see it now.

28
00:01:23,320 --> 00:01:24,180
I actually get a light bulb.

29
00:01:24,300 --> 00:01:25,380
It's like, it actually shows up.

30
00:01:25,460 --> 00:01:26,140
I can see it.

31
00:01:26,140 --> 00:01:26,620
It's great.

32
00:01:27,900 --> 00:01:33,140
But we, I met a grown adult the other day who was like half my age and said like,

33
00:01:33,220 --> 00:01:34,720
oh, I watched a podcast the other day.

34
00:01:34,840 --> 00:01:37,160
And I was like, no, that's what, what's happening.

35
00:01:37,560 --> 00:01:37,680
Yeah.

36
00:01:38,940 --> 00:01:41,580
Like, do you know the etymology of the word podcast?

37
00:01:41,700 --> 00:01:42,940
I just want to tell you.

38
00:01:43,340 --> 00:01:44,940
I am of an age that I do.

39
00:01:45,200 --> 00:01:45,360
Yeah.

40
00:01:45,660 --> 00:01:48,460
And so they did not, I'm not that old.

41
00:01:48,780 --> 00:01:50,480
And so it's a little bit of gray.

42
00:01:53,160 --> 00:01:56,920
But if you remember when the video iPod came out, then you're old enough to know.

43
00:01:57,820 --> 00:01:59,360
You know what's funny though?

44
00:01:59,540 --> 00:02:02,780
Like this, this grown adult I was talking to was born in 2006.

45
00:02:03,240 --> 00:02:05,520
And like, dude, see the, see the problem.

46
00:02:05,880 --> 00:02:10,040
Every time my little brother reminds me like what, like how, like when he was born,

47
00:02:10,140 --> 00:02:12,560
like I was graduating high school in 2006.

48
00:02:12,560 --> 00:02:15,900
I'm like, why are you like, you didn't have to do me like that.

49
00:02:18,260 --> 00:02:19,140
I know it hurts.

50
00:02:19,620 --> 00:02:19,820
Yeah.

51
00:02:19,820 --> 00:02:26,180
So, uh, on the topic of being old, um, David, uh, we wanted to talk about, uh,

52
00:02:26,400 --> 00:02:32,200
clouds being old, not you, but the industry and technology, uh, just being too old and

53
00:02:32,200 --> 00:02:34,880
needing, needing a revamp on how things work.

54
00:02:34,900 --> 00:02:39,220
And so you are one of the co-founders of tail scale, which in my opinion is like

55
00:02:39,220 --> 00:02:45,040
rebuilding the network for the internet or what networks should be on, on a, on a

56
00:02:45,040 --> 00:02:47,780
giant amorphous thing that we call the internet.

57
00:02:48,420 --> 00:02:49,620
That's how I feel about it.

58
00:02:49,620 --> 00:02:50,860
Network should be better.

59
00:02:51,900 --> 00:02:53,060
That scale improves the situation.

60
00:02:53,220 --> 00:02:53,320
Yeah.

61
00:02:53,420 --> 00:02:54,560
I'm a huge fan of tail scale.

62
00:02:54,580 --> 00:02:58,220
Can you tell us what you, what you, what tail scale is from your point of view?

63
00:02:58,280 --> 00:03:02,440
Cause I think sometimes our like listeners don't always know all the things Justin

64
00:03:02,440 --> 00:03:04,060
does, so that way.

65
00:03:04,380 --> 00:03:07,760
I spent six years trying to come up with an elevator pitch for tail scale.

66
00:03:07,900 --> 00:03:09,680
I still don't really have it right.

67
00:03:09,840 --> 00:03:11,480
So, you know, it's, can I, can I do it?

68
00:03:11,680 --> 00:03:12,740
Well, no, but I can try.

69
00:03:12,860 --> 00:03:15,040
Like tail scale is a network for my computers.

70
00:03:15,280 --> 00:03:17,920
Like they connected when I installed tail scale on them.

71
00:03:17,980 --> 00:03:19,220
Like they can ping each other.

72
00:03:20,120 --> 00:03:22,960
And without tail scale on them, they mostly can't ping each other.

73
00:03:23,080 --> 00:03:26,320
Like my phone is like on some Verizon thing.

74
00:03:26,540 --> 00:03:28,000
I can't ping that without tail scale.

75
00:03:28,100 --> 00:03:31,860
But once I'm, once I did, once I can, then, you know, I can open up, you know,

76
00:03:31,860 --> 00:03:34,140
port 8080 on my, on my Mac or whatever.

77
00:03:34,240 --> 00:03:35,040
I can just look at stuff.

78
00:03:35,460 --> 00:03:39,200
And like, when I started using computers, that's how they worked

79
00:03:39,200 --> 00:03:40,420
because we had a bunch of PCs.

80
00:03:41,060 --> 00:03:44,140
Uh, we didn't have the internet yet, but we set up a LAN between them.

81
00:03:44,340 --> 00:03:46,720
This was my father's medical practice back in the nineties.

82
00:03:47,460 --> 00:03:52,620
And, you know, we, it was IPX originally, but eventually we switched to TCP IP and

83
00:03:52,620 --> 00:03:55,420
they all had IP addresses and all the machines could talk to each other.

84
00:03:55,580 --> 00:03:57,160
And that to me is what networking is.

85
00:03:57,540 --> 00:04:01,320
Uh, and now we live in 2026 and everything's on the internet, but

86
00:04:01,320 --> 00:04:02,540
nothing can talk to each other.

87
00:04:02,840 --> 00:04:04,360
And that's kind of weird.

88
00:04:04,800 --> 00:04:07,300
And so tail scale is like, what if everyone could talk to each other?

89
00:04:08,300 --> 00:04:11,880
Oh, that was such a good, your elevator pitch is better than you think it is.

90
00:04:11,880 --> 00:04:14,720
Because I feel like a lot of times people can't explain what

91
00:04:14,720 --> 00:04:16,060
problem they're solving, you know?

92
00:04:16,060 --> 00:04:17,340
I really appreciate that.

93
00:04:17,520 --> 00:04:17,600
Also.

94
00:04:17,740 --> 00:04:20,260
I just totally winged that and I've never said that before, so

95
00:04:21,580 --> 00:04:24,140
you know what, save this podcast for later.

96
00:04:24,280 --> 00:04:25,720
So you can then use it again.

97
00:04:25,820 --> 00:04:26,580
Cause it was good.

98
00:04:26,800 --> 00:04:28,220
I'll enjoy watching this on TikTok.

99
00:04:31,600 --> 00:04:31,960
Perfect.

100
00:04:32,600 --> 00:04:34,980
Actually, we did close our, do we close the TikTok account?

101
00:04:35,080 --> 00:04:35,700
I think I did.

102
00:04:36,220 --> 00:04:36,580
Yeah.

103
00:04:36,680 --> 00:04:41,300
The fact that I tried to redownload it again the other day and I went to go turn

104
00:04:41,300 --> 00:04:45,460
off the location services after asked me a bunch of very scary questions about like

105
00:04:45,460 --> 00:04:50,920
we get your data, but like not passively, like actively, like every five minutes.

106
00:04:50,920 --> 00:04:52,500
I know what room of your house you're in.

107
00:04:52,660 --> 00:04:56,120
Like this is like not my neighborhood, but like I know exactly where you are.

108
00:04:56,220 --> 00:04:59,600
And then I went to in the settings and there's no turn off

109
00:04:59,600 --> 00:05:00,920
location services settings.

110
00:05:01,060 --> 00:05:02,060
It does not exist.

111
00:05:02,560 --> 00:05:02,740
Yeah.

112
00:05:02,880 --> 00:05:05,040
Well, I was like, no, I shut it down.

113
00:05:05,180 --> 00:05:05,940
I was like, you know what?

114
00:05:06,060 --> 00:05:06,420
Nevermind.

115
00:05:06,580 --> 00:05:07,340
That's I did too.

116
00:05:07,360 --> 00:05:08,920
But I was like, well, everybody's still on it.

117
00:05:09,000 --> 00:05:10,240
Maybe I should go like, look.

118
00:05:10,600 --> 00:05:12,800
And I was like, I feel like I'm losing, missing so much news.

119
00:05:12,800 --> 00:05:15,260
Like looked at this, like the agreement.

120
00:05:15,440 --> 00:05:16,220
And I was like, not today.

121
00:05:16,300 --> 00:05:17,700
Say nevermind.

122
00:05:19,440 --> 00:05:22,500
Which, which also by like a complete aside on the tail scale stuff.

123
00:05:22,500 --> 00:05:24,240
And we're going to move into the next thing.

124
00:05:24,280 --> 00:05:25,980
Tail scale up, I think is in August.

125
00:05:26,620 --> 00:05:28,620
This podcast is coming out in June.

126
00:05:28,780 --> 00:05:30,480
I am, I will be speaking at tail scale up.

127
00:05:30,520 --> 00:05:31,640
So if anyone's going, it's in San Francisco.

128
00:05:31,820 --> 00:05:33,060
I think the tickets are open now.

129
00:05:33,120 --> 00:05:35,120
You can sign up at least when this podcast is out for sure.

130
00:05:35,240 --> 00:05:36,820
Tickets are out and available.

131
00:05:36,860 --> 00:05:37,620
Maybe it's sold out.

132
00:05:37,640 --> 00:05:41,020
I have no idea, but if you're in the area, come stop by because I will be there.

133
00:05:41,280 --> 00:05:42,440
So yeah, that's fun times.

134
00:05:42,880 --> 00:05:44,460
We know that's not fun because you'll be there.

135
00:05:44,840 --> 00:05:46,700
Yeah, I will.

136
00:05:46,780 --> 00:05:47,860
I will build something ridiculous.

137
00:05:48,400 --> 00:05:48,800
Absolutely.

138
00:05:49,100 --> 00:05:52,320
On top of it, I think last year I spoke at tail scale up

139
00:05:52,320 --> 00:05:54,680
and I built my own game streaming service

140
00:05:55,800 --> 00:05:57,920
using moonlights from an AWS instance.

141
00:05:58,220 --> 00:06:00,540
I'm always like, what is he going to put in the like

142
00:06:01,140 --> 00:06:02,600
kitty backpack today?

143
00:06:02,780 --> 00:06:04,420
I actually did the entire

144
00:06:05,120 --> 00:06:08,440
entire presentation from a steam deck, which was which was also fun

145
00:06:09,160 --> 00:06:11,740
just just because it was a new toy and it was fun to do.

146
00:06:12,340 --> 00:06:14,300
But let's talk about the next thing you've been working on,

147
00:06:14,360 --> 00:06:15,920
because you're building a brand new cloud.

148
00:06:16,080 --> 00:06:17,960
And that's the thing that is too old now.

149
00:06:18,640 --> 00:06:19,780
Existing clouds suck.

150
00:06:20,520 --> 00:06:24,080
And I am a huge fan for anyone that builds a service

151
00:06:24,080 --> 00:06:26,820
that has no app or unique interface.

152
00:06:26,940 --> 00:06:31,280
And it's just like just when I saw SSH into exe.dev, I was like, I'm in.

153
00:06:31,880 --> 00:06:33,460
I want to I want to see how this works.

154
00:06:33,900 --> 00:06:35,980
Can you explain exe.dev to us?

155
00:06:36,480 --> 00:06:37,520
Yeah, exe.dev is a cloud.

156
00:06:38,200 --> 00:06:39,820
Yeah, it's computer elevator pitch right there.

157
00:06:39,820 --> 00:06:44,000
Yeah, I know it took us took us a while, but we finally got around to like,

158
00:06:44,140 --> 00:06:47,240
like it's very tempting to try and not say you're building a cloud

159
00:06:47,240 --> 00:06:48,300
because like that's too big.

160
00:06:48,480 --> 00:06:49,620
You can't possibly do that right.

161
00:06:49,720 --> 00:06:50,720
I'm doing some smaller thing.

162
00:06:51,060 --> 00:06:53,920
But like, let's just go right at it and say what we're doing.

163
00:06:54,440 --> 00:06:56,060
So, yeah, we put computers in data centers

164
00:06:56,060 --> 00:06:58,760
and then we let you use those computers for money.

165
00:06:59,360 --> 00:07:00,400
And that's the service.

166
00:07:00,500 --> 00:07:01,640
And I think that's what a cloud is.

167
00:07:02,480 --> 00:07:05,440
And yeah, clouds are not in great shape right now.

168
00:07:05,500 --> 00:07:06,460
And that's why we're building one.

169
00:07:06,640 --> 00:07:09,020
Like every time I try to use one, I don't enjoy the process.

170
00:07:09,020 --> 00:07:11,840
I gave them all a try before building this one and said,

171
00:07:11,920 --> 00:07:12,880
yeah, I don't want any of these.

172
00:07:13,580 --> 00:07:14,700
I've always thought that.

173
00:07:14,780 --> 00:07:16,520
Actually, I thought that for at least 10 years now.

174
00:07:17,840 --> 00:07:22,800
It's gotten slightly worse over that time as clouds continue to,

175
00:07:23,060 --> 00:07:25,360
you know, change exactly who they're for.

176
00:07:25,680 --> 00:07:28,300
I would claim clouds have never really been designed for developers

177
00:07:28,980 --> 00:07:33,400
because fundamentally the people buying resources and clouds are not developers.

178
00:07:33,920 --> 00:07:36,240
They're companies that are looking for compute.

179
00:07:36,240 --> 00:07:40,940
And, you know, if they can save a certain amount of money on a cloud product

180
00:07:40,940 --> 00:07:43,880
by making developers do a bit more work to use an even worse API,

181
00:07:44,520 --> 00:07:46,240
they will, if like the dollars add up.

182
00:07:46,560 --> 00:07:48,680
And so, you know, the purchaser is not the developer.

183
00:07:48,860 --> 00:07:50,260
And like it really shows with clouds.

184
00:07:50,700 --> 00:07:56,800
You know, I have a command to like LS my AWS VMs so I can see them in EC2.

185
00:07:56,920 --> 00:08:00,580
It's like four lines long and involves like parsing JSON several times.

186
00:08:00,840 --> 00:08:02,320
I'm like, why?

187
00:08:02,540 --> 00:08:03,620
Like, I just want a list.

188
00:08:03,740 --> 00:08:04,420
Could I have a list, please?

189
00:08:04,420 --> 00:08:06,460
I don't understand why this stuff is hard.

190
00:08:06,860 --> 00:08:07,680
And, you know, I kind of do.

191
00:08:07,960 --> 00:08:08,840
There's a lot of path dependence.

192
00:08:09,000 --> 00:08:09,600
The stuff is old.

193
00:08:10,300 --> 00:08:13,040
And so I wanted to build a cloud for a long time, and it's never made any sense

194
00:08:13,040 --> 00:08:17,980
because for exactly the reason I said, the developers don't buy clouds.

195
00:08:18,500 --> 00:08:19,800
You know, they're not the big spenders.

196
00:08:20,160 --> 00:08:21,320
And so why would you build one?

197
00:08:22,140 --> 00:08:26,120
And the answer is things are changing and they're changing because of agents.

198
00:08:27,380 --> 00:08:28,880
We're going to write a lot more software.

199
00:08:29,140 --> 00:08:32,260
It's going to be agent driven and agent written and agent deployed.

200
00:08:33,260 --> 00:08:36,980
And what is best for agents is not the clouds that we have right now.

201
00:08:37,120 --> 00:08:39,080
What is best for agents is the big question.

202
00:08:39,540 --> 00:08:42,440
And my co-founder Josh and I spent a while working on this problem

203
00:08:42,440 --> 00:08:43,760
and we finally figured it out.

204
00:08:44,080 --> 00:08:46,820
What's best for agents is actually just what's best for developers.

205
00:08:47,980 --> 00:08:50,700
And there's a really simple reason for that, which is

206
00:08:53,640 --> 00:08:57,980
models that agents are powered by are trained on transcripts of developers working.

207
00:08:58,940 --> 00:09:00,760
That's that reinforcement learning cycle.

208
00:09:00,980 --> 00:09:02,000
That's that's where it comes from.

209
00:09:02,260 --> 00:09:06,680
And so, you know, it's whatever is best for a developer is best for an agent.

210
00:09:07,200 --> 00:09:10,500
And so none of the clouds are best for agents because they're not made for developers.

211
00:09:10,920 --> 00:09:13,200
So the idea is if we build something we actually want,

212
00:09:13,580 --> 00:09:15,900
we're also building something that agents actually want.

213
00:09:16,560 --> 00:09:20,520
And so we have this, you know, great reason to go and build all this infrastructure.

214
00:09:20,700 --> 00:09:23,360
And so the real focus initially is developer environments.

215
00:09:23,780 --> 00:09:26,780
You know, the idea is use XE.dev for writing software.

216
00:09:28,720 --> 00:09:31,300
So that's the that's what we're doing and why we're doing it.

217
00:09:31,800 --> 00:09:34,060
Now, I feel like there's a.

218
00:09:34,520 --> 00:09:38,000
I don't know, a shift in what people were thinking

219
00:09:38,000 --> 00:09:40,540
maybe last year or the year before, what agents wanted, right?

220
00:09:40,580 --> 00:09:45,280
Like agent wanted APIs and clouds made sense there because you get APIs

221
00:09:45,280 --> 00:09:50,260
and like MCP is basically like, what if my my shell commands were an API?

222
00:09:50,760 --> 00:09:53,420
Right. Like that's basically what MCP kind of is doing for a lot of folks.

223
00:09:53,440 --> 00:09:54,260
They're just like, let me just show it.

224
00:09:54,280 --> 00:09:57,080
And like agents talking to MCP servers was all about

225
00:09:57,080 --> 00:10:00,040
how do I get this agent to talk to an API that is better

226
00:10:00,040 --> 00:10:02,200
than the output of FFmpeg slash help?

227
00:10:02,640 --> 00:10:04,660
Right. Which is which is sometimes hard to parse.

228
00:10:05,440 --> 00:10:09,720
But agents are generally or at least LLMs in general kind of figure it out.

229
00:10:09,800 --> 00:10:13,080
But in a lot of cases, like there's an FFmpeg MCP server

230
00:10:13,080 --> 00:10:16,700
that gives you more verbose information around that.

231
00:10:16,900 --> 00:10:20,540
Why isn't that the right solution of exposing

232
00:10:21,660 --> 00:10:23,240
LLMs or agents to APIs?

233
00:10:25,720 --> 00:10:27,580
Because it's not what developers are trained on.

234
00:10:28,140 --> 00:10:32,540
You know, it just comes back to those those those reinforcement learning loops.

235
00:10:33,480 --> 00:10:37,420
Developers type stuff into the command line and the command line is a great API.

236
00:10:38,020 --> 00:10:39,680
And you can see this with agents today, right?

237
00:10:39,700 --> 00:10:42,500
Like if you take the playwright MCP and plug it into your agent

238
00:10:42,500 --> 00:10:44,960
so that it can, you know, for cloud code or something

239
00:10:44,960 --> 00:10:48,120
so that it can use Chrome while you're doing some web work,

240
00:10:48,180 --> 00:10:49,540
which is really good, by the way.

241
00:10:49,680 --> 00:10:53,320
Like if you're if you're ever trying to like get cloud code to create some CSS

242
00:10:53,320 --> 00:10:55,260
for you, it's just like a person.

243
00:10:55,260 --> 00:10:58,380
It'll like make some CSS and then you try looking at it and it doesn't work

244
00:10:59,220 --> 00:11:00,680
because, you know, the alignment's wrong.

245
00:11:01,700 --> 00:11:05,280
And with a playwright plugged into it, with Chromium plugged into it,

246
00:11:05,900 --> 00:11:09,160
cloud code can open it in Chrome, take a screenshot,

247
00:11:09,840 --> 00:11:12,620
look at the screenshot and say, oh, yeah, nothing's aligned correctly

248
00:11:12,620 --> 00:11:13,740
and then go and fix it.

249
00:11:13,880 --> 00:11:16,020
And so you get this wonderful feedback loop from plugging playwright

250
00:11:16,020 --> 00:11:18,940
into cloud code or, you know, some other equivalent Chromium system.

251
00:11:19,760 --> 00:11:21,700
And yeah, you can plug the playwright MCP in,

252
00:11:21,780 --> 00:11:25,180
but you can also plug the playwright CLI in and it works just as well.

253
00:11:25,640 --> 00:11:28,920
And, you know, if you try measuring like how many tokens

254
00:11:28,920 --> 00:11:30,540
does it take to solve a problem, either way,

255
00:11:30,800 --> 00:11:33,340
it's about the same once you both get them working.

256
00:11:33,520 --> 00:11:36,100
Only I can get the playwright CLI working faster.

257
00:11:36,780 --> 00:11:40,960
And so, you know, I don't see the fundamental advantage

258
00:11:42,160 --> 00:11:44,620
for agents in MCP from a

259
00:11:45,520 --> 00:11:46,340
easier use.

260
00:11:46,560 --> 00:11:49,960
The advantages of MCP is that for us, not for agents,

261
00:11:50,080 --> 00:11:51,800
it can be easy to put auth on that stuff.

262
00:11:52,540 --> 00:11:54,500
And so it can be easier to expose services

263
00:11:54,500 --> 00:11:56,380
that are otherwise just really difficult to expose.

264
00:11:56,540 --> 00:11:58,760
And so, you know, MCP potentially has a lot of value.

265
00:11:58,960 --> 00:11:59,940
I'm not saying don't use it,

266
00:12:01,020 --> 00:12:04,240
but it's not fundamentally better than a command line interface.

267
00:12:04,620 --> 00:12:08,220
I actually think that using AI and command line interface is easier

268
00:12:08,220 --> 00:12:09,580
for me just in general.

269
00:12:11,220 --> 00:12:13,320
Me too. It's easy for me to know what's going on, right?

270
00:12:13,520 --> 00:12:13,820
Exactly.

271
00:12:14,760 --> 00:12:17,760
Well, none of those command line interfaces were ever designed for

272
00:12:18,400 --> 00:12:21,500
something like an agent or like there's no auth built into

273
00:12:21,500 --> 00:12:24,520
pretty much any Linux CLI tool or at least standard CLI tool.

274
00:12:24,660 --> 00:12:28,280
So the way you're getting around that is basically more sandboxes, right?

275
00:12:28,340 --> 00:12:31,120
You just say, actually, I'm going to give you full access in a VM

276
00:12:31,880 --> 00:12:35,600
and go have fun because there's no files there for you to destroy or.

277
00:12:35,740 --> 00:12:36,860
Which is like really hard, too,

278
00:12:36,900 --> 00:12:39,700
when you're trying to interact with multiple things like what I'm.

279
00:12:40,360 --> 00:12:43,880
I feel like that's like one of the biggest barriers to using agents

280
00:12:43,880 --> 00:12:46,940
because I'll be committing to open source, but then wanting to commit

281
00:12:46,940 --> 00:12:50,000
to like a proprietary repo or like my own personal repo.

282
00:12:50,000 --> 00:12:53,960
And then the off between all those three of those things is so difficult.

283
00:12:54,160 --> 00:12:59,800
And then you're trying to like be secure and not just give it access to everything.

284
00:12:59,800 --> 00:13:03,220
And then sometimes you can't give it access to the one thing you want to give it to.

285
00:13:03,560 --> 00:13:05,300
Yeah, absolutely. I agree.

286
00:13:05,460 --> 00:13:08,820
That's actually the hardest problem with agents is setting them up so they can work.

287
00:13:09,340 --> 00:13:14,400
Yes, it's it's hard because agents have to be able to do it themselves.

288
00:13:14,400 --> 00:13:17,760
They have to have their own feedback loop without you in the middle saying,

289
00:13:17,920 --> 00:13:19,420
I ran the tests and here's the error.

290
00:13:19,420 --> 00:13:23,220
Because otherwise you just sit there doing that 30 times a day,

291
00:13:23,300 --> 00:13:26,040
but on four minute intervals, which is the worst.

292
00:13:27,040 --> 00:13:29,440
That's and yeah, so you've got to figure out how to set that up.

293
00:13:29,480 --> 00:13:31,500
And right now it's a ton of work to do all that stuff.

294
00:13:31,640 --> 00:13:35,300
And it's it's, you know, we've spent actually

295
00:13:35,300 --> 00:13:39,100
we've spent 18 months thinking about this problem and trying various things.

296
00:13:39,520 --> 00:13:41,920
And I don't think there's any easy solution here.

297
00:13:42,060 --> 00:13:45,040
I think we have to rebuild a bunch of stuff and it's going to be pretty painful.

298
00:13:46,460 --> 00:13:49,060
That's where do you see people using agents?

299
00:13:49,420 --> 00:13:51,460
You know what I mean? Like, where do you see?

300
00:13:51,580 --> 00:13:54,100
Because everybody wants to be like AI is going to replace people.

301
00:13:54,300 --> 00:13:56,340
And then people are like, we hate AI.

302
00:13:56,480 --> 00:13:57,560
Then they're like, we love AI.

303
00:13:57,760 --> 00:13:58,460
And like, you know what I mean?

304
00:13:58,960 --> 00:14:02,320
Yeah, like you have a more measured approach from what I've seen on your blogs.

305
00:14:03,020 --> 00:14:06,700
Because I think my opinion of AI even has changed over the last year.

306
00:14:07,640 --> 00:14:10,560
Where with like currently, like where do you see people

307
00:14:10,560 --> 00:14:13,520
like using this technology that you're building the cloud for?

308
00:14:13,860 --> 00:14:16,060
I mean, it's good your opinion has changed over the last year

309
00:14:16,060 --> 00:14:18,800
because AI has changed a lot over the last year.

310
00:14:18,800 --> 00:14:20,880
I feel like it's changed a lot in the last couple of months.

311
00:14:21,180 --> 00:14:23,520
Yeah, it was it was somewhere near the end of last year.

312
00:14:23,620 --> 00:14:27,120
So like October, November, depending who you ask, where models got really,

313
00:14:27,320 --> 00:14:29,540
really good at the tool calling loops for agents,

314
00:14:30,120 --> 00:14:35,560
where agents went from like writing 20% of my code to almost all of my code.

315
00:14:35,920 --> 00:14:37,940
You know, it was a really huge shift.

316
00:14:38,220 --> 00:14:40,480
And, you know, it changed the way I think about them.

317
00:14:40,740 --> 00:14:43,520
Are you ever scared that you're going to like lose your muscle to write code?

318
00:14:43,600 --> 00:14:47,380
Because like, I think like not to plug the company I work for,

319
00:14:47,380 --> 00:14:48,620
because it's not about that.

320
00:14:48,700 --> 00:14:52,120
But I've been using GitHub CLI more because of the CLI interface

321
00:14:52,120 --> 00:14:54,920
and being able to like see what it's actually doing,

322
00:14:55,040 --> 00:14:58,260
because I feel like I learn the commands that I don't know,

323
00:14:58,420 --> 00:15:01,900
but also just kind of knowing and tracking where it is and what it's doing.

324
00:15:02,020 --> 00:15:04,060
And I feel like it's slightly less scary.

325
00:15:04,620 --> 00:15:08,740
Like, do you feel like like do you ever worry about that, I guess?

326
00:15:09,300 --> 00:15:11,900
I mean, I think it's a real risk.

327
00:15:12,100 --> 00:15:15,700
Like I've been using agents to do all of my Git work for me

328
00:15:15,700 --> 00:15:20,440
because driving the Git API, it's the CLI itself has always been too hard for me.

329
00:15:20,900 --> 00:15:22,360
I've never been very good at it.

330
00:15:24,240 --> 00:15:27,280
And I'm never going to rebase again my hand ever.

331
00:15:27,540 --> 00:15:29,540
And I've already forgotten half of how to do it.

332
00:15:29,800 --> 00:15:31,320
It's so much better than I am.

333
00:15:31,480 --> 00:15:34,300
Like me resolving merge conflicts is just a disaster.

334
00:15:34,580 --> 00:15:37,380
Like if you if you ever wanted to make like a

335
00:15:37,380 --> 00:15:40,500
like a ridiculous video for the Internet of embarrassing programming,

336
00:15:40,500 --> 00:15:42,380
it would be me handling a merge conflict.

337
00:15:42,620 --> 00:15:44,040
Like I get it wrong like four times.

338
00:15:44,460 --> 00:15:48,440
But it's rad that you'll actually admit that, because I love how people are like,

339
00:15:48,540 --> 00:15:50,720
you can't be a developer unless you know all the Git,

340
00:15:50,780 --> 00:15:52,920
but we're never going to teach Git in college or do it.

341
00:15:52,960 --> 00:15:53,540
You know what I mean?

342
00:15:53,740 --> 00:15:56,540
Like, and there's never any room for people to be like, I hate it.

343
00:15:56,580 --> 00:15:57,320
And it was hard.

344
00:15:58,560 --> 00:16:01,680
I feel like if I became a Git expert and really knew how to use it,

345
00:16:01,740 --> 00:16:03,640
there'd be no space left in my head for programming.

346
00:16:04,280 --> 00:16:06,640
And so like I'd be really good at merge conflicts,

347
00:16:06,640 --> 00:16:08,440
but I wouldn't know how to write a computer program.

348
00:16:08,740 --> 00:16:11,600
So don't you think that's something that's like not talked about a lot?

349
00:16:11,600 --> 00:16:14,700
Like it's like I think sometimes we want to like posture to be like,

350
00:16:14,860 --> 00:16:17,940
I know everything and it is impossible to know everything.

351
00:16:18,300 --> 00:16:22,020
Like you're just not going to be very good at something if you can't,

352
00:16:22,040 --> 00:16:23,180
like if there's no room to grow.

353
00:16:23,280 --> 00:16:25,400
But then I also kind of wonder, do you think that like

354
00:16:25,400 --> 00:16:29,180
we'll get to the point where all the knowledge will be like behind a paywall

355
00:16:29,180 --> 00:16:30,820
because we're going to have to use AI?

356
00:16:31,000 --> 00:16:31,600
Like, you know what I mean?

357
00:16:32,220 --> 00:16:33,840
Yeah, I mean, I don't know.

358
00:16:33,840 --> 00:16:35,360
That was the cool thing about development.

359
00:16:35,640 --> 00:16:37,140
You had all these websites, right,

360
00:16:37,140 --> 00:16:39,260
that you could go to and you could learn code for free.

361
00:16:39,520 --> 00:16:40,440
Like, is this now?

362
00:16:40,860 --> 00:16:41,940
I've never thought about that.

363
00:16:42,200 --> 00:16:45,900
You know, I haven't played through the idea of like knowledge

364
00:16:45,900 --> 00:16:48,460
being locked behind one of the models,

365
00:16:48,780 --> 00:16:51,940
because right now the models are desperate for any knowledge they can find.

366
00:16:52,080 --> 00:16:52,200
Right.

367
00:16:52,240 --> 00:16:55,580
So they're only good at things that are very public and very on the Internet.

368
00:16:56,040 --> 00:16:56,500
But you're right.

369
00:16:56,520 --> 00:16:58,960
There is I think there is a long term risk there of some sort.

370
00:16:58,980 --> 00:17:00,920
But it's it's years out, I would say.

371
00:17:01,700 --> 00:17:04,940
Well, one of the things you mentioned that like end of 2025,

372
00:17:05,780 --> 00:17:07,120
agents got better at writing code.

373
00:17:07,120 --> 00:17:11,000
And I feel like in in May, when we're recording this of 2026,

374
00:17:11,960 --> 00:17:16,000
the open source models are getting better at like they're catching up.

375
00:17:16,180 --> 00:17:21,320
And so all the price hikes and changes in every AI company

376
00:17:21,320 --> 00:17:24,440
and people are just like, oh, you know, like we we can't go back to the old way.

377
00:17:24,960 --> 00:17:27,080
But I need to be able to have some control over this

378
00:17:27,080 --> 00:17:30,360
or at least be able to just like buy a model, like buy box software.

379
00:17:30,360 --> 00:17:31,580
That's like an element, right?

380
00:17:31,660 --> 00:17:32,560
Like, because like,

381
00:17:33,420 --> 00:17:36,440
you know, like just being like somebody who came from the graph design world,

382
00:17:36,440 --> 00:17:40,540
like be usable to buy Adobe like suite and you pay for it once

383
00:17:40,540 --> 00:17:41,540
and you use it for years.

384
00:17:41,680 --> 00:17:45,580
And then it got to the point where it's like having to pay for it every month.

385
00:17:45,720 --> 00:17:49,520
Like some like some artists don't make the same amount of money every month,

386
00:17:49,560 --> 00:17:53,840
you know, so it's like like and I wonder like a lot of what we're training on

387
00:17:53,840 --> 00:17:57,140
and what, you know, developers used with Stack Overflow in these forums

388
00:17:57,140 --> 00:18:00,220
that were public and that everybody talked about the problems they were having.

389
00:18:00,260 --> 00:18:05,160
And it gets to a point like if those like if those forums disappear,

390
00:18:05,160 --> 00:18:07,800
like what are we even going to train models and developers on, you know?

391
00:18:08,340 --> 00:18:10,040
Well, there's lots to train models on.

392
00:18:10,420 --> 00:18:12,240
That's what that's what the subscriptions are.

393
00:18:13,320 --> 00:18:16,460
Like when you buy a subscription to one of the major platforms,

394
00:18:16,460 --> 00:18:19,820
you're agreeing in the terms of service that you can be trained on.

395
00:18:20,620 --> 00:18:21,620
And that's true. Yeah.

396
00:18:21,780 --> 00:18:24,060
And so there's a there's a lot of training data out there.

397
00:18:24,660 --> 00:18:25,940
It's harder to come by.

398
00:18:26,020 --> 00:18:28,020
It's not all nice and open like Stack Overflow anymore.

399
00:18:28,880 --> 00:18:32,520
Fortunately, the open models seem to be able to train on the closed models.

400
00:18:33,300 --> 00:18:35,360
You know, I'm not sure they don't want to.

401
00:18:35,580 --> 00:18:36,480
Yeah, that's right.

402
00:18:36,620 --> 00:18:40,280
This this whole distillation process, it seems it seems quite easy.

403
00:18:41,240 --> 00:18:44,940
And, you know, it's, you know, they're very upset about this.

404
00:18:45,420 --> 00:18:47,260
But but they also stole the world.

405
00:18:47,340 --> 00:18:48,260
So screw you.

406
00:18:48,460 --> 00:18:50,200
You know what I mean? Like, it's so funny.

407
00:18:50,320 --> 00:18:53,560
It's like they're just like and this is like egregious.

408
00:18:53,640 --> 00:18:57,320
And you're like, but you like did it to like artists and writers.

409
00:18:57,600 --> 00:18:59,380
You know how many lawsuits are you like?

410
00:18:59,380 --> 00:19:02,560
And you you just, you know, rolling through every YouTube video

411
00:19:02,560 --> 00:19:05,660
and not just that, but our answers on Stack Overflow and questions.

412
00:19:05,920 --> 00:19:06,300
You know what I mean?

413
00:19:06,300 --> 00:19:09,400
Like I spent like a lot of my early career at AWS

414
00:19:09,400 --> 00:19:12,120
answering people Stack Overflow for the service I worked for.

415
00:19:12,240 --> 00:19:14,800
You know, nobody's paying me for all of that work.

416
00:19:16,040 --> 00:19:20,940
Yeah, I don't know what the correct moral and legal outcome of all of this is.

417
00:19:21,020 --> 00:19:22,700
But my violin is very small.

418
00:19:23,920 --> 00:19:25,720
So so that's.

419
00:19:27,260 --> 00:19:28,780
David, you're really good at software,

420
00:19:28,780 --> 00:19:33,360
but you're really funny, like you've had me like laughing the entire time.

421
00:19:33,460 --> 00:19:35,180
I want to. I wish my kids agreed with you.

422
00:19:35,340 --> 00:19:37,760
No, my kids think I'm so not cool.

423
00:19:37,800 --> 00:19:38,820
Kids never know.

424
00:19:39,040 --> 00:19:42,100
Like our kids have to talk to each other and then they'll think each other's funny

425
00:19:42,100 --> 00:19:43,560
because a lot of kids thinks I'm hilarious.

426
00:19:43,560 --> 00:19:44,720
But my kids are deadpan.

427
00:19:44,720 --> 00:19:47,320
So annoying. They don't think I'm funny or cool at all.

428
00:19:47,400 --> 00:19:50,780
They get on like the like they get on FaceTime with each other

429
00:19:50,780 --> 00:19:52,160
and Justin will be in the background

430
00:19:52,160 --> 00:19:55,120
and they think he's the coolest thing since Slice Bread.

431
00:19:55,220 --> 00:19:57,740
And I'm like, well, how do you think you know him?

432
00:19:57,740 --> 00:19:59,900
My kids looking at me like, what do you mean?

433
00:20:00,040 --> 00:20:01,460
Him? No, this is my dad.

434
00:20:01,760 --> 00:20:04,900
Yeah, I don't think my kids even know what I do.

435
00:20:05,340 --> 00:20:06,000
That's fine.

436
00:20:07,860 --> 00:20:10,820
So I want to take a step back here on on building a cloud.

437
00:20:12,900 --> 00:20:14,660
Everyone has known for a long time.

438
00:20:14,840 --> 00:20:17,840
The wrong way about going, building anything is going and buying

439
00:20:17,840 --> 00:20:19,360
and racking and stacking a bunch of servers, right?

440
00:20:19,380 --> 00:20:22,800
Yeah, not a capital and time and stuff you have to do there is just

441
00:20:22,800 --> 00:20:25,500
you're going to you're going to die and someone's going to surpass you.

442
00:20:25,820 --> 00:20:27,060
Why did you go that routes?

443
00:20:27,060 --> 00:20:28,260
How did you go that routes?

444
00:20:28,480 --> 00:20:29,720
And like, what does that look like today?

445
00:20:30,260 --> 00:20:32,620
Yeah, it's I mean, you're right.

446
00:20:32,740 --> 00:20:36,500
Like you can't compete with a hyperscaler on racking and stacking computers.

447
00:20:36,660 --> 00:20:37,640
Like that's actually true.

448
00:20:38,160 --> 00:20:41,980
You get economy of scale by buying billions at a time.

449
00:20:42,480 --> 00:20:47,920
And so the machines are necessarily more expensive than they they'll sell.

450
00:20:48,220 --> 00:20:51,300
And that is a certain deal size.

451
00:20:51,300 --> 00:20:51,900
That's the killer.

452
00:20:52,420 --> 00:20:54,100
That's that's what makes it all not work.

453
00:20:54,660 --> 00:20:57,160
But the deal size is very large is the key thing, right?

454
00:20:57,240 --> 00:20:59,900
Like we actually started XeDev on AWS.

455
00:21:00,700 --> 00:21:03,500
And that's that's when we launched in late December.

456
00:21:03,760 --> 00:21:05,480
We launched, we silently launched around Christmas

457
00:21:05,480 --> 00:21:07,560
and then like got overwhelmed.

458
00:21:08,080 --> 00:21:11,160
And, you know, we were off AWS, we were mostly off AWS by the end of January.

459
00:21:12,400 --> 00:21:16,720
But like our AWS bill got to 100 grand a month before that.

460
00:21:17,480 --> 00:21:20,060
That is not enough for an AWS sales rep to call you.

461
00:21:20,200 --> 00:21:20,920
They don't care.

462
00:21:21,240 --> 00:21:22,860
100 grand a month, that's nothing to them.

463
00:21:22,860 --> 00:21:24,980
Like there's no discounts at that price.

464
00:21:25,580 --> 00:21:27,960
Which is like wild, because I think that the bar is going up.

465
00:21:28,040 --> 00:21:31,100
I remember when we used to give out discounts at like a certain scale.

466
00:21:31,600 --> 00:21:35,620
And if you think about the workloads with AI, that scale has got to get bigger

467
00:21:35,620 --> 00:21:39,560
and bigger because people are consuming more compute power and their bills are bigger.

468
00:21:39,680 --> 00:21:42,540
And the gap between the top end and the bottom end is wide.

469
00:21:42,560 --> 00:21:43,060
That's what I'm saying.

470
00:21:43,200 --> 00:21:44,280
It is just getting ginormous.

471
00:21:44,300 --> 00:21:45,460
They don't care about anyone in the middle there.

472
00:21:45,700 --> 00:21:46,080
Yeah. Yeah.

473
00:21:46,220 --> 00:21:49,540
And not caring about anyone in the middle is why there's a chance to try

474
00:21:49,540 --> 00:21:54,460
and build something and why the economics of this can work out in the middle size.

475
00:21:54,600 --> 00:21:59,200
Because right now, if I try to run on AWS, I pay their vast margins on top of their machines.

476
00:21:59,820 --> 00:22:02,240
And then how can I charge users on top of that?

477
00:22:02,440 --> 00:22:03,840
Like I can't pay for everything.

478
00:22:04,340 --> 00:22:08,440
But if you rack your own machines, they're dramatically cheaper than AWS's list prices.

479
00:22:09,560 --> 00:22:13,720
And so that's at a medium sized scale to build one of these things.

480
00:22:13,800 --> 00:22:17,000
You have to rack your own machines because they're not going to give you the discounts you need.

481
00:22:17,000 --> 00:22:22,000
It's either that or go and raise all the money in the world and then give it all to AWS.

482
00:22:22,760 --> 00:22:26,340
Which is, I mean, if you're not wrong, look at what people are doing.

483
00:22:26,440 --> 00:22:31,220
People are either doing colos because that's the only way that can control their bills.

484
00:22:31,660 --> 00:22:36,180
Because, you know, if AWS decides to like charge a certain amount for data coming in and out

485
00:22:36,180 --> 00:22:39,840
and they change something like you're now stuck because that's what you built it on.

486
00:22:39,960 --> 00:22:43,360
You know, so it's like, that's what I'm saying, like, oh, guess what?

487
00:22:43,380 --> 00:22:44,960
I get free bandwidth. What the hell?

488
00:22:44,960 --> 00:22:49,300
That's something that like, and think about the fact that for a long time, that was like a secret, right?

489
00:22:49,340 --> 00:22:53,240
Like people did not know and they didn't take that into consideration when doing these budgets.

490
00:22:53,460 --> 00:22:56,580
Right. So, but like, you're not wrong.

491
00:22:56,620 --> 00:22:59,360
Like how many times have we heard from like Blue Sky and all these new places,

492
00:22:59,560 --> 00:23:02,400
these new like companies that are scaling at this extent.

493
00:23:02,400 --> 00:23:05,900
And they're like, we are running in colos because that's their options.

494
00:23:06,240 --> 00:23:12,040
Yeah. Blue Sky is too small for AWS to care to give them proper discounts, which, you know, is a little astonishing, right?

495
00:23:12,040 --> 00:23:16,560
And they only have like, well, they have like 10 machines and like, like three of those are Raspberry Pis, right?

496
00:23:16,880 --> 00:23:18,740
Even if they had a hundred, they'd be too small.

497
00:23:18,960 --> 00:23:19,080
Yeah.

498
00:23:19,180 --> 00:23:19,860
You know, it's a...

499
00:23:19,860 --> 00:23:22,320
But that's what I'm saying, like, that is the threshold.

500
00:23:22,580 --> 00:23:27,320
Like, think about how much Blue Sky has scaled over the last like few years, you know,

501
00:23:27,360 --> 00:23:29,240
and they're still too small for that discount.

502
00:23:29,500 --> 00:23:33,880
I mean, what you're saying is you can do a lot with one computer, which is very true.

503
00:23:34,220 --> 00:23:35,240
But isn't it like cool?

504
00:23:35,300 --> 00:23:37,760
Like if you mentioned the two things that you've built, right?

505
00:23:37,760 --> 00:23:41,120
You're talking about Tailscale and XEDev.

506
00:23:41,360 --> 00:23:46,960
Those are, it's almost like solving problems more in a way that we used to solve them a while ago.

507
00:23:47,140 --> 00:23:52,140
You know, like actually like having, and not just that, but it's kind of like more control over your network,

508
00:23:52,280 --> 00:23:54,200
more control over your data and your cloud.

509
00:23:55,140 --> 00:24:01,660
Like that's all things that we kind of gave up some control of things for like ease.

510
00:24:01,780 --> 00:24:03,440
And now it's not easier or cheaper.

511
00:24:03,560 --> 00:24:07,540
And it just seems like a lot of this is like going back to almost the basics, you know?

512
00:24:07,540 --> 00:24:08,980
Yes, that's a hundred percent.

513
00:24:09,100 --> 00:24:10,840
My whole philosophy with building everything.

514
00:24:10,920 --> 00:24:13,260
You're saying your third startup is going to be a storage provider.

515
00:24:14,180 --> 00:24:15,120
Blob storage.

516
00:24:17,780 --> 00:24:20,400
I mean, that's not a terrible idea.

517
00:24:21,020 --> 00:24:22,540
You know, yeah.

518
00:24:23,420 --> 00:24:23,940
Yeah.

519
00:24:24,720 --> 00:24:31,220
I mean, I would say one other thing, which is one of the reasons why I was willing to jump so quickly into Colo is

520
00:24:32,000 --> 00:24:37,560
building on top of another cloud means inherently accepting some of their design decisions.

521
00:24:38,900 --> 00:24:39,460
Identity.

522
00:24:39,760 --> 00:24:39,980
Yeah.

523
00:24:40,300 --> 00:24:40,860
Yeah.

524
00:24:41,180 --> 00:24:48,060
And it would be different if you knew exactly what you were signing up for when you were accepting those design decisions,

525
00:24:48,480 --> 00:24:55,000
but a lot like just different regions and different clouds and just different ways that those services were built.

526
00:24:55,160 --> 00:24:58,920
Like sometimes you have no idea what you're truly building on top of.

527
00:24:59,260 --> 00:24:59,700
Absolutely.

528
00:25:00,720 --> 00:25:07,720
It's always, and that's particularly true at the PaaS level, when like you're using those services, you read the docs, you say,

529
00:25:07,780 --> 00:25:13,720
I can build my software like four weeks in, you discover some new constraint of the PaaS that ruins everything, your design is invalid.

530
00:25:13,940 --> 00:25:15,700
Oh, that's happened to me several times.

531
00:25:15,900 --> 00:25:17,660
That's why I won't touch PaaS anymore.

532
00:25:18,160 --> 00:25:21,280
I don't think people like realize like what you're being sold.

533
00:25:21,460 --> 00:25:28,720
Like this is somebody else building a technology on top of other constraints that you are now bound to that you don't even know exist.

534
00:25:28,960 --> 00:25:29,260
That's right.

535
00:25:29,260 --> 00:25:32,580
I mean, and AWS Lambda, one of those PaaSs, it's built on EC2.

536
00:25:33,100 --> 00:25:34,620
Yeah, it's all VMs under the hood, right?

537
00:25:35,040 --> 00:25:36,620
It's computers at the end of the day.

538
00:25:37,080 --> 00:25:39,680
All of it is a Linux server in the cloud.

539
00:25:41,140 --> 00:25:41,580
Exactly.

540
00:25:42,080 --> 00:25:48,780
There's one physical constraint of AWS machines and GCP and all the rest.

541
00:25:49,120 --> 00:25:49,960
They're all very similar.

542
00:25:50,080 --> 00:25:57,820
That really interests me about doing our own machines that I'm very interested in trying to do something new in, which is they're all very big

543
00:25:57,820 --> 00:25:59,160
into remote block storage.

544
00:26:00,140 --> 00:26:05,520
And this made a lot of sense when they built it 15, 20 years ago.

545
00:26:05,560 --> 00:26:15,160
And in fact, I was working on big storage systems at Google or adjacent to the storage team because we had a lot of storage back in 2010.

546
00:26:16,420 --> 00:26:20,020
Like remote storage was exactly the correct solution for everything back then.

547
00:26:20,260 --> 00:26:22,120
And it was because we were using hard drives.

548
00:26:22,680 --> 00:26:31,500
And with hard drives, a seek to random read or random write takes 10 milliseconds to move the head and the hard drive physically.

549
00:26:32,480 --> 00:26:35,020
And then you add a millisecond of Ethernet roundtrip time.

550
00:26:35,060 --> 00:26:35,820
It's not that big a deal.

551
00:26:36,600 --> 00:26:38,020
And so we built all these remote systems.

552
00:26:38,160 --> 00:26:47,080
And then at some point we swapped everything out for SSDs and SSDs have 10 microsecond seek times and then one millisecond of Ethernet roundtrip time on top of it.

553
00:26:47,420 --> 00:26:50,820
And so now we've taken these really fast SSDs and put them behind very slow network.

554
00:26:51,840 --> 00:26:56,260
And, you know, we've taken away a lot of the advantage of NVMe.

555
00:26:57,360 --> 00:27:01,920
And I think that there's an opportunity to try and revisit that.

556
00:27:02,060 --> 00:27:03,740
And like clouds don't want to revisit that.

557
00:27:03,800 --> 00:27:07,640
Like you can get machines with NVMe in them on AWS, but they're odd.

558
00:27:07,960 --> 00:27:12,280
They're odd in the sense of the NVMe is relatively small compared to the capabilities of the machine.

559
00:27:12,280 --> 00:27:17,720
Or you've gone to one of their more exotic instances that are clearly designed for running really big database servers.

560
00:27:18,260 --> 00:27:19,240
And they're very large.

561
00:27:19,640 --> 00:27:21,880
And you're probably much more expensive also.

562
00:27:22,180 --> 00:27:28,220
You're looking at five or six figures per month to rent something.

563
00:27:28,560 --> 00:27:29,900
It's that's right.

564
00:27:30,000 --> 00:27:32,000
It's a very expensive machines.

565
00:27:32,900 --> 00:27:36,680
And even on top of that, they're unusual and really interesting ways, right?

566
00:27:36,720 --> 00:27:42,600
Like if you rack a traditional machine with a bunch of NVMe in it, like maybe a drive will fail and you've set up some sort of RAID 5 like thing.

567
00:27:43,160 --> 00:27:45,300
And, you know, the drive fails.

568
00:27:45,600 --> 00:27:47,020
You have an ops person you're paying.

569
00:27:47,120 --> 00:27:48,200
They go in, they swap the drive.

570
00:27:48,200 --> 00:27:50,460
The machine is running integrated mode the rest of the time.

571
00:27:50,640 --> 00:27:51,180
Swap is good.

572
00:27:51,940 --> 00:28:00,180
In the cloud provider, when one of those drives fails, they have you reset the machine and you immediately come up on a new machine with all the drives there.

573
00:28:00,200 --> 00:28:01,120
But you lost all the data.

574
00:28:02,460 --> 00:28:04,440
You know, your drives get wiped between those events.

575
00:28:04,560 --> 00:28:14,220
And then you have to restore from your remote block storage, which is a multi-hour procedure, which is totally fine if you're doing postgres replicas because your postgres primary fails.

576
00:28:14,280 --> 00:28:15,500
You fail over to a replica.

577
00:28:15,660 --> 00:28:16,540
You bring up a new replica.

578
00:28:16,540 --> 00:28:21,320
You spent eight hours filling, you know, 50 terabytes of data into the replica.

579
00:28:21,740 --> 00:28:22,300
No big deal.

580
00:28:22,900 --> 00:28:31,580
But it also means there's a whole stack of software you can't write that is easier and more reliable to build on physical machines than it is to buy it from a cloud provider.

581
00:28:31,620 --> 00:28:32,660
And that's very counterintuitive.

582
00:28:32,860 --> 00:28:35,700
I think of the cloud provider as the most reliable source out there.

583
00:28:36,000 --> 00:28:37,720
And they are, except in this one instance.

584
00:28:38,880 --> 00:28:44,140
And so I'm really interested in this problem because I have multiple times written a program on my laptop and then shipped it up to the cloud.

585
00:28:44,180 --> 00:28:44,980
And it's really slow.

586
00:28:44,980 --> 00:28:46,480
I'm like, why is it really slow?

587
00:28:46,520 --> 00:28:49,020
And it's like, oh, well, my server is using the disk.

588
00:28:49,300 --> 00:28:50,120
Yeah, exactly.

589
00:28:50,440 --> 00:28:54,120
The server in the cloud's got 3,000 IOPS and my local laptop's got 500,000 IOPS.

590
00:28:54,240 --> 00:28:56,280
I'm like, OK, yeah, there's the problem right there.

591
00:28:56,560 --> 00:29:00,240
So, you know, we're experimenting with that and seeing if we can do fun things with it.

592
00:29:00,720 --> 00:29:06,360
It also depends on if you're on like something that's hosting multiple workloads or just your workload, too.

593
00:29:06,740 --> 00:29:09,400
The noisy neighbor problem is always fun.

594
00:29:10,900 --> 00:29:14,880
Going back to January, like we're out of the AWS.

595
00:29:15,520 --> 00:29:16,800
What did that practically look like?

596
00:29:16,820 --> 00:29:19,120
I want to because like this audience is like really interested in the details.

597
00:29:19,600 --> 00:29:23,440
And at that point, you say, OK, I need you're like I need you need four locations.

598
00:29:23,600 --> 00:29:24,580
You're like, let me find some colos.

599
00:29:24,640 --> 00:29:26,380
Let me just go like rent some servers from them.

600
00:29:26,860 --> 00:29:27,700
What does that actually look like?

601
00:29:27,720 --> 00:29:29,980
I've like core like how do you calculate?

602
00:29:30,420 --> 00:29:34,840
We have a hundred thousand dollars in AWS and we're going to move to that's whatever.

603
00:29:35,580 --> 00:29:38,180
Maybe a hundred machines, let's say, or whatever it was.

604
00:29:38,180 --> 00:29:44,160
And you're moving to and you're going to move to what, like 20 physical servers?

605
00:29:44,300 --> 00:29:46,220
Is that like what the calculation ended up being?

606
00:29:46,280 --> 00:29:46,940
Like, how did that look?

607
00:29:47,640 --> 00:29:52,180
Yeah, it we when you're doing startup things, you always think about growth.

608
00:29:52,760 --> 00:29:56,240
And so you immediately say to yourself, look, we're spending this on AWS for these users.

609
00:29:56,340 --> 00:29:57,300
We're growing at this rate.

610
00:29:57,680 --> 00:29:59,840
Let's build something that the users can grow into.

611
00:30:00,040 --> 00:30:02,520
And then we'll think about the previous users and moving them off.

612
00:30:02,720 --> 00:30:04,620
And that's very much the model.

613
00:30:04,840 --> 00:30:06,960
You know, we still have a bit of an AWS built from users.

614
00:30:06,960 --> 00:30:07,980
We haven't finished moving off.

615
00:30:08,560 --> 00:30:10,760
That's because you can move me if I'm there.

616
00:30:10,860 --> 00:30:11,960
I don't remember when I signed up.

617
00:30:12,040 --> 00:30:17,180
But yeah, we we really should move a lot of users because we have more regions now.

618
00:30:17,240 --> 00:30:19,080
And so we can move the machines physically closer to you.

619
00:30:19,140 --> 00:30:20,080
You get a better experience.

620
00:30:20,180 --> 00:30:21,160
The machines are bigger.

621
00:30:21,720 --> 00:30:23,480
They're they're they've got less load on them.

622
00:30:23,620 --> 00:30:25,100
You know, there's a lot of nice things about them.

623
00:30:25,420 --> 00:30:27,020
But we have a stack of users to move.

624
00:30:27,080 --> 00:30:30,500
And so it's you know, we've got lists and we're going to start sending people emails

625
00:30:30,500 --> 00:30:32,740
and like trying to semi automated because there's a few.

626
00:30:33,720 --> 00:30:39,420
But the we've been we're using it as an excuse to build good live migration machinery

627
00:30:39,420 --> 00:30:45,100
for moving VMs between regions without restarting them, which, you know, is going pretty well.

628
00:30:45,240 --> 00:30:50,040
But it's a there's a long tail of things we run into, including just networks going bad in the middle.

629
00:30:50,180 --> 00:30:53,440
And like suddenly there's a lot of packet loss and all of our live migrations go slow.

630
00:30:53,520 --> 00:30:54,220
It's astonishing.

631
00:30:55,120 --> 00:31:02,440
But yeah, we started by reaching out to providers who sell bare metal in in data centers.

632
00:31:02,440 --> 00:31:05,240
But do quite a lot of management around it.

633
00:31:05,280 --> 00:31:07,800
So they've pre racked them and you're purchasing machines that are already there.

634
00:31:08,060 --> 00:31:08,760
Yeah, we could service.

635
00:31:09,280 --> 00:31:10,940
Yeah. And there's there's several of those.

636
00:31:11,300 --> 00:31:13,760
And that's you can do that very quickly.

637
00:31:14,000 --> 00:31:15,460
You can get machines up in days doing that.

638
00:31:15,520 --> 00:31:16,160
And so we did that.

639
00:31:16,680 --> 00:31:20,300
Those services are very busy right now because a lot of people interested machines.

640
00:31:20,960 --> 00:31:23,900
Also, they're all going through price pain right now because.

641
00:31:25,720 --> 00:31:29,380
Computers have got a lot more expensive, and so they're all busy revamping all their prices,

642
00:31:29,480 --> 00:31:30,720
and so everything's very messy right now.

643
00:31:30,720 --> 00:31:32,540
But there's several very good services out there.

644
00:31:33,000 --> 00:31:39,300
And then the step after that is you start talking to a separate set of companies who basically run ops and data centers.

645
00:31:39,900 --> 00:31:42,860
And for this, you can actually go to some data centers, do this directly.

646
00:31:43,020 --> 00:31:46,660
You can talk to someone like Equinix and they have op services they offer.

647
00:31:46,820 --> 00:31:54,880
But basically, you want to have some company on retainer who is employing someone who is there at 2 a.m.

648
00:31:54,880 --> 00:32:01,640
who can swap out the disk when it breaks, because I'm not, you know, if I have to fly to London to swap out a disk at 2 a.m.,

649
00:32:01,640 --> 00:32:03,080
you have to wait for the plane to get there.

650
00:32:03,160 --> 00:32:05,420
And that's going to suck all night and I'm going to be tired.

651
00:32:06,560 --> 00:32:12,360
And so it's very much and, you know, it doesn't make sense for us to employ a full time person in London to maintain just our machines

652
00:32:12,360 --> 00:32:14,900
because there's not very much maintenance on a small set of machines.

653
00:32:15,020 --> 00:32:16,760
And so this is a great thing to share between companies.

654
00:32:16,860 --> 00:32:18,660
And this this is done at several levels.

655
00:32:18,660 --> 00:32:22,000
And there's a whole ecosystem of companies that offer these services in different forms.

656
00:32:22,580 --> 00:32:25,260
And so we just spoke to a whole set of them and picked one.

657
00:32:26,100 --> 00:32:28,840
And then we are racking machines with them.

658
00:32:29,080 --> 00:32:33,920
And then it's a matter of ordering machines from a machine provider and you have to price them out with several people.

659
00:32:34,680 --> 00:32:38,200
And then there's a question of how much extra do you order and where do you store it?

660
00:32:38,420 --> 00:32:42,780
And again, a lot of these intermediate services will do the dropshipping of the machines for you, too,

661
00:32:42,840 --> 00:32:44,740
and the racking and like help manage all of this.

662
00:32:45,540 --> 00:32:49,840
And so we're in the middle of processes there right now, you know, proving out machines.

663
00:32:51,440 --> 00:32:53,760
What what kind of scale were you at in January?

664
00:32:53,920 --> 00:32:56,260
I mean, if you just launched it like soft launch in December.

665
00:32:56,440 --> 00:33:00,320
Yeah, you're not you're not doing hundreds of physical machines in all these locations.

666
00:33:00,760 --> 00:33:02,440
No, no, we're doing dozens.

667
00:33:03,120 --> 00:33:04,660
Yeah, this is more than more than number.

668
00:33:05,060 --> 00:33:08,500
And we're building out some extra capacity as part of it all.

669
00:33:09,260 --> 00:33:16,160
One of the interesting challenges with us is we're trying to, you know, again, we're trying to make VMs easier to use.

670
00:33:16,160 --> 00:33:22,220
And the big thing for me is and this is if you want to get technical, I don't sell people VMs.

671
00:33:22,380 --> 00:33:25,660
I sell people a cgroup in which they can run VMs.

672
00:33:26,900 --> 00:33:28,200
And, you know, that's probably too technical.

673
00:33:28,400 --> 00:33:29,800
But the basic idea is great for the audience.

674
00:33:29,940 --> 00:33:31,760
You know, it's right here.

675
00:33:32,420 --> 00:33:38,800
Yeah. I mean, I want to give you CPU and RAM and disk and you run as many VMs in it as you like,

676
00:33:38,820 --> 00:33:41,200
because from my perspective, a VM is a Linux process.

677
00:33:41,960 --> 00:33:43,820
And if you want two of them, that's OK.

678
00:33:44,290 --> 00:33:45,920
You have two processes. They're in a cgroup.

679
00:33:46,060 --> 00:33:47,420
The resources are assigned to the cgroup.

680
00:33:47,660 --> 00:33:49,140
You want a hundred of them. That's fine.

681
00:33:49,600 --> 00:33:50,980
You might run out of resources.

682
00:33:51,220 --> 00:33:52,360
Like you need to buy more resources.

683
00:33:52,620 --> 00:33:54,560
That's that depends on your machines.

684
00:33:54,880 --> 00:34:01,020
And that's the other thing I want to talk about was your pricing model is difference where it's not a easy two.

685
00:34:01,140 --> 00:34:04,440
I need I need five micros and I get charged for each one.

686
00:34:04,540 --> 00:34:06,520
You're selling a pool of resources, right?

687
00:34:06,540 --> 00:34:11,040
Like your pricing model is how much CPU, how much memory, how much disk space.

688
00:34:11,040 --> 00:34:17,920
I don't know if if network is counted in there, but I know that AI tokens are at something you can buy more of each of those things.

689
00:34:17,960 --> 00:34:27,400
Right. These are all knobs I can turn and say I need more of this thing because whatever my workload is or my constraints are, I just need the constraint to go away so I can pay for more and divvy it up however I want.

690
00:34:27,460 --> 00:34:29,240
Right. Yeah. Network is in there.

691
00:34:29,360 --> 00:34:35,760
We just dropped the price because the original price was AWS's bill plus five percent for my accounting errors.

692
00:34:36,460 --> 00:34:38,580
And now which was enormous, of course.

693
00:34:38,580 --> 00:34:48,780
And I've dropped it a bunch, but I haven't dropped it as far as I can because we have some interesting things we want to launch and we're trying to decide how complicated to make network pricing.

694
00:34:49,280 --> 00:34:52,340
Like, you know, it'll it'll I think it'll make sense when it comes out.

695
00:34:52,680 --> 00:34:56,460
But we can do the nice, cheap networking that you would expect in a colo.

696
00:34:58,180 --> 00:35:01,580
Because, like you said, it's like a 10x bill for just immediately going into a cloud.

697
00:35:01,940 --> 00:35:03,080
So why? Why go that route?

698
00:35:03,080 --> 00:35:04,160
Why pool? Yeah.

699
00:35:04,340 --> 00:35:18,860
So this is entirely motivated by the fact that agents have just changed what programming is to me because I'm one of these programmers who always has like 10 things they want to do and can do maybe half of one of them and like not as well as I wish I could.

700
00:35:19,320 --> 00:35:20,820
But, you know, it's something.

701
00:35:21,440 --> 00:35:28,040
And so I've often ended up with like lists in an Apple note on my phone of like one liners of a program I wish I could write.

702
00:35:28,080 --> 00:35:31,440
And like I write it down just so that I can then forget about it, never think about it again.

703
00:35:32,680 --> 00:35:44,640
And the nature of an agent now is I can take that one line, drop it into an agent and like a good quarter of the time the program I want it pops out or something close to it, especially with a few more minutes of work.

704
00:35:46,080 --> 00:35:49,660
And, you know, quarter time is not great, but it's a lot better than zero, which is what I had before.

705
00:35:50,120 --> 00:35:51,540
And it's getting better all the time.

706
00:35:51,800 --> 00:36:00,700
So, you know, agents are, you know, creating this is what I mean by every program is going to have so many more programs now because it's just so easy to like build these small things.

707
00:36:01,290 --> 00:36:07,760
And so instead of an Apple note, what I do now is exe dev slash new, a little prompt box appears.

708
00:36:07,880 --> 00:36:09,780
I type the one liner in there and hit create VM.

709
00:36:09,960 --> 00:36:11,020
And then I forget about it.

710
00:36:11,080 --> 00:36:15,200
I look at it again a few days later and like sometimes it's useful or else sometimes I just delete it.

711
00:36:15,820 --> 00:36:25,900
And for that flow to make sense where it just makes me the program and keeps it running every time I create it, the marginal cost of the VM to me has to be zero.

712
00:36:26,520 --> 00:36:30,920
Because my idea that I just put an Apple note is worth less than a cup of coffee to me.

713
00:36:31,100 --> 00:36:32,840
I'm not going to go and buy a micro VM.

714
00:36:33,540 --> 00:36:36,020
It's like, you know, it's like, oh, three dollars a month.

715
00:36:36,100 --> 00:36:38,380
No, I'm just not going to do it is the actual answer.

716
00:36:38,680 --> 00:36:43,220
So that, you know, because most almost all of my ideas are terrible, as you would expect.

717
00:36:43,280 --> 00:36:43,460
Right.

718
00:36:43,520 --> 00:36:46,500
You know, I have a lot of them and I obviously the quality has to be low.

719
00:36:46,940 --> 00:36:48,380
I make it up in quantity, though.

720
00:36:49,300 --> 00:36:50,580
And it's the bulk buy.

721
00:36:50,700 --> 00:36:51,340
It's the bulk buy.

722
00:36:52,180 --> 00:36:59,340
But when I was reading your blog, that was one of the most relatable parts when you're like, I always have all these ideas, but like rarely do I get to them all.

723
00:36:59,420 --> 00:37:00,120
And I was like, me too.

724
00:37:00,220 --> 00:37:01,520
Do you know how many domains I have?

725
00:37:02,860 --> 00:37:03,580
Yes, exactly.

726
00:37:03,880 --> 00:37:04,040
Right.

727
00:37:04,280 --> 00:37:12,040
So I feel like it was like, and honestly, the only time I've ever used those domains was when I was using Copilot to get that, like to get those websites up faster.

728
00:37:12,200 --> 00:37:13,340
So that way, you know what I mean?

729
00:37:13,340 --> 00:37:15,380
Because I don't have time to do all these things.

730
00:37:15,380 --> 00:37:21,160
And I feel like you're solving a really interesting problem that we all have and experienced.

731
00:37:22,020 --> 00:37:23,220
I'm glad you feel that way.

732
00:37:23,480 --> 00:37:26,000
The yeah, it's very, it's very hard to explain that.

733
00:37:26,120 --> 00:37:30,820
Like, yes, the innovation in my software is the pricing model has reduced the marginal cost of VM to zero.

734
00:37:30,980 --> 00:37:34,720
It's like one of those technically correct statements that everyone's like scratching their heads around.

735
00:37:34,800 --> 00:37:35,500
Why would I want that?

736
00:37:35,960 --> 00:37:38,120
But I think people, I actually want that.

737
00:37:38,440 --> 00:37:39,440
That's not crazy, though.

738
00:37:39,560 --> 00:37:46,740
I mean, if you look at it, a lot of AWS's success was being able to lower the prices when they were offering things to people because they had excess.

739
00:37:46,940 --> 00:37:48,320
Like, you're not wrong.

740
00:37:48,320 --> 00:37:54,140
I think AWS's benefit early days was lowering the time to get to something.

741
00:37:54,280 --> 00:37:59,220
Right. Because it was like the process of I have an idea is like, OK, we'll go price out.

742
00:37:59,720 --> 00:38:01,640
Wait for the next procurement cycle.

743
00:38:02,080 --> 00:38:05,260
Go price out what your budget is going to be for that idea for the next year.

744
00:38:05,460 --> 00:38:06,540
Then we're going to rack and stack.

745
00:38:06,660 --> 00:38:07,500
It's like three months later.

746
00:38:07,520 --> 00:38:09,120
You're like, oh, yeah, I don't think I want that idea.

747
00:38:09,280 --> 00:38:09,940
That's part of it.

748
00:38:10,000 --> 00:38:10,880
But think about it.

749
00:38:10,880 --> 00:38:18,300
When you went in as an essay to most conversations, you were talking about how you were going to like offer more and continue to make it less or to

750
00:38:18,300 --> 00:38:19,720
be more simple. You know what I mean?

751
00:38:19,880 --> 00:38:20,380
I mean, that was just lies.

752
00:38:20,500 --> 00:38:20,960
That was just marketing.

753
00:38:24,300 --> 00:38:25,720
The price will go down.

754
00:38:26,000 --> 00:38:28,320
Like, yeah, he said the quiet part out loud.

755
00:38:29,200 --> 00:38:35,080
I will say that with the right architecture, I've saved people a lot of money, but that's also because they were building it wrong.

756
00:38:36,800 --> 00:38:41,820
When we shipped the carpenter, like we saw like people's bills go down by like double digits.

757
00:38:42,020 --> 00:38:43,100
And we're like, that's exactly what we wanted.

758
00:38:43,160 --> 00:38:45,520
That's great. And we helped you do that.

759
00:38:45,520 --> 00:38:49,460
We didn't drive the the price of the AWS instance or the network bandwidth down.

760
00:38:49,580 --> 00:38:51,440
We drove their consumption down.

761
00:38:51,720 --> 00:38:52,000
Exactly.

762
00:38:52,360 --> 00:38:53,440
Yeah, no, that makes perfect sense.

763
00:38:53,540 --> 00:38:58,980
And like, you know, how would I do this if I wasn't, you know, I hadn't built XCDEV and I wanted this, what would I do?

764
00:38:59,060 --> 00:39:02,000
I'd go and get a VM, VPS like thing somewhere.

765
00:39:02,400 --> 00:39:08,520
And I'd set up something like Docker or some other, you know, nested VM thing on it.

766
00:39:08,620 --> 00:39:12,460
And then I'd set up a reverse proxy that routes to all the things.

767
00:39:12,700 --> 00:39:13,540
That sounds horrible.

768
00:39:13,540 --> 00:39:17,620
I would basically build XCDEV is the idea.

769
00:39:17,740 --> 00:39:19,480
And I was like, well, what if someone built that for you?

770
00:39:19,580 --> 00:39:20,040
Oh, yeah.

771
00:39:20,100 --> 00:39:20,320
All right.

772
00:39:20,380 --> 00:39:21,820
So that's what it is.

773
00:39:22,820 --> 00:39:24,380
And that's what it is for individuals.

774
00:39:24,580 --> 00:39:36,080
And then for businesses, it's that at a larger scale of like, instead of you bring up three EC2 VMs and you've got to size them appropriately, you just purchase a pile of CPU and RAM from us.

775
00:39:36,300 --> 00:39:40,440
And then you run the VMs you need to run and call it done.

776
00:39:40,440 --> 00:39:44,100
And, you know, size your pool appropriately so that all your VMs can run.

777
00:39:45,040 --> 00:40:00,840
That's actually rad because it's the same premise as the cloud of not me not having to go and rack and stack my own stuff and not having to like think ahead and buy and like, you know, procure all these things, but at the same and but have the ability to use like compute power.

778
00:40:01,080 --> 00:40:04,920
So that's actually like the best of both worlds, but much cheaper.

779
00:40:05,500 --> 00:40:07,320
I think it could come out cheaper.

780
00:40:07,320 --> 00:40:10,920
So like, in a sense, AWS has two kinds of margins on their product.

781
00:40:11,220 --> 00:40:15,780
One margin is like they charge you more for the machine than it costs them, which is fine.

782
00:40:16,040 --> 00:40:16,860
Like, that's correct.

783
00:40:17,040 --> 00:40:18,080
That's good for them.

784
00:40:19,360 --> 00:40:24,100
The second is like on, you know, I buy machines that are too big for the thing I'm doing.

785
00:40:24,900 --> 00:40:33,680
And that's traditionally one of the arguments for the PaaS systems is like, you know, don't go and buy a bunch of, you know, vCPUs and leave them lying around on a VM for spikes.

786
00:40:34,380 --> 00:40:40,180
But, you know, another way to approach that is like you have a pool of vCPUs that you buy for all of your machines.

787
00:40:40,960 --> 00:40:42,000
And, you know, you handle it there.

788
00:40:42,840 --> 00:40:47,800
There's interesting open questions about like, how regularly do we let you dynamically resize your pool?

789
00:40:48,240 --> 00:40:54,240
Like, you know, I price it per hour for the business case.

790
00:40:54,500 --> 00:40:57,900
Like, it seems reasonable that we'd have APIs that let you resize the pool hourly.

791
00:40:58,080 --> 00:40:59,620
You know, we have to figure that out.

792
00:40:59,620 --> 00:41:02,060
That's, you know, it gets more complicated there.

793
00:41:02,100 --> 00:41:08,060
But, you know, there's a stack of spike demand things I'd love to support somehow down the road.

794
00:41:08,300 --> 00:41:18,200
Do you think you'll ever get to the point where you do like spot pricing when like compute is like really not being used and then it gets cheaper and then more expensive when it gets really expensive?

795
00:41:18,640 --> 00:41:20,220
I would love to do that.

796
00:41:20,520 --> 00:41:23,040
I think you need large numbers to make that work.

797
00:41:23,040 --> 00:41:34,060
But it's a fundamentally possible thing in the sense that if you're racking machines in London for a UK audience, then at 2 a.m. in London, those machines are going to be less used.

798
00:41:34,500 --> 00:41:35,660
There's going to be extra resources.

799
00:41:36,000 --> 00:41:38,200
And so you should be able to spot price them at that time of day.

800
00:41:39,020 --> 00:41:40,440
Like, that just makes sense.

801
00:41:40,820 --> 00:41:42,400
And it's just complicated.

802
00:41:42,920 --> 00:41:45,660
And so it's not, you know, we're a team of eight right now.

803
00:41:45,800 --> 00:41:46,680
I said nine yesterday.

804
00:41:46,760 --> 00:41:47,140
I was wrong.

805
00:41:47,240 --> 00:41:48,460
We're eight because I can't count.

806
00:41:48,800 --> 00:41:49,700
You counted the agent.

807
00:41:50,180 --> 00:41:53,220
It's not meant to do that.

808
00:41:53,260 --> 00:41:53,780
They're not people.

809
00:41:55,120 --> 00:41:57,640
Actually, that brings up another question I had.

810
00:41:57,720 --> 00:41:59,040
It was like you integrated Shelley.

811
00:41:59,200 --> 00:42:03,680
You have this agent built into your like VM interface with a proxy available.

812
00:42:03,820 --> 00:42:08,160
Can you explain technically how that works and why you decided to go do that?

813
00:42:08,720 --> 00:42:09,040
Yeah.

814
00:42:10,220 --> 00:42:12,580
It's very easy to build an agent harness.

815
00:42:12,740 --> 00:42:16,920
Honestly, it's not super easy, but it's much easier than writing a language compiler.

816
00:42:16,920 --> 00:42:20,520
You know, in terms of like from a software engineering perspective, it's like everyone

817
00:42:20,520 --> 00:42:25,460
should do an agent harness 101 course in computer science.

818
00:42:25,780 --> 00:42:28,780
And honestly, if you spend a whole semester building an agent harness, it's going to come

819
00:42:28,780 --> 00:42:30,240
out about as well as Cloud Code.

820
00:42:30,380 --> 00:42:31,400
Honestly, there's nothing in there.

821
00:42:31,880 --> 00:42:36,680
Like, I don't know why they built a bunch of stuff in there for like sandboxing and

822
00:42:36,680 --> 00:42:36,920
stuff.

823
00:42:36,980 --> 00:42:41,460
But if you skip all that and just let it be root on a VM, it's really easy to write.

824
00:42:41,500 --> 00:42:42,320
Your sandbox is outside the harness.

825
00:42:42,720 --> 00:42:42,940
Exactly.

826
00:42:42,940 --> 00:42:43,500
This is easier.

827
00:42:43,780 --> 00:42:44,440
Yeah, that's right.

828
00:42:44,440 --> 00:42:48,220
So like almost all the machine, you know, it exposes a couple of tool calls and it's a

829
00:42:48,220 --> 00:42:49,940
big while loop that calls an LLM.

830
00:42:50,020 --> 00:42:51,000
That's just easy to write.

831
00:42:51,640 --> 00:42:53,080
And it's not just easy to write.

832
00:42:53,140 --> 00:42:54,440
It's really fun to customize.

833
00:42:55,160 --> 00:42:59,920
Like, I actually think, you know, I suspect those things will end up open source because

834
00:42:59,920 --> 00:43:03,120
engineers are going to want to customize those more than they ever wanted to customize a

835
00:43:03,120 --> 00:43:03,600
text editor.

836
00:43:03,960 --> 00:43:06,860
And like we have no end to customization of those.

837
00:43:07,960 --> 00:43:09,740
So it's quite easy to build one.

838
00:43:09,880 --> 00:43:10,700
So that's that's nice.

839
00:43:10,840 --> 00:43:12,880
That's your first thing is like, what's the cost?

840
00:43:12,880 --> 00:43:14,380
Then there's a question of what's the benefit?

841
00:43:15,420 --> 00:43:18,420
And like one of our problems was like, oh, what if someone shows up to XeDev on their

842
00:43:18,420 --> 00:43:18,720
phone?

843
00:43:19,100 --> 00:43:20,140
Because people do that.

844
00:43:20,300 --> 00:43:21,020
I do that.

845
00:43:21,120 --> 00:43:22,220
I learn about things on my phone.

846
00:43:22,500 --> 00:43:24,700
Be cool if they could make a VM and do something useful there.

847
00:43:25,400 --> 00:43:30,880
And like with all these CLI agents, what's your story for like how they would log in and do

848
00:43:30,880 --> 00:43:31,140
something?

849
00:43:31,240 --> 00:43:32,860
It's like, oh, well, we need a web based agent.

850
00:43:33,640 --> 00:43:35,160
What web based agent can we use?

851
00:43:35,300 --> 00:43:36,020
Like, oh, crickets.

852
00:43:36,320 --> 00:43:37,020
You know, there's nothing.

853
00:43:37,580 --> 00:43:39,100
For some reason, it's a thing.

854
00:43:39,100 --> 00:43:41,880
So, you know, we built Shelly, which is a web based agent.

855
00:43:42,040 --> 00:43:46,240
And we said we'll include some LLM tokens as part of the base plan because that's super

856
00:43:46,240 --> 00:43:47,160
useful to individuals.

857
00:43:47,280 --> 00:43:47,980
Let them buy more.

858
00:43:48,780 --> 00:43:54,060
I'd love to just plug the the subscriptions in and maybe we can now with the open AI one.

859
00:43:54,140 --> 00:43:57,920
I need to be really cool because I'm just thinking like I could be like sitting there

860
00:43:57,920 --> 00:44:03,320
with my son going to bed and like like just orchestrating a bunch of programs to be written

861
00:44:03,320 --> 00:44:04,420
while I'm like doing that.

862
00:44:04,600 --> 00:44:04,940
That's rad.

863
00:44:05,040 --> 00:44:08,340
And then by the time you're done putting your kid to bed, you can go and like code review

864
00:44:08,340 --> 00:44:10,000
and see what it actually made.

865
00:44:10,120 --> 00:44:10,780
That's pretty rad.

866
00:44:11,340 --> 00:44:15,920
Yeah, I, you know, I use Shelly for actually all the XCDEV programming I do these days

867
00:44:15,920 --> 00:44:20,460
precisely because it's easy to use for my phone and it works just as well as codex or

868
00:44:20,460 --> 00:44:22,160
code, code, code, code, code.

869
00:44:22,360 --> 00:44:23,440
Always have trouble saying that.

870
00:44:23,480 --> 00:44:25,580
I love a good efficiency tool like that.

871
00:44:25,660 --> 00:44:29,360
I can do my life, but also be like being productive at the same time.

872
00:44:29,760 --> 00:44:36,120
Yeah, that's it's also, you know, you know, we're trying to it's just it's super useful

873
00:44:36,120 --> 00:44:38,500
to have there because like, oh, I want to set up a VM.

874
00:44:38,660 --> 00:44:39,700
I want to put something on it.

875
00:44:39,800 --> 00:44:43,200
I say to myself, it's like, well, I've got an app, get update and app, get install a

876
00:44:43,200 --> 00:44:47,200
bunch of stuff or you can write a one line prompt to an agent running on the machine

877
00:44:47,200 --> 00:44:48,000
and let it do it.

878
00:44:48,440 --> 00:44:50,020
And how do you, how do you do that today?

879
00:44:50,040 --> 00:44:55,220
Like you mentioned, you have an idea you, you know, XCDEV new and then give it a prompt

880
00:44:55,220 --> 00:44:56,140
and then you just go away.

881
00:44:56,220 --> 00:45:00,800
Like I, I, my first experience was basically that with interacting with Shelly and saying

882
00:45:00,800 --> 00:45:05,020
like, okay, I want to try this thing, but I spent probably three, four hours like prompting

883
00:45:05,020 --> 00:45:06,260
it again to tweaking things.

884
00:45:06,300 --> 00:45:10,080
And it wasn't like a fully autonomous, you know, dangerously skip off permissions.

885
00:45:10,280 --> 00:45:16,160
I mean, the, if I'm doing an idea just out of the blue, yes, I just type something into

886
00:45:16,160 --> 00:45:17,800
XCDEV new and let it go.

887
00:45:18,640 --> 00:45:23,740
Uh, and the first thing I see is like, is there anything potentially useful here?

888
00:45:23,880 --> 00:45:28,840
You have to iterate to get any, you know, to actually get it to a good place, but that's

889
00:45:28,840 --> 00:45:29,760
the nature of agents today.

890
00:45:29,780 --> 00:45:30,920
They can't do it all themselves.

891
00:45:31,700 --> 00:45:35,340
Honestly, most of my ideas these days are in our code base and they're like, how do

892
00:45:35,340 --> 00:45:39,160
I, uh, uh, you know, I want to make a change to how XCDEV works.

893
00:45:39,900 --> 00:45:44,980
Uh, and for those, I've gone through several stages of like what I do.

894
00:45:46,400 --> 00:45:51,500
Uh, the thing, the, the thing that's most reliable right now is I have a whole set of

895
00:45:51,500 --> 00:45:53,740
VMs and I treat them as like reusable Git branches.

896
00:45:54,020 --> 00:45:56,760
And like, I just drop into one, reset the thing.

897
00:45:56,760 --> 00:46:00,580
I have a little reset button, uh, and then, uh, uh, type something into Shelley and let

898
00:46:00,580 --> 00:46:02,020
it, let it rip and see what happens.

899
00:46:02,120 --> 00:46:05,020
Then it brings up all of XCDEV inside one of our VMs.

900
00:46:05,060 --> 00:46:08,940
And I just take, play with it and like start creating VMs inside this little, you know,

901
00:46:08,980 --> 00:46:09,840
embedded universe.

902
00:46:11,220 --> 00:46:14,400
But, uh, the thing I've been working on is the same thing my colleagues have all built.

903
00:46:14,500 --> 00:46:17,940
We've each, each of us is building one of these things, which is an orchestrator.

904
00:46:18,660 --> 00:46:23,140
And, you know, one of the, uh, the basic idea is I have one VM and I say, I want to do a

905
00:46:23,140 --> 00:46:26,420
thing and then it starts a new VM, gets the Git repo checked out.

906
00:46:26,420 --> 00:46:27,620
It has like a standard play.

907
00:46:28,120 --> 00:46:33,060
And then, uh, it takes the results and drops them into some sort of task list like thing.

908
00:46:33,160 --> 00:46:36,100
It's every time I open the orchestrator, there's like all the ideas I've tried are in

909
00:46:36,100 --> 00:46:36,460
this thing.

910
00:46:36,920 --> 00:46:40,240
And we've thought about building a generic one of these, and there's several companies

911
00:46:40,240 --> 00:46:44,620
out there that have built one of these, but what we've seen so far is like each one of

912
00:46:44,620 --> 00:46:47,720
us that's built one has built such a radically different orchestrator.

913
00:46:48,240 --> 00:46:53,420
And like, they're also, it takes like an hour to build one, uh, that, uh, we're not ready

914
00:46:53,420 --> 00:46:55,940
to commit to building one of those generically for everyone.

915
00:46:55,940 --> 00:46:57,520
So what we're trying to do is build all the machinery.

916
00:46:57,740 --> 00:47:03,400
And so things are about to launch, uh, that, uh, a couple of our customers using, we're

917
00:47:03,400 --> 00:47:08,780
going to make these public in a moment, uh, is, uh, a scoped, uh, SSH key for Exedev,

918
00:47:09,260 --> 00:47:13,160
uh, that every VM gets so that every VM in Exedev can make new VMs.

919
00:47:13,160 --> 00:47:17,280
It gets its own little like sub account universe, uh, where a tag gets applied.

920
00:47:17,620 --> 00:47:21,900
And so like it LSs and it can't see all your other VMs, like the, you know, it's its

921
00:47:21,900 --> 00:47:24,240
universe starts empty, but then it can make as many as it wants.

922
00:47:24,260 --> 00:47:25,340
And they're part of your account.

923
00:47:25,340 --> 00:47:32,260
Uh, and that lets, uh, that lets you write programs that use VMs and, uh, and that is

924
00:47:32,260 --> 00:47:33,600
like the basis of an orchestrator.

925
00:47:33,700 --> 00:47:39,080
And then if you check out our integrations page, this is our, this is our version of

926
00:47:39,080 --> 00:47:39,760
secrets management.

927
00:47:40,280 --> 00:47:45,400
And so one of the things we've realized is almost all of the secrets we have, and I'm

928
00:47:45,400 --> 00:47:50,420
not all of them, but almost all of them are basically HTTP headers that you send to a

929
00:47:50,420 --> 00:47:50,820
service.

930
00:47:50,880 --> 00:47:53,640
There are token that get embedded in a HTTP request.

931
00:47:54,200 --> 00:47:59,000
Uh, so we set up HTTP proxies, uh, outside the VM with the token in them.

932
00:47:59,480 --> 00:48:04,700
And so from inside the VM, you can send a request to like to GitHub and you can pull

933
00:48:04,700 --> 00:48:06,740
private repos and push them and everything.

934
00:48:07,220 --> 00:48:10,720
But what's actually going on is you did a GitHub OAuth flow outside and that's what

935
00:48:10,720 --> 00:48:12,260
the token lives outside the VM.

936
00:48:12,400 --> 00:48:14,440
So the agent can't X fill it or anything like this.

937
00:48:14,520 --> 00:48:18,580
And the neat thing is you can set up all those integrations on tags, uh, so that, you

938
00:48:18,580 --> 00:48:20,200
know, your orchestrator can go and make a new VM.

939
00:48:20,260 --> 00:48:23,480
It's immediately got GitHub access to whatever repos it wanted to give it to.

940
00:48:23,480 --> 00:48:27,400
It's got access to your Clickhouse logs because it's a Clickhouse integration, you

941
00:48:27,400 --> 00:48:29,480
know, whatever external services you want.

942
00:48:29,920 --> 00:48:33,820
And so our whole goal is to build all of the infrastructure, to let teams write their

943
00:48:33,820 --> 00:48:38,220
own orchestrators, because my guess is for the foreseeable future, engineers have to

944
00:48:38,220 --> 00:48:39,520
build that stuff themselves.

945
00:48:40,080 --> 00:48:42,980
And if you try buying one off the shelf, you're going to find a really awkward piece

946
00:48:42,980 --> 00:48:46,560
of software that almost solves your problem that you'd have to customize a bunch.

947
00:48:46,700 --> 00:48:51,880
And like the work of customizing it, it's easier just to open a, open a, uh, uh, an

948
00:48:51,880 --> 00:48:52,860
agent and start building it.

949
00:48:53,320 --> 00:48:57,780
Uh, if someone gives you all the pieces, you know, I mean, this is the idea behind like

950
00:48:57,780 --> 00:48:58,440
Gastown, right?

951
00:48:58,440 --> 00:49:01,840
Like Gastown idea is an orchestrator for all these agents.

952
00:49:01,920 --> 00:49:08,360
And so like layering these things in pieces of like how they get contained or like the

953
00:49:08,360 --> 00:49:13,260
harness at the very bottom of like the tools calling LLM, we have an orchestrator on top

954
00:49:13,260 --> 00:49:13,520
of that.

955
00:49:13,580 --> 00:49:17,820
That's calling all the, all the harnesses and you're boxing all that into container or

956
00:49:17,820 --> 00:49:24,520
into, into VMs, a VM sandbox that also can do more VMs so that each agent is, is contained

957
00:49:24,520 --> 00:49:25,180
inside of that.

958
00:49:25,320 --> 00:49:26,780
Each, I guess at that point it's a harness, right?

959
00:49:26,780 --> 00:49:29,620
So you have the orchestrator on top of a bunch of VMs.

960
00:49:29,780 --> 00:49:29,960
Yeah.

961
00:49:30,320 --> 00:49:33,180
That's, and there's several, there's several ways you can arrange this.

962
00:49:34,340 --> 00:49:36,520
Someone yesterday told me about type two agents.

963
00:49:36,640 --> 00:49:37,800
I was like, what's a type two agent?

964
00:49:38,260 --> 00:49:39,480
You know, I thought I knew about this stuff.

965
00:49:39,620 --> 00:49:40,880
Is that like type two fun?

966
00:49:42,620 --> 00:49:44,480
No, no, because I'm really into type two fun.

967
00:49:44,700 --> 00:49:45,520
This was something else.

968
00:49:45,520 --> 00:49:52,060
This is, I mean, isn't programming just type two fun?

969
00:49:52,200 --> 00:49:54,000
Like you just stay yelling at a computer for hours.

970
00:49:54,040 --> 00:49:57,300
Because like the whole time you're just like, this is horrible and it's the worst.

971
00:49:57,340 --> 00:49:58,940
And then all you're like, but I love it.

972
00:49:58,940 --> 00:50:00,240
When you like something good happens.

973
00:50:00,400 --> 00:50:01,080
Oh my God.

974
00:50:01,340 --> 00:50:02,460
I just had a revolution.

975
00:50:02,680 --> 00:50:06,800
I like, I usually judge people that like type two fun, but you just made me look at

976
00:50:06,800 --> 00:50:07,260
myself.

977
00:50:07,260 --> 00:50:07,960
It's you Autumn.

978
00:50:08,180 --> 00:50:08,400
It's you.

979
00:50:08,400 --> 00:50:11,140
Wait, are you, are you going to, are you going to make me Google type two fun?

980
00:50:11,260 --> 00:50:11,900
I don't know this.

981
00:50:11,900 --> 00:50:15,280
You have to like, you need to know this.

982
00:50:15,480 --> 00:50:15,680
Okay.

983
00:50:18,420 --> 00:50:18,820
Pause.

984
00:50:19,120 --> 00:50:22,480
I need to know David, whether it's your type two fun, because just in the like

985
00:50:22,480 --> 00:50:28,100
brief hour of knowing you, your type two fun tracks, like you're like, let me go

986
00:50:28,100 --> 00:50:29,560
build all the hard things.

987
00:50:30,780 --> 00:50:31,720
You mean my type two fun?

988
00:50:31,780 --> 00:50:32,460
That's not programming.

989
00:50:33,420 --> 00:50:34,500
What else do I do?

990
00:50:34,660 --> 00:50:36,960
Uh, I mean, exercise is my type two fun.

991
00:50:37,060 --> 00:50:37,900
It's just, it's awful.

992
00:50:38,240 --> 00:50:39,760
You know, I, I hate it.

993
00:50:39,820 --> 00:50:41,880
It's, it's the worst, but, uh, you know, I do it.

994
00:50:41,880 --> 00:50:43,140
You know, running all those sorts of things.

995
00:50:43,220 --> 00:50:43,580
It's great.

996
00:50:44,080 --> 00:50:46,280
I just feel so called out and attacked right now.

997
00:50:46,380 --> 00:50:50,380
Like I've always been like judging the people that do like type two fun.

998
00:50:50,500 --> 00:50:53,380
And I was like, Oh my God, my life is built around type two fun.

999
00:50:53,480 --> 00:50:55,080
And this whole time I didn't realize it.

1000
00:50:55,660 --> 00:50:58,800
No, it just, you just, uh, found a whole new group of people to love.

1001
00:50:58,880 --> 00:51:02,040
You know, now you get them all those, all those people, you know, when you, when

1002
00:51:02,040 --> 00:51:04,960
you see them like doing an iron man or something, we're going to be friends,

1003
00:51:05,020 --> 00:51:08,980
but like, I just need to like, somehow, like, I need you to find a way to put

1004
00:51:08,980 --> 00:51:13,140
like your facial expressions and the hilarious, like jokes in between.

1005
00:51:13,360 --> 00:51:14,620
Cause like they're fire.

1006
00:51:19,580 --> 00:51:20,760
That's very kind of you.

1007
00:51:20,960 --> 00:51:24,200
Uh, two days ago, someone told me I, uh, had a face made for radio.

1008
00:51:25,680 --> 00:51:26,120
Jeez.

1009
00:51:27,100 --> 00:51:28,160
Who said that?

1010
00:51:28,240 --> 00:51:29,240
They were a hater.

1011
00:51:29,500 --> 00:51:31,040
They said it themselves too.

1012
00:51:31,120 --> 00:51:31,760
And I was like, they're right.

1013
00:51:31,960 --> 00:51:32,140
I get it.

1014
00:51:32,140 --> 00:51:33,800
The ones that also watch the podcast.

1015
00:51:35,440 --> 00:51:37,980
No, this is someone who knew what radio is.

1016
00:51:38,460 --> 00:51:40,660
See, I, oh, wow.

1017
00:51:40,780 --> 00:51:41,820
Does that mean we're old?

1018
00:51:43,740 --> 00:51:44,160
Sorry.

1019
00:51:48,640 --> 00:51:49,880
Oh, I'll stop now.

1020
00:51:50,220 --> 00:51:53,860
Also, I feel like we've lost like that part of storytelling and I think that's

1021
00:51:53,860 --> 00:51:56,320
what people are obsessed with TikTok, you know, like the being able to see

1022
00:51:56,320 --> 00:51:59,900
people's facial expressions and react to like saying something funny.

1023
00:52:00,040 --> 00:52:03,740
Like that's like such a missed art because sometimes you like drop a one

1024
00:52:03,740 --> 00:52:06,660
liner and it's hilarious and you know, it's hilarious, but you wish you saw

1025
00:52:06,660 --> 00:52:09,780
the other person's face and then like every time you guys say something and

1026
00:52:09,780 --> 00:52:13,160
it's like too real and you both start laughing before it even gets out of

1027
00:52:13,160 --> 00:52:14,160
your mouth, you know what I mean?

1028
00:52:14,200 --> 00:52:16,900
I'm like, I miss seeing people's facial expressions.

1029
00:52:18,500 --> 00:52:21,420
I've always struggled with jokes for that reason because I find them really funny.

1030
00:52:22,380 --> 00:52:23,620
Getting them out is hard.

1031
00:52:24,000 --> 00:52:26,560
But like me too, but like, it makes it so much funnier.

1032
00:52:26,560 --> 00:52:28,360
Cause you're just like, oh, it's going to be so good.

1033
00:52:28,360 --> 00:52:29,300
Like I can see it.

1034
00:52:30,420 --> 00:52:31,080
I agree.

1035
00:52:31,180 --> 00:52:31,480
I agree.

1036
00:52:31,600 --> 00:52:31,880
It's fun.

1037
00:52:32,220 --> 00:52:36,520
So there's a British TV show I just watched recently called Last One Laughing.

1038
00:52:37,560 --> 00:52:40,180
But why is British comedy so much funnier?

1039
00:52:41,040 --> 00:52:44,140
It's because it's different and comedy is always funny when it's different.

1040
00:52:44,360 --> 00:52:44,840
And you're right.

1041
00:52:44,880 --> 00:52:46,340
It's all specialized in sarcasm.

1042
00:52:46,480 --> 00:52:47,340
Like nobody's business.

1043
00:52:47,780 --> 00:52:48,300
Like, yeah.

1044
00:52:48,420 --> 00:52:49,100
And misery.

1045
00:52:49,680 --> 00:52:54,380
That's a, that's a, I mean, if you lived in England, wouldn't

1046
00:52:54,380 --> 00:52:55,520
you have to be funny too?

1047
00:52:55,980 --> 00:52:57,580
It's like, it's just a, sorry.

1048
00:52:57,680 --> 00:52:59,000
I live in Seattle.

1049
00:52:59,200 --> 00:53:00,580
It's cold and rainy here too.

1050
00:53:01,360 --> 00:53:02,640
I really like Seattle.

1051
00:53:02,760 --> 00:53:03,200
It's nice.

1052
00:53:03,680 --> 00:53:04,400
I like it too.

1053
00:53:04,460 --> 00:53:05,620
It's a nice kind of rain.

1054
00:53:06,040 --> 00:53:07,440
Then you have a, you have a Harbor.

1055
00:53:07,560 --> 00:53:07,980
It's really good.

1056
00:53:08,200 --> 00:53:08,580
The water.

1057
00:53:09,000 --> 00:53:12,380
But yeah, Last One Laughing is great because it's a bunch of comedians in a room and

1058
00:53:13,020 --> 00:53:15,140
they lose and have to leave if they laugh.

1059
00:53:15,360 --> 00:53:17,380
And so they were just telling jokes to each other and having

1060
00:53:17,380 --> 00:53:18,820
to deadpan it the whole time.

1061
00:53:19,940 --> 00:53:21,680
Like it's, it's so hilarious.

1062
00:53:21,680 --> 00:53:24,320
And also sometimes I'm watching him like, am I meant to laugh now?

1063
00:53:24,380 --> 00:53:25,260
I don't even know.

1064
00:53:25,400 --> 00:53:25,940
Like what's happening.

1065
00:53:26,320 --> 00:53:28,600
Was that a joke deliberately or accidentally?

1066
00:53:31,400 --> 00:53:32,040
I swear.

1067
00:53:32,500 --> 00:53:36,380
Wait, like I liked your, actually like liked your joke about like laughing

1068
00:53:36,380 --> 00:53:38,960
through misery does not seem like sometimes that's the only way you

1069
00:53:38,960 --> 00:53:40,160
get through like a shitty time.

1070
00:53:41,420 --> 00:53:45,160
You're like, if I, like, if you can't like cry, you gotta laugh about it.

1071
00:53:45,200 --> 00:53:50,360
Cause if not, like a hundred percent, you know, that's a, I think that's why

1072
00:53:50,360 --> 00:53:54,120
like when you see like a developer meme, it hits so hard cause you're like,

1073
00:53:54,220 --> 00:53:55,900
Oh my God, somebody else struggles.

1074
00:53:58,180 --> 00:53:59,660
That's why I laugh so much.

1075
00:53:59,660 --> 00:54:01,880
It's still with the tears.

1076
00:54:04,320 --> 00:54:04,800
Yeah, exactly.

1077
00:54:05,200 --> 00:54:05,940
You got it.

1078
00:54:06,580 --> 00:54:09,100
So, uh, that's uh, but yeah.

1079
00:54:09,220 --> 00:54:10,060
Uh, let's see.

1080
00:54:10,180 --> 00:54:10,800
What are we talking about?

1081
00:54:10,820 --> 00:54:12,320
We're talking about computers.

1082
00:54:12,320 --> 00:54:13,460
I was going to ask what's next.

1083
00:54:14,040 --> 00:54:14,440
Next.

1084
00:54:14,780 --> 00:54:15,020
Yeah.

1085
00:54:15,360 --> 00:54:15,920
What's next.

1086
00:54:16,000 --> 00:54:19,960
I mean, that's one direction we're going in is like your VMs need VMs.

1087
00:54:20,040 --> 00:54:21,700
You know, it's like, you just need a computer.

1088
00:54:21,900 --> 00:54:23,280
It turns out your computer needs a computer.

1089
00:54:23,420 --> 00:54:24,740
Like we just need more and more computers.

1090
00:54:24,800 --> 00:54:25,600
That's absolutely it.

1091
00:54:26,080 --> 00:54:29,540
Uh, but those, uh, there's several other things we're working on simultaneously.

1092
00:54:29,540 --> 00:54:33,620
One is those integrations that secrets management, like we have to help as much

1093
00:54:33,620 --> 00:54:35,360
as we can there because it's hard.

1094
00:54:35,800 --> 00:54:39,940
And like I said, I don't think there's an easy answer to any of this, but we can

1095
00:54:39,940 --> 00:54:42,680
build a whole bunch of machinery to make it easy for you.

1096
00:54:42,760 --> 00:54:48,420
Like if you want, uh, every VM you start to just be on your tail net as part

1097
00:54:48,420 --> 00:54:50,680
of a tail scale integration, we should just make that work.

1098
00:54:50,960 --> 00:54:52,220
And so that's on our list.

1099
00:54:52,520 --> 00:54:56,520
I literally was trying that the other night where I was trying to run the, um,

1100
00:54:56,520 --> 00:54:58,580
the apertures, their new like AI proxy.

1101
00:54:58,740 --> 00:55:01,140
Cause like, again, I'm running local models and I was like, Hey, I like

1102
00:55:01,140 --> 00:55:03,220
Shelly, but I also want to be able to use my local models.

1103
00:55:03,680 --> 00:55:07,920
And so I, but I didn't want to go through the whole tail scale off flow of doing

1104
00:55:07,920 --> 00:55:08,280
this thing.

1105
00:55:08,300 --> 00:55:12,240
And it looks like there's a, like a tail scale aperture proxy that they're working

1106
00:55:12,240 --> 00:55:14,180
on, um, which will probably help there.

1107
00:55:14,280 --> 00:55:16,760
Um, so yeah, that was, those were like the things that I was kind of looking at.

1108
00:55:16,780 --> 00:55:23,380
I'm like, how would I get this XC dev VM to talk to my aperture so that I can, again,

1109
00:55:23,420 --> 00:55:24,680
all the auth is stored separately.

1110
00:55:24,720 --> 00:55:25,700
It's outside the VM, right?

1111
00:55:25,700 --> 00:55:28,720
That bootstrapping problem is always hard where I'm like, where do I put all these

1112
00:55:28,720 --> 00:55:29,100
secrets?

1113
00:55:29,600 --> 00:55:31,820
And then I have to like copy and paste them every time.

1114
00:55:31,840 --> 00:55:34,100
I'm like starting something new and go through new OAuth flows.

1115
00:55:34,200 --> 00:55:34,900
I was like, no, I don't want to do that.

1116
00:55:34,960 --> 00:55:37,940
I want to do it once and then scope it for the thing that I'm working on.

1117
00:55:38,240 --> 00:55:38,460
Yeah.

1118
00:55:38,660 --> 00:55:42,340
Well, tail scale has some good OAuth flows built into it that we can leverage for an

1119
00:55:42,340 --> 00:55:46,200
integration into XC dev, which lets us, you know, set it up on a tail net when you

1120
00:55:46,200 --> 00:55:47,760
create the thing without any extra steps.

1121
00:55:48,480 --> 00:55:52,760
And lots of other, uh, larger, more mature services have that too.

1122
00:55:52,760 --> 00:55:56,900
So it's actually possible for us to set it up so that if you go through a flow once,

1123
00:55:57,000 --> 00:56:02,560
that's not easy, but you do it, uh, then, uh, you can actually do an AWS integration

1124
00:56:02,560 --> 00:56:07,240
where each VM you bring up with a tag can automatically reach some S3 buckets you

1125
00:56:07,240 --> 00:56:08,620
have without any extra work.

1126
00:56:08,680 --> 00:56:12,960
You do the same with GCP, lots of other services, uh, Slack integration.

1127
00:56:13,000 --> 00:56:13,600
Where's that?

1128
00:56:13,680 --> 00:56:14,800
You know, there's so much to build.

1129
00:56:14,980 --> 00:56:20,220
Like our problem is really like our, you know, we have so much to do, uh, and you

1130
00:56:20,220 --> 00:56:21,420
know, only so many hours in the day.

1131
00:56:21,420 --> 00:56:25,580
That's kind of exciting though, because I feel like a lot of times people that have

1132
00:56:25,580 --> 00:56:29,100
AI startups, it's like, well, I wouldn't say that you have an AI startup, but let's

1133
00:56:29,100 --> 00:56:33,280
just like in this realm and this time, it's always like these really random,

1134
00:56:33,560 --> 00:56:36,400
obscure niche problem solutions.

1135
00:56:36,780 --> 00:56:40,440
And they're like, I'm just like, dude, can you do something to make my life easier

1136
00:56:40,440 --> 00:56:44,500
with AI and like yours is making lives easier so you can go build things.

1137
00:56:44,520 --> 00:56:47,860
But like, it's also so exciting because you have all these problems to solve.

1138
00:56:47,920 --> 00:56:51,280
And like, I feel like right now people are just making up problems to solve and

1139
00:56:51,280 --> 00:56:55,040
like, that's a legitimate, like you have like a very clear roadmap of all

1140
00:56:55,040 --> 00:56:56,020
these cool things you can add.

1141
00:56:56,160 --> 00:57:00,100
Like, is it exciting to be working on something that like you can actually,

1142
00:57:00,620 --> 00:57:02,080
you know, it's so much fun.

1143
00:57:02,380 --> 00:57:05,600
And, you know, we have, we have some really great users who like, you know,

1144
00:57:05,600 --> 00:57:09,120
there's a discord where people hang out and like tell us when we break things and

1145
00:57:09,120 --> 00:57:11,280
give us ideas and like ask for features.

1146
00:57:11,600 --> 00:57:15,340
And like the feedback we get from users is part of why we have an endless roadmap.

1147
00:57:15,620 --> 00:57:17,880
Like they keep coming up with good stuff we have to build.

1148
00:57:17,880 --> 00:57:22,120
That's so rad because a lot of people do not do feedback and they don't realize

1149
00:57:22,120 --> 00:57:25,600
like the, like what a community can offer like their company.

1150
00:57:25,800 --> 00:57:29,700
So that's rad that you are intentionally building that community and getting that

1151
00:57:29,700 --> 00:57:34,700
feedback because like, that's just like, it's so much, it adds so much to your

1152
00:57:34,700 --> 00:57:38,880
process of becoming like this really awesome product that people enjoy using.

1153
00:57:39,160 --> 00:57:43,240
Because developers love to build something that they like because it's cool and they

1154
00:57:43,240 --> 00:57:45,040
built it and nobody wants to use it.

1155
00:57:45,100 --> 00:57:46,160
And it's absolutely horrible.

1156
00:57:46,160 --> 00:57:49,520
You know, like you've almost created a way to have a better software

1157
00:57:49,520 --> 00:57:50,760
development loop, you know?

1158
00:57:51,820 --> 00:57:57,040
I feel like you're exactly right with, especially in the, the age of, of idea to

1159
00:57:57,040 --> 00:57:59,220
execution is cheaper, right?

1160
00:57:59,220 --> 00:58:02,220
Like just the, like I have a list of ideas and I can execute on them.

1161
00:58:02,320 --> 00:58:06,200
I am buying more domains because I'm actually executing some of them.

1162
00:58:06,240 --> 00:58:10,180
But be real people bought, people made crappy software because they thought they

1163
00:58:10,180 --> 00:58:11,920
were smarter than everybody for forever.

1164
00:58:11,920 --> 00:58:16,620
Now you can just do it faster and like do it at a bigger scale.

1165
00:58:16,820 --> 00:58:21,340
I think the problem I having is that sharing of the thing I built or the thing

1166
00:58:21,340 --> 00:58:25,500
I created from an idea that I didn't invest time into the thing that I want you to try

1167
00:58:25,500 --> 00:58:29,420
is really weird to me because it's, you know, some people call it slop.

1168
00:58:29,680 --> 00:58:32,600
Some people are like, if I didn't do the AI prompts, then I don't know how

1169
00:58:32,600 --> 00:58:33,780
yours works, so I don't want it.

1170
00:58:33,780 --> 00:58:36,140
And I'm like, that's weird too, because it's like, it's still software at the

1171
00:58:36,140 --> 00:58:40,400
end of the day, but I think the lack of taste in a lot of it of just like, Oh,

1172
00:58:40,400 --> 00:58:41,400
well, this model did it.

1173
00:58:41,460 --> 00:58:44,980
So I'm, I'm relying on Claude to have taste in what not to include.

1174
00:58:45,020 --> 00:58:48,400
And I see these people building these things that are more complicated than

1175
00:58:48,400 --> 00:58:50,880
they need to be because they didn't put any thought into, like,

1176
00:58:51,400 --> 00:58:53,800
come back to the software development life cycle.

1177
00:58:53,800 --> 00:58:58,180
Like, I think a lot of times the people saying that things are AI slop also

1178
00:58:58,180 --> 00:59:02,500
because like what, whether you're coding with an agent or not, you should be

1179
00:59:02,500 --> 00:59:07,360
thinking about how it's built, having like opinionated, like, you know, facts

1180
00:59:07,360 --> 00:59:10,200
about how you want to build this and your architecture and figuring out

1181
00:59:10,200 --> 00:59:11,000
these different things.

1182
00:59:11,160 --> 00:59:14,560
And that is what happens before coding ever starts, right?

1183
00:59:14,640 --> 00:59:18,420
Like the planning and the requirements and thinking about what you want to

1184
00:59:18,420 --> 00:59:19,840
build and how you're going to break it down.

1185
00:59:20,060 --> 00:59:23,440
And you can do that with an agent and you can do it with regular coding, but

1186
00:59:23,440 --> 00:59:25,180
that's the whole doing that first.

1187
00:59:25,280 --> 00:59:25,500
Right.

1188
00:59:25,920 --> 00:59:28,240
And I think that's a problem that we've always had.

1189
00:59:28,300 --> 00:59:29,460
It's just exasperated.

1190
00:59:29,940 --> 00:59:30,740
Yeah, that's, that's right.

1191
00:59:30,820 --> 00:59:32,840
I was writing slop long before we had AI.

1192
00:59:32,980 --> 00:59:33,520
That's what I'm saying.

1193
00:59:33,640 --> 00:59:34,420
Like, you know what I mean?

1194
00:59:34,760 --> 00:59:37,560
I can write terrible code all on my own, right?

1195
00:59:37,640 --> 00:59:38,120
This is not a.

1196
00:59:38,120 --> 00:59:42,740
This is just the point where you either continue to like, are you iterating on

1197
00:59:42,740 --> 00:59:44,100
the first thing that you got?

1198
00:59:44,240 --> 00:59:45,620
Where do you planning it properly?

1199
00:59:45,860 --> 00:59:49,980
Like, to me, it's like the same, but people could just can do it cheaper and.

1200
00:59:50,460 --> 00:59:50,820
Faster.

1201
00:59:51,280 --> 00:59:54,760
The weird thing is I'm just seeing if two people have the same idea and they, they

1202
00:59:54,760 --> 00:59:57,580
both, they could work together and like build a thing, right.

1203
00:59:57,620 --> 01:00:00,680
Or, but in most cases they both go off to their own agents and they build it

1204
01:00:00,680 --> 01:00:02,160
themselves in their own prompting style.

1205
01:00:02,160 --> 01:00:04,280
And they're like, actually, I just want this flag to be a little bit different.

1206
01:00:04,840 --> 01:00:08,080
And, and when someone else slot something and send it to me, I'm like, no,

1207
01:00:08,080 --> 01:00:09,160
I'm just going to build that myself.

1208
01:00:10,140 --> 01:00:13,560
To me, I was equated to like, when you fart and like, at the end of the day,

1209
01:00:13,580 --> 01:00:15,320
like, like you're like, this is why my kids love you.

1210
01:00:15,400 --> 01:00:15,620
Okay.

1211
01:00:15,840 --> 01:00:20,580
Like, are you, why are you like, like, like someone else's fart is so terrible

1212
01:00:20,580 --> 01:00:22,940
that you don't, you don't want anything to do with it.

1213
01:00:22,940 --> 01:00:23,100
Right.

1214
01:00:23,100 --> 01:00:24,860
But like your own, you're like, you can, it's fine.

1215
01:00:24,860 --> 01:00:26,040
Like, it's just like, I did this.

1216
01:00:26,080 --> 01:00:26,600
It's okay.

1217
01:00:26,960 --> 01:00:27,860
Uh, it's terrible.

1218
01:00:27,900 --> 01:00:30,380
It's a bad thing, but it's still mine.

1219
01:00:30,740 --> 01:00:33,080
And, and at the end of the day, like someone else's

1220
01:00:33,080 --> 01:00:34,700
killing your ugly children.

1221
01:00:34,760 --> 01:00:37,680
Like when you like, cause you spend so much time on a piece, right.

1222
01:00:37,680 --> 01:00:40,920
That you become so like romanticized and in love with it.

1223
01:00:40,980 --> 01:00:42,480
Like it becomes like your child.

1224
01:00:42,580 --> 01:00:45,900
So you can't take constructive feedback anymore because all you see is this

1225
01:00:45,900 --> 01:00:47,520
thing that you spent so much time on.

1226
01:00:47,660 --> 01:00:50,580
And you have, and you know the process, you know what, like why you made some

1227
01:00:50,580 --> 01:00:52,540
decisions, but at the end of the day, they're both, you have like two farts.

1228
01:00:52,940 --> 01:00:54,380
And they're both basically the same thing.

1229
01:00:54,780 --> 01:00:58,560
I think that's like the argument of if SaaS is dead or not, it can't be dead

1230
01:00:58,560 --> 01:01:02,420
because people like, if it's one thing to have a good idea and then contribute

1231
01:01:02,420 --> 01:01:06,040
it to like GitHub and then make it an open source project, but if you're like

1232
01:01:06,040 --> 01:01:09,760
two people are always going to be better at code reviewing and looking at something

1233
01:01:09,760 --> 01:01:11,200
and figuring out if it's good or not.

1234
01:01:11,300 --> 01:01:14,400
And just how David said with the whole having a discord and people giving you

1235
01:01:14,400 --> 01:01:16,920
feedback, like software is easy.

1236
01:01:17,080 --> 01:01:18,540
The fun part is writing the code.

1237
01:01:18,720 --> 01:01:22,420
It's not the fun, the not fun part is maintaining it and like how users are

1238
01:01:22,420 --> 01:01:25,600
going to use it and breaking it and making it reliable and updating it.

1239
01:01:25,660 --> 01:01:25,780
Right.

1240
01:01:25,860 --> 01:01:28,740
Like, and AI is not going to do all of that.

1241
01:01:29,180 --> 01:01:29,360
Yeah.

1242
01:01:29,480 --> 01:01:34,380
I think there's actually some pretty subtle stuff we can tease apart here

1243
01:01:34,380 --> 01:01:39,520
that I don't know if I can do it in a short enough time for a podcast, but let

1244
01:01:39,520 --> 01:01:44,500
me give it a try because I think not all programs are the same, you know, and like

1245
01:01:44,500 --> 01:01:48,580
there's a, there's different kinds of programs and like what's ideal has

1246
01:01:48,580 --> 01:01:52,880
shifted, so what's, what's gone on is agents are a force multiplier on our

1247
01:01:52,880 --> 01:01:59,500
ability to program and that means we can create more code and that means some

1248
01:01:59,500 --> 01:02:03,300
decisions we made about how we write software have changed some of the time.

1249
01:02:03,300 --> 01:02:08,920
And so to give you an example, there's always a tension when you're dealing

1250
01:02:08,920 --> 01:02:14,980
with a web server, for example, between, do I write my own web server using a

1251
01:02:14,980 --> 01:02:18,760
library that does almost all of the work, or do I take an Apache or

1252
01:02:18,760 --> 01:02:20,920
Nginx style thing and configure it?

1253
01:02:22,100 --> 01:02:32,300
And, and five, the generally accepted norm was, oh, you take the Apache server.

1254
01:02:32,300 --> 01:02:34,660
I don't even remember if you had Nginx at that point and you configure it.

1255
01:02:34,720 --> 01:02:37,200
And like, here's the 800 line config file and you get it just right.

1256
01:02:37,220 --> 01:02:38,200
CGI bin all day.

1257
01:02:38,260 --> 01:02:38,860
Yeah, exactly.

1258
01:02:39,040 --> 01:02:41,040
Here's a whole bunch of mods you add to it, but it's one program.

1259
01:02:41,160 --> 01:02:41,900
We all share it.

1260
01:02:42,220 --> 01:02:45,700
We've added a billion flags to it to handle the billion use cases.

1261
01:02:46,100 --> 01:02:53,920
And then the Go standard library came out and Brad built a, an HTTP library

1262
01:02:53,920 --> 01:02:57,360
into it that's just a full web server that you can start in like five lines of code.

1263
01:02:57,920 --> 01:03:01,260
And so you've got this other way of building, of doing a web server

1264
01:03:01,260 --> 01:03:05,780
configuration, which is, oh, write a binary that's like 50 lines long with a

1265
01:03:05,780 --> 01:03:07,780
web server, just the way you want it using this library.

1266
01:03:08,240 --> 01:03:10,060
And so there were these two ways to solve problems.

1267
01:03:10,160 --> 01:03:13,820
And in one of these ways, there's one web server that we all work on, that we all

1268
01:03:13,820 --> 01:03:16,680
configure, and we built lots of stuff in it to handle the cases.

1269
01:03:17,000 --> 01:03:20,100
There's another case where there's 10,000 different web server binaries out there.

1270
01:03:20,180 --> 01:03:21,320
They're all like slightly tuned.

1271
01:03:22,560 --> 01:03:27,240
And both of those are kind of workable, but which of those is the correct way to

1272
01:03:27,240 --> 01:03:32,100
do something has shifted because of agents, because it is now so easy to use

1273
01:03:32,100 --> 01:03:34,020
the library to build your own custom binary.

1274
01:03:34,240 --> 01:03:37,600
It's almost certainly the correct option now, whereas before you could have a

1275
01:03:37,600 --> 01:03:39,960
legitimate argument over, should we all pile into this one thing?

1276
01:03:40,440 --> 01:03:44,960
And so I think the question of whether we all pile into one piece of code is how

1277
01:03:44,960 --> 01:03:48,960
much do our needs overlap and how much configuration do we have to add to it

1278
01:03:48,960 --> 01:03:50,100
to handle everyone's cases?

1279
01:03:50,480 --> 01:03:55,400
If we can write a program that meets all of our needs and we don't have to

1280
01:03:55,400 --> 01:03:59,800
customize it dramatically for each person, then we should totally do that.

1281
01:03:59,860 --> 01:04:00,780
And we should keep doing that.

1282
01:04:01,380 --> 01:04:04,040
And I think the Linux kernel is a good example of this, even though it's very

1283
01:04:04,040 --> 01:04:07,940
customizable, it's everywhere because it's fundamentally, we all need device

1284
01:04:07,940 --> 01:04:11,200
drivers and we all need a pretty stable Unix API on top of that.

1285
01:04:11,540 --> 01:04:15,280
And so we shouldn't each write a new Linux kernel replacement in the world of

1286
01:04:15,280 --> 01:04:17,520
agents. We should keep sharing that stuff.

1287
01:04:18,160 --> 01:04:23,240
But there's probably a whole set of programs out there that we shared before and had

1288
01:04:23,240 --> 01:04:28,040
awful configuration on top of that should now be decomposed because of the force

1289
01:04:28,040 --> 01:04:28,900
multiplier of agents.

1290
01:04:29,880 --> 01:04:33,660
So that's my attempt to split hairs there on what kind of program.

1291
01:04:35,320 --> 01:04:38,120
Actually, I think that was a really good summary.

1292
01:04:38,340 --> 01:04:41,400
And if you think about it, like you said about the Linux kernel or just different

1293
01:04:41,400 --> 01:04:46,580
ways that we use open source, like I made a joke that Linux is like a religion and a

1294
01:04:46,580 --> 01:04:47,520
cult at the same time.

1295
01:04:47,520 --> 01:04:50,880
But it is because people want to do their own little differences.

1296
01:04:50,880 --> 01:04:57,320
But sometimes it's like when I use my Mac as my daily driver, but my dev environment is

1297
01:04:57,320 --> 01:05:03,520
Linux. Sometimes it's depending on what your risk or just how much work you want it

1298
01:05:03,520 --> 01:05:08,580
to be. So I think that the different levels of what you want that software to do and is

1299
01:05:08,580 --> 01:05:10,540
this just for fun? Is this like in production?

1300
01:05:10,820 --> 01:05:14,040
They're all different. So I think that was actually a really, really good summary of

1301
01:05:14,040 --> 01:05:20,680
that. I think one of my fears right now in 2026 is even

1302
01:05:20,680 --> 01:05:25,800
this week, in the last week, we've had multiple critical CVEs against Linux kernel

1303
01:05:25,800 --> 01:05:30,020
things that have come out and like, here you go, here's a root shell for nothing and

1304
01:05:30,020 --> 01:05:33,880
you can do whatever you want. And the complexity of that kernel and what it's been

1305
01:05:33,880 --> 01:05:39,140
doing at this Apache style shared interface, right, like it does everything for a lot of

1306
01:05:39,140 --> 01:05:44,440
people that there's going to be a lot more of these micro tuned kernels for specific use

1307
01:05:44,440 --> 01:05:47,000
cases that are like, actually, I don't need the whole Linux kernel.

1308
01:05:47,160 --> 01:05:48,360
I'm only doing agents.

1309
01:05:48,880 --> 01:05:52,320
And so like for me, like at work, like we only run Kubernetes, right?

1310
01:05:52,340 --> 01:05:53,600
Like our Linux distro only runs Kubernetes.

1311
01:05:53,740 --> 01:05:56,020
So we just disable a bunch of stuff that we're like, we know we don't need this stuff,

1312
01:05:56,060 --> 01:05:59,240
right? Like we're never going to have interactive users at this level.

1313
01:05:59,500 --> 01:06:01,300
And so but I think that's happening.

1314
01:06:01,440 --> 01:06:04,300
That's going to be happening more as as that decomposes.

1315
01:06:04,400 --> 01:06:09,800
The problem is it is infinitely harder to track when something is vulnerable, when it's

1316
01:06:09,800 --> 01:06:13,800
a tweaked your own implementation of like I can go to the library.

1317
01:06:14,200 --> 01:06:15,300
But that's like maintenance, right?

1318
01:06:15,300 --> 01:06:19,420
That's why everybody like when people get all excited about like writing a bunch of

1319
01:06:19,420 --> 01:06:20,300
code and I'm like, cool.

1320
01:06:20,440 --> 01:06:22,260
And so you have to figure that out.

1321
01:06:23,020 --> 01:06:26,300
No one's been in any of the stuff in like the one thing that I've always like come back

1322
01:06:26,300 --> 01:06:31,160
to that like enterprises need to centralize on is security for the reporting side of it

1323
01:06:31,160 --> 01:06:32,300
just to say like, are we vulnerable?

1324
01:06:32,760 --> 01:06:36,280
And if you're going at the go standard library, that's a lot of stuff.

1325
01:06:36,360 --> 01:06:39,140
And you say, oh, I need this patch level to avoid this CVE.

1326
01:06:39,540 --> 01:06:41,620
But also maybe you still implement it a different way.

1327
01:06:41,620 --> 01:06:46,460
And like those are too coarse, those like jumps from go versions when a CVE comes out

1328
01:06:46,460 --> 01:06:48,260
or just a hat comes out.

1329
01:06:48,440 --> 01:06:51,720
Can we just talk about how hard Go is when it comes to like how they do all of their

1330
01:06:51,720 --> 01:06:54,060
dependencies? Like it just makes me want to puke sometimes.

1331
01:06:54,340 --> 01:06:56,640
I have hated Go for a very, very long time.

1332
01:06:56,800 --> 01:06:58,400
Their dependencies are ridiculous.

1333
01:06:58,760 --> 01:07:00,680
But I think you should have existed before GoMod.

1334
01:07:00,800 --> 01:07:02,620
This was oh my gosh, it was trash.

1335
01:07:02,680 --> 01:07:06,680
I hated Go. Agents for me today, like solve so many of that problem.

1336
01:07:06,780 --> 01:07:08,240
I was like, actually, like I don't care about this anymore.

1337
01:07:08,360 --> 01:07:11,220
And GoMod does help. They do all the things that I absolutely like hate.

1338
01:07:11,220 --> 01:07:13,040
That's what I use agents for.

1339
01:07:13,620 --> 01:07:14,420
I think rebasing.

1340
01:07:14,980 --> 01:07:19,700
Yes. No, but honestly, like doing the little like tedious thing so I can do the like more

1341
01:07:19,700 --> 01:07:24,000
bigger thinking. I like I think that you're on to something, though, like because we are

1342
01:07:24,000 --> 01:07:29,560
going to have to give agents kind of more of room to build and to help us.

1343
01:07:29,760 --> 01:07:34,780
Right. Like I think that just like we were talking about privacy on our last episode

1344
01:07:34,780 --> 01:07:38,400
and making things more private by default, I almost think that we're going to have to

1345
01:07:38,400 --> 01:07:39,920
make things more secure by default.

1346
01:07:39,920 --> 01:07:44,620
Yes. So I think that a lot of we're going to have to really cut down on what we're

1347
01:07:44,620 --> 01:07:50,760
offering, whether it be a Linux distribution or, you know, whatever it is, like it should

1348
01:07:50,760 --> 01:07:54,560
be almost the least privileges or the least amount like what you need to actually get

1349
01:07:54,560 --> 01:08:00,300
the job done. And then kind of using whether it be like a smaller image and more secure

1350
01:08:00,300 --> 01:08:04,840
image when you're not really needing to do anything complex with it or you know, because

1351
01:08:04,840 --> 01:08:06,700
I think that's what we're that's where we're getting to.

1352
01:08:06,700 --> 01:08:10,820
That's why people are using change guard or people are wanting those smaller images,

1353
01:08:10,980 --> 01:08:15,100
because when you're letting an agent loose or just junior developers, right, like we've

1354
01:08:15,100 --> 01:08:18,780
been trying to cheat. We've been trying to teach people security for years.

1355
01:08:19,000 --> 01:08:24,220
It's hard. So now they're letting an agent go and these agents are breaking out of boxes

1356
01:08:24,220 --> 01:08:28,480
all the time, you know, so I think we're going to have to start building intentionally

1357
01:08:28,480 --> 01:08:35,720
like default security where we're releasing distros with less and if you need more, add

1358
01:08:35,720 --> 01:08:40,040
it to more, you know, and and even distro is too big of a like I saw a pull request

1359
01:08:40,040 --> 01:08:44,980
today for the Linux kernel to dynamically disable functions in the kernel where you

1360
01:08:44,980 --> 01:08:48,760
can just like do a kill switch to like this function name doesn't don't don't do it

1361
01:08:48,760 --> 01:08:52,420
because even if the CV can't be like I have no patch for this, I just need to disable

1362
01:08:52,420 --> 01:08:55,160
this function call completely Linux kernel.

1363
01:08:55,420 --> 01:08:58,900
Right. You need to build it dynamically at runtime across the board.

1364
01:08:58,960 --> 01:09:02,980
And even if your kernel says my you know, I have a CV against this thing like, no, we've

1365
01:09:02,980 --> 01:09:04,380
disabled those functions. Right.

1366
01:09:04,400 --> 01:09:10,360
Like you need it more fine grained at this sort of like hyper speed of how things are

1367
01:09:10,360 --> 01:09:12,680
coming out and we can't really software and we can't patch it fast enough.

1368
01:09:12,780 --> 01:09:14,420
We won't we never will be able to.

1369
01:09:14,660 --> 01:09:19,920
Just looking at the last big CV that was zero day, it was patched so so differently, like

1370
01:09:19,920 --> 01:09:24,920
a time frame over all of the distros and think about how bad that CV was and just trying

1371
01:09:24,920 --> 01:09:27,200
to roll that out, you know, like.

1372
01:09:27,940 --> 01:09:32,180
I think short term we're in a lot of pain around the fact that models have made it a

1373
01:09:32,180 --> 01:09:35,420
lot cheaper to find exploits and software.

1374
01:09:37,000 --> 01:09:40,620
And I don't know whether Mythos is really significantly better than Opus at this or

1375
01:09:40,620 --> 01:09:44,680
not. It's very hard for me to see through that fog, but let me use Mythos and maybe

1376
01:09:44,680 --> 01:09:47,720
I'll answer it. But Opus is really good at it.

1377
01:09:48,120 --> 01:09:49,900
And so there's clearly a problem right now.

1378
01:09:49,960 --> 01:09:54,000
And like we see this and all of the bugs we're seeing right now, long term, it's not

1379
01:09:54,000 --> 01:10:00,340
necessarily going to be a bad outcome because if the models stop getting better, if it

1380
01:10:00,340 --> 01:10:03,940
turned out what there was, was there was a step function into like, now we can find

1381
01:10:03,940 --> 01:10:04,240
exploits.

1382
01:10:04,260 --> 01:10:05,580
We found all the CVEs for a while.

1383
01:10:05,900 --> 01:10:09,300
It's kind of interesting, though, because is it like is it a bad thing or is it just

1384
01:10:09,300 --> 01:10:12,940
because now we have this ability to find them faster that we wouldn't have found them

1385
01:10:12,940 --> 01:10:13,640
previously?

1386
01:10:14,140 --> 01:10:15,740
Well, they were there. They were already there.

1387
01:10:15,860 --> 01:10:17,160
Right now we can find them. That's good.

1388
01:10:17,600 --> 01:10:22,120
And if the models don't keep getting better at finding exploits or they only get a

1389
01:10:22,120 --> 01:10:26,880
little bit better, then we're in a great place because right now at work, we have every

1390
01:10:26,880 --> 01:10:32,920
time I push a commit to the system, Opus sits there for 20 minutes in a fresh context

1391
01:10:32,920 --> 01:10:36,280
being asked security questions about it and being like, are there flaws, are there

1392
01:10:36,280 --> 01:10:39,800
exploits in this? And sometimes it finds them, which is very off-putting.

1393
01:10:40,480 --> 01:10:44,780
But we're finding those bugs in the software before we're pushing the binary to

1394
01:10:44,780 --> 01:10:49,760
production. And so if the Linux kernel starts going through that process of every

1395
01:10:49,760 --> 01:10:55,560
change that comes in is being analyzed by an Opus or a Mythos or whatever it is to find

1396
01:10:55,560 --> 01:10:59,340
these exploits so that we can catch them before they're deployed, we actually end up in

1397
01:10:59,340 --> 01:11:01,720
a much better place in terms of software security.

1398
01:11:02,060 --> 01:11:06,380
I think that's true. It's like finding all the pain up front and then getting through that

1399
01:11:06,380 --> 01:11:08,120
painful cycle of being able to do better.

1400
01:11:08,360 --> 01:11:10,020
There's no question we're in pain right now.

1401
01:11:10,360 --> 01:11:12,860
And I think it's going to get worse before it gets better, actually.

1402
01:11:13,120 --> 01:11:14,740
Yeah, I think there's some pain in our future.

1403
01:11:16,240 --> 01:11:19,060
Yeah. So everyone laugh through the pain.

1404
01:11:19,280 --> 01:11:20,500
That's what we're telling you to do.

1405
01:11:20,620 --> 01:11:22,740
Just if you're here, you should be laughing through the pain.

1406
01:11:22,740 --> 01:11:26,560
And if you're listening to this show, go SSH to exe.dev.

1407
01:11:26,920 --> 01:11:30,180
There's literally no account sign up ahead of time.

1408
01:11:30,500 --> 01:11:32,720
There's no request. You just get a shell and it's amazing.

1409
01:11:32,980 --> 01:11:37,680
So thank you, David, for coming on the show and explaining that and building the exe.dev.

1410
01:11:38,260 --> 01:11:39,760
Thank you. Thank you so much for having me.

1411
01:11:39,800 --> 01:11:40,500
It's been a ton of fun.

1412
01:11:40,920 --> 01:11:44,720
I'm so excited to see all the things that you build, just listening to you talk about

1413
01:11:44,720 --> 01:11:49,580
it. I'm so excited to kick the tires and try all the stuff out because it sounds like it's

1414
01:11:49,580 --> 01:11:51,980
really going to be extremely useful.

1415
01:11:51,980 --> 01:11:57,280
And it's exciting to hear a really useful startup idea that is going to make it easier

1416
01:11:57,280 --> 01:11:58,300
for us to build things.

1417
01:11:58,600 --> 01:12:02,500
Thank you. I'll try to remember to actually post the features when we release them,

1418
01:12:02,700 --> 01:12:04,140
because that's our biggest problem right now.

1419
01:12:05,340 --> 01:12:10,180
One of my colleagues last Monday, he was telling me, he's like, oh, yeah, I take the

1420
01:12:10,180 --> 01:12:14,620
Discord thing and I paste it into my orchestrator and it produces a PR.

1421
01:12:14,820 --> 01:12:18,220
And I look at it and I hit submit and the continuous deployment deploys and I fix it.

1422
01:12:18,480 --> 01:12:20,100
I was like, do you tell them that you fix this?

1423
01:12:20,100 --> 01:12:21,360
And he's like, oh, no.

1424
01:12:24,240 --> 01:12:26,700
Now you need to build an agent to go and tell people what you fix.

1425
01:12:28,540 --> 01:12:31,800
We need a Discord bot, yeah, it's on the to-do list.

1426
01:12:33,060 --> 01:12:34,940
Well, thank you very much. I really appreciate this.

1427
01:12:35,000 --> 01:12:35,500
This has been great.

1428
01:12:35,780 --> 01:12:36,660
Isn't that crazy?

1429
01:12:36,820 --> 01:12:40,140
Like, we're going to change the whole cycle in which all of this, you know what I mean?

1430
01:12:40,200 --> 01:12:42,700
Like, it's so rad to see it playing out in real life.

1431
01:12:52,720 --> 01:12:55,920
Thank you for listening to this episode of Fork Around and Find Out.

1432
01:12:56,280 --> 01:13:00,200
If you like this show, please consider sharing it with a friend, a co-worker, a family

1433
01:13:00,200 --> 01:13:01,460
member or even an enemy.

1434
01:13:01,760 --> 01:13:05,980
However we get the word out about this show helps it to become sustainable for the long

1435
01:13:05,980 --> 01:13:10,800
term. If you want to sponsor this show, please go to fafo.fm

1436
01:13:10,800 --> 01:13:15,340
slash sponsor and reach out to us there about what you're interested in sponsoring and

1437
01:13:15,340 --> 01:13:19,900
how we can help. We hope your systems stay available and your pagers stay

1438
01:13:19,900 --> 01:13:22,080
quiet. We'll see you again next time.