[Cold open β Oswald] Even big models are moving away from it in favor of cloud design more and more. We end every week with so much more money in the bank. Like the business is very healthy and we can't [music] spend money fast enough to service all of the demand that we have.
[Harry intro] Today we have Oswald Nitski, CPO at Mcore, in the hot seat. Today it's a really open conversation in a way that I don't think has been had with someone from Mcore before about what happens if frontier models actually do what the data providers are going to do. How does synthetic data cannibalize that business? Does open source help or hurt data providers because their biggest customer, oh yeah, it's the closed frontier models. This and so much more in our conversation with Oswald today.
[Cold-open teaser 1 β Oswald] Get a real internship as soon as possible because whatever you learn in school is probably going to be updated quickly.
[Cold-open teaser 2 β Oswald] I don't think there's an ROI problem right now. I think we're in a period of ...
Ready to go?
[Harry] Oswald, it is so good to have you on the show, dude. I've heard so many good things from Brandon. So thank you so much for making this happen, man.
[Oswald] Thanks for having me. Super excited.
[Harry] Dude, I am seeing open open open. Everyone claiming that we will see the mass migration from frontier closed to open. Kimmy very recently came out with that new model. And I literally I didn't really know. Does [snorts] open cannibalize Mcore's core business?
I wouldn't say that open source model improvements cannibalize our core business because data is most valuable on the frontier of model performance. Open models just raise the floor of what people are interested in. As long as customers still have new capabilities that they want to get better at, our business still continues to grow. Open source models just mean that nobody's buying anything that K3 can already do.
[Oswald] I wouldn't say that open source model improvements cannibalize our core business because data is most valuable on the frontier of model performance. So each of our customers has their own unique goals and is purchasing eval training data sets to uh fill gaps in current model capabilities. Open models just raise the floor of what people are interested in. Uh as long as uh customers still have new capabilities that they want to get better at, um, our business still continues to grow. Uh, open source models just mean that nobody's buying anything that can be K3 can already do.
[Harry] So, if like 90% of enterprise workflows can be done with open models, which more and more people say they can be, and that 10% is really where you serve your customers and provide data. I'm naive, does that not make it harder and harder to make huge amounts of revenue if that 10% and frontier moves further and further away?
[Oswald] I'm not convinced that 90% of enterprise enterprise workflows can be handled by open models or frontier models right now. Uh, we think that the these calculations might be based off of existing demand or things that, um, come top of mind when uh, current model users are thinking about models could do, but there's a whole category of latent demand that people aren't even, these are things that people aren't even trying to do with models yet. Uh, most commonly we think these are like long horizon tasks like setting up a procurement agent to fully automate your procurement team for months on end. You only check on it maybe once a week. We think that's just not even captured in these calculations when someone says, you know, enterprise workflows are being handled because nobody's trying to do these things yet. The market for data to support those use cases is growing and that's where we see a lot of the the leaders moving to.
[Harry] Okay, so we see a lot of leaders moving that and seeing new capabilities that they never thought existed. But then we have like Alex Karp in I thought a rather sedate performance. Normally he jumps up and down much more, but I mean it was still rather energetic. Uh, where he said about the incredible skepticism we see from large enterprises towards data and sharing data with the frontier model providers. To what extent do you see skepticism and fear from large enterprises in working with frontier model companies?
It depends on the specific workflow and how core it is to the business. HR, procurement can be less sensitive. It's the core work vital to the business that differentiates it from competitors where we see more sensitivity β like the actual legal memos a law firm writes, the advice it gives its clients.
[Oswald] We see it depend on the uh specific workflow and how core it is to the business. Things that are just like general things that every company needs to do like HR, procurement, it can be less sensitive and enterprises are more open to putting these workflows on proprietary models. It's the core work that the um company is doing that's that's vital to its business that differentiate it from competitors where we see more sensitivity. So, you can imagine this being the actual um legal services that a law firm provides. Like what what are the actual like memos that it's writing? What is the the advice that it's giving to to its clients?
[Harry] Am I the only one who sees the irony in we put the sensitive sensitive data on open-source, most likely Chinese models and we put the HR and procurement data on the closed model? Am I a [__]?
[Oswald] Well, it depends on where you run the open models, right? Whether or not uh that's a bad idea. So, um the the beauty about open weights models is that the uh the inference can happen in multiple places. So, you could be you could make mistakes using them, but um you have more control.
[Harry] When you look at that dispersion, what do you think is inaccurate? You said you didn't really believe the 90/10. What do you believe a more accurate representation is?
[Oswald] Well, in our uh Apex uh benchmarks, we're getting closer to around uh 50% um of long horizon workflows. Um top models are scoring around around around that much. But, I think that um the percentage for there's [snorts] a there's a class of workflows that are just sufficiency-based where you do it and it's done and you're you're good. This is something like updating a CRM. Um you couldn't really get much better at it. And then there's a class of workflows that we shouldn't even be thinking about in terms of, you know, binary, like, can the models do it or not. Um and these can be things like legal arguments or uh to an extent medical advice, where you could always get better. Um and in those cases, I think that the percentage framing is is just totally off and we need to be thinking more about continuous uncapped rewards.
[Harry] When we think about it could be better, I had Lynn Qual, the founder of Filecoin, on the show the other day, and she was like, "Exactly that is why we'll have specialized models for every single company." Because it could be better, depends entirely on the company. One company wants to focus on growth, one company wants to focus on margin, another wants to focus on I don't know, if we're in Europe, uh work-life balance. Um and and so you need individual specialized models for every company. Do you buy that we will have specialized models for every company, or is that a little bit self-serving towards Filecoin? [laughter]
[Oswald] Uh I buy it. I think it's also self-serving towards Mekor um in that we think that every specialized model will need uh enterprise uh specific eval training data to show the model how to perform in its setting. Uh and I think it depends on uh I think the diversity and the market for this depends on um the value that customers can get from the specialized models. So, there will be cases where the ROI is really justified and I think that'll the those cases will increase over time. Uh but we certainly believe in this future, yeah.
[Harry] Switching back, Alex Corp's second point in that show was ROI questionability. You mentioned the word ROI there, which made me think of it. Is very present, and enterprises maybe have questionability around the ROI that they're getting. Do you think we have an enterprise ROI problem with AI today?
We're in a period of exploration and experimentation where there's more tolerance, more patience to get that ROI calculation right now. Things are moving so quickly that the ROI calculation might shift too dramatically still.
[Oswald] Exploration and experimentation where there's more tolerance, more patience to get that ROI calculation right now. There's a lot of different projections around where token prices will go, where performance will go, and right now we're we're starting to see some amount of tightening of the screws on spend here and there. But I think the paradigm we're in is still let's see what happens because things are moving so quickly that the ROI calculation might shift too dramatically still.
[Harry] Two ways I want to go in this. I'll take the first way. We saw Aaron from ClickHouse say that he 6X his spend and that's what they need to do cuz we need to be at the frontier. And then you see Uber and Microsoft and some forms of I think it was Grok or X or one of Elon's companies put budgets on per user head. What do you think is the right way to be navigating this cycle? If I'm a founder listening, what would your advice be on how I should think about optimizing the balance between performance and budget?
[Oswald] It totally depends on the use case. So I've I've mostly worked at hypergrowth companies where growth matters at all costs, right? And there is a willingness to spend for growth as long as the unit economics are fine. When you're looking at coding agent spend for your software engineers, that's not always like that's not COGS for your work, you know? That doesn't like if that's really high, that could still be giving you compounding gains. If you're looking at like a customer service agent that has massive token spend and you know, the revenue you're getting from the customers being served is like way lower than the token spend, then you're then you're definitely in a bad position. Um but in my experience has been just these growth stage companies and I think for a lot of founders um considering um you know, their token spend if it's on if it's for growth, if it's for improving the efficiency of your head count, that's just uh what you need to do to uh to service um large amounts of demand when you're starting up.
[Harry] Mr. Benioff from Salesforce said that he spends 300 million a year on Anthropic, which works out to be about 3.8% of developer salaries if you average the salaries. Do you think that is the going rate moving forward? Do you think that will be 20%? Or do you think it'll be 100%? Or will it be way less?
[Oswald] I hope that we can move towards a future of better accounting of the uh outcomes being driven by token spend because even here I think what in a company like Salesforce um we have so many a company of that size, certainly you're getting you should have different spend profiles depending on what the team is doing. Again here you have teams that might be more like solutions engineering or forward deployed where the um you have to think in terms about unit of unit economics and teams doing R&D where you can be more you can have more tolerance for spend, so um I think at the large companies you have to consider like which parts of your organization are doing what and um how much tolerance should you have in different areas. I think macro the percentage will increase um over time to to more than 3%.
[Harry] Huh. Do you want to hear something funny? Brandon said on the show that it would hit 100% and he said that you already spend more today than you do on salaries.
Yeah, we do. 100% sounds reasonable. Mercari is hypergrowth and continues to be, more so every day. The company is more than 10x in head count since I joined. The revenue has also commensurately increased. We're just in a race nonstop to service our insatiable customer demand.
[Oswald] Yeah, yeah, we we do. Um and uh 100% sounds reasonable. We're um So, as I said, like we're uh I've only worked at hypergrowth companies, and that's what Mercari is and uh continues to be um more so on you know, more so every day uh as the growth uh just accelerates. And for us, it makes sense because the demand that we have is so high. Um we're just um like the the company is more than 10x in head count since I joined. The revenue has also commensurately increased. We're we're just in a race nonstop to service our insatiable customer demand. So, um for us, it it makes sense because we're we can't spend...
[Harry] Dude, do we just build 10x more products quicker? Like, help me understand. Do we have smaller engine product teams? Do we just build much more than we ever used to? How do you think about that?
This paradigm makes the job of product management a lot harder because we're trying NOT to build 10x more product surface area. We're constantly in this battle to simplify. And we see a higher ratio of PMs to engineers because engineering is less bottlenecked.
[Oswald] I think this paradigm makes the job of product management a lot harder because we're trying not to build 10x more product surface area. It makes things incredibly chaotic. We have moments in time where product surface area uh rapidly expands because people think, "Oh, I can make all these features really quickly. This is like I could you know, like let me just like push these multi-thousand line PRs." Um but we have uh we're constantly in this battle to try to simplify our product surface area and find the interactions and the workflows that are most scalable. So, the trend that we see is we're uh as a as a product team constantly fighting to reduce surface area and simplify things. Um and we also see a uh higher ratio of PMs to eng uh because engineering is less bottlenecked. So, there's much, much more work to be done in uh like understanding the uh workflows of users, the needs of users, and what products actually drive revenue the most becomes the bottleneck now to uh servicing more demand for us.
[Harry] If we think about the kind of pre-AI era, how has what it takes to be a great PM changed for this new world?
One: you don't need to learn as many tools. Even Figma we're moving away from in favor of cloud design more and more. Two: everyone needs to up-level and think about business impact much more. Skill issues have almost gone away β now it's all about judgment.
[Oswald] There's two major changes. One is that don't really need to learn as many like tools anymore, you know? You just have to be able to use uh coding agents like a couple tools will do everything you need, you know? Um I don't I like even even Figma is uh we're moving away from it in favor of cloud design more and more. Um so more uh you know, less less tool diversity for us. Uh and then the other is everyone needs to up-level a lot and think about business impact much more. I think that all work is starting to look like higher level. So the the kind of like the minutiae and the details get sorted out way faster. Um and all the PMs at Remarque are to think way more about is what I'm focusing my time on the right thing, right? I can do things very quickly now, you know? It's like there's a you know, skill issues have almost gone away. So now it's all about uh judgment and am I doing what is going to drive the most business value?
[Harry] Dude, I have to ask. You said that like a cool job is retaining simplicity and deciding what to do versus what not to do. What did you do in product that with the benefit of hindsight you wish you hadn't done? And what did you learn?
[Oswald] So one uh interesting thing that happened this year was um our annotation platform serves a lot of different workflows. And the demand for human data is so large and it's so heterogeneous that and our delivery team is so good at um delivering projects and selling projects that we um supported I think too many workflows for human data projects. And we built a tool that was extremely flexible in supporting all sorts of different research experiments that customers might want to do. So, the uh shape of data has changed a lot since it started um with InstructGPT for for gen AI from supervised fine-tuning to preference ranking to all these environment-type projects. There's a lot of multimodal projects that have totally different formats, and your annotation tool needs to support these and different workflows. And customers will ask for all sorts of stuff. We tried to serve every ask. Um we made a tool that's maximally flexible, has all sorts of We had like hundreds of different projects running on it. That's just chaos to manage. Um and what we needed to do sooner was to put guardrails on the type of services that we support uh and work closer with our operations team to say like, "Hey, here's the best practices." You know, customers are going to ask for everything. Like, we can do it, but should we do it? If there's no enduring demand for certain workflows, maybe it's not worth the investment. So, putting guardrails, narrowing down the services that we support was something we should have done a lot sooner um that we did it uh recently.
[Harry] How do you determine enduring demand?
[Oswald] This is uh what makes Merkle a hypergrowth company is that we're incredibly tapped into the market and the ecosystem. It's really a judgment from leadership, I think. It's very hard to say kind of like what will data look like in a year or two. And the best way to figure it out is to stay in constant touch with uh leaders from a diverse set of labs um and constantly be validating hypotheses. Um I think we I think Brandon does it very well. I think our operations team does it very well. But ultimately, it's kind of like uh it's kind of a guess.
[Harry] Which lab has the most advanced and sophisticated data team?
[Oswald] I can't speak too much to customer details, but [laughter] they're all super good. Everyone is everyone's sophisticated. Everyone blows me away in in different ways.
[Harry] That is such an unfair question. Okay, I totally agree. The other question to ask is which has the worst team? My question to you is you mentioned another element that which actually didn't shock me, but I thought it was interesting was the movement away from Figma. Can you talk to me about that because I hear more and more companies doing the same. As a product leader today, how do you think about that? And what was the thinking there?
The team do whatever is best for them and this is a trend I've just observed amongst almost everybody is that cloud design's done a great job. People really like using it. It's easy to use and we've just had a natural movement towards it.
[Oswald] The team do whatever is best for them and this is a trend I've just observed amongst almost everybody is that cloud design's done a great job. People really like using it. It's easy to use and we've just had a natural movement towards it. It's also been easier to not have too many tools, not manage too many licenses and because cloud is like making all these other great great features, people just gravitate towards it and then it's it's, you know, a bit less friction to have the procurement team issue licenses for Figma for every single person.
[Harry] We were talking about the ROI earlier for enterprises and we're seeing Microsoft set up a services department. We're obviously seeing Palantir, you know, skyrocket and services becoming an increasing part of everyone's business. Is that the future of AI enterprise deployment and how do you think about the incredible rise of services in deployment?
Hot take: it's the future for the short term as the knowledge of how to use AI gets disseminated throughout industry. We have a concentration of people in SF who really know how to deploy agents. Eventually it'll become more of a job function like software engineering. Maybe a decade-long change.
[Oswald] Yeah, so I I have a bit of a hot take here. I think it's the future for the short term as the knowledge of how to use AI gets disseminated throughout industry. We have basically a concentration of a bunch of people in San Francisco who really know how to deploy agents, eval agents, be AI-first in engineering and in in other areas. And that knowledge just isn't out there yet. And eventually it will be, and maybe you won't need at that point teams to go and set things up, set up AI agents for every enterprise, and it'll become more of like a a job function similar to software engineering.
[Harry] And so in the short term it enables deployment, in the long term products become more and more sophisticated that they're able to sit themselves. Because Matt from Factory said to me, "You know what? [__] this. Services, they're just an excuse for crap product."
[Oswald] I think that it's a knowledge dissemination problem. So I think that that's one way to look at it. The other way is why not hire someone to just do this agent deployment at your own company. And I just don't think the skill is out there yet. I don't think there's enough I don't think the talent is available for every enterprise to have their own expertise in it at this point in time. But that'll change over over the long run. This is I think like a you know, maybe a decade-long change.
[Harry] Question, do good engineers really want to be FDs though?
[Oswald] Uh I think good engi- There are a lot of different types of good engineers. There's a lot of things a lot of ways to be a good engineer. And one way to be a good engineer is being a great communicator and cutting through to the source of a problem and simplifying. And I think that those engineers are great fits for FDs. And I think that those engineers are also great fits to eventually become founders. And I think that that is a different profile of person who's incredibly valuable. And that's what a lot of people are looking for when they're looking for FDs. And it's also like a profile that we look for generally, which is why we have so many alumni go off and start uh companies.
[Harry] Do you like that? I spoke to Brendan about this, but is it a good thing to have some of a core mafia? Because you also want to retain talent.
Proud that of the people I work closest with on my teams, I've only had attrition to founding. It's a lot better to lose someone to starting a company than to taking another job. It's tough β makes management harder β but I'd rather be in an environment like this than one where everyone's soft.
[Oswald] Proud that of the people I work closest with on my teams, I've only had attrition to founding. And we've had quite a bit of it. It's a lot better to lose someone to starting a company than to uh you know, taking another job. It's interesting from a personal level because I like these people. I wish the best for them. I really enjoy seeing it. Um, it is tough though. It makes the job of management a lot harder because it you we just have so many high agency people who are very ambitious and uh it it's difficult, but I like it and I'd rather be in an environment like this than one where everyone's a goal, you know, soft and you know, oh, I don't want to work.
[Harry] No wonder you left Europe. Um, how has hiring changed in a post-AI new world? When you look at the people that you add to your team today, especially in product, what do you ask today or look for today that you didn't before?
[Oswald] I think touching on the earlier point of everybody needing to up-level and think closer to business impact, we've biased towards more senior hires who are better at understanding what drives the business forward, finding kind of like really grokking how we operate, how we how we make more revenue, how we deliver better services to our customers, how we keep our customers happy. I found that more senior candidates just get that a lot faster. And like I said, all these like kind of like tool like can you use the tool? Can you do all these other kind of like more more junior things are becoming less relevant. So, the hiring for us is biased towards more more senior candidates.
[Harry] Do you worry that you're just falling for the kind of classic, "I'm so sorry to be like the fast growth founder mode." which is like your VCs come in and say, "Oh, you need to hire this person from Facebook." and you know, you get the seasoned operator who fits exactly that rubric. And it never works. It never works.
[Oswald] We're not quite doing, you know, seasoned here is a spectrum, right? We're not I'm not saying um we're we're hiring people who have uh you know, are in like formerly in executive positions. We are treating everything as an executive search, but we want to find someone who's at the sweet spot. They're still hungry. They've done the job that we want them to do for a few years. Um and they're right in kind of like really hitting their prime.
[Harry] When do you think people hit that prime?
[Oswald] [sighs] I think 25 to 35.
[Harry] Oh, I just turned 30. I'm bang in the middle. Perfect.
[Oswald] Good timing for you.
[Harry] Perfect timing for me. Okay, in terms of like the questions, what we look for in the take-home assignments, has that changed?
[Oswald] We've moved away from take-home assignments. We do one take-home assignment which is like, "Can you just like, you know, use an agent to go You're on your own for a bit of time. Go use an agent, you know, give produce this artifact for me and we'll look at it." Do that once, you know that the person's AI fluent. Uh and then we move towards a lot of whiteboarding uh because we want to avoid like we we'll do one round where we know where we find out if the person is just familiar with AI tools.
[Harry] Don't laugh. Okay, so cool. We do that. I'm familiar with AI tools and now you're like coming to my room. We've got a whiteboard. What are we going to do? What do you want to see? What would impress you?
We care a lot about being able to set up good experiments and understanding statistics, having good judgment, and then systems design as well. With AI tools it's very easy to offload a lot of thinking. We want to make sure people still have the ability to have good judgment and not just regurgitate what comes out of Claude.
[Oswald] We care a lot about being able to set up good experiments and understanding statistics, having good judgment, and then systems design as well. The reason is these are just skills that are so easy to kind of like be asked.
[Harry] Sorry. I'm so sorry to interrupt you. Good experiments and systems design. It feels quite wordy. What does that actually mean?
[Oswald] We ask people a lot I don't want to give away too much about our interview process, but there are we need to run a lot of experiments uh as a product team. We need to make sure that our um our team knows how to run a good experiment that actually reveals information and isn't just totally like fudged. And with AI tools, it's very easy to offload a lot of thinking, a lot of judgment. We want to make sure that people still have the ability to have good judgment and know what they're doing and not just like regurgitate what comes out of of Claude.
[Harry] That's so interesting. I completely agree with you. I have it with my team, which is like we do scripts for for content, for reels, for uh yeah, I do all questions myself. I would never use AI and I'm very concerned about it cuz you lose the muscle to me. Can I ask you, how do you retain thinking, thought, creativity when so many people are so freaking hooked already?
[Oswald] I think it's kind I tell my team it's kind of like phones, you know, they kind of fry your brain, they turn it into goop. But I love I do a lot of stuff on my phone, like I use my phone all the time. Uh you just have to learn personally where that boundary is of like what's when is a good time to scroll through reels and when's a bad time, you know? Um in a meeting, try not to scroll through it. For work, um that boundary I think is between kind of like the judgment and decision-making and the execution. Right? So, I was very careful never to delegate judgment or decision-making um to Miles because it's it's They make you think that it's doing the right thing, but you have to be paranoid with them still. Right? You still have to like double-check everything. Um and that's what I I I tell my team is that like don't delegate your decision-making, like your actual job, to a model because you're going to you're going to lose that ability and then you're going to get psychosist.
[Harry] Totally agree with that. When you look at the experiments that you've run, does the data correlate to the outcome? Like I often think in investing, sometimes I do no work and no diligence and I make loads of money. [snorts] And sometimes I do lots and it I make terrible investments to lose all the money. Do the inputs correlate to the outputs?
[Oswald] It varies, you know, that's it varies because we run a lot of experiments. Um but uh sometimes they do, sometimes they don't. We want to get more um that actually uh show good results and move the business forward and that's really the job of the team is to find the right experiments to run and and make the narrative around, you know, this hey these these changes to to our product have have impacted the business in a positive way. Um that's that's a lot of the core job right now. So, you know, it's it's week to week, month to month we get different [music] different results, but we try to trend in the right direction over time and and people start to learn um learn the dynamics of the product, learn the dynamics of like the the users better and better to um improve over time.
[Harry] When you think about like product and engine running experiments that you mentioned there, I'm running good experiments, how do you structure the teams today? And what does that like meeting look like?
[Oswald] We have um a few different groups that do experimentation. So, we have two major product areas where this is most relevant, our marketplace, which matches experts to jobs, and our annotation and Eval platform, which is where experts log in to um do annotation um for eval or training data sets, where our operations team also logs in to run those uh projects, and our customers will log in to see their data and run evals. So, annotation platform called studio marketplace, call it the marketplace. Um these two groups um they they're kind of like self-contained in trying to do um it's it's to make their individual like product offering better. And we have two modes main modes of engagement within human data, talent only, which is when we just send experts to our customers, and they'll they'll run the project. So, this is like a a lab needs a doctor, a lawyer, whatever, and they're like, "We're just going to use them." And like, you know, thanks for thanks for finding the best person for the job. You'll need to pay them, performance manage them, but like we'll do the we'll run the project. And then a managed service project, where we give our customers data. So, for the talent only model, we just use the marketplace. For the managed service, we use the marketplace to send people to our annotation platform, and then we'll give them we'll run the project and give them the whole data set.
[Harry] Totally. Are they two separate product teams?
[Oswald] They are two separate product teams.
[Harry] How big are the product teams?
[Oswald] Around two to three per um per product area uh with uh also data scientists and a design team. Uh data scientists dedicated to each and a design team that flexes between them of uh just a few, yeah.
[Harry] So, you have like pods of like four, five.
[Oswald] That's fair, yeah.
[Harry] Gosh, you totally. Okay, that makes absolute sense. Will those ratios change over time, do you think, between PMs and design, or that stay the same?
[Oswald] I think that the ratio of PM to end will change over time to have fewer engineers per PM as engineering uh velocity increases with uh better uh coding agents. And we will be bottlenecked by understanding business needs, user needs, uh um and that's more of a PM job. We need to be very careful as a hyper growth company to grow the teams in lockstep um because as the head counts increase, you know, like more than 10x in the last year, we just uh want to be careful not to grow one faster than the other. The trend will be to uh hire PM to engineer ratio, though.
[Harry] So, when we talk about the the good experiments and and making sure that we're running a really tight process, what does that look like in terms of the meetings? You have a weekly product team meeting? Like what is the right way to approach cadence of product team meetings and how to run them today?
[Oswald] We break it down into uh so, within these product areas, we'll have like a whole whole PA like weekly and this product and engine a lot of other stakeholders as well. And this one is just like everyone, you know, kind of like there it's broken down into pods. So, as I mentioned, within that product area, there might be like let's say three product managers. They'll all have like a pod of these parts of the product that are we can naturally kind of like segment uh work into. Our marketplace, for example, has expert-facing side and a hiring manager-facing side. These are naturally two distinct pods. Um there's some other uh kind of um pods within here as well, like uh managing the expert experience, making sure that everyone has great customer support, there's never any issues with with any um you know, with working for Recur. Um each of these pods will do their own sprint planning. They'll come together in the weekly kind of like product area meeting um and we try to keep it efficient, but maintain like a lot of visibility between the pods cuz they all need to have their roadmaps well aligned. yeah, but we need to have weekly uh meetings to maintain accountability, right? So, uh do them on Friday a a later in the day, make sure no one's uh leaving early on the weekend.
[Harry] Love it. Um what do you not do in your product meetings that you should do to make them better?
[Oswald] It varies by by product area. So the challenges in like the marketplace versus like the annotation platform are a bit different. Um the main challenge as we grow quickly is having the right amount of communication and feedback from other teams. So like our marketplace and our studio team need to get information from each other, right? There's cases where like something's wrong in one and it's affect it's it's popping up in the other. Um if something's wrong with one product and it's affecting like the expert experience when they're like on the other one somehow. Uh and that communication just like the because the head count and the team's grown so quickly, like the communication channels like just explode very quickly. So we need to do more kind of like cross product area uh collaboration. Keeping it efficient is just really hard as the team grows because we're um you know, the nodes just keep moving around and there's more of them.
[Harry] What has been the secret to scaling supply on the marketplace side so efficiently? Like that's [__] hard. How have you guys done that so well?
[Oswald] I probably put it down to three things. Um the first one is a great expert experience. So uh experts get paid on time, they get paid well, uh transparently. Um the everybody involved in what the expert experiences cares deeply about um whether or not like whether or not they're having any challenges and whether or not the work is um dignified and well paid and fairly paid. And that is a requirement for a great referrals program because nobody's going to refer their friends to you know their colleagues to some kind of job that's like that sucks, right? So, everybody caring about expert experience drives a great referral program and additionally a great sourcing team that's able to find people in every corner of the world with very specific skills helps us fill the gaps when you know we have spiky demand for a specific skill set.
[Harry] Are people as short-sighted as just being wanting to be paid the most? I've heard that Mckool pays the most. Is that the secret?
[Oswald] It's not I wouldn't say it's like you know short-sightedness because we want to retain the top experts as well, right? So, if you get paid a lot on like one project and it's like you know there's like some kind of like I I know there are a lot of other competitors in the space who will do some crazy like bonus payouts and stuff and for short-term sprints. Um, that's not that doesn't get you to come back as much as a great experience with a lot of work, visibility into like what future work is coming up, the feeling of like I'm growing my skill set, I have the ability to pick between a few different jobs, I have I'm doing interesting work, I have great communications right from the people running the project. It's really hard to sign up for online work and then you just like have get hit with this like 100-page instruction document. It's a very foreign kind of job. That's part of the experience as well. And get knowing that you're going to get paid highly for a long time for something that you can do for a long time is what keeps people interested.
[Harry] Have you seen your margin improve over time or is it one where actually margins relatively fixed given the complexity?
Margins are decided after the fact based on consideration of costs. For a lot of these projects the costs are driven equally from paying experts and LLM spend on things like synthetic data and automatic quality control.
[Oswald] So, so margins are an interesting thing in this business. We try to think about as a product team, how do we deliver the best value for our customers. And that is independent of how do we how do we price the project. So, there are cases where you could have automatic quality control and synthetic data improvements to make the delivery better. There are cases situations where you could, you know, think about the staffing on the project to to change the the cost of of the service. All of that like we as as a product team we want to make sure that we could deliver the best value to our customers and we can have the best experience for our experts. Margins are decided after the fact based on, you know, consideration of costs and, you know, now for a lot of these projects it's a the costs are driven from equally from paying experts and LLM spend on things like synthetic data and automatic quality control.
[Harry] Does the "It's not revenue. It's not revenue." shouting from the crowd throwing peanuts, does that annoy you? And is there anything that hasn't been said that you think people are just like not getting?
It doesn't annoy me, because we end every week with millions more in the bank. I've seen interesting financial engineering and accounting elsewhere, but we end every week with so much more money in the bank. The business is very healthy and we can't spend money fast enough. The cash flow is insane.
[Oswald] Um, it doesn't annoy me, no, cuz you know, we end every week with like millions more in the bank, right? So, it's it's it's funny how you can have I've been at other companies where I've seen, you know, interesting financial engineering and accounting and, you know, people can have all these different metrics, but we end every week with so much more money in the bank like the business is is very healthy and we can't we can't spend money fast enough. So, what people, you know, want to call it is, you know, up to them, but like the the cash flow is insane.
[Harry] Does it matter that you have such high revenue concentration? You know, the frontier model providers are your biggest customers by far. Some would say, "Woof, that's a lot of concentration." How do you think about that?
[Oswald] So, I can answer this from a kind of like a how it affects the product team. Yeah. We would love to move like our biggest challenge is moving down market so that every single enterprise can efficiently run human data projects for eval and training. And that'll diversify our revenue for sure because there's many more enterprises than there are labs. And that's a harder product to build. And that's the direction that we are taking taking our products, taking the company is to be able to self-serve projects very efficiently, have like AI project managers so that it's a lot easier to do this work for smaller customers cuz running a human data project for a lab is incredibly hard. It's a white glove service that requires a lot of people on the operations team. As we make that more efficient with better products, better processes, we can do smaller projects that are more heterogeneous for more customers. It's the direction we have been heading which has reduced concentration and it's the direction that we'll continue to head as every enterprise begins to have human data work for their proprietary use cases.
[Harry] What's so hard about it? Making it really simple, explaining it? What is the challenge with not dumbing down but democratizing?
Running a human data project is just hard. All the edge cases matter. The process is basically continually surfacing these edge cases which requires insanely fast alignment between customers, maybe their customers, other experts, and the annotators. And a huge amount of paranoia from operations to make sure every data point is perfect.
[Oswald] Running a human data project is just hard. Um there's so much information that needs to be transmitted from the customer as the end users of our customers to experts. And all the edge cases matter, right? So people will try to write a guideline that says like here's how you here's how you make a data point, but the experts will have like some edge case that gets bubbled up and like what you do on that edge case matters a lot. So the process of making a human data project is basically like continually surfacing these edge cases which requires insanely fast alignment between customers, maybe their customers, maybe other experts in the field, and the experts who are doing the annotation. And it also requires a huge amount of paranoia from the operations team to make sure that every data point is perfect. It fits whatever guidelines are the customers have, and the projects are running on time, all the bottlenecks are removed. Um it's just an operationally intense process because it necessarily deals with um edge cases and things that haven't seen before and are outside of model capabilities. The data types also change very frequently. So we're we've moved from supervised fine-tuning to preference ranking to uh rubric-based um annotation to now RL environments across a whole bunch of different modalities. It's There's a lot of complexity within each project and then between projects. Um so I would I would boil it down to those two things of like the need for paranoia and the need for very crisp communication that make it challenging.
[Harry] What data type is not hugely in demand today that you think will be hugely in demand next year?
The data type growing fastest for us is environments β simulations of apps you want your agent to use, plus rich start state representing all the data on your machine, plus tasks. If you want to learn Salesforce, you need a high-fidelity mock that acts exactly like Salesforce. This is the frontier right now.
[Oswald] The data type that's growing the fastest for us is environments. People you know that you might have seen a lot about these RL environments on on Twitter. It's kind of like a you know hype term. Um every company kind of like has a different definition for it. Um but we um are are certainly the leader um in the category and view it as basically these like simulations of apps that you might want your agent to use. Um and also as per rich start state, which we call like the world that is basically representative of all the data you might have on your machine like your laptop. And then we have tasks that train agents how to use those tools to accomplish something that's useful. It's a bit of a It's a bit of a complicated annotation process because the agent has to like interact with this simulated world. We have to make that start state, which can be hundreds of files, thousands of files. Um and the shift here is that the data that the models are now the agents are being evaled and trained on looks a lot closer to what they see in deployment. Right? So, if you want to learn how to use something like Salesforce, you need a pretty high-fidelity mock that acts exactly like Salesforce in your eval and training. And it's complicated to get this set up. Just like years ago preference ranking was really hard to get set up. Uh, SFT was really hard to get set up when instruction when Instruct GPT first came out. So, this is the frontier right now. Um, labs are figuring out new labs are figuring it out. Eventually, it'll get so smooth that enterprises can do it, too.
[Harry] Are labs price sensitive on data acquisition?
[Oswald] By data acquisition, um...
[Harry] Well, when they go on a project with you, are they price sensitive? Like, are they haggling going, "Oh, well, you know, Edwin at Surge gave me a 10% discount. Can I have that?" Or are they like, "Just give me the [__] data."
[Oswald] Well, there's always the, you know, aspect of negotiation and the procurement team trying to get a better deal. Um, but we're we've chosen a great business where our work directly affects the business outcomes of our customers, right? So, we we have a great setup where if you're making an eval set, like any other lab, you're evaluating something that your customers want to do. If you could just do it better, right? You would make more revenue. If we're if they're buying a training set, they're now hill climbing that eval set that they've said, you know, that represents what their customers want to do. So, as long as like the amount of money they're spending on data is less than the revenue that they're going to get, um, they're happy to crank the lever. Like, people want to crank it harder and harder because spend on record directly translates to more revenue for our...
[Harry] Do you think we'll have a unbundled data provider world? You know, I I'm a venture investor, and I see so many people that are like, "Oh, we're like McCaw, but for like, you know, uh, domestic robotics. And you're like, "Okay, cool." Good. Okay, I get it. But, do you think we will see this kind of specialized data provider world where niches have thousands of players?
We're facing what looks like a cottage industry of founders doing annotation themselves. Labs love this because it's just totally mispriced β VC-subsidized work. But the problem is scaling beyond a few data points. It doesn't scale, and our customers know that when you want to 10x throughput. But it's indicative that the field needs higher-skilled experts.
[Oswald] To an extent, we're already in this world. I wouldn't uh it's not that successful though for the small players always. So, how I would describe it is we're facing what looks like a cottage industry of founders doing annotation themselves. Right? So, you have all of these small startups where as the skill bar for annotation gets higher and higher as models get better, you have startups where the founders are actually just making the data. Right? And labs love this because it's just like totally mispriced, you know? They get like someone raises a bunch of money. Um they have loads of cash to blow, and they they go to these labs, and they're like, "I I need to like I need to get your business. Like, please let me work for you." And then they're they're smart people. They're they're founders. They're formerly like great technical employees. But, they're running the projects themselves. They're doing the annotation themselves. And this is just like VC-subsidized work that labs love. The problem is scaling it beyond a few data points or what what one founder or full-time employees can do. Um and this is the position that we're in is we're having to compete against basically founder-led annotation where some of them are even running it as cash flow businesses, and they're just like taking the profits home themselves. It doesn't scale though, and vendors our customers know this that it won't scale when you want to 10x the throughput, 10x the amount of projects. Um but, it is indicative of the direction the field's heading in in that we need higher-skilled experts. We need the best people in the world to be doing this annotation.
[Harry] Don't laugh. I'm I have a bit of an ego, and so I like to feel like a special snowflake. Um And what I mean by that is, I would be like, "Oh, when Meta or OpenAI or you name your large company is buying data from multiple people, it feels like you're being promiscuous and cheating on me." Do you mind? And do you monitor budget and percent of budget that gets spent with you versus another provider?
[Oswald] Of course we do a lot of competitive intelligence. Um, and uh our customers like us, so you know, we uh they'll often and share information with us, but everybody just wants models to get better, right? So, we're happy for uh to have this kind of competitive pressure that tells us like where to go. If someone else is able to do something better than us, um we'd love to hear about it and then do it better than them, right? It's it's healthy to have um you know, uh vendor bake-offs. It pushes us to make our services better. Um we do stay on top of it because we want to pre-deliver better services to our customers. We want to know who's doing better than us and then we want to surpass them. So, uh it's a totally healthy thing to happen as long as Mercor's winning.
[Harry] I'm you said models getting better there. We said Frontier earlier. I'm an investor in Legora and everyone's like, "Oh, your real competition is is actually Anthropic." And I'm like, if if Anthropic go after legal and winning Cooley and Goodwin, something's gone very wrong with the world cuz they should be solving cancer and climate change. To what extent am I right and how do I balance between Anthropic are coming for Legora and Figma and Anthropic's also working on the frontier problems that humanity faces today?
[Oswald] I would look to precedents for from other big tech companies who have had a lot of different efforts like Google, Microsoft, who um you know, coincidentally also try to uh solve climate change and cancer, but it's not their main business. And they have their hands in like a lot of different areas, but competitors still emerge. So you seen like you know, you remember Google Plus, right? That didn't go anywhere, right? It probably maybe it freaked some people out when it happened. You probably remember threads. I don't know the current state of of threads. But...
[Harry] Apparently 400 million users according to their marketing team.
[Oswald] That's very interesting. I won't comment too much on on that because I...
[Harry] I'd love to see the engagement.
[Oswald] I genuinely don't know anything about this. [gasps] But yeah, so it's in in I think if you look to precedents here, large companies often try to you know, make new bets, diversify, but they lose to companies that have intense focus on their market. So we'll see how it plays out, but I would I would wonder if there's anything to learn from you know, history with Google and Microsoft um having having many business units, many efforts, but a core business that has driven all of their revenue.
[Harry] You know, I love Brendan. I remember texting him when there was the hack. It's tough when there's a hack cuz you're like don't know what to say, but like I'm here for you, you know, and thumbs up. And I felt like it's such a VC cuz you're like I'm here for you. Good luck. [__] all help that is. Um my question to you, how did that change your mindset and approach to product? It is a really hard thing to go through. I remember you were under intense pressure and stress and I I seriously am sorry if that cuz it's horrible to go through. Um how did it change your product mindset?
[Oswald] I'm not an expert in security. Uh but we hired a lot of experts in security and I listen to them. And that's the main uh change is just larger investment and learning from uh the experts that we've brought in house.
[Harry] Are we entering a golden age for cyber? And what I mean by that is we're seeing a huge amount of uh AI-generated code, which in a lot of cases has holes. we're seeing a lovable and a Replit and a you name it produce a huge amount of uh output. And the threat is going to increase much more significantly than we're anticipating.
Most likely yes. It's an interesting data type because it's competitive and you can have these AlphaGo-type situations for cyber offense and defense, with uncapped rewards and performance and the field constantly moving. There's never going to be that 90% for security because the goal posts are always going to move.
[Oswald] Uh most likely, yes. Where we see it the most is it's an interesting data type because it's competitive and you can have these AlphaGo-type situations for for cyber offense and defense, where you can have uh uncapped rewards and performance and the field's constantly moving. So, we love this kind of stuff because it's like a it's like a game for from like a data perspective. Um and we see very rapidly increasing demand for um cyber defensive capabilities via data and very interesting data types. And this is an example of something where sufficient...
[Harry] You help me understand? What data types do people want around security that they maybe didn't want before there was this explosion in demand?
[Oswald] I have to be careful not to reveal too much about customer uh work. The category is growing very quickly and the nature of a lot of security work is that it's it's adversarial, right? So, it's not this sufficiency-style work like update a CRM and and then you're good. It's it there's a constant uh cat-and-mouse game between like the offensive capabilities and the defensive capabilities, which um you know, to your point earlier about like the 90% of enterprise workflows that can already be completed. Like, there's never going to be that 90% for security because the goal posts are always going to move. Um so, most cyber as a category is growing and the nature of the data types is it's um much more kind of like uncaped evolving adversarial in terms of the where the goal posts are.
[Harry] Can you help me out here? I you're you're Estonian by kind of heritage. I say to European founders SF is the worst place to start a company. It is impossible to acquire talent. It is impossible to afford it. And then it's a impossible to retain it. Is the talent war in SF as brutal as it seems?
[Oswald] Yeah, it's pretty brutal. It is it is very difficult to hire. It is difficult to retain. It's difficult. I think it's harder than before, but it's easy when you're on a rocket ship, right? It's always easy when like when you're on a rocket ship to to get someone. It's it's hard to make the right decisions about who you want to hire.
[Harry] When you've made a bad hire, what did you not see that you wish you'd seen?
[Oswald] It's really hard to assess agency and ownership uh in the interview process.
[Harry] I am super freaking talented. I'm super talented. I'm a bit of an [__] I'm not like a total [__] but I'm a bit of a douche. Are you okay with that?
If you're super talented, yeah. The company culture is high agency, high performance, high ownership. Personalities can change. But we care about hiring people who give a f***. That's a lot harder to coach than smoothing out colleague friction β a few drinks together and you smooth that out. It's really hard to make someone give a f***.
[Oswald] If you're super talented, yeah. we're yeah, I mean we're the company culture here is of high agency, high performance, high ownership. Personalities can change. You can learn how to work with people better. And but but we we care about like growth and we care about like we want to hire people who give a [__] That's a lot harder to coach into someone than you know, smoothing it out with with your colleagues, getting some you know, making sure that we have happy hours, people all get along. Like that kind of that's easy to work. I you know, like you got to you can have a couple [__] You they get drinks together a few times and then you you smooth it out. It's really hard to make someone give a [__]
[Harry] Yeah, also if you hire multiple [__] they can just hang out together. Um it's fine. It's [laughter] a group hug.
[Oswald] We don't We don't hire a lot of [__] Like my my...
[Harry] Uh no no I I can be also like like happy hours like really?
[Oswald] We had a great offsite just recently actually with our annotation team. We went to Tofino in in Canada. it's on the west coast of Canada. It's the only place you can surf. And everyone did surfing lessons. We went to a floating sauna. And it was a great time. I thought it was actually great for the team and it was a great use of uh of money and everybody loved it. And I think that doing these like outdoor activities where people are being active is is good.
[Harry] Are you ready for a quick fire round, dude?
[Oswald] Sure. Yeah.
[Harry] Uh what have you changed your mind on most in the last 12 months?
[Oswald] Honestly, I think it's probably the environment uh our environment market. Because when we were starting it off last year, it was so complicated to do these deliveries. And it was so hard um to get it to work that I just thought it wasn't going to work out. I thought it wasn't going to scale. But then it did. Um so I was like I was pretty surprised.
[Harry] What changed?
[Oswald] Uh the demand was very high and we just we got it to work, right? We just had to try like a lot of different things to get environments to actually improve model performance. Um so we just kept going at it and it ended up working.
[Harry] I'm your little brother and I'm studying computer science at university today. You sit me down and say, "Little brother, you should know this." What what should I know?
[Oswald] Get a real internship as soon as possible because whatever you learn in school is probably going to be uh updated quickly.
[Harry] Interesting. Why should I get a real internship? I know that sounds stupid, but like should I start my own company? Should I join a fast-growing company? Should I join a super established company where there's, you know, adults in the room so to speak?
[Oswald] Maybe I'm biased, but join a fast-growing company in San Francisco. Um doesn't need to have adults in the room, but somewhere on the frontier that's indicative of where the field is going. A bit larger than, you know, 10 people, not super early. Uh just to kind of like filter out the companies that might not go anywhere.
[Harry] Would you say that you're too late for me?
[Oswald] No. No, we still act like a startup.
[Harry] How many people do you have?
[Oswald] Maybe 500. It's a cult. We're culturally we're a startup. We're paranoid, we're in office all the time, we're fast moving. We want to hold on to that as long as possible.
[Harry] I love it. That's amazing. Uh totally. Uh absolutely. Yes. which competitor do you most respect and why them?
[Oswald] I don't think about competitors too much. They're all kind of even in that they're all behind Brex. It's a bit of an odd answer, but uh we we really try not to think about them as much as we try to think about our customers. So, I respect our customers a lot. I love the work that they're doing. Um we stay on top of what competitors are doing, but every time I look at one of their websites, they're just doing something we did like a week or a month ago. You know, they write a We write a blog, someone else writes a blog like a week later that's the exact same thing. We make an update to our website, someone else makes an update to their website that's the exact same thing. Um so, we I spend a lot more time...
[Harry] That about Surge?
[Oswald] It's happened before. They're a bit out there. We honestly, like I don't spend that much time thinking about them because I spend more time thinking about customers. We've seen it They're a bit out there in in that they they don't copy us as much um and they do seem a bit different from others in the field. Hard to say why. They're very secretive. Yeah, you kidding me? Um yes, absolutely. Um I totally get that. Can you please paint the bull case for how Brex is a $200 billion company.
We're a tech-enabled services company. Our services drive revenue gains for customers primarily through better model capabilities. Evals and training data are the primary bottlenecks in model performance right now. Even if capabilities start to saturate, the eval serves as the PRD for exactly what you want AND the optimization objective. As long as better models are valuable to the economy, there will be demand for eval sets and training sets.
[Oswald] It looks like we we're we sell services. We have we're basically like a tech-enabled services company. Our services are incredibly valuable in driving you know revenue gains for our customers primarily through better model capabilities. Evals and training data are the primary bottlenecks in model performance right now. If every enterprise needs to have specialized proprietary models, even if the capabilities start to saturate, the eval serve as the PRD for kind of like exactly what you want, but also the optimization objective for better performance. As long as more and more as long as better models are valuable to the economy, there will be demand for eval sets and training sets. If we can make that process faster and faster, we can serve a growing demand for human data free eval training. And then we also have a growing agent deployment enterprise arm as well.
[Harry] What line of revenue do you not have today that you think will be very significant in 3 years' time?
[Oswald] I think that real-world, like physical data, is going to grow significantly over the next 3 years. Robotics is an interesting area for us. The data market for robotics is nascent relative to GenAI, relative to you know, like autonomous vehicles as well. And we think that's going to grow a lot.
[Harry] Do you scale supply ahead of demand?
[Oswald] We at times retain exceptional talent to do work that might be valuable in the future. And we can do like off-the-shelf data creation to basically like, you know, make use of supply when demand is is low, and then um and then resell that data later. Um in that case, we do. Otherwise, we don't.
[Harry] What's the best piece of advice you've ever been given?
[Oswald] I got a lot of advice to join small companies, join startups, move to San Francisco. I grew up in Canada. I I went to school in Toronto. Um I followed that advice. I think it was great. I've loved uh living out here, and I I like small companies. Uh I like fast-growing companies. It's been super fun and great for uh my career.
[Harry] Final one for you. What are you most excited about that you don't think enough people are talking about?
[Oswald] Probably the same answer as before in that like the 3 years out um opportunity um of robotics. Um I think there's a lot of discussion around robotics on on Twitter and in certain...
[Harry] I'm sorry, dear. Can you just help me out here? And this is where I get in trouble. It's Friday afternoon. It's past 6:00. [__] it. I can say what I want. I don't get it. Okay? Like whenever you watch a robotics talk like the yeah demo, they're like, you know, you see this kind of terribly moving robot around a home. And then after watching it like take one water out of a fridge in 15 minutes, it goes, "And Brandon was in the other room all along." Or And you're like, "Are you [__] kidding me? I had this absolute spaco in my kitchen for 15 minutes getting a water, and Brandon was in my bathroom doing it? That's where we're at?" What am I not seeing? How help get excited.
If you go back a decade, self-driving cars had the people in them all the time. Cruise had a person in it for years. But now I take Waymo more than Uber. It might play out similar to driverless cars where it's really hard to scale physical things vs software β more of a Waymo/robo-taxi moment than a ChatGPT moment. But the progress will be there.
[Oswald] Yeah, I think if you go back like uh you know a decade or so, like self-driving cars um had you know, they had the people in them all the time. You would see Cruise driving around San Francisco, and there was like a person in it for years. For years, right? Um but now I take Waymo more than I take Uber. Um there's someone...
[Harry] I'm thrilled for you. Welcome to London. We still have these people in cars.
[Oswald] I love it, but it's in one city. It can't deal with like very ambiguous data. It's like pretty irrelevant. It's made leaps and bounds in the past decade at least in in uh you know, San Francisco and in Austin, Phoenix. Uh it's tough because yeah, it's I guess it is the distribution is unequal, but uh it's an incredible service here in San Francisco and people here use it a lot. So, like technically, it works and um it might be you know, there might be like regulatory challenges or other challenges with scale scaling, but uh...
[Harry] Do you think we'll hit a chat GPT moment with robotics which will cause an inflection in usage and adoption?
[Oswald] I think so. Yeah, I think so. But, I think it might play out similar to driverless cars where it's really hard to scale physical uh things as opposed to to software. Um so, it might be more of a more of like a Waymo robo-taxi Cruise type moment than a than a chat GPT moment. Um but, I think the uh the progress will be there, yeah.
[Harry] Dude, you have been fantastic. Thank you so much for putting up with this like incredibly wayward, poorly structured conversation which was brilliant, and I so appreciate you putting up with it.
[Oswald] Thanks for having me. Yeah, it was super fun.