Josh Gray
Co-founder and CEO of Artemis
Episode with Josh
How to Revolutionize Data Engineering with Diana, the AI-Powered Jr Data EngineerArtemis CEO Josh Gray explains Diana, an AI junior data engineer, and argues that AI agents will add to data teams rather than replace them.Josh GrayCo-founder and CEO of ArtemisIn Josh’s words
An AI agent in one sentence: a helper that uses your software and does the work.
I think the easier way to put it is actually just agents are like smart little helpers that can actually use the technology that you have and do the work rather than you having to do it yourself.If AI makes your 10 engineers 50% more productive, firing half of them gets you nowhere.
Now, with AI, what's powerful is let's say you have 10 engineers, right? And you're able to get 50% more done. That doesn't mean you therefore, you know, you can double your output. That doesn't mean therefore you have to fire, you can fire half your engineers, right? Because if you fire half your engineers, you're just going to get done what you were currently getting done.The CEO of an AI agent company says it plainly: his agent will mess up.
And sometimes, you know, here's the reality of it. AI agents operate like humans. They don't get it right every single time. They're going to mess up. It's just part of the game. It's non-deterministic software. It's non-deterministic technology, which means that you can't actually predict the outcome.Most SaaS tools are just databases you fill out by hand. Agents could end that.
since the PC, the last 20, 30 years has really been all about, moving and inputting data into software. Like when you think about most SaaS tools, what are they? They're databases that you can put software, like that you have to fill out, right?Filling in a CRM was never the valuable part of your job.
I'm a big believer that the value of a worker isn't in the fact that they can digitally push paper and fill out forms, you know, for, you know, the CRM. You know, the value is in that person deploying and understanding, okay, here's a problem that I'm trying to solve. How do I want to go solve it?Every data team has one engineer who knows everything. What if that knowledge never left?
It's like talking, you know, to somebody who has full access. You know, it's like talking to the first engineer that set up the entire infrastructure, right? They all know it, but if they were to leave the company, there'd be a lot of issues, right? We've essentially built Diana to be like that person.A knowledge graph is the Google Maps that tells an AI agent where to drive.
And so it's kind of, you know, we like to think of the knowledge graph as almost like, you know, the Google Maps, right? It's plotting out the neighborhoods and the streets that can tell the self-driving car where to drive.An AI agent that can't hide what it did may be more transparent than a human colleague.
if we're being honest, probably more transparent than some employees, right? You know, if you ask Diana to do a task, she's going to tell you the second she's done. She's not going to, you know, do half of it, go for a coffee break, come back, you know what I mean, and not say how she did it.Afraid AI will replace you? Use it to make your manager wonder how you did so much.
Let's have your manager scratching their head going, like, how in the world did they get all this work done? Right? Like, that's not going to get you fired. That's going to get you a promotion or anything.There's plenty of AI hype, but real value is being built too, and it may take five years to show.
There's a lot of hype. There's a lot of, you know, not real value being accrued. Absolutely. There's a ton of real value being built right now. It might not be recognized today. It might take, you know, five years for it to really be, you know, seen. But this is, you know, an internet-like event.This AI startup pays $50 if you can find a data task its agent can't do.
we basically say, give us the most complex data engineering tasks you have. And if you give it to us and Diana can't do it, we'll send you a $50 gift card of your choice. Because we want to be testing and pushing Diana to the absolute brink.Delegate five to ten tasks a week to an AI agent and save about 15 hours.
Yeah, we typically see if you can delegate about five to ten, you know, ten tasks a week, that saves you about 15 hours of work.Software engineering agents have benchmarks. Data engineering agents mostly don't.
There's a lot of evaluations for software engineers, but there's not a lot of evaluation for data engineering because there's not as many data engineering agents.