How to Revolutionize Data Engineering with Diana, the AI-Powered Jr Data Engineer

Automatic transcript of the recording (English); speaker labels are automatic. Timestamps open YouTube at that moment.

0:00Sacha: Hey, so today I have a very special guest with me, Josh Gray, the co-founder and CEO of Artemis, a company empowering data analysts to work faster and automate data transformations. Earlier this year, Josh was recognized as the top youth entrepreneur of the year in Canada by the League of Innovators. I'm really excited to have you on today, Josh. Welcome.

0:18Guest: Thanks so much for having me. Excited to be here.

0:20Sacha: Josh, let it start. So for those who don't know about Artemis yet, could you give us like a little bit of a pitch for the company? What does Artemis do and who is it designed to help?

0:31Guest: Totally. So Artemis is a data startup based here in Vancouver. And really what we're doing is we're building an AI junior data engineer. So really what that means, and it's our main product, it's called Diana. And Diana acts as what we call an AI agent, which can actually do work for the users. And really the problem that we're trying to solve is that data engineers have a lot of context that they need to keep at all times, whether it be what models they should be working on, where data is stored, how it's been worked on, and they need a lot of help with that. And so what our junior data engineer can do, Diana, is she can help those engineers work faster by bringing that context to them faster, helping them maintain that context so it's actually consistent, and really just help them actually do the work. And that's really how it's different. Rather than just being another tool that you have to fill in and manually work with, Diana does the work for the user rather than having to be told what to do. So I've been building that for about a year now, and it's been a lot of fun.

1:19Sacha: That's great. And you've mentioned the term AI agents. A lot of people are saying 2024 is like the year of AI agents. For some people, you know, it might be a new term, something that they've not heard of it yet, right? So could you explain what they are, what's an AI agent, and what kind of tasks they can handle?

1:37Guest: Yeah, totally. So the complex way to kind of talk about it is that agents, you know, leverage multiple tools like LLMs and maybe proprietary tools like Salesforce and things like that. And they provide much more fluid ways to interact with software and they can do and perform tasks on, you know, on behalf of the user. I think the easier way to put it is actually just agents are like smart little helpers that can actually use the technology that you have and do the work rather than you having to do it yourself. So a great example, let's go through a simple sales example. Your sales, you know, you're a BDR on a sales team. You have to fill out your, you know, your CRM every single day, right? With all the different lead information. Well, hopefully an agent can actually fill that out for you. And rather than you having to go and say, hey, I talked to this lead based on this, the contract size of their phone number, this is their name, this is their company. An agent can actually maybe, let's say, read your transcript of, let's say, the call that you had with the user and then go in and fill all that information for you. So right there, you're taking a piece of software that really would have been hard to automate earlier and, you know, pre-AI, let's say, because of how inflexible that data is. However, LLMs make that, make data really today, whether it's structured or unstructured, super flexible, and they can basically take that and perform work on top of it for you.

2:42So the way I like to think about it is more of a, you know, a little helper that can help you out, you know, do the work with you.

2:47Sacha: Yeah, that's a great explanation. And why do you think they're becoming so increasingly important these days?

2:54Guest: I mean, there's a whole bunch of different ROI case studies about it. I think for me, what's exciting and kind of a sentiment is, you know, since the PC, the last 20, 30 years has really been all about, moving and inputting data into software. Like when you think about most SaaS tools, what are they? They're databases that you can put software, like that you have to fill out, right? A CRM, it's a customer database that you put in all your information about your customers. It gives you a nice UI to do that. So it's maybe not a spreadsheet. And then it can help you understand what's happening in your business. You know, same thing for, you know, blog posts, things like that. A lot of that is just data that you're moving around and writing and putting out. What's really exciting about agents is that that digital paper pushing, so to speak, can actually be automated and taken away from us, which means that we can get really into the value of what we want. I'm a big believer that the value of a worker isn't in the fact that they can digitally push paper and fill out forms, you know, for, you know, the CRM. You know, the value is in that person deploying and understanding, okay, here's a problem that I'm trying to solve. How do I want to go solve it? If you have more time to do that, what you can do for your business is astronomically more. Mm

3:54Sacha: -hmm. That's a great take. And so I want to get into how Artemis actually leverages AI agents. So you've talked about Diana. Could you explain how these agents interact with, like, a data engineer's workflow, for example?

4:07Guest: Totally, yeah. So we're pretty opinionated in how we built Diana. And really what we believe is that Diana should be working with the engineer hand-in-hand where the engineer works. And so because of that, you know, while we do have a web UI, we also work within tools like VS Code, like Slash, like GitHub, where our engineers are already working, where our users are already working. And you interact with Diana just like you would any teammate. You can tag Diana and ask her the question. And then what Diana's going to do is take that, break that down, and actually then start to do the actions of that task and show you step-by-step exactly what task she's doing in order to solve that problem. So let's go through a simple, super simple, bare-bones use case. I'm a data engineer that wants to, you know, fix a broken pipeline, right? A pipeline of mine is broken. How do I fix it? Well, if you ask Diana, what Diana can do is first go and investigate. So we can go and open up the file in the error logs. Hey, why did this pipeline break? It can read that and say, hey, we think that broke because of X. Because the error logs show X and Y, therefore, this is how you need to fix it. So Diana is now able to investigate. She can read the file. She can then understand why it's broken. Then she can say, hey, well, because it's broken because of X, we can actually fix X by doing Y.

5:10And then actually start to go through the steps to fix that. And in the end, she sends the user a PR request in GitHub to prove that change. And so right there, you have a full step-by-step breakdown of exactly what Diana did, which is super transparent. It's not black box. But it's able to take in super flexible inputs, a random question or natural language, and then take that, break it down into a series of tasks and provide an output that's useful to the user.

5:32Sacha: That's great. And you said that Diana works hand-in-hand with the data engineer. I'm curious, how steep is the learning curve for a data engineer like me to start leveraging Diana?

5:43Guest: I mean, as long as you can speak to people, you're good. Diana is super easy to work with, right? You know, LLMs have really given the world an easy interface. Now, we've built Diana a way that it actually takes care of a lot of the things on the back end for you. So you can actually have a super high level. But also, we want to give you the juicy details, right? As a data engineer, you want to know what Diana is doing. You want to be able to test and validate the actions that she's taking. So we give you everything that you want to see in the UI that's really easy to work with and easy to use. But, you know, you interact with Diana in a natural language way. So you just ask and, you know, delegate work. And Diana is able to take that and run with it.

6:16Sacha: That's amazing. It reminds me of the trend that we're seeing with, for example, Databricks that's democratizing access to data, for example. So that's amazing. How do you envision Diana or Artemis to transform the day-to-day of a data engineer, for example?

6:32Guest: For us, we want you to crush your weekly tickets. That's what we want to do. We want to make data engineers get promoted. Like, when you think of, like, what, you know, gets us up and out and out of bed into the offices, we want data engineers to be working at 5x what they currently are right now. And not to be bogged down by things that, in reality, they don't even want to do, right? There's so many little pieces of manual processes and tedious activities that data engineers put up with because they always have had to where we don't believe they need to anymore. And so if we can get an engineer to, you know, output, you know, close 10 more tickets at the end of the day on Friday and actually enjoy their weekend, that's a huge win for us. So really what we're trying to do is build Diana in a way that she can handle more and more complex use cases so that you can actually work hand-in-hand with this, you know, with this, you know, AI agent. The other benefit as well, I think, to flip on the other side is what's really powerful, too, is Diana has full context of all your data. So it understands your models and how they interact with each other. It understands your warehouse by pulling in metadata. It's like talking, you know, to somebody who has full access. You know, it's like talking to the first engineer that set up the entire infrastructure, right?

7:33They all know it, but if they were to leave the company, there'd be a lot of issues, right? We've essentially built Diana to be like that person. And what's great about that is that you can work with Diana to get your answers faster. You can work with her to solve problems, you know, and debug issues faster. And so we don't see it at all as a replacement. If anything, we see it as a huge accelerant to what engineers can get done in a day and, you know, what they can accomplish over a year.

7:55Sacha: That's awesome. And you kind of answered that, my next question already. But I still want to get into it because, you know, AI for a lot of people that are not as familiar with the technology as us, it kind of scares people, right? They're scared that it may eliminate their job or replace them fully. So I'm curious for you to elaborate a little bit more on, okay, so is Diana going to replace my job as a data engineer or is it going to augment and really enhance the work that I'm doing?

8:24Guest: Totally. So I think that's a really see Diana as a replacement. Now, Aaron Levy, who is the founder of Box, huge enterprise company, he has an amazing tweet. Actually, I wrote a blog about it. You can find on our website. And it really kind of walks through this calculus of, you know, what does it mean to bring on AI productivity? And the point he makes is every company in the world is capital constrained, basically. Unless you're like Microsoft or Amazon, capital is usually the number one dictator of how companies are run, right? Do they have a large headcount? Do they have a lower headcount? How many teams do they have? Do they even have a data team? Based on the amount of revenue and income a company makes, that's going to impact that question. Now, with AI, what's powerful is let's say you have 10 engineers, right? And you're able to get 50% more done. That doesn't mean you therefore, you know, you can double your output. That doesn't mean therefore you have to fire, you can fire half your engineers, right? Because if you fire half your engineers, you're just going to get done what you were currently getting done. But if we can get done double with the same amount, we can actually bring in more revenue. We can get more done on our roadmap. We can accelerate that much faster.

9:25We can pull that revenue forward. Well, what's that going to mean? It means we make more money, which means that we need a scale factor, which means that we need more people to help us get there, right? Like, it's a very iterative process that we, I'm a huge believer that will actually snowball to more people. You know, sure, there might be smaller teams at some organizations that can make do without. But that was happening regardless whether people wanted to say that or not. Agents have huge potential to really accelerate what teams are doing currently. And yeah.

9:51Sacha: That's awesome. Can you elaborate on the pre-built knowledge graphs? I think I've read that on your website or one of your blogs. Yeah. And I'm curious how they accelerate data cleaning.

10:01Guest: Totally. Yeah. So really what knowledge graphs are and, you know, their piece of technology, my co-founder was really working with them early, I'd say, you know, a few years back now. And what's exciting about them is that it's really the way of how we're structuring and giving context to LMS today. You know, you look at companies like, you know, amazing Canadian companies like Neo4j and a few others who are really leading the charge. But for us, it's a great way for us to contextualize and give Diana a roadmap of how and what to look at. And so for us, how we operate is when we integrate with a company, we integrate with their data warehouse and we integrate with their GitHub. And in GitHub, we just integrate with their model. We go for their data models because we don't want to access anything else. And so because of that, using our knowledge graph that we've already built and built and can take in that context, we're basically able to build out this really rich graph that Diana can traverse to actually understand what's happening. So when you ask it, hey, investigate this pipeline bug or, hey, go write this model on X or, hey, go look at Y, Diana knows what to do and where to go. And so it's kind of, you know, we like to think of the knowledge graph as almost like, you know, the Google Maps, right? It's plotting out the neighborhoods and the streets that can tell the self-driving car where to drive.

11:02But, you know, yeah, there's a lot of opportunity there and it's a really cool technology.

11:06Sacha: It is. Do you see a future where Diana can handle more complex data engineering tasks like building data pipelines, for example?

11:15Guest: Absolutely. I mean, we've already done it. So 100%. We actually, it's funny, we have a promo on right now. So if you're listening to this, go check it out, where we basically say, give us the most complex data engineering tasks you have. And if you give it to us and Diana can't do it, we'll send you a $50 gift card of your choice. Because we want to be testing and pushing Diana to the absolute brink. So if you have one, come let us know and we'll see if Diana can handle it.

11:36Sacha: But can you give me a few examples of different tasks that Diana can accomplish?

11:42Guest: Totally. So one of the examples I gave earlier, which was that idea of fixing pipelines. So taking the error logs of saying, hey, you know, this pipeline keeps on breaking. Maybe it's an issue in the warehouse. Maybe it's an issue, you know, in the actual model, right? In the DBT model, Diana can basically look at that, investigate it, and then actually write the code to fix that in real time. Looking at upstream and downstream impacts. Giving you that root cause analysis and the fix while you're sleeping is a huge value add to companies, right? It's a very complex thing and can save a whole lot of time and a whole lot of issues. You know, if you have a pipeline that's been off or not running for 12 hours. So it can help kind of with that downtime, that monitoring. Another big one as well that we've really seen is investigation. So, you know, pair programming with an engineer when they're going through maybe building out a really complex data model or, let's say, a new product that they brought online. Pair programming with that user to say and actually, you know, get the right documentation in place, get the right unit tests. All that sort of work can be done with Diana.

12:38Sacha: So now I want to get into a bit of ethical, you know, conservation. So AI is taking more and more responsibility in different tasks like data engineering. What ethical considerations should companies be aware of? And how does Artemis ensure transparency in its agent decisions?

12:57Guest: So by ethics, can you kind of elaborate on that?

13:00Sacha: For example, responsible AI. You know that, for example, AI models, usually they're considered as black box. You know, I think that's a lot of people who are talking about earlier in this conversation. How do you ensure that this is not the case with Diana? That we know what's going on, how it takes decisions, how it, you know.

13:17Guest: 100%. I think for us, we build a platform that our users want to use. And users today don't want black box solutions, right? They don't want that. They want clear output with clear understanding of how they found that output. Especially when it comes to calculations. You know, if you want an LLM to write you a blog post, yeah, I don't care, like, you know, the statistical background of that LLM to write that blog post. That's whatever. You're looking at how modeling, you're changing code. Yeah, absolutely. I want to know how you did that, right? I want to know the sources. I want to know the references you pulled. I want to understand, you know, what models you used on that. We highlight all of that in our UI. So, like I was saying earlier, every single task that Diana does for you is shown in our UI, which means that you can go through and see exactly what Diana was looking at. Even if Diana messes up a task and fixes it herself, you'll see that. We're not going to cover that up. Even if Diana makes a mistake. And sometimes, you know, here's the reality of it. AI agents operate like humans. They don't get it right every single time. They're going to mess up. It's just part of the game. It's non-deterministic software. It's non-deterministic technology, which means that you can't actually predict the outcome. It's going to be different slightly every single time.

14:18And so, because of that, you need to be able to have an interface that, if it does make a mistake, you can fix it really, really fast and really, really easily. And so, we are really transparent in how we provide the insight into exactly what Diana does in a way that, if we're being honest, probably more transparent than some employees, right? You know, if you ask Diana to do a task, she's going to tell you the second she's done. She's not going to, you know, do half of it, go for a coffee break, come back, you know what I mean, and not say how she did it. She's going to give you, you know, step-by-step breakdown of exactly what she did. So, it's really easy to follow along. It's really easy, actually, funny enough, a great use case we've seen is teams, when they're trying to solve a problem, they'll give it to Diana, and they'll walk through the solution of the team and see how Diana would have done it. You know, it's a great learning tool as well.

14:57Sacha: That's awesome. And adopting technology is usually a lot of change management within a company, right? What advice would you give to a data team that may be a little bit hesitant to adopt, like Diana, for example?

15:10Guest: Yeah, I mean, obviously, it depends on why you're hesitant, right? Like, you know, if you're at a hospital and it's security, then maybe be hesitant, you know what I mean? Not from the perspective of it's your duty to make sure that everything's secure, right? You know, it's like, you know, maybe not rush into that. However, I think for most teams that are looking to explore, I think a lot of teams right now are afraid that they're going to get either replaced or that they're not going to have any value. And for us, the way we look at it is like, you know, if anything, you know, you have quarterly goals. Let's get those quarterly and six-month roadmaps collapsed in the next month and a half. And let's blow the socks off your boss, right? Like, let's have your manager scratching their head going, like, how in the world did they get all this work done? Right? Like, that's not going to get you fired. That's going to get you a promotion or anything. And kind of like what I was saying earlier, for us, we want our users to look like superstars. That's the goal of Diana, so that when you're using it, it looks like there's 10 of you everywhere, right? Because it's just you're able to do so much more faster. And so for us, like, you know, we totally understand that hesitation. But, you know, we really encourage teams to dive in and really start to experiment because there's so much opportunity in what you can do. And there's so many ways that you can make your life and job so much more enjoyable.

16:13Sacha: Would you be willing to share, like, some success stories that you've had with clients that benefited from Diana?

16:19Guest: Yeah, totally. So, you know, we've had a couple of users, and I won't say names for now just because it depends on the contract. But, yeah, absolutely. We've had some. You know, we've had one in the healthcare space that used Diana to really optimize and scale their team. You know, they had a smaller data team. I think it was only three or four engineers. And they were really falling behind on their annual metrics that they were trying to report. You know, they had a huge conference coming up, and they were able to leverage Diana to basically get all the reporting done for year-end and then actually scale and start to do more work on that conference work that they need to get done. That's just a quick example. But others who use Diana to basically automate how they fix and maintain their pipelines. So, you know, looking at mistakes and issues and bugs and saying that report to them, they wake up now knowing this will be, you know, this has been investigated as to why it breaks. And they can go in and make those changes right away in the morning. So, really practical, simple use cases that save a lot of time. And, honestly, a lot of fear of what if this breaks in the middle of the night or three days ago and I don't ever know. A lot of that can kind of be taken away.

17:15Sacha: That's amazing. Do you have, like, a specific number for how much more productive I can become using Diana?

17:20Guest: Yeah, we typically see if you can delegate about five to ten, you know, ten tasks a week, that saves you about 15 hours of work. That's

17:27Sacha: awesome. What are the most exciting developments or features that you're working on or that are, like, on your horizon for your company? Oh,

17:35Guest: it's a tough question. There's a lot. But I think one of the things that really has me excited is, and, you know, stay tuned because we'll be putting some more information out about this, is to, you know, to build and understand how Diana works. We built up this really comprehensive evaluation, this really big benchmark that we use internally to understand what tasks Diana can get done, how well she can complete it, and then how we can improve Diana, basically. It's kind of how we use it. However, there's not really many in the data world that show. There's a lot of evaluations for software engineers, but there's not a lot of evaluation for data engineering because there's not as many data engineering agents. And so we're really excited. We're going to be launching and releasing kind of some of the insights on that. In the next week in a bit. So stay tuned for that. But we're really excited to see, you know, how kind of the analytics and data engineering communities respond to that evaluation, how they feel about it, and kind of where they can see that value.

18:20Sacha: Josh, I want to give you the floor to share some last words, whatever you'd like. The floor is yours.

18:26Guest: I just think, you know, get into this stuff as fast as you can. I think there's so much opportunity in this world right now. There's a lot of hype. There's a lot of, you know, not real value being accrued. Absolutely. There's a ton of real value being built right now. It might not be recognized today. It might take, you know, five years for it to really be, you know, seen. But this is, you know, an internet-like event. It comes every once, every once in a while. There's a ton of opportunity. It's a really cool space to be in. It's super exciting. So if you're kind of, you know, on the sidelines of it, get into it, dive in, and learn as much as you can.

18:58Sacha: Thank you, Josh. It was a real pleasure.

19:01Guest: Thanks so much. Appreciate it.