T-Shaped Analytics Engineers, Curiosity and Proving Impact — Oleg Agapov

sure. Yeah. yeah, first of all, thanks, thanks for having me. I believe this is going to be a great conversation. yeah, and like you basically highlighted my career, and especially when you said that, oh, you know, this was the pivotal moment for me when I realized that there is analytics engineering, there is a technology, you know, there is a framework. And at, at that moment, it actually all aligns in my head. because for, like you mentioned, you know, I worked as data analyst for like majority of my career. And one thing always bothered me is how, you know, how messed up analytics is. You know, like all those ad hocs and, and SQL flying around and some ad hoc Python scripts, right?

It, it all felt like a cause, right? And at some point, I was like, "No, there should be a better way," right? And this is when I discovered, and it was like around two thousand and eighteen, I believe. This is where I discovered that, oh, there is this new kinda, you know, quote-unquote, "new" data engineering field. And it felt like it's something that could, you know, I should do, right? Because it's kinda structuring your inputs, structuring your outputs. And this felt like, oh, you know, this is something where I should explore more, right? Learn more. and maybe this will help me kinda improve my, you know, analytics skills in, in the sense that like engineering, analytics engineering skills, right?

And only then, you know, in like a few years, I discovered that, oh, there is actually analytics engineering, right? And it's called like this. And, technology actually allowed us to do so, right? Because you mentioned that at GOG I worked with Hadoop and, and Spark. And it was-- The thing that didn't work for me back then is I couldn't understand how I can, you know, take this technology like Spark, right, or Hadoop and, and like data modeling. How can I close the gap between like engineering part, right, and analytics part? Because I was still data analyst, right? So my main focus was producing insights, you know, looking at the data. It's just I felt like it's always inefficient.

It's always, you know, chaotic, you know, this whole process. And, and then I discovered that, oh, you know, there is this great technology called Snowflake, and I learned how distributed data warehouses work. And I was like, "Okay, this is actually closing the loop because now I can use vast amounts of data, right? Store them in data warehouse, use SQL because it's like standard, it's easy to use, it's easy to understand, it's easy to store and produce, right?" And at this moment, I was like, "Okay, now everything clicked together," like dbt for transformation, you know, for testing, for documentation. and now all those data modeling paradigms, right, they also like clicked together because okay, now I'm using the tool, but I'm moving towards like dimensional modeling, and this is how it all fits together and work together.

So... And from that moment, I was like, "Okay, yeah, so analytic engineering is what I actually wanted to do this whole time," right? I, I don't want to go like too deep into data engineering, right? Because it's still like a bit more technical, a bit more involved in like coding, you know, infrastructure, all those like scaling issues. While analytic engineering is more lightweight, you know, data engineering. I, I still like that, you know, we use all those software engineering best practices, right? We use Git, we use, you know- Like DevOps practices, we use lifecycle. So this is all working great for analytics, right? And that's why, like, I, I want to stay here.

I want to develop this, you know, field. I want to teach this field. I want to learn this field. Yeah, that's, that's pretty much how, how I got into this

Right. Great question. Okay. All right. Yeah. so I usually try to explain it as, as a contrast. so when we take the analytics process, we usually think about it in, like, two big areas. So one, one area is how you prepare the data, so the technical part, right? And the other area is how you present the data, how you find insights, right? How you work with this data. So there is, like, a preparation and, and, actually, activation, right, you could say, of this data. So, and those... There, there are two titles, right, which is data analyst and data engineer, which actually grade, like, they, they map great on those two areas, right? So data analyst is working with the data to provide insights, while data engineer work on, data to gather it, to store it, to process it, right, and can provide some inputs for data an- data analysis.

The thing which was always missing is data engineers, they are technical, yes, but they are not too deep into the business, you know, into the analysis part of this data, right? They for sure can know, like, they know a lot of details about, oh, you know, how should we structure the data, how we should model the data. they for sure should know about d-dimensional modeling, right? But because they do not communicate with the business as close as data analyst, right, they have this, a little bit of disconnection. And on the other hand, we have data analyst whose, like, main job is to provide insights, right? And to do this efficiently, they need to, to access the data as efficient as possible, right?

Ideally, without skipping all the nitty-gritty of how can I derive my metric from all the source data that is available, like, in the data warehouse, right? Or, like, data lake. And usually, data analysts, they are more on the business side, and they are less on the engineering side, right? And this is where we're starting to see that there is the gap that ideally we would want someone who is technical enough to understand dimensional modeling, to understand how data is flowing, to understand data pipelines, right? and on the other hand, is close, is pretty much close to the business so they can understand what is the best output of my work, like, of my, of my engineering, like, right, that will be useful for everyone downstream.

Like data analysts, BI tools, maybe stakeholders through, you know, AI tools, right? And this is, to me, it's, like, very natural that analytics engineering appeared because it's actually the role that closes the gap, right? Analytic engineer are technical enough. They use software engineering best practices to deliver analytics, yet they are very hands-on with, with analytics part. For example, I can, I could speak for myself. You know, I, I build, data models in dbt, but I also build self-serve analytics in, in BI, right? So I work with semantic layers. I work with dashboards. I teach our business how to use self-service, how to use dashboards, right?

So I'm very hands-on with both parts while not going very deep in both of them. So I would say that's... And, and for myself, I think this is for me the sweet spot because this is what I actually like to do, right? I'm, I'm technical enough, and I'm also close to the business enough that, you know, I could combine those two worlds.

Right. Yeah. Obviously, AI is, like, accelerating a lot. All, like, probably all, you know, IT roles, like engineering roles. So, so I usually-- I like this concept of T-shaped person, if you ever heard of this. So T-shaped means that you have a vertical, which is where you, like, your deep knowledge is, right? And also horizontal. This is, like, all the adjacent kinda disciplines that you, kinda a-at least understand how they work, right? So AI really helps with, I would say, kind of both, you know, both lines because, like, if you look at your, like analytics, let's take analytic engineering and let's take, like, core responsibilities, right?

I can do... AI here is really enabler because I can do my work a bit faster, right? Maybe a bit more efficient. maybe I need to Google less, you know. this all sounds like maybe it's, like small things because this is my core, like, this is my foundation, right? I can do this even without AI, right? But I'm using AI to kinda multiply my outputs, right? And then when I go to this horizontal line where, oh, you know, like I'm working on, let's say, CI/CD for, you know, dbt pipelines, and DevOps is not my specialty, right? I don't know all the intricacies of, like, GitHub Actions or like other, you know, DevOps frameworks and, and, and tools. And this is where I, I would rather turn to, you know, AI to give me some guides, right?

Again, it, it is en- like enabling at some, you know, at some level, right? Where I probably, I don't have deep expertise, but it helps me speed up, and it helps me, depend less on others, right? but, but again, to, to kinda reiterate and going one step back, we still need to know foundations, right? And we go horizontally only when we need to, and now we can do this because AI helps us a lot, right? I don't have to know how to set up, you know, I don't have to remember how to set up permissions in Snowflake because it's just one prompt away, right? otherwise I would look at Google, you know, or go to, like, data engineers or, like, kinda work my, way around it, right?

Because it's not my core kind of responsibility, right? but it enables me to do so, you know, to do that and not depend on much on, external parties as much as before, right? so it's definitely, like, it's hard to predict where it's going, right? And at what point we will stop and at what point, you know, obviously lines are blurred. And they were before, before AI blurred to some extent, right? But now it's even blurrier because, oh, you know, what is, what is analytics engineer? What is data analyst like? Because data analyst now can also, you know, a lot of people in my comment not argue, but like, they give examples where they work as data analyst, but they have to set up the whole s- data stack, right?

And like, so they can do, you know, they can set up Snowflake, they can set up ingestion, they can set up everything even though their, their title is data analyst. AI really kinda enable them to do the other part and struggle less, you know, and be more efficient. but should we call them analytic engineer at this point? I don't know, right? It, it really depends on where, you know, the specific person wants to go or the specific company requires them to do. so if, if their main responsibility is producing insights, then all this infrastructure engineering is just a side quest, right? it's just AI really helps them to get there faster and, like, more efficient.

Yeah. One thing is still important for now is obviously engineering culture. So we are not skipping fundamentals in a way that, oh, you know, now we just give it to AI, and AI just, you know, Code Slop us a lot of, you know, configurations and instructions, and we just don't care what is under the hood. We still, we still want to have engineering culture where we have structured inputs, structured outputs. We control the process. We test the process. We document the process. and it actually, it actually works pretty well with AI as well because we all know that the respond-- like AI response really depends on how much context you give it. But also Also the quality of this context, right?

Because if you just, just provide it like your dbt project, you know, dump all the code in there and ask it to do some stuff, it will produce some stuff. What is going to be the quality of this output? I don't know. Like no one knows, right? It's, it, it could be a pure slop, right? that's why th-this is where engineering culture is kinda helping us because, one, one example I could give you where it worked very good for us, like at, at my company, even before this whole AI madness and, you know, AI systems and, like, AI-enabled, work, we always maintained pretty good standard of how we do dbt projects. So we always had, you know, how, documentation on, on how we write code, how we interact with other developers, how we produce pull requests, how we document changes, how we should, like even to the level like, oh, how we should write SQL, you know, what are our conventions or s- of SQL, right?

What are our conventions for how we write documentation? So we have, we have convention on how to write YAML files, which is kind of crazy, right? and then so when we started looking into AI-assisted, workflows, right? We just fed all this information into agent, into, into Claude, right? And it was able to produce results that are very similar to what we would produce ourselves, right? Just because it had structured and very quality content, right? Context. and it helped us a lot transitioning from purely, you know, human-based outputs to kinda assisted outputs where, you know, human is steering the AI, but AI does most of the work. Thanks to the context we provided, the output is usually, you know, even from the first run, it's like seventy, eighty percent of what we would do ourselves, right?

So it get us there f- much faster.

I hire across the board, so it doesn't have to be seniors. f-for me, like for, for majority of my career, because I was data analyst, I was mostly hiring data an- other data analysts, right? But then with recent changes, I also hired a couple of data engineers, BI engineers, and with like last couple of years, I only hire analytics engineers, and across the board. So, one example I could give you, like last year, it was my first time where I hired an intern. So I never hired intern, interns before, right? And it was like pretty new experience because it's the level which is below juniors, right? So it's like it was interesting experience.

Right. to, to answer actually the second question, like the, the senior and, and the progression and ac- and actually the, the juniors as well. So one thing which really helps me understand that this person will be successful is when they show curiosity. Because this is like a single driving, you know, force that helps you to grow, to grow faster, you know, to grow in the correct direction. Because as long as you are curious, as long as you, you know, you, you don't have this tunnel vision where you, "Oh, you know, I want to learn and master SQL to the extent that, you know, I will solve LeetCode Hard in 10 seconds," right? This is, this is obviously impressive, but it has nothing to do with the real job, right?

but if I see that the person is curious, in, in the work that he does, right, in the technology, in the business, it all signals me that this is actually like even if person is not, maybe is not as good on technical levels where I know I, I expected them to be, but the person is curious and I can see that there is a potential, I would rather hire this person than someone who knows, you know, SQL, completely like all the, all the functions and stuff, but has very little, you know, interest in like how the business is working, how, how technologies are evolving, what is going on with the role, et cetera, right? So this is, this is for me the biggest indication I would say.

when it comes to hire like a specific, grade, right? Let's say if we go for juniors, I want for them to be, to be independent obviously, right? So this is where I understand that, you know, if I give them a task and I understand that like obviously independent in the way, independent in their responsibility range, right? Obviously, right? I, I, I don't believe that juniors, you know, should run projects, right? But they could, could be independent in some very small area where well-defined and this is probably where, you know, managers or like team leads should, help them. So I, I look at them whether they can produce the output, from start to end, right?

So if I give them a task like, "Hey, we need to", whatever, let's say, you know, "We need to add a new data about users to our dashboards", right? And I will ask them, "What would you do", right? And for me, the ideal answer would be, you know, I would, like I would start to trace the data. I would ask business how we define this data. You know, I would look into like dbt. So, so juniors are still very hands-on and very technical. So because I don't want to hire a junior who understands, like who is curious about the business, but like knows very little about SQL and stuff, right? Because I don't want to spend too much time teaching technical parts.

Even though like I understand obviously there is, there is some level where I accept that, oh, you know, if, if you know how to do select star, but you don't know how to do, I don't know, window functions, this is still enough, right? This is still enough for me because I understand that this is just a technical skill that can be acquired while doing the work. This is how I done myself. So when I, when I started as data analyst, I only know how to do select star, and it was fine for me, right? I exported everything to Excel and then did, did the rest of the job, right? only then I learned, oh, you know, you can do joins, you can do window functions, you can do some crazy, stuff with SQL.

so, so there is obviously some level that I expect even from juniors, right? So they need to understand how, you know, what is the database, how to create the database. You know, what is Excel, how to manipulate data in Excel. So like basic literacy should be there, but I obviously don't expect them to be, you know, even like, if, if we translate to LeetCode, so probably LeetCode easy is what I, I'm looking, for in, juniors, right? But as I said, if they are curious, if they can prove me that they are-- they have like attention to details, right? They don't try just, you know, dump question to, Claude to get a response and give this response back to me, right?

This is useless because I could, I could do this myself, right? So but I'm fine if they dump question to AI, get the response, process the response themselves, and give me like one, two sentence output of this response, right? Then I know that this person actually went through exercise. You know, they, they learned, the output. They, they kind of processed it, right? So that I can consume it better. as for like more senior roles, obviously I want to see like more independence where on a level of projects. So if I say like, "Oh, you know, we are working on new set of dashboards for finance. What would you do?" So this is my, you know, question on, on the interview.

And then I see how they interact with like how their response to this. Are they going to, you know, ask me questions back? You know, this is very important. are they going to find some, you know, any existing work, like past work on this? Are they going to talk to business stakeholders back and forth? Are they going to produce some artifacts like, you know, design doc, right? so when I see then this, if, if this like groundwork is inside of them, then I understand that, okay, yeah, they are probably senior enough. And I don't really even care that much about their technical abilities because I understand that if they can lead the project, they, the, the technical kinda skills, they should be there already, right?

Because you, you cannot get there, you know, with skipping all those parts. So we obviously, we obviously do, take home, assignments for all grades. Ev- like even, even for like as I mentioned, for, for the intern, we gave a very small take home, you know, assignment where we could see, you know, the output. Because, take home assignment is very similar in a way to how you would work, right, on your daily job. Because you, you have the assignment, you have some time, you know, you s- have some head space to actually think through it. You know, uh- Process it, produce the result, verify the result, and then give me the output, right? So, and this is where it's usually ob-obviously take-home assignments, they are technical, but this is usually the second thing I, I check when, when I look at them, right?

First thing I check is how they structure it, what is the output? Is there a summary? You know, is there, a c-a clean insight out of, of their work, right? Because if they just try to feed me with whatever AI, you know, produced, like again, I don't need that, right? This is just a... This is-- I, I can do this without this person, right? For me, the most, the most valuable is when person actually takes the output, process themselves, right, and give me like the golden nuggets out of it. so this is what I am looking for, like in more senior roles. But again, just to, just to repeat myself, curiosity. So when I see that, you know, if they ask me questions like, "Oh, you know, is this a real data?

is this the actual scenario you got on your work, right?" This is, this is when I know that they are actually went through the ex-exercise. They got excited about the data or, you know, the, the technical parts, and they actually did the job. So, yeah.

G-great question. so the impact can be measured on, I believe, like on different, like on many levels, depending on obviously organization, how they... what was your involvement. So for me, I would go like this. So first of all, try to, especially if you're like more on the junior side of, of, of the role, try to see parts where you improved any kind of process technical-wise. So my, my personal example, while I was a data analyst and we had this thing called, like a weekly reporting where I had to manually run some SQLs and then put them in, in Excel spreadsheet, maybe do some like final touches, right? And, and then give to, like to C-level.

And because I was like, you know, young and like I had this energy, I was like, I was-- every time I was very excited doing this weekly, even though it's like very repetitive, you know, boring job. Back then I, I felt like, "Oh, you know, this is, this is the work, right? I'm doing the work." but then at some point I realized, wait, I'm, I'm wasting like every week, like three, four hours of my time on stuff that is easily automatable, right? And I started to see, can I, can I improve it, right? So can I automate it? And this is where I first time, you know, spoke to like our data engineers. We didn't have data engineers. We had like database administrators, right?

So I talked to them, "Hey, I have this, this task. Can it be somehow automated?" And they told me, "Oh yeah, use stored procedures." Like, whatever, right? So it was at least some kinda s-step forward. And, and from that moment, I actually started to see that, oh, you know, a lot of work that I do can be automated. And this automation to me is very, very material thing to show to my, manager, right? Because if they would ask me, "Hey, like, what we're doing this last year that, you know, help us, in some way?" I say, "Hey, remember we had this da-da-this dashboard, which was like manual and sometimes had errors and sometimes was delayed. I actually went and automated it, and now you have this artifact produced for you, on time every week, you know, with correct numbers."

So this was very s-- it's, it's like automation, right? It's a very small impact for juniors. Like, I mean, not, not, not for juniors. For juniors, it's actually a big impact, right? Because they released like three, four hours a week that they could spend on more meaningful work, right? but then for me, it's, for me as, for example, as a manager, if I would see this progression, I would say, "Oh, you know, okay, this person has something in it because instead of doing this job, you know, over and over again and just doing this, meaningless work in a way, right? They actually went a bit far and automated it. And now we have, you know, they could say, "Oh, you know, I actually did this automation, so this is my impact."

And then as you grow, as you grow, you start participating in more projects, right? And you start seeing parts where, oh, you know, last week-- sorry, last year, for example, we released a big feature, a big project, right? And for me, as data analyst, it was always, interesting to see where I participated in this project. Was I delivering the initial numbers? Were my numbers actually affected the output of the project, right? So one thing, a-again, this is from my d-like data analyst, years. we were at this point where we had to decide what kind of feature we should build next, and my analysis actually helped shape the product because, because the initial, So initial gut feeling of our, stakeholders was not entirely correct, and I proved it with numbers.

I could say, "Hey, this is, this is what your hypothesis and these are numbers for your hypothesis, right? And this is my hypothesis, and I checked it with numbers." So, so I basically gave them like two, one-pagers, and we decided to go with my solution, like with, with my suggestion just because numbers, you know, you, you cannot argue with numbers, right? And this is where I could say that I shaped the product, right? I made an impact. I actually produced something that, that changed the direction of the product. And now if we go to like very probably like senior level, right? you have way more, Now you, you stop thinking in projects, and you start thinking in like company impact.

So what I can say, for example, for, for myself again right now as analytics engineer, like what was my impact on, on the company? last year we, we rolled out a new BI tool, Omni, and it has like self-serve capabilities. And one thing we measured is how many people actually using our BI tool on a daily basis, and we increased like weekly active users of our BI tool, like I think like two point five times, right? So we enabled that many people to actually use our analytics, right? And obviously we kinda, we know that if people are using analytics, they are kinda more aligned with the, with, with, with, with the business, right? With the product.

So our kinda North Star is to see, like is to enable more people to use analytics. And it was very clear to me that, oh, you know, we increased weekly active users of our BI tool. We actually can see what they are like producing. It, it's super, it's super rewarding to see when you like, you know, walk through the office like some-- you see at some screens and you see the dashboardy that you've never seen before, and you know that no one from data team helped business to build this dashboard, right? You see that like you impacted this person to actually go to our BI tool, figure it out, build the metrics and numbers that they want, and use it in their work, right?

So for me, impact is, is in here, right? I, I can see that I, I produced something that other people are using, and other people are benefiting from my work. So for me, this is the probably the biggest impact I could imagine, right? And, and now as you can see, it, it, it kind of, scales from as a junior, for me impact was where I just automated a report which freed me from like a couple hours a week, right? From to now, where I'm senior and I can impact com-like company level, employees that they go and they do the work on their own, and they change how they, how they work because of me. So this is where I see my impact. This is, this is a very...

You're right. This is, this is a very tricky question in my opinion because, obviously not everything is quantifiable. And I would say not everything should be quantifiable because, you know, if I, if I really want to, you know, I could go and say, "Oh, you know, since now we have, whatever, like two times more weekly active users on our BI tool, let's take the job of those people and see, how they affect on the business." Let's say they're like from sales team, and they close like whatever millions of, of deals. And this is like there is a chain, right? So we kinda can do this, but should we? I don't think that we should. Like, and not everything is quantifiable.

Like, obviously, sometimes it's easy when you say, "Oh, you know, I optimized our Snowflake bill and it's like two times less." Right? You give away-- Like, you give back money to your company, and this is very quantifiable. When you say, you know, twice more people are using our BI tool, it's less quantifiable. But, you know, when you tell this to C-level that, "Oh, you know, we are becoming actually data-driven company where people are using data to, to, you know, to provi- to, to, to kinda produce some decision," right? It's hardly quantifiable, but even on this level where I, I could say we are twen- t-two, twice many people are using our BI tool, for me is a great result even though it's not quantifiable in money, you know, currency, et cetera.

so I would say it, it really depends. A-and you shouldn't try to always, bring it to, to money. It's, it's not always make sense.

Right. well, obviously I'm, I'm still, I'm still posting on, on LinkedIn. I, I think that this is my platform because it's, it's not as short as, you know, Twitter or X. It's not as long as Substack, even though I have a Substack. but you know, I always have some ideas I want to share with the community, so that's why, you know, I, I'm posting a lot on LinkedIn. And yeah, my, my next thing is I'm, this is a bit of a teaser. I'm working on a new bootcamp. so I have this idea of full stack analytics engineers where I see that, you know, using AI ana-analytics engineers, they can actually do more like across the stack, right? And, I want to show this that it's actually possible with, with modern developments of tools and AI, and how to learn stuff.

So yeah, I, I, I'm planning to build... I actually, I'm building the new bootcamp that would teach how to be analytics engineer like in the modern world with AI, with modern tools, modern data stack. All right.

Yeah, thanks. Thanks for having me. Yeah, it was, it was a great conversation. All right.