How Sequentum Scrapes the Web at Scale — Sarah McKenna

Thanks, Tony. That was a great intro, very flattering. I ended up in automation. When I first joined it.com, I took the job that nobody wanted, the thankless job where you're crossing T's and dotting I's. I was doing very large-scale automation in the context of software test automation, quality assurance. Automating the stand-up, what became the field of DevOps, automating, release engineering, what became the field of release engineering following agile practices, sometimes defining those practices. At some point, I had so much experience with all different kinds of technologies that I was recruited into scraping, which is really just the flip side of the coin from test automation.

That's when I really realized, wow, this is an enormous field. Instead of being someone in the back office managing the most thankless area, I could go into the front office and go into business. I've always wanted to go into business. My original job was actually at hedge fund, and I left there to join dot.com thinking I was going to be an entrepreneur. When I went full circle and ended up in scraping, I realized the all data world and how decision intelligence was driven by data and by web data, and how all of a sudden I had this tremendous expertise to influence an entire industry and an entire movement in the tech world. Not just at the time, it wasn't all about driving AI, it was about driving decision intelligence.

But now that we have AI agents, it's really all about driving this new agenda world. It's a very exciting place to be. Thank you.

I mean, I think way back in the beginning of my career, we used to advocate for agile practices and test driven development and all of these validations and all of this automation to minimize risk and to eliminate all of these firestorms that these teams were living through, which were so common back in the day. And when I came into data, I was amazed to realize that this sort of shift left philosophy and approach that we always took was something that was just a very big rigor required upfront. And there was an entire industry of data engineers and data professionals who were equally obsessed with all of those best practices and all of those principles that we held so close.

So it was just really, really great to come into a field where what we always advocated for and what we tried so hard to push, and what we've seen is the annoying quality assurance people, right? It was something that was front and center, mission critical. And now we've come full circle where trust, transparency and auditability and all of those best practices, even if IT security and governance, those are all front and center and driving everyone's decisions in procurement and in business. It's really, really exciting.

It's very hard to drive trust, right? If you're thinking about decision intelligence, and basically that's what we're all after now, right? We want these AI agents to be able to make choices. We want to enable them to make decisions that improve our lives, that drive our productivity. But we want to do it in a way that we can trust. And that's sort of the basis of it all, right? And in order to trust it, it has to have guardrails and it has to have acceptance criteria, and it has to have all of these best practices that we built sort of principle by principle over the years are what is most important today. I mean, it's really, you know, people, it's kind of, you know, it's not the most exciting, you know, conversation in the world to talk about these sort of compliance and these kinds of things.

But really trust is if you can trust that agent to go and get the best price on that ticket to the destination that you really care about when it finds that price at three in the morning and to actually grab that airplane ticket and that amazing beach spot for you, you know, inside your budget and to trust that it's going to be able to do the right thing, like that's huge. And that's what we're all, I think that's what we're all after. I think that's what we're hoping. We're going to get out of this.

Such a good analogy, Juan. Like, yes, exactly. Right. We were so excited about cars. Everyone wanted one. You know, it really got us to places that we'd never been able to get to before. You know, but then, of course, there were all of the risks and the dangers. Same with self-driving cars, you know, we, you know, there's, there's a lot of a lot of room for improvement there. There's also pretty exciting developments. But, you know, are we really going to get there? And that's the thing about AI also and AI adoption is we do have a problem in our industry with trust. And it's not just that there are known ways that these things are falling down and, you know, real problems with data quality or the way that the data is sourced or missing guardrails or whatever it is.

People do fear the changes coming from AI. And so the trust is again just front and center. It's so important.

I mean, I think you need to see the big picture, like the business is the most important priority, right? But then you also need to be able to sort of bring to bear all of those details so that you can pull together a complete risk profile and make sure that you're doing right by the business by permitting and fueling and speeding and greasing the wheels of, you know, exactly what that business is trying to do. And that's where, you know, the real governance comes to play. Like you want to make sure the business is fully enabled to operate safely and sustainably in a way that's not going to bring risk and liability. So it's the big picture and the ability to sort of bring all of those small details together and to bear.

So yeah, if anybody listening to this hears that, I think governance experts are the true unicorns in the data world. I actually think that it's taking over everything, right? We have AI coding for us, right? So that's sort of AI is most successful in the coding space, right? It's text and you have developers and people who understand, you know, how these things fall down and they have all these governance things already. We already built, you know, the SDLC roles and code reviews and how to deal with DevOps and builds and, you know, all of these security reviews. We already have all that stuff in place, so it's pretty well defined. So AI adoption for coding is pretty broad. So what's the problem with AI coding?

The problem is that it's just producing too much code too quickly and humans can't review it. And, you know, I was just having a conversation with someone yesterday. Again, it feels like this ability to see the big picture and all the details is really, really critical at this time because the way that you review a tremendous amount of code produced by AI is by making sure that it hears to all those best practices, produces all the documentation, produces all the unit tests, passes all the unit tests, integrates with the testing frameworks and the governance frameworks, OASP rule, you know, etc. All the kinds of things that we used to manually write test plans for, you know, the AI tools can create those frameworks and create those requirements and create that sort of governance that then gets automatically generated with the code.

And that just speeds the human comprehension and review and approval. And so again, it's like the AI, I think, is taking a lot of the brute force manual mind numbing work that is not really human work, human friendly, putting into the hands of the machines. But then, again, under reinforcing the human in the loop role of ultimately deciding if this makes sense or not, because the AI obviously, it's not really there in terms of common sense, it makes pretty big errors, which are surprising sometimes.

It's the same theme everywhere I look. It just feels like the convergence of everything, where governance and automation and all those best practices are all coming together. I mean, if you look at something like co-work and how co-work is structured, it's literally piggybacking off all the best practices of automation, large scale automation, having all of those atomic pieces of that automated universe, all sort of defined living in context files. And then you have all different empty files for different aspects of ultimately what's governance and creating those guardrails and shape and enabling that productivity and taking it from, when it's raw, it's like 70% correct.

But when you start to add in all of this governance and all of the structure and all these guardrails, you can really get up to incredible accuracy, which is, you know, and it's also self reinforcing because it's also going, those structures are also feeding back into the training of the AI, and so that it's more accurate over time as well. So it, to me, I mean, I guess it's almost like, you know, the paradox of, you know, perspective is what you see everywhere. So I think that's probably what's going on too. I see it all as one giant governance framework.

So one thing I will say about, like when you say the hubris of people who are so excited about all this automation, back in the day, we called that fragile. Instead of agile practices, they were fragile practices. People got really gung-ho and excited, and they thought that this automation was great, and they were actually missing all of the, you know, the main problems that the automation was supposed to surface, and it ended up being very fragile. And so if you didn't do it the right way, you would end up with tremendous lapses, which is how I, you know, when I look at AI at 70% correct, I'm like, well, they haven't, they haven't really put in all the structures that they need to make sure that it doesn't do that.

And so what we do at Sequentum to try to implement the best practices and make sure that we're not going off the rails in our automation is, first of all, we have a platform, and it has all of the features built into the platform that are sort of reusable commands. That's one big best practice in in automate large scale automation is to make sure that all your commands are sort of atomically maintained, so that if this little thing changes over here, and you need to edit it, you don't have to edit thousands of agents, you only edit it in one place, and everything else benefits from that change. So for example, we deal with a lot of bot blocking and in web scraping.

And that's just a matter of, you know, anonymizing a lot of objects in the browser is what it's come down to really. And so, you know, if we're blocked, maybe there's a new, a new variable that somebody is some bot blocking company is introduced, and we need to introduce in our end, but we'll put it in one place, and in the agents themselves, you can use AI now to decide how to do that on block, or you can manually configure and explicitly say, I'm willing to do this, I'm not willing to do that, you know, etc. But these are sort of built in features and atomic controls, and we have that for everything loading a URL. If the internet changes. If the itf comes out with new HTTP standards or whatever it is, we will maintain that command in one place, and all agents will benefit from that. So that's a huge sort of, you know, best practice that we have this reusable platform, and then we have a low code interface. And that allows you to, you know, point and click and very

quickly and visually, write your agents and review them, so that any stakeholder even the non technical ones can see exactly what's happening if this, if this workflow is driving decision intelligence for business, it's driving that business decision making. There's a lot of stakeholders that need to trust that and building that transparency is a huge best practice in and of itself, allowing people who are not technical to go in and see exactly how something changed, who changed it when and how really goes a long way to drive trust. So we built that in as well. A third best practice is we have custom command templates, we have command templates and we have agent templates. And what this does is if there are any sort of routines that are specific and custom to a business, maybe not a command like loading a URL.

But maybe it's something about how you map hotel data internally, right, and it or how you write out your data internally. Right, those are specific to your organization, or maybe even your team, you can write those in command templates that are versioned independently. So, we have commands that are reusable that are built into the platform, but you can write your own custom command templates that you can also bring to bear, and their versions separately. And similarly with agent templates as well if you have a specific template for collecting job data for example it has specific schema specific way of mapping and classifying. You can write an agent template for that, and then you can use that for all of your job, job data collection agents. And there's just so many best practices that we've built into our platform, obviously, logging and troubleshooting has to be extremely easy.

You have to make it so easy to understand that something is not working the way that you expect. So we have built in acceptance criteria we have all of these automated checks at the level of every field of every row of data that comes across you can configure, you know, just innumerable numbers of rules and validations. But to cut down on the noise of having too many to strict rules, we also have sort of a, you know, overall, sort of tolerance thresholds that you can set in terms of the number of errors that you would expect, you know, how many rows of data, compare it to most more recent runs, you know, am I getting similar, you know, volumes of data, am I getting similar numbers of, you know, Honda's or like whatever the thing is that you're collecting. You have this tolerance threshold so it's not too noisy.

And then the last thing is you need to really make it easy for people to not only take ownership of problems as they arrive, but to know the moment something has gone wrong. So you have this built in validation and tolerance threshold so it's not too noisy. But then you have alerts and notifications that are meaningful. Right. So the right person gets notified in the right timeframe that something has changed. We work in web data, we work with web data, right. websites change, Cloudflare takes the internet down. There's all kinds of things that go on. Right. If this data is driving your mission critical business decision making you better know immediately when it's not coming across as expected.

So we have all the alerts and notifications built in, where you can communicate exactly what's not meeting acceptance criteria. And then, every little change, because, because we're in web data and the mission and the data is so critical. What we do instead of putting a lot of restrictions on the ability of agent developers to fix something. We do the opposite. We make it extremely easy. So we're track all the versions of every agent that's in production. Right. And anyone can go and see what's where and what changes have been done. And the moment something needs to be adjusted. The right person is notified and that person can go in, download the latest point click fix verify against all the existing rules that are in there, and they can deploy.

And the check ins and tracking is under that named human who approved that change to go live. Right. Humans make mistakes. Okay, they're not infallible. But if you're talking about a process that requires these just in time changes and fixes in production. And you have the audit trails and governance built in to show who did what where when that goes a long way towards trust. So we've built all that in.

I mean, this is again back to sort of agile best practices and you know you want to make sure that somewhere you have, you know, how it is that the system should behave. You know cloud cloud I think has distinguished itself with its, what is it 800 page constitution or something. Maybe, maybe your code base doesn't need an 800 page constitution but you should have some documentation of how you expect it to behave. And then you should have some testing and validation that makes sure that it's actually meeting those, you know, that those expectations or acceptance criteria. Right. I mean you kind of need it. There's work involved in defining all of that and building the guardrails, but it pays back in spades, because you don't have all those fires.

It's much easier to introduce a change and an enhancement improvement, you stop focusing on fixing things that broke. Right things that you already built value you already add it, you stop focusing on that because you automate the, the processes to make sure that that's still there with each new release and you just focus on the new functionality speeds everything up exponentially. So it's worth the investment.

But that's comes down to sort of the basics of requirements right we've always had that problem. Like the, you know, the user says I wanted to do this and the developer understand something completely different. And the reality is that as they're implementing it there are all these micro choices that the developer is making. And maybe, you know, they're making some assumptions about how this, this feature needs to work. Right, that's always been a problem with software. Right, but with with with these new tools. You can really put a lot of information in terms of how the business user expects it to work and behave, you can really put that in writing, and you can just have the tools develop a lot of that stuff for you, which is a huge luxury, like it's obviously a token cost.

But again, I think it's, it pays back in spades. And also with security, by the way, the security security, the IT security testing that these tools give you is just phenomenal. It's fantastic. And boy that is really thankless work for humans to do manually.

So we, we use a lot of different AI tools, and we switch back and forth depending on who's, you know, ahead in the race lately it's been, it's, it's been a lot of cloud, but not exclusively. But it doesn't really matter and I don't want to talk about the AI tools, I think just so when your license. I mean, when you have AI coding tools doing code reviews from a security perspective, for example, they catch so many things sort of instantly that, you know, a developer is focused on the positive case. My user said that they, my business user said they want a B and C, and they have checked in something that achieves that. But they didn't think about the thousand different security use cases that, you know, could end up being a disaster in production, AI can do that so easily.

And it's just great, and they'll go in and it'll write all your unit tests and they'll document everything and, you know, tell you, this is why I'm doing what, and it's our developers, when they first upgraded to enterprise GitHub that has this code review capability. And, you know, they immediately had me knocking on the door, because we went from like, I don't know, $20 a month or something to like $900 a month. I was like, what's this, what are we doing, what have we done. And they just, you know, they know me, they walked me through what these tools are doing from catching all of these, you know, potential security, you know, issues. And, you know, I was sold immediately. I mean, it's such a low cost to protect against, you know, those kinds of issues and we're SOC 2 type two and you know we're kind of obsessed with security as well as, you know, so we take this stuff very seriously.

This is we're producing data that's driving major, major decisions for regulated industries so it's really, really important to us. But yeah, I mean, use the tools, invest, spend the tokens, get the licenses.

I mean, I are head of AI and engineering when I when we had that original GitHub discussion, you know, what he said was, you know, what you're what you're spending it is like having another engineer on the team that's going and doing all of this, but the cost is of that engineer is so low compared to the risks. That it's, you know, totally worth it, but there does have to be budget there for sure. We're spending more on AI than I think we ever thought we would. We're really, really appreciating it.

That's right. It's a very open debate and I'm very involved. So I, you know, as you mentioned, I, you know, worked on the web data collection considerations for the all data council. And that was in the finance industry. And then I went and joined the Alliance for Responsible Data Collection teaming with Bright Data and a lot of other companies like Common Crawl and a lot of the sort of public interest groups and a lot of the big, big tech companies. And what we're doing is we're trying to make sure that the parties who normally don't raise their hands, the scrapers, are actually represented in the conversation because of seasoned assists and, you know, the types of lawsuits that happen in our field.

Most people don't raise their hand to say, you know, we're scrapers and we're doing this for very valid reasons, like background check companies. That's a scraping use case. Fighting human trafficking is a scraping use case. You know, anything to do with, you know, investment decision making or, you know, price algorithms or yield, you know, optimization algorithms. These are all scraping use cases in every industry is is engaged in its web data powered, you know, mission critical decision making. And so I made a decision when they launched the Web Butthoff working group, and it was launched by a bunch of people from Cloudflare, you know, their mission is to make the internet a better place.

And their customers, they represent about 20% of websites on the internet, their customers are having some major issues right after chat GPT came out, it became very easy to have these dumb crawlers go to websites and collect all their data, and they are incentivized to do this because there's tremendous value created from training AI with this data. But the problem is they're really dumb crawlers. So if they go to a website that's getting hammered by a lot of these AI crawlers, and it's giving 503 server errors, right, they're just going to be like, Come on, give me the page, give me the page, and the site's never going to be able to come up again.

It's like a denial of service attack, right? Companies like sequentum, like we would never ever do that. We would, you know, we would notice immediately the sites having a problem we would face back we have all of these throttles and rate limits and all these things in place to be respectful to the websites. And, you know, but these crawlers don't. And, and so what's happened so several things have happened for publishers of data, which we depend on these publishers and so we don't want anything to happen to them. We don't want anything to happen to the open internet. We're sort of two sides of the same coin. But what's happened is that economy that used to feed them, the ad economy has broken down because it's very clear that companies like Cloudflare who are bot blocking companies, they can't block bots.

Right? We've always known that they can't block bots because we're a sophisticated platform. But once the AI crawlers came out, the publishers realized there's no way that all of these eyeballs are human. Cloudflare, your bot blocking doesn't work. Right? So there, it used to be that a website would just pay for Cloudflare, and then they could sell ads, because they could say to the publishers to the sorry the advertisers, look, we have Cloudflare. All these eyeballs that are looking at your ads are human. And after a while the advertisers were like, no, they're not. This is not human, and we're not paying for these ads. So there's sort of a crisis. Right? Because the open that the internet sites need some sort of business model.

And this is why this, this, this conversation is so important. We encourage anyone who has a voice at stake in this in this field to get involved, because these sites deserve to have some economy, right, they're getting especially these public interest sites, they're, they're getting absolutely slammed. They have to pay the cost of serving all of this traffic. Right? And then the the AI firms that are profiting maybe off of consuming all this data are not paying anything. And so, you know, but then again, all the traditional industries that are driving decision intelligence, they need to keep going. Right? You can't not have background checks. You can't not have competitive intelligence. Right?

You can't wish these things away. They're not going away. And so we're having a big discussion. So in, so there's a global standards body called the Internet Engineering Task Force. And they have a working group called Web Bot Off. And a lot of the conversations are happening in there. And what we're, you know, we have a lot of the publishers in there. We have a lot of we have all the big tech companies. There's a lot of lawyers too. It's not just, it's not just site owners or or or engineers. But we, we have the EU AI act pointing at some of this ITF work. There's sort of a number of working groups. There's AI preps which which is trying to define the vocabulary of how you might define the assets on your site.

And then Web Bot Off is supposed to cryptographically self identify the agent and would feed into sort of some concept of toll booths on the internet, where you could actually pay for this data. And then there's some other things like privacy paths fits into this framework as well so that researchers or human rights workers, you know, investigative journalists could fall outside of it without being blocked and without being required to pay. And, you know, and there's, there's, there's, I can't get into all of it but there's a lot of really really good discussion going on. And the interesting thing from Squint of point of view is we believe that a large number, unlike the public interest groups who don't have budget, our customers would love, I think most of them would just love to pay to pay to scrape the data that they need to drive their decision making.

They would love to be legitimized and not have to worry about potentially being sued by some website for collecting their data on a daily basis. And so we've actually submitted, we've actually integrated a digital wallet into our platform. And we're sort of unusual that way, because we're a web automation platform, and we do web scraping. But we ended up getting pulled into this agenda commerce world because we're not blocked and our web automation works really well it's 18 years mature, and we have all this trust and governance and audit ability built in. And so we were pulled in by the KY pay sort of. I don't know if you want to call it a cabal there's all these competing protocols for agenda commerce, but we integrated with them and we are the hands of the checkout, which is kind of an interesting place to be because that's actually a much bigger market than web scraping.

So it's exciting. But yeah, so there's a lot of conversation going on there so we're already cryptographically self identifying our agents for agenda commerce, but we're still, you know, sort of bypassing all the bot blocking controls. Even with those agenda checkouts because the sites don't know how to turn it off for some and not for others. They don't have that capability and probably won't for many many years. And then of course, you know, for for other data collection like background check companies like they have to fight bot blocking in order to get data that the government is mandated to provide. You know, so we're, we're engaged in that as well, sort of defeating the bot blocking and from our sophisticated platform in order to facilitate these regulated industries and and and good practices.

So it's a very interesting place to be.

The thing you have to understand is just like entities that are scraping get cease and desist platforms and publishers that facilitate scraping are also getting cease and desist. And so far we don't have any websites that are willing to allow us to pay them to scrape. So we've got the capability, but we don't have any publishers who are willing. It's very interesting. They want to sell their data directly, but they can they only have so many humans to negotiate. You know, like Reddit, for example, think about Reddit, like, they can negotiate a deal with open AI. Sure. But what about the long tail of everyone on the internet that wants, you know, some little tiny slice of data to feed into whatever it is that they're building.

They have no way of doing that. So we're ready. If if that starts to emerge, we're ready. And in the meantime, we're immediately seeing this upsurge in consumption for the agenda checkouts, which we weren't expecting. It's not something that we were thinking of or or planning for.

That's a really funny situation to to think about it. Absolutely fascinating. I mean, literally, everything is changing. AI has disrupted everything, turned everything on its head. And we are in this whole new world and finding our way. And obviously, you know, you need to adapt quickly because everything is changing. And so we're adapting to, you know, we are we are an 18 year mature platform and we've integrated so that we have, you know, AI agent generation. Now you can use natural language to generate an agent, but that agent is still deterministic, right? Our customers who are regulated, they do not want AI at runtime deciding what the price is and what the tax is and whether or not this person, you know, has a felony record.

No, they don't want that. They want a deterministic agent that does the exact same thing every time. It's completely defensible and auditable. So we are using the AI to drive productivity. We're using the AI to speed development of agents, to automatically configure the unblocker. You know, we have a visual editor that makes it very easy to see what's in the agent, exactly how it's running and also to point and click and customize it. So it's very, very easy to customize. You don't have to go back and forth to the prompt. And then we have a magic wand in the visual editor. If there's another page that it didn't add and you actually want that page, you can go navigate there using these reusable commands and then the magic wand to generate that section of your workflow.

So we're doing a lot with AI and we're adapting a lot. Obviously we have a wallet now. That's a big shift for us. But yeah, I mean, I think it's just a reflection of customer demand and what's happening in the market.

Yeah, I mean, so we have this advantage in that we have an 18 year mature platform and the agents themselves are really just a text file that runs on the platform. So really we have all of the ability to define what an agent does in our platform. So what we're able to do is to give it instructions to generate an agent that basically to generate it's not really the code for the agent. It's really just the instructions to our platform for exactly what the agent needs to do. So that's why it's deterministic. I mean, we can include through custom commands, AI decision points. But most of our customers don't do that. They're really just using the AI to speed the development of agents.

And they just want it completely deterministic at runtime. It's also so much cheaper like an AI decision process in the middle of your web scraper like it's very slow to read the page. It's very slow to comprehend. You know, again, it's 70% wrong. I'm sorry, 70% right. It's 30 to 30% wrong. You know, so it's expensive, it's slow. It hallucinates, right, you just don't want it in at your at runtime. So we haven't included it in runtime. And so far, we do have a lot of competitors that use AI for self healing right when a website changes and you have to adjust. And we're not doing that. So far, none of our customers have said they absolutely want it.

Because again, they have trust problems. But I think we may end up adding in, like a feature to automatically generate a new version of the agent and store the store the data collected off to the side. Just in anticipation that a human will come in and approve that self healed edit. I think we may do something like that. Instead of directly self healing and just having the agent doing whatever the AI told it to adjust to. But yeah, it's a very interesting, very interesting thing and and oh and the deterministic agents are like fractions of a penny to run versus dollars of using AI.

Yeah, healthcare is really, really tough. We have customers in healthcare to they use our software on prem. I don't envy them. It's really tough. But that's your that's your world, right.

Well, hopefully, you know, these new AI tools make the governance easier because that's the thing like they're so good at pulling together all these details as long as you've got some human who's making sure it's not doing things that are too, you know, overbearing or, you know, really hampering the business or, you know, just, you've got to have that human, you know, sophistication. And that's a really, really, really complicated sort of common sense. You know, I don't think they tools really have that yet, but they've got tremendous ability to pull together all those details in a way that is just incredibly thankless for human to do manually.

We joke about the hotch hotch watcher, you know, the Dr. Seuss book where they talk about the hotch hotch watcher. It's a big long rhyme and it's basically you know, so much more than you think about it. And then it's like there's someone who's watching the watcher and there's this hilarious Dr. Seuss thing where there's like basically all these people like standing there like Oh, but yeah, I mean, it's hopefully we can, we can, you know, use the tools to help us define all those guardrails and align on what matters. And then, you know, limit you know, if there have to be all those watchers like let the AI be the watchers but let them all be discreetly defined and in a way that the human can kind of oversee.

Yeah, well I think you can do that with guardrails right I mean that's like the idea of the agents checking code review and checking against all of the security best practices. That's, you know, it's just so brilliant that was always so hard.

They're biased and thinking that. But, you know, who knows a lot of good stuff going on out there.

Well, we're at sequentive.com you can write to sales at sequentive.com you can contact me Sarah dot McKenna at sequentive.com you can reach out to me on LinkedIn. And, you know, we're quite active actually we have a newsletter there and we post almost daily. You know what we're up to and and what our thoughts are and what's happening. Happy to engage DMA.