chatbots. What you give is a description of your data, some particular tools for using that data, and some example prompts. And then, armed with that information, and the end user’s question, the chatbot can operate with your content much more easily.”
catalogue, or fetching content from ser- vices like
Legislation.gov.uk. Alongside this inhuman rate of requests the AI can also improvise workarounds to any obstacles. “I’ve had this experience myself,” says John, “and it’s kind of remarkable to see. I was asking Claude to answer a question and it couldn’t get the answer from The National Archives through search results. But it still thought that the resources it needed were there on the website. I found it making me not just one, but two crawl- ers. One using something called React in the browser and then a Python script. It said it could see there was information there – 2,000 relevant PDF documents that are not accessible through search – adding ‘I’ve written you a crawler, now run this code’.”
He said: “So now you’ve got bots that are just trying to grab all your content to train a model, and you’ve got bots that are grabbing specific content to meet an individual’s needs. But if the individual, aided by AI, is also deploying crawlers there isn’t any way to distinguish them.
John Sheridan.
This is a really big problem. The internet protocols we’ve got are not very good at helping publishers manage this, and we’re seeing massive numbers of bots across all of our public-facing digital services.”
Lowering defences
One solution John likes is letting the AI models into the system: “We’ve been experimenting and there are approaches that we much prefer. One of them is called MCP, the model context protocol, originally developed by Anthropic but it’s now being supported by all of the big AI systems. We like it because it allows you to directly connect an AI chatbot with a known source in a managed way, and you get much better results.”
“It’s a way for organisations like us, who have a lot of rich content, to make it available in a way that is better suited for consumption by large language model
John said: “It was while comparing Claude without the MCP connected against Claude with the MCP that I saw Claude busily writing me crawlers. With- out the MCP it made the decision to use other means to take information that was not accessible through search. Whereas with the MCP server, it found the right tool to get the information directly. And the difference was that with the MCP Claude answered the question correctly in a few seconds. Without the MCP it took minutes, wrote a crawler, and it still got the wrong answer.
“We like it because it’s moving to a world that’s a bit less ‘Wild West’. It is properly engineered, more controlled, and because we can see a big improvement in the quality of the result – although that’s largely anecdotal. We’ve not done a thor- ough evaluation.”
Orders of magnitude
Another reason for exploring different solutions is that the problem will keep changing. Even if the change appears to only be volume. John said: “We can see there’s been this growth and we anticipate that this demand for high quality content will grow and grow and grow. “A good rule of thumb is the order of magnitude rule. When things change by an order of magnitude, they become a dif- ferent thing. And here we’re looking not at just one order of magnitude in change in demand and consumption, we’re looking at several successively.”
He says an order of magnitude change of a hundred or a thousand, saying: “I don’t know over what time period. Whether it’s over three years or five years, but with this model of AI we’re on a fast-accel- erating shift in the demand for content. That’s for this model of AI. It is very content intensive, both in training, and then fetching content to answer human requests. Anthropic’s Claude, in research mode, will mint hundreds of requests in a few seconds. The computation involved is enormous. That’s why you see conver- sations about building data centres and concerns about environmental impact. It is all driven by these projections of growth in AI, which means massive demands for data. This is not a temporary problem. It is a fundamental change. We expect this growth in demand for content to continue for as far ahead as we can see.
Cost problem The National Archives. August-September 2026
Orders of magnitude changes in numbers of bots means creates new costs. “If it’s
INFORMATION PROFESSIONAL 25
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56
