One of the things I’ve been experimenting with is turning entire documentation websites into AI-powered knowledge bases.
By creating a pipeline that takes a documentation site and converts it into a structured knowledge base that can be used by tools such as ChatGPT, Google NotebookLM, Claude Projects, Microsoft Copilot, and similar AI systems.
Many modern platforms have excellent documentation, but finding the right answer can still be surprisingly slow.
Even when you know the answer exists, you often end up jumping between pages, searching multiple sections, reading forum posts, or opening support tickets for questions that are already documented.
At the same time, AI tools are becoming increasingly capable of answering questions from uploaded knowledge sources.
The challenge is that documentation sites don’t come packaged in a format that these systems can consume efficiently.
A typical documentation site might contain hundreds or thousands of pages. Some AI platforms impose limits on the number of files that can be uploaded, while others perform better when information is organised into logical topics rather than thousands of disconnected documents.
Building the Pipeline
To explore this, I built a small Python project that converts documentation websites into AI-ready knowledge bases. You can download from Github.
The process starts with a crawl of the documentation website using Screaming Frog. The pipeline uses two exports:
- URL data from the crawl
- The internal links report
The process looks like this:
Documentation URLs → Crawl export → Content extraction → Topic clustering → Knowledge base generation → AI assistant
The script extracts the content, cleans it, analyses the internal linking structure, groups related pages together, and generates a set of structured knowledge files.
Instead of uploading hundreds of individual pages, the output becomes a collection of topic-based files plus an index file that acts as a map of the documentation.
One of the more interesting challenges was deciding how to group pages together. Simply merging documentation pages in URL order creates a knowledge base that jumps from topic to topic with little context.
Instead, the pipeline uses internal linking and semantic similarity to keep related topics together. The goal is to preserve as much of the original information architecture as possible while reducing thousands of pages into a manageable set of files.
A Practical Test
To test the approach, I crawled the Schema.org documentation and used the generated knowledge base to build a Schema Helper assistant.
The files of the Helper can be found also in a forlder on Github.
The assistant can:
- Explain schema types
- Suggest relevant structured data
- Identify required and recommended properties
- Generate JSON-LD examples
- Explain relationships between entities
- Help troubleshoot implementation questions
The value comes from having the official documentation available through a conversational interface.
Instead of searching through multiple pages, the documentation becomes immediately accessible.
Questions that might normally require searching across multiple sections can often be answered in seconds.
For example, I have shared the product Magnum White Chocolate Ice Cream On Tesco and ask for improvements and opportunities.
It’s Not Really About GPTs
Although I used ChatGPT to build the Schema Helper, the interesting part of the project isn’t tied to a specific AI platform.
The output of the pipeline is a structured knowledge base.
That same knowledge base could be used in:
- ChatGPT custom GPTs
- Google NotebookLM
- Claude Projects
- Microsoft Copilot
- Internal enterprise AI assistants
- Future retrieval-based AI systems
The AI interface is simply the final layer.
The real challenge is transforming a documentation website into something that an AI system can navigate efficiently.
Documentation sites are built for humans and knowledge bases for AI often benefit from a different structure.
Pages need to be grouped logically, related topics should stay together, and the system needs a way to understand where information lives.
That’s why most of the effort in this project went into the pipeline itself rather than the assistant.
The Schema Helper is really just a demonstration of what becomes possible once the documentation has been transformed into a structured knowledge system.
Beyond Single Products
What I find more interesting is where this could go next.
Most software vendors only document their own products. Shopify documents Shopify. Contentful documents Contentful and so on..
Each documentation set explains its own world extremely well. But real implementations rarely operate in isolation. A content platform connected to an ecommerce platform. An ecommerce platform connected to a search engine.
Many implementation questions live in the space between those systems and not inside either product individually.
When something breaks, support teams often understand their own product very well, but they may not understand the complete interaction between multiple systems.
The answer may exist across several documentation sources rather than inside a single one.
By creating knowledge bases from multiple documentation sets, it becomes possible to explore those interactions from a single place because the documentation itself has been brought together.
Imagine combining documentation from:
- Shopify AND Contentful AND Algolia
Or:
- Shopify AND Google Merchant Center AND Google Analytics
Or any other combination of platforms that regularly exchange data.
The assistant can reason across documentation silos that were never intended to be consumed together.
That opens up interesting possibilities for troubleshooting, implementation planning, integration design, and discovering capabilities that may not be obvious when reading each documentation set separately.
Thinking Outside the Box
One thing this project reinforced is that useful AI applications don’t always require new models.
Sometimes they require a different way of organising existing knowledge.
The challenge was turning hundreds of disconnected pages into something that could be searched, navigated, and reasoned about more efficiently.
The Schema Helper was simply the first proof of concept.
The more interesting idea is a repeatable process that can transform documentation websites into structured knowledge systems, ready to be used by whichever AI platform makes sense for the task.
