Related MCP Server Resources

Explore more AI models, providers, and integration options:

  • Explore AI Models
  • Explore AI Providers
  • Explore MCP Servers
  • LangDB Pricing
  • Documentation
  • AI Industry Blog
  • Powertools MCP Search Server
  • Python Codebase Analysis RAG System
  • MkDocs MCP Search Server
  • Perplexity AI MCP Server
  • MCP Pokemon Server
Back to MCP Servers
docmcp

docmcp

Public
visheshd/docmcp

Index and semantically search documentation using AI-generated embeddings with PostgreSQL and pgvector, enabling seamless integration with AI IDEs via Model Context Protocol for efficient crawling, processing, and querying of up-to-date docs.

typescript
0 tools
May 29, 2025
Updated Jun 4, 2025

Supercharge Your AI with docmcp

MCP Server

Unlock the full potential of docmcp through LangDB's AI Gateway. Get enterprise-grade security, analytics, and seamless integration with zero configuration.

Unified API Access
Complete Tracing
Instant Setup
Get Started Now

Free tier available • No credit card required

Instant Setup
99.9% Uptime
10,000+Monthly Requests

DocMCP: Index the latest doc for LLMs on PostgreSQL using pgvector and expose to AI IDEs

A system for crawling, processing, and querying documentation with AI-powered embedding generation and semantic search capabilities.

Features

  • Documentation Crawling: Automatically crawl documentation sites with customizable depth and rate limiting
  • Content Processing: Convert HTML to clean Markdown with metadata extraction
  • Vector Embeddings: Generate embeddings using AWS Bedrock for semantic searching
  • Job Management: Track and manage document processing jobs with detailed progress reporting
  • MCP Integration: Built-in MCP tools for AI agent integration

Roadmap

  • SPA support: currently the crawler doesn't support SPAs
  • Caching: crawled urls are directly added to the DB

Getting Started (Development Setup)

Prerequisites

  • Docker (Install Guide)
  • Docker Compose (Install Guide)
  • Node.js 16+
  • Git
  • AWS Account with Bedrock access
  • AWS CLI configured with appropriate credentials

Quick Start Steps

  1. Clone the Repository:

    git clone https://github.com/visheshd/docmcp.git cd docmcp
  2. Configure Environment:

    • Copy the example environment file:
      cp .env.example .env
    • Edit the .env file:
      • Set DATABASE_URL to postgresql://postgres:postgres@localhost:5433/docmcp
      • Configure AWS Bedrock:
        • Set AWS_REGION to your AWS region (e.g., us-east-1)
        • Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your AWS credentials
        • Or ensure your AWS CLI is configured with appropriate credentials
      • Adjust other settings like LOG_LEVEL if needed
  3. Start the Development Environment:

    # Make the script executable chmod +x dev-start.sh # Start the development environment ./dev-start.sh

    This script will:

    • Start PostgreSQL with pgvector in a Docker container
    • Install project dependencies
    • Run database migrations
    • Import seed data automatically
    • The database will be accessible on port 5433
  4. Add Documentation: Use the add-docs script to crawl and process documentation:

    # Basic usage npm run add-docs -- --url https://example.com/docs --max-depth 3 # With additional options npm run add-docs -- \ --url https://example.com/docs \ --max-depth 3 \ --tags react,frontend \ --package react \ --version 18.0.0 \ --wait

    Available options:

    • --url: Documentation URL to crawl (required)
    • --max-depth: Maximum crawl depth (default: 3)
    • --tags: Comma-separated tags for categorization
    • --package: Package name this documentation is for
    • --version: Package version (defaults to "latest")
    • --wait: Wait for processing to complete
    • --verbose: Enable detailed logging
    • See npm run add-docs -- --help for all options
  5. Query Documentation: Once documentation is added, you can query it using the MCP tools. See the "Querying Documentation" section below.

  6. Stop the Development Environment:

    docker-compose -f docker-compose.dev.yml down

This setup provides a lightweight development environment with just the required PostgreSQL database and pre-loaded seed data. For production deployments or if you prefer a fully containerized setup, see the "Production Docker Setup" section below.

Cursor Setup

To use DocMCP with Cursor IDE, you'll need to configure the MCP transport. Add the following configuration to your Cursor settings:

{ "docmcp-local-stdio": { "transport": "stdio", "command": "node", "args": [ "/dist/stdio-server.js" ], "clientInfo": { "name": "cursor-client", "version": "1.0.0" } } }

Replace `` with the absolute path to your DocMCP installation directory.

For example, if DocMCP is installed in /home/user/projects/docmcp, your configuration would be:

"args": ["/home/user/projects/docmcp/dist/stdio-server.js"]

After adding this configuration, restart Cursor for the changes to take effect.

Architecture

The system consists of several core services:

  • CrawlerService: Handles documentation site crawling with robots.txt support
  • DocumentProcessorService: Processes documents (HTML→Markdown, chunking, embedding)
  • JobService: Manages asynchronous processing jobs with detailed status tracking
  • ChunkService: Stores and retrieves document chunks with vector search capabilities
  • MCP Tools: Agent-friendly interface for adding and querying documentation

Document Processing Pipeline

The DocMCP system processes documentation through the following pipeline:

  1. Documentation Input

    • User provides a URL through the add_documentation MCP tool
    • System creates a Job record with "pending" status
    • Job is assigned tags for categorization and future filtering
  2. Web Crawling (CrawlerService)

    • Crawler respects robots.txt restrictions
    • Follows links up to specified maximum depth
    • Captures HTML content and metadata
    • Creates Document records linked to the parent Job
  3. Document Processing (DocumentProcessorService)

    • Cleans HTML and converts to structured Markdown
    • Extracts metadata (package info, version, document type)
    • Establishes parent-child relationships between documents
    • Updates Job progress as processing continues
  4. Chunking & Embedding (ChunkService)

    • Splits documents into semantic chunks for better retrieval
    • Generates vector embeddings using AWS Bedrock
    • Stores embeddings in PostgreSQL with pgvector extension
    • Preserves chunk metadata and document references
  5. Job Finalization (JobService)

    • Updates Job status to "completed"
    • Calculates and stores document statistics
    • Makes documents available for querying
  6. Querying & Retrieval

    • User sends query through query_documentation MCP tool
    • System converts query to vector embedding
    • Performs similarity search to find relevant chunks
    • Returns formatted results with source information
    • Supports filtering by tags, status, and metadata

This pipeline enables efficient storage, processing, and retrieval of documentation with semantic understanding capabilities. All steps are tracked through the job system, allowing detailed progress monitoring and error handling.

Project Structure

docmcp/
├── prisma/                  # Database schema and migrations
│   └── schema.prisma        # Prisma model definitions and database configuration
├── src/
│   ├── config/              # Application configuration
│   │   └── database.ts      # Database connection setup
│   ├── generated/           # Generated code (Prisma client)
│   ├── services/            # Core service modules
│   │   ├── crawler.service.ts     # Website crawling functionality
│   │   ├── document.service.ts    # Document management
│   │   ├── document-processor.service.ts # Document processing and transformation
│   │   ├── job.service.ts         # Async job management
│   │   ├── chunk.service.ts       # Document chunking and vector operations
│   │   └── mcp-tools/       # MCP integration tools
│   │       ├── add-documentation.tool.ts    # Tool for adding new documentation
│   │       ├── get-job-status.tool.ts       # Tool for checking job status
│   │       ├── list-documentation.tool.ts   # Tool for listing available documentation
│   │       ├── query-documentation.tool.ts  # Tool for querying documentation
│   │       ├── sample.tool.ts               # Example tool implementation
│   │       └── index.ts                     # Tool registry and exports
│   ├── types/               # TypeScript type definitions
│   │   └── mcp.ts           # MCP tool interface definitions
│   ├── utils/               # Utility functions
│   │   ├── logger.ts        # Logging utilities
│   │   └── prisma-filters.ts # Reusable Prisma filtering patterns
│   └── __tests__/           # Test files
│       └── utils/           # Test utilities
│           └── testDb.ts    # Test database setup and teardown
├── .env                     # Environment variables
└── package.json             # Project dependencies and scripts
Publicly Shared Threads0

Discover shared experiences

Shared threads will appear here, showcasing real-world applications and insights from the community. Check back soon for updates!

Share your threads to help others
Related MCPs5
  • Powertools MCP Search Server
    Powertools MCP Search Server

    Model Context Protocol server enabling efficient local search of AWS Lambda Powertools documentation...

    2 tools
    Added May 30, 2025
  • Python Codebase Analysis RAG System
    Python Codebase Analysis RAG System

    Analyzes Python codebases using AST to extract and vectorize code elements, enabling advanced queryi...

    Added May 30, 2025
  • MkDocs MCP Search Server
    MkDocs MCP Search Server

    Enables Model Context Protocol integration for efficient, version-specific search of MkDocs-powered ...

    Added May 30, 2025
  • Perplexity AI MCP Server
    Perplexity AI MCP Server

    Provides seamless integration with Perplexity AI via Model Context Protocol, enabling chat, search, ...

    5 tools
    Added May 30, 2025
  • MCP Pokemon Server
    MCP Pokemon Server

    An MCP server implementation enabling interaction with the PokeAPI to fetch dynamic Pokémon data and...

    Added May 30, 2025