stable-diffusion.cpp-rest

stable-diffusion.cpp-rest

A C++ based REST API wrapper for the stable-diffusion.cpp library, providing HTTP endpoints for image generation with Stable Diffusion models.

✨ Features

• REST API - Complete HTTP API for Stable Diffusion image generation
• Web UI - Modern, responsive web interface (automatically built with the server)
• Queue System - Efficient job queue for managing generation requests
• Model Management - Support for multiple model types with automatic detection
• CUDA Support - Optional GPU acceleration for faster generation
• Authentication - Multiple authentication methods including JWT, API keys, and PAM

Table of Contents

• Project Overview
• Web UI
• Architecture
• Model Detection and Architecture Support
• Technical Requirements
• Project Structure
• Model Types and File Extensions
• API Endpoints
• Authentication
• Build Instructions
• Usage Examples
• Development Roadmap

Project Overview

The stable-diffusion.cpp-rest project aims to create a high-performance REST API server that wraps the functionality of the stable-diffusion.cpp library. This enables developers to integrate Stable Diffusion image generation capabilities into their applications through standard HTTP requests, rather than directly using the C++ library.

Objectives

• Provide a simple, RESTful interface for Stable Diffusion image generation
• Support all parameters available in examples/cli/main.cpp
• Implement efficient resource management with a generation queue system
• Support multiple model types with automatic detection and loading
• Ensure thread-safe operation with separate HTTP server and generation threads

Web UI

A modern, responsive web interface is included and automatically built with the server!

Features:

• Text-to-Image, Image-to-Image, Inpainting, and Upscaler interfaces
• Real-time job queue monitoring
• Model management (load/unload models, scan for new models)
• Light/Dark theme with auto-detection
• Full parameter control for generation
• Interactive mask editor for inpainting

Quick Start:

# Build (automatically builds web UI)
mkdir build && cd build
cmake ..
cmake --build .

# Run server with web UI
./src/stable-diffusion-rest-server --models-dir /path/to/models --checkpoints checkpoints --ui-dir ../webui

# Access web UI
open http://localhost:8080/ui/

See WEBUI.md for detailed documentation.

Architecture

The project is designed with a modular architecture consisting of three main components:

HTTP Server

• Handles incoming HTTP requests
• Parses request parameters and validates input
• Returns generated images or error responses
• Operates independently of the generation process

Generation Queue

• Manages image generation requests
• Processes jobs sequentially (one at a time)
• Maintains thread-safe operations
• Provides job status tracking

Model Manager

• Handles loading and management of different model types
• Supports automatic model detection from default folders
• Automatically detects diffusion model architecture and selects appropriate loading method
• Supports traditional SD models (SD 1.5, SD 2.1, SDXL Base/Refiner) using ctxParams.model_path
• Supports modern architectures (Flux Schnell/Dev/Chroma, SD3, Qwen2VL) using ctxParams.diffusion_model_path
• Includes fallback mechanisms for unknown architectures
• Applies optimal parameters based on detected model type
• Manages model lifecycle and memory usage

• Provides type-based model organization

  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
  │   HTTP Server   │───▶│ Generation Queue│───▶│  Model Manager  │
  │                 │    │                 │    │                 │
  │ - Request Parse │    │ - Job Queue     │    │ - Model Loading │
  │ - Response      │    │ - Sequential    │    │ - Type Detection│
  │   Formatting    │    │   Processing    │    │ - Memory Mgmt   │
  └─────────────────┘    └─────────────────┘    └─────────────────┘
  

Technical Requirements

Core Technologies

• C++17 or later
• CMake 3.15 or later
• Threading support (std::thread, std::mutex, std::condition_variable)
• CUDA support (optional but recommended for performance)

Dependencies

• stable-diffusion.cpp library (automatically downloaded via CMake)
• HTTP server library (to be determined based on requirements)
• JSON library for request/response handling
• Build tools compatible with CMake

Platform Support

• Linux (primary development platform)
• Windows (planned support)
• macOS (planned support)

Project Structure

stable-diffusion.cpp-rest/
├── CMakeLists.txt              # Main CMake configuration
├── README.md                   # This file
├── src/                        # Source code directory
│   ├── main.cpp                # Application entry point
│   ├── http/                   # HTTP server implementation
│   │   ├── server.h/.cpp       # HTTP server class
│   │   ├── handlers.h/.cpp     # Request handlers
│   │   └── responses.h/.cpp    # Response formatting
│   ├── generation/             # Generation queue implementation
│   │   ├── queue.h/.cpp        # Job queue management
│   │   ├── worker.h/.cpp       # Generation worker thread
│   │   └── job.h/.cpp          # Job definition and status
│   ├── models/                 # Model manager implementation
│   │   ├── manager.h/.cpp      # Model manager class
│   │   ├── loader.h/.cpp       # Model loading logic
│   │   └── types.h/.cpp        # Model type definitions
│   └── utils/                  # Utility functions
│       ├── config.h/.cpp       # Configuration management
│       └── logger.h/.cpp       # Logging utilities
├── include/                    # Public header files
├── external/                   # External dependencies (managed by CMake)
├── models/                     # Default model storage directory
│   ├── lora/                   # LoRA models
│   ├── checkpoints/            # Checkpoint models
│   ├── vae/                    # VAE models
│   ├── presets/                # Preset files
│   ├── prompts/                # Prompt templates
│   ├── neg_prompts/            # Negative prompt templates
│   ├── taesd/                  # TAESD models
│   ├── esrgan/                 # ESRGAN models
│   ├── controlnet/             # ControlNet models
│   ├── upscaler/               # Upscaler models
│   └── embeddings/             # Textual embeddings
├── tests/                      # Unit and integration tests
├── examples/                   # Usage examples
└── docs/                       # Additional documentation

Model Detection and Architecture Support

The stable-diffusion.cpp-rest server includes intelligent model detection that automatically identifies the model architecture and selects the appropriate loading method. This ensures compatibility with both traditional Stable Diffusion models and modern architectures.

Supported Architectures

Traditional Stable Diffusion Models

• SD 1.5 - Stable Diffusion version 1.5 models
• SD 2.1 - Stable Diffusion version 2.1 models
• SDXL Base - Stable Diffusion XL base models
• SDXL Refiner - Stable Diffusion XL refiner models

These models are loaded using the traditional ctxParams.model_path parameter.

Modern Architectures

• Flux Schnell - Fast Flux variant
• Flux Dev - Development version of Flux
• Flux Chroma - Chroma-optimized Flux
• SD 3 - Stable Diffusion 3 models
• Qwen2VL - Qwen2 Vision-Language models

These models are loaded using the modern ctxParams.diffusion_model_path parameter.

Detection Process

1. Model Analysis: When loading a model, the system analyzes the model file structure and metadata
2. Architecture Identification: The model architecture is identified based on key signatures in the model
3. Loading Method Selection: The appropriate loading method is automatically selected:
   • Traditional models → ctxParams.model_path
   • Modern architectures → ctxParams.diffusion_model_path
4. Fallback Handling: Unknown architectures default to traditional loading for backward compatibility
5. Error Recovery: If loading with the detected method fails, the system attempts fallback loading

Benefits

• Automatic Compatibility: No need to manually specify model type
• Optimal Loading: Each architecture uses its optimal loading parameters
• Future-Proof: Easy to add support for new architectures
• Backward Compatible: Existing models continue to work without changes

Model Types and File Extensions

The project supports various model types, each with specific file extensions:

Model Type	Enum Value	Description	Supported Extensions
LORA	1	Low-Rank Adaptation models	.safetensors, .pt, .ckpt
CHECKPOINT	2	Main model checkpoints	.safetensors, .pt, .ckpt
VAE	4	Variational Autoencoder models	.safetensors, .pt, .ckpt
PRESETS	8	Generation preset files	.json, .yaml, .yml
PROMPTS	16	Prompt template files	.txt, .json
NEG_PROMPTS	32	Negative prompt templates	.txt, .json
TAESD	64	Tiny AutoEncoder for SD	.safetensors, .pt, .ckpt
ESRGAN	128	Super-resolution models	.pth, .pt
CONTROLNET	256	ControlNet models	.safetensors, .pt, .ckpt
UPSCALER	512	Image upscaler models	.pth, .pt
EMBEDDING	1024	Textual embeddings	.safetensors, .pt, .ckpt

Model Type Enum Definition

enum ModelType {
    LORA = 1,
    CHECKPOINT = 2,
    VAE = 4,
    PRESETS = 8,
    PROMPTS = 16,
    NEG_PROMPTS = 32,
    TAESD = 64,
    ESRGAN = 128,
    CONTROLNET = 256,
    UPSCALER = 512,
    EMBEDDING = 1024
};

API Endpoints

Planned Endpoints

Image Generation

• POST /api/v1/generate/text2img - Generate image from text prompt
• POST /api/v1/generate/img2img - Transform image with text prompt
• POST /api/v1/generate/inpainting - Inpaint image with mask
• GET /api/v1/generate/{job_id} - Get generation status and result
• DELETE /api/v1/generate/{job_id} - Cancel a generation job

Model Management

• GET /api/v1/models - List available models
• GET /api/v1/models/{type} - List models of specific type
• POST /api/v1/models/load - Load a model
• POST /api/v1/models/unload - Unload a model
• GET /api/v1/models/{model_id} - Get model information

System Information

• GET /api/v1/status - Get server status and statistics
• GET /api/v1/system - Get system information (GPU, memory, etc.)

Example Request/Response

Generate Image Request

POST /api/v1/generate
{
    "prompt": "a beautiful landscape",
    "negative_prompt": "blurry, low quality",
    "model": "sd-v1-5",
    "width": 512,
    "height": 512,
    "steps": 20,
    "cfg_scale": 7.5,
    "seed": -1,
    "batch_size": 1
}

Generate Image Response

{
    "job_id": "uuid-string",
    "status": "completed",
    "images": [
        {
            "data": "base64-encoded-image-data",
            "seed": 12345,
            "parameters": {
                "prompt": "a beautiful landscape",
                "negative_prompt": "blurry, low quality",
                "model": "sd-v1-5",
                "width": 512,
                "height": 512,
                "steps": 20,
                "cfg_scale": 7.5,
                "seed": 12345,
                "batch_size": 1
            }
        }
    ],
    "generation_time": 3.2
}

Authentication

The server supports multiple authentication methods to secure API access:

Supported Authentication Methods

1. No Authentication (Default)

   • Open access to all endpoints
   • Suitable for development or trusted networks

2. JWT Token Authentication

   • JSON Web Tokens for stateless authentication
   • Configurable token expiration
   • Secure for production deployments

3. API Key Authentication

   • Static API keys for service-to-service communication
   • Simple integration for external applications

4. PAM Authentication

   • Integration with system authentication via PAM (Pluggable Authentication Modules)
   • Supports LDAP, Kerberos, and other PAM backends
   • Leverages existing system user accounts
   • See PAM_AUTHENTICATION.md for detailed setup

Authentication Configuration

Authentication can be configured via command-line arguments or configuration files:

# Enable PAM authentication
./stable-diffusion-rest-server --auth pam --models-dir /path/to/models --checkpoints checkpoints

# Enable JWT authentication
./stable-diffusion-rest-server --auth jwt --models-dir /path/to/models --checkpoints checkpoints

# Enable API key authentication
./stable-diffusion-rest-server --auth api-key --models-dir /path/to/models --checkpoints checkpoints

# Enable Unix authentication
./stable-diffusion-rest-server --auth unix --models-dir /path/to/models --checkpoints checkpoints

# No authentication (default)
./stable-diffusion-rest-server --auth none --models-dir /path/to/models --checkpoints checkpoints

Authentication Methods

• none - No authentication required (default)
• jwt - JWT token authentication
• api-key - API key authentication
• unix - Unix system authentication
• pam - PAM authentication
• optional - Authentication optional (guest access allowed)

Deprecated Options

The following options are deprecated and will be removed in a future version:

• --enable-unix-auth - Use --auth unix instead
• --enable-pam-auth - Use --auth pam instead

Authentication Endpoints

• POST /api/v1/auth/login - Authenticate with username/password (PAM/JWT)
• POST /api/v1/auth/refresh - Refresh JWT token
• GET /api/v1/auth/profile - Get current user profile
• POST /api/v1/auth/logout - Logout/invalidate token

For detailed authentication setup instructions, see PAM_AUTHENTICATION.md.

Build Instructions

Prerequisites

1. CMake 3.15 or later
2. C++17 compatible compiler
3. Git for cloning dependencies
4. CUDA Toolkit (optional but recommended)

Build Steps

1. Clone the repository:

   git clone https://github.com/your-username/stable-diffusion.cpp-rest.git
   cd stable-diffusion.cpp-rest
   

2. Create a build directory:

   mkdir build
   cd build
   

3. Configure with CMake:

   cmake ..
   

4. Build the project:

   cmake --build . --parallel
   

5. (Optional) Install the binary:

   cmake --install .
   

CMake Configuration

The project uses CMake's external project feature to automatically download and build the stable-diffusion.cpp library:

include(ExternalProject)

ExternalProject_Add(
    stable-diffusion.cpp
    GIT_REPOSITORY https://github.com/leejet/stable-diffusion.cpp.git
    GIT_TAG master-334-d05e46c
    SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/stable-diffusion.cpp-src"
    BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/stable-diffusion.cpp-build"
    CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/stable-diffusion.cpp-install
    INSTALL_COMMAND ""
)

PAM Authentication Build Options

PAM authentication support is enabled by default when PAM libraries are available. You can control this with CMake options:

# Build with PAM support (default when available)
cmake -DENABLE_PAM_AUTH=ON ..

# Build without PAM support
cmake -DENABLE_PAM_AUTH=OFF ..

# Check if PAM support will be built
cmake -LA | grep ENABLE_PAM_AUTH

Note: PAM authentication requires the PAM development libraries:

# Ubuntu/Debian
sudo apt-get install libpam0g-dev

# CentOS/RHEL/Fedora
sudo yum install pam-devel

Usage Examples

Starting the Server

# Basic usage
./stable-diffusion.cpp-rest

# With custom configuration
./stable-diffusion.cpp-rest --config config.json

# With custom model directory
./stable-diffusion.cpp-rest --model-dir /path/to/models

Client Examples

Python with requests

import requests
import base64
import json

# Generate an image
response = requests.post('http://localhost:8080/api/v1/generate', json={
    'prompt': 'a beautiful landscape',
    'width': 512,
    'height': 512,
    'steps': 20
})

result = response.json()
if result['status'] == 'completed':
    # Decode and save the first image
    image_data = base64.b64decode(result['images'][0]['data'])
    with open('generated_image.png', 'wb') as f:
        f.write(image_data)

JavaScript with fetch

async function generateImage() {
    const response = await fetch('http://localhost:8080/api/v1/generate', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
            prompt: 'a beautiful landscape',
            width: 512,
            height: 512,
            steps: 20
        })
    });

    const result = await response.json();
    if (result.status === 'completed') {
        // Create an image element with the generated image
        const img = document.createElement('img');
        img.src = `data:image/png;base64,${result.images[0].data}`;
        document.body.appendChild(img);
    }
}

cURL

# Generate an image
curl -X POST http://localhost:8080/api/v1/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "a beautiful landscape",
    "width": 512,
    "height": 512,
    "steps": 20
  }'

# Check job status
curl http://localhost:8080/api/v1/generate/{job_id}

# List available models
curl http://localhost:8080/api/v1/models

Development Roadmap

Phase 1: Core Infrastructure

• Basic HTTP server implementation
• Generation queue system
• Model manager with basic loading
• CMake configuration with external dependencies
• Basic API endpoints for image generation

Phase 2: Feature Enhancement

• Complete parameter support from examples/cli/main.cpp
• Model type detection and organization
• Job status tracking and cancellation
• Error handling and validation
• Configuration management

Phase 3: Advanced Features

• Batch processing support
• Model hot-swapping
• Performance optimization
• Comprehensive logging
• API authentication and security

Phase 4: Production Readiness

• Comprehensive testing suite
• Documentation and examples
• Docker containerization
• Performance benchmarking
• Deployment guides

Future Considerations

• WebSocket support for real-time updates
• Plugin system for custom processors
• Distributed processing support
• Web UI for model management
• Integration with popular AI frameworks

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• stable-diffusion.cpp for the underlying C++ implementation
• The Stable Diffusion community for models and examples
• Contributors and users of this project