Scenario
Across the industry, manufacturing organizations spend significant engineering effort converting tribal knowledge and assembly demonstrations into standardized work instructions. This process is traditionally manual, inconsistent, and difficult to scale across plants, product lines, and geographies. By integrating multimodal Generative AI directly into supply chain workflows on Azure, this solution automates the transformation of raw assembly videos into reviewable, illustrated work instructions—dramatically reducing documentation effort, accelerating worker onboarding, and preserving critical operational knowledge.
Solution Components and Architecture
Assembly Video to Manufacturing Document Execution Flow
- Video Ingestion and Job Creation
The user selects an assembly video from the web application and starts a pipeline run. The API creates a job record and places a transcription request onto a queue for asynchronous processing.
Components
- Web Review Application: Lists available videos and starts pipeline runs.
- Azure Blob Storage (video): Stores source assembly videos.
- Azure Functions (StartJob): Creates and initiates the job.
- Azure Cosmos DB (assembly-pipeline/jobs): Records job status, source video, timestamps, and progress.
- Azure Queue Storage (transcription-jobs): Delivers the transcription request to the worker.
Output: A tracked pipeline job and queued transcription request.
- Audio Extraction and Transcription
The queue-triggered worker downloads the source video, extracts mono 16-kHz WAV audio, divides long recordings into chunks, and submits each chunk to Whisper.
The resulting segments are merged into a timestamped transcript. Repeated phrases, chunk overlaps, and common transcription artifacts are removed.
Components
- Azure Functions (TranscriptionWorker): Coordinates transcription processing.
- FFmpeg: Extracts and divides the audio.
- Azure OpenAI Whisper: Converts speech into timestamped text.
- Azure Blob Storage (audio): Stores extracted audio.
- Azure Blob Storage (transcripts): Stores transcript JSON.
- Azure Cosmos DB: Records progress and transcript metadata.
Quality controls
- Retry handling for temporary service failures and rate limits.
- Transcript cleanup and boundary deduplication.
- Recommended enhancement: benchmark transcription models using evaluations.
Output: Clean, consolidated, timestamped transcript.
- Transcript Review and Approval
The transcript is presented to a human reviewer. The reviewer can approve it or provide corrections. GPT-5.1 applies submitted feedback and stores a revised transcript for another review cycle.
The pipeline cannot proceed until the transcript is approved.
Components
- Web Review Application: Displays the transcript and captures feedback.
- Azure Functions (SubmitFeedback, ApproveStage): Processes feedback and approval.
- Azure OpenAI GPT-5.1: Applies transcript corrections.
- Azure Blob Storage (transcripts): Versions revised transcripts.
- Azure Cosmos DB: Maintains approval status and the current preview.
Quality Gate: Human transcript accuracy check.
Output: Human-approved transcript.
- Assembly Step Identification
After transcript approval, GPT-5.1 analyzes the timestamped content and converts it into a structured manufacturing procedure.
The generated JSON includes:
- Document and product information
- Numbered assembly instructions
- Part numbers and quantities
- Required tools and fixtures
- Safety warnings
- Step-specific notes
- Source-video timestamps
- Training requirements
- Technical glossary
Components
- Azure Functions (IdentifySteps): Constructs the prompt and coordinates model execution.
- Azure OpenAI GPT-5.1: Generates the structured procedure.
- Azure Blob Storage (procedures): Stores procedure JSON.
- Azure Cosmos DB: Stores the procedure preview and workflow status.
Recommended automated evaluations
- Step completeness and sequence
- Safety-instruction coverage
- Part-number accuracy
- Groundedness against the transcript
- Unsupported-instruction and hallucination detection
- JSON schema and structure validation
Output: Structured assembly-procedure JSON.
- Step Review and Approval
The generated procedure is presented to a reviewer for validation. The reviewer checks the instructions, sequence, parts, tools, timestamps, and safety information.
Feedback is applied through GPT-5.1, and the revised procedure remains in review until approved.
Components
- Web Review Application: Presents structured instructions.
- Azure Functions (SubmitFeedback, ApproveStage): Coordinates revision and approval.
- Azure OpenAI GPT-5.1: Revises the procedure using reviewer feedback.
- Azure Blob Storage (procedures): Stores revised procedures.
- Azure Cosmos DB: Tracks feedback and approval state.
Quality Gate: Human procedure-quality check.
Output: Human-approved assembly procedure.
- Reference Frame Extraction
For every approved step, the system locates the associated video timestamps and extracts several nearby candidate frames.
OpenCV calculates a sharpness score using Laplacian variance. The sharpest candidate is selected to avoid blurry or transitional images.
Components
- Azure Functions (ExtractFrames): Builds and executes the extraction plan.
- FFmpeg: Captures candidate video frames.
- OpenCV: Scores image sharpness.
- Azure Blob Storage (frames): Stores selected PNG images.
- Azure Cosmos DB: Records stage progress.
Quality controls
- Multiple candidates per timestamp
- Configurable action-time offset
- Sharpness-based selection
- Video-duration boundary checks
Output: Best-quality reference images for each assembly step.
- Manufacturing Document Generation
The approved procedure and selected images are assembled into technician-ready manufacturing instructions.
The documents contain product information, revision details, parts and tools tables, safety warnings, illustrated instructions, notes, and a technical glossary.
Components
- Azure Functions (GenerateDocumentation): Coordinates document creation.
- python-docx: Generates the editable Word document.
- ReportLab: Generates the PDF.
- Azure Blob Storage (reports): Stores final DOCX and PDF files.
- Azure Cosmos DB: Records report paths and completion status.
Output: Draft manufacturing document in DOCX and PDF formats.
- SME Validation and Publication
A subject-matter expert reviews the generated document for technical correctness, operational usability, safety coverage, and image relevance.
The SME can approve the document, reject it, or request corrections before publication.
Validation areas
- Instruction accuracy and completeness
- Correct assembly sequence
- Part-number and tool accuracy
- Safety-content coverage
- Image-to-step alignment
- Terminology and formatting
- Readiness for shop-floor use
Output: Approved manufacturing work instruction.
The current application implements transcript and procedure approval, but a separate post-generation SME approval state is not yet implemented. It is a recommended workflow enhancement.
Component Table
|
Stage |
Primary activity |
Components |
Stored output |
|
Ingestion |
Select video and create job |
Web UI, Functions, Cosmos DB, Queue Storage |
Job record |
|
Transcription |
Extract audio and generate text |
Functions, FFmpeg, Whisper |
Audio and transcript |
|
Transcript review |
Correct and approve transcript |
Web UI, GPT-5.1, Functions |
Approved transcript |
|
Step identification |
Generate structured instructions |
GPT-5.1, Functions |
Procedure JSON |
|
Step review |
Validate instructions |
Web UI, GPT-5.1, Functions |
Approved procedure |
|
Frame extraction |
Select reference images |
FFmpeg, OpenCV, Functions |
PNG frames |
|
Document generation |
Create manufacturing documents |
python-docx, ReportLab |
DOCX and PDF |
|
SME validation |
Final technical approval |
SME review workflow |
Approved document |
Cross-Cutting Components
- Azure Cosmos DB: Tracks status, progress, approvals, errors, and artifact locations.
- Azure Blob Storage: Stores all input, intermediate, frontend, deployment, and output artifacts.
- Managed Identity and RBAC: Authenticate backend access to Storage, Cosmos DB, and AI services.
- Application Insights and Log Analytics: Capture telemetry, failures, and operational diagnostics.
- Retry and Recovery: Resume failed jobs from their last completed processing stage.
- AI Quality Assurance: Combines automated evaluation with human approval gates.
Demo
Security Considerations
Azure Blob Storage: Use separate quarantine, processing, and approved-output containers for source videos, extracted media, transcripts, and generated manuals. Keep uploads quarantined until Azure Functions validates the file and Azure AI Content Safety assesses representative frames; the application must enforce the result and release only reviewed outputs. Restrict access through a private endpoint, disable public and anonymous access, and assign each function a least-privilege managed identity at container scope. Require TLS 1.2+, encryption at rest, and short-lived user-delegation SAS tokens only when needed. Enable Defender for Storage, diagnostic logging, versioning, soft delete, lifecycle retention, and Azure Policy to detect threats, remove temporary artifacts, and recover source or generated content without publishing unapproved instructions.
Azure Cosmos DB: Store only pipeline metadata, transcripts, review status, and Blob references—not media files. Restrict access to the Azure Functions VNet through a private endpoint and least-privilege managed identities. Partition by job or assembly ID, validate AI-generated fields, and never store secrets. Require TLS 1.2+ and encryption at rest, and enable Defender for Cosmos DB, logging, TTL, continuous backup, and Azure Policy for threat detection, retention, recovery, and compliance.
Azure Functions: Use separate pipeline functions with least-privilege managed identities for Blob Storage, Cosmos DB, Azure OpenAI, and Key Vault. Keep service traffic private, require HTTPS and Entra ID for operator APIs, and validate uploads, AI outputs, and approval status at every stage. Never log media, secrets, tokens, or sensitive transcripts. Use queue retries, poison-message handling, idempotency, and correlation IDs to prevent duplicate processing or unreviewed publishing. Monitor failures and publishing events with Application Insights, Azure Monitor, and Defender for Cloud, and enforce security standards with Azure Policy.
Microsoft Foundry: Use a dedicated project with approved models and least-privilege access for Azure Functions, developers, reviewers, and administrators. Keep connections to Blob Storage, Cosmos DB, and Key Vault private. Minimize data sent to models, defend against prompt injection, validate generated parts, timestamps, safety warnings, and steps, and require expert approval before publishing. Encrypt data, avoid sensitive content in logs, and use Defender for AI Services, AI Security Posture Management, Purview, and Azure Policy to detect threats and enforce governance.
AI Model: Only verified and approved models should be deployable, with centralized registration, security validation, RBAC controls, auditability, and policy-based enforcement. AI applications should be designed with layered defenses, including multi‑stage content filtering, safety meta‑prompts, and least‑privilege permissions for agents and plugins to reduce the risk of prompt injection, data leakage, and unintended actions. Enforce model provenance, integrity validation, approval workflows, policy-based deployment controls, and lifecycle auditability.
Related Scenarios
Equipment Maintenance: Organizations often rely on experienced technicians to perform complex maintenance and repair procedures, yet much of this knowledge remains undocumented or difficult to scale. By recording maintenance activities and technician narration, multimodal AI can automatically generate standardized work instructions, troubleshooting guides, and training materials. This helps preserve institutional knowledge, accelerate onboarding, and improve consistency across maintenance operations.
Healthcare Training: Healthcare educators and clinical experts spend significant time creating training materials and documenting procedures. By converting recorded demonstrations into structured documentation, healthcare organizations can automatically generate training guides, equipment setup instructions, and step-by-step procedural references. Human review checkpoints ensure accuracy while reducing the manual effort required to produce educational content.
Consumer Products: Manufacturers of consumer products often create assembly and setup instructions manually, requiring extensive documentation effort for each new product and revision. By analyzing product assembly recordings and narrated demonstrations, AI can generate user manuals, assembly guides, parts lists, and visual instructions automatically. This accelerates product launches while helping maintain consistency and quality across documentation.
Hobby Communities: Many hobbyists share expertise through video tutorials covering activities such as woodworking, model building, electronics, and crafting. Multimodal AI can transform these demonstrations into structured, searchable guides that include step-by-step instructions, reference images, required tools, and material lists. This allows creators to expand the reach and accessibility of their knowledge beyond video content alone.
DIY and Home Improvement: Home improvement professionals and skilled tradespeople frequently create instructional videos to demonstrate repairs, installations, and maintenance tasks. AI-powered documentation generation can convert these demonstrations into detailed how-to guides complete with images, safety considerations, required tools, and step-by-step procedures. This makes technical knowledge easier to consume while reducing the effort needed to create high-quality instructional content.
Contributors
This article is maintained by Microsoft. It was originally written by the following contributors.
Principal authors:
- Manasa Ramalinga | Senior Principal Cloud Solution Architect
- Oscar Shimabukuro | Senior Cloud Solution Architect
- Matt Kazanowsky | Cloud Solution Architect
- Abed Sau | Principal Cloud Solution Architect
- Yagneswari Kanadam | Senior Cloud Solution Architect