How to Run an AI Video Generation Agent with OE Runtime

Generate short cinematic video clips from text descriptions automatically. OE Runtime connects to Runway, crafts detailed scene prompts, submits the generation job, polls for completion, and returns the video URL — no video production skills required.

The Agent YAML

Create a file called agent.yaml inside a video-generation/ directory:

name: Video Creator
description: Generate short video clips from text or image prompts
instructions: |
  You are a video generation agent. Create short video clips using Runway.
  Describe motion, style, and scene clearly. Videos are generated asynchronously —
  poll for completion before returning the URL.
  Complete all steps fully before writing your report.
steps:
  - name: Design Video Prompt
    content: |
      Craft a detailed Runway video prompt for a 5-second product reveal clip.
      Scene: a sleek smartphone emerging from mist on a dark reflective surface.
      Motion: slow upward drift with a rotating reveal.
      Style: cinematic, high contrast, blue accent lighting.
      Camera: slow push-in, shallow depth of field.
  - name: Generate Video
    content: |
      Submit the prompt to the Runway connector.
      Poll every 15 seconds until generation status is "SUCCEEDED".
      Retrieve the video URL once complete.
  - name: Report
    content: |
      Summarize the result:
      - Prompt used for generation
      - Generation time taken
      - Output video URL
      - Duration and resolution
      - Recommended use cases for this clip
connectors:
  - connection_name: Runway
    connection_type: runway

The Config File

Create oe-config.json in the same directory:

{
  "llm": {
    "provider": "openai",
    "model": "gpt-4o",
    "apiKey": "YOUR_OPENAI_API_KEY"
  },
  "server": {
    "enabled": false,
    "port": 3333,
    "apiKey": "your-secret-api-key"
  },
  "connectors": [
    {
      "connection_name": "Runway",
      "connection_type": "runway",
      "apiKey": "YOUR_RUNWAY_API_KEY",
      "model": "gen3a_turbo",
      "duration": 5
    }
  ]
}

Get your Runway API key from app.runwayml.com/settings. The model gen3a_turbo is the fastest generation option. The duration field accepts 5 or 10 seconds. Video generation is asynchronous — the agent polls until the job completes before returning the URL.

Download OE Runtime

Run the Agent

Windows

# Run the agent
oe-runtime-win.exe video-generation/agent.yaml --config video-generation/oe-config.json

macOS

# Make executable (first time only)
chmod +x oe-runtime-macos

# Run the agent
oe-runtime-macos video-generation/agent.yaml --config video-generation/oe-config.json

Linux

# Make executable (first time only)
chmod +x oe-runtime-linux

# Run the agent
oe-runtime-linux video-generation/agent.yaml --config video-generation/oe-config.json

Running as an API Server

Add --serve to start OE Runtime as an HTTP server on port 3333:

oe-runtime-win.exe --serve --config video-generation/oe-config.json

Then send a request:

curl -X POST http://localhost:3333/run \
  -H "Content-Type: application/json" \
  -d '{"yaml": "video-generation/agent.yaml", "params": {}}'

Or import the Postman collection (download above) to test all endpoints visually.

Use Cases

• Product reveal and teaser videos — Generate cinematic product reveal clips from text descriptions for use in social media, email campaigns, and landing pages. The agent crafts detailed prompts with lighting, motion, and camera direction to produce professional-looking clips without a production budget.
• Social media content creation — Produce short video content for Instagram Reels, TikTok, LinkedIn, and Twitter at scale without a video production team. Describe the scene, mood, and style and the agent generates a unique clip ready for posting in minutes rather than days.
• Marketing campaign visuals — Create compelling background video loops for landing pages, digital ads, and video banners that align with your campaign's visual direction. Swapping visual concepts is as simple as changing the text prompt, enabling rapid creative iteration.
• Explainer and concept visualisation — Turn abstract product concepts, architectural ideas, or technology descriptions into visual clips that communicate what words alone cannot. The agent's ability to render motion and atmosphere makes it ideal for early-stage concept communication before physical assets exist.
• E-learning and training content — Generate illustrative video clips to accompany instructional content, making abstract concepts more engaging and easier to understand. Video components increase learner engagement and knowledge retention compared to text and static images alone.
• Real estate and architectural visualisation — Produce cinematic exterior or interior video walkthroughs of properties or architectural concepts from descriptive prompts. These clips can be used in property listings, investor pitches, and planning presentations before construction begins.
• Game and app trailers — Generate stylised video clips showcasing game environments, characters, or app interfaces for use in store listings, trailers, and promotional material. Multiple visual styles can be explored quickly by iterating on the prompt without engaging a production studio.
• News and editorial illustration — Produce illustrative video segments to accompany news articles, blog posts, or editorial content that lack original visual assets. Publications can generate contextually appropriate video to accompany written stories without sourcing stock footage.