From YAML to AI: How Platform Engineers Are Embedding LLMs into Internal Developer Platforms

Platform engineering spent years turning tribal knowledge into YAML: catalog descriptors, scaffolder templates, scorecards, policies, and deployment workflows. Now LLMs are becoming the conversational layer over that machinery. The useful shift isn't replacing YAML with chat. It's giving developers a safer way to discover and invoke the platform capabilities already encoded underneath.

YAML solved repeatability, not discoverability¶

A mature internal developer platform usually has the right answer somewhere. The problem is finding it.

The service owner might be in catalog-info.yaml. The production-readiness rules might be in a scorecard. The deployment path might be a Backstage template, a Port action, an Argo workflow, or a Terraform module.

Backstage 1.53.0 still models catalog entities as YAML and uses Software Templates to collect inputs, render skeletons, and publish components to systems such as GitHub or GitLab.¹²

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  description: Processes customer payments
  tags:
    - go
    - pci
spec:
  type: service
  lifecycle: production
  owner: group:payments
  system: checkout

That's good machine-readable context. It doesn't answer a developer who asks:

Why can't payments-api deploy to production, and what should I fix first?

A useful platform assistant has to join the catalog record with scorecards, deployment history, policy results, runbooks, and the user's permissions. The LLM is the interface; the IDP remains the system of record and control plane.

The new IDP stack has five layers¶

Embedding an LLM in a portal isn't a chat-widget project. The working architecture has five distinct layers.

This separation matters. If the model is allowed to invent shell commands and execute them directly, you've built a remote-command interface with probabilistic input validation. That's not an IDP.

A safer request path looks like this:

Developer request
      |
      v
Portal / IDE assistant
      |
      v
Identity + authorization
      |
      v
Context builder ------> Catalog + docs + scorecards + live state
      |
      v
LLM plans a tool call
      |
      v
Schema validation + policy + approval
      |
      v
Existing platform action / workflow
      |
      v
Audit log + result returned to developer

Let the model choose from approved capabilities. Don't let it manufacture capabilities.

Pattern 1: Turn the software catalog into grounded context¶

The catalog is the best starting point because it already contains structured facts: owner, lifecycle, dependencies, environment, scorecard status, and links to runbooks.

Port's current AI architecture makes this explicit. Port AI acts as an MCP client, uses tools exposed by Port's MCP server, and grounds responses in its Context Lake. Its documentation lists catalog queries, scorecard analysis, and self-service actions as supported use cases.³

A developer can ask:

Show production services owned by Payments that are below Gold,
then explain the failing checks and link each runbook.

The important part isn't the wording. The assistant should translate that request into typed queries against authoritative data, not guess from whatever documentation happened to fit in the model's context window.

For a Backstage-based platform, the same pattern can be built with the Catalog API, Search, TechDocs, and permission-aware backend plugins. Backstage's next-version documentation now includes AI extension points for MCP actions and skills, but these pages aren't part of the stable 1.53.0 documentation yet. Treat them as preview APIs, not a production contract.⁴

Context quality beats model size¶

A larger model won't repair a weak catalog. Before adding chat, check the data it will depend on:

• Every production component has an owner.
• Dependencies are modeled, not buried in diagrams.
• Scorecards explain why a check failed.
• Actions have descriptions, schemas, and bounded inputs.
• Runbooks have stable URLs and service metadata.
• Deployment and incident data include timestamps and environment names.

If those fields are missing, the assistant will produce polished uncertainty.

Pattern 2: Put MCP between agents and platform capabilities¶

The Model Context Protocol gives agents a standard way to discover context and tools. MCP uses a client-host-server architecture built on JSON-RPC. Servers can expose resources, tools, and prompts; the host manages connections, consent, authorization decisions, and context aggregation.⁵

For an IDP, the mapping is clean:

MCP resources -> catalog records, docs, scorecards, deployment state
MCP tools     -> create service, deploy, rollback, open incident
MCP prompts   -> approved incident triage or service onboarding playbooks

Port now ships a remote MCP server that lets external clients such as Claude, Cursor, or GitHub Copilot query the catalog and invoke portal actions. Port is explicit about the boundary: the IDE or chat client runs the LLM; Port exposes the catalog tools.⁶

That separation avoids writing a custom integration for every model and IDE. It doesn't remove the hard security work.

An MCP tool should have a narrow schema:

{
  "name": "deploy_service",
  "description": "Deploy an approved artifact to an environment",
  "inputSchema": {
    "type": "object",
    "properties": {
      "service": { "type": "string" },
      "artifactDigest": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
      "environment": { "enum": ["dev", "staging", "production"] }
    },
    "required": ["service", "artifactDigest", "environment"],
    "additionalProperties": false
  }
}

The tool handler still has to verify ownership, artifact provenance, environment access, change windows, and approval policy. MCP standardizes the connection. It doesn't make a dangerous tool safe.

Pattern 3: Move from chat to bounded agents¶

A chat assistant answers questions. An agent can plan and call tools. The second is much more valuable and much more dangerous.

Current Port agents can be limited to selected catalog-query tools and specific self-service actions. Each selected action can run automatically or require approval. Agents may also receive individually selected tools from external MCP servers.⁷

That is the right shape for platform engineering: one narrow agent per operational job.

Avoid the "platform super-agent" with access to every cluster, repository, secret store, and CI system. It creates a single confused-deputy path across the whole engineering estate.

Start with a task a junior platform engineer could perform from a documented checklist. Then encode the checklist as tools, validation, and policy rather than hiding it in a 4,000-word system prompt.

Pattern 4: Add AI where the operational evidence lives¶

Not every LLM capability belongs in the portal UI. Sometimes the useful reasoning happens closer to the system.

K8sGPT 0.4.36, released on July 10, 2026, scans Kubernetes resources for known problems and can ask an AI backend to explain its findings.⁸ The project currently documents 11 backends, including OpenAI, Amazon Bedrock, Azure OpenAI, Google Gemini, Hugging Face, IBM watsonx.ai, LocalAI, and Ollama.⁹

A basic flow is:

k8sgpt auth add --backend openai --model gpt-4o-mini
k8sgpt analyze --explain

K8sGPT's getting-started example creates a Pod with an invalid image tag, observes ErrImagePull, then runs k8sgpt analyze to surface the issue.¹⁰

In an IDP, don't stop at pasting that explanation into a dashboard. Attach the result to the catalog entity, correlate it with ownership and recent deployments, then offer a controlled remediation action.

K8sGPT finding
  -> map namespace/workload to catalog entity
  -> fetch owner + recent deploy + runbook
  -> summarize evidence
  -> offer "retry rollout" or "open incident"
  -> apply normal RBAC and approval policy

This is where the platform earns its keep: connecting cluster evidence to organizational context.

A practical first implementation¶

Don't begin with autonomous production changes. Build a read-only assistant, measure it, then add one low-risk action.

Phase 1: Read-only catalog assistant¶

• Choose 20 real questions developers ask the platform team.
• Expose catalog, scorecard, documentation, and deployment-history queries as typed tools.
• Pass the user's identity and authorization scope into every retrieval call.
• Require citations or record identifiers in every answer.
• Return "I couldn't verify that" when tools return no evidence.
• Record tool calls, latency, token use, and user feedback.

Good evaluation questions are specific:

Who owns checkout-api in production?
Why is inventory-worker below Silver?
Which Tier-1 services have no rollback runbook?
What changed before the last payments outage?

Phase 2: One reversible action¶

Pick an action with bounded inputs and an easy undo path, such as creating a ticket or starting a non-production template.

• Use the existing IDP action instead of a new AI-only backend.
• Validate tool arguments against a JSON Schema.
• Re-check authorization at execution time.
• Show the exact action and inputs before confirmation.
• Make the action idempotent.
• Return the workflow run ID and audit link.

Phase 3: Approval-gated production operations¶

Only move here after the read-only assistant is accurate and the low-risk action is boring.

• Separate plan from execute.
• Require approval from an authorized human, not simply the requester.
• Pin artifacts by digest rather than mutable tags.
• Enforce policy outside the LLM.
• Put timeouts and rate limits on every tool.
• Define a kill switch for the agent and each write-capable tool.

The security model must assume hostile context¶

Catalog descriptions, pull-request titles, incident comments, and documentation are all untrusted input. Any of them can contain text designed to redirect an agent.

Treat prompt injection like data injection: isolate instructions from retrieved content, allowlist tools, validate parameters, and enforce authorization after the model has finished reasoning.

The MCP specification says implementations should build consent and authorization flows, access controls, data protection, and privacy safeguards. It also notes that MCP itself cannot enforce those principles at the protocol layer.¹¹

Port's current controls show what production guardrails look like: standard RBAC for AI access, approval options for actions, invocation records, audit trails, tool-execution logs, and organization-wide monitoring.¹²

Use this minimum checklist before enabling writes:

• Identity propagation: every tool receives the requesting user's identity.
• Least privilege: agents get only the tools and entities required for one job.
• Server-side authorization: never trust the model to honor permissions.
• Human approval: production and destructive actions stop for review.
• Secret filtering: credentials and sensitive fields never enter prompts or traces.
• Untrusted-content labeling: retrieved documents can't override system instructions.
• Complete audit: record model, prompt version, retrieved object IDs, tool arguments, approver, and result.
• Failure containment: cap retries, time, cost, and blast radius.

Measure platform outcomes, not chat activity¶

Message count is a vanity metric. A busy assistant may simply be confusing.

Track outcomes tied to platform work:

Keep an evaluation set of real, permission-sensitive questions. Test it whenever the prompt, model, catalog schema, retrieval logic, or tool definition changes.

A model upgrade is a production change. Treat it like one.

Summary¶

The path from YAML to AI isn't a rewrite of the internal developer platform. It's an interface upgrade over the catalog, scorecards, templates, workflows, and policies that platform teams already maintain.

The implementation rules are simple:

1. Ground answers in structured platform data.
2. Expose existing capabilities as narrow, typed tools.
3. Keep authorization, policy, and approvals outside the model.
4. Start read-only, then add reversible actions.
5. Audit and evaluate every step before increasing autonomy.

The best platform agent doesn't bypass the paved road. It helps developers find it, understand it, and travel it safely.

Questions or discussion? Connect on LinkedIn, X, or email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.