External Worker Runtime for Non-Java Plugins #43

New issue

Open

opened 2026-05-24 02:29:58 +02:00 by FTMahringer · 0 comments

FTMahringer commented

2026-05-24 02:29:58 +02:00

(Migrated from github.com)

Problem / Motivation

Synapse is planned as a backend-first AI platform with a strong Java/Spring Boot core and a plugin ecosystem. Java-native plugins are useful for deeply integrated extensions, but not every plugin should run inside the Synapse JVM.

Future plugins may be written in Python, Go, Node.js or other languages. Running those directly inside the main backend process would make the core harder to isolate, secure and operate.

A worker-based runtime would allow Synapse to support non-Java plugins while keeping the core stable and controlled.

Related issues:

#38 Plugin API & Ecosystem Architecture Refactor for Long-Term Extensibility
#39 Official SDK Strategy for Python, TypeScript and Go
#36 Plugin validation entrypoints for CLI and dashboard upload checks
#30 Plugin Store Website

Proposed Solution

Introduce an external worker runtime model for non-Java plugins.

The runtime should allow plugins to run outside the main Synapse backend and communicate through a stable API layer.

Possible communication options:

HTTP REST
WebSocket
gRPC
Redis/queue-based task execution
event-stream based communication

Possible runtime model:

Synapse Core
 ├── Plugin Registry
 ├── Capability & Permission System
 ├── Event Bus
 ├── Task Scheduler
 └── Worker Runtime Gateway
      ├── Python Worker Plugin
      ├── Go Worker Plugin
      ├── Node.js Worker Plugin
      └── Future runtimes

Worker plugins should be able to:

register themselves with Synapse
declare capabilities
receive events/tasks
return structured results
expose health/status information
be started, stopped or disabled by Synapse where possible
use scoped tokens instead of full backend credentials

Capability / Permission Model

Worker plugins should not automatically receive full access to Synapse.

Each worker should declare required capabilities, for example:

capabilities:
  - provider.chat
  - mcp.server
  - server.proxmox.readonly
  - filesystem.read.scoped
  - channel.discord.send

Synapse should approve and enforce these capabilities centrally.

This fits the controlled plugin approach instead of a free Minecraft-style plugin model where plugins can do anything inside the process.

Security Requirements

Workers run outside the main JVM process
Worker authentication via scoped tokens or mTLS
Capability-based access control
Install-time permission prompt
Audit logging for worker registration and task execution
Rate limits and timeouts
Health checks and automatic disable on repeated failures
Secret access only through Synapse secret references, not raw secrets
Optional network restrictions for local deployments

Developer Experience

The worker runtime should later integrate with SDKs from #39.

Possible SDK helpers:

worker registration
event subscription
task handler registration
typed request/response models
authentication helpers
local dev mode
test harness / simulator

Example future usage:

worker = SynapseWorker("my-python-plugin")
worker.capability("provider.embedding")
worker.on_task("embedding.create", handle_embedding)
worker.run()

Alternatives

Only support Java-native plugins
- simpler, but limits the ecosystem
Load Python/Node.js directly into the backend process
- risky and hard to isolate
Let every plugin define its own API style
- flexible, but hard to govern and validate

Priority

High / Long-Term Architecture

This should be considered before external runtimes become stable and before the public plugin ecosystem grows too much.

## Problem / Motivation Synapse is planned as a backend-first AI platform with a strong Java/Spring Boot core and a plugin ecosystem. Java-native plugins are useful for deeply integrated extensions, but not every plugin should run inside the Synapse JVM. Future plugins may be written in Python, Go, Node.js or other languages. Running those directly inside the main backend process would make the core harder to isolate, secure and operate. A worker-based runtime would allow Synapse to support non-Java plugins while keeping the core stable and controlled. Related issues: - #38 Plugin API & Ecosystem Architecture Refactor for Long-Term Extensibility - #39 Official SDK Strategy for Python, TypeScript and Go - #36 Plugin validation entrypoints for CLI and dashboard upload checks - #30 Plugin Store Website --- ## Proposed Solution Introduce an external worker runtime model for non-Java plugins. The runtime should allow plugins to run outside the main Synapse backend and communicate through a stable API layer. Possible communication options: - HTTP REST - WebSocket - gRPC - Redis/queue-based task execution - event-stream based communication Possible runtime model: ```text Synapse Core ├── Plugin Registry ├── Capability & Permission System ├── Event Bus ├── Task Scheduler └── Worker Runtime Gateway ├── Python Worker Plugin ├── Go Worker Plugin ├── Node.js Worker Plugin └── Future runtimes ``` Worker plugins should be able to: - register themselves with Synapse - declare capabilities - receive events/tasks - return structured results - expose health/status information - be started, stopped or disabled by Synapse where possible - use scoped tokens instead of full backend credentials --- ## Capability / Permission Model Worker plugins should not automatically receive full access to Synapse. Each worker should declare required capabilities, for example: ```yaml capabilities: - provider.chat - mcp.server - server.proxmox.readonly - filesystem.read.scoped - channel.discord.send ``` Synapse should approve and enforce these capabilities centrally. This fits the controlled plugin approach instead of a free Minecraft-style plugin model where plugins can do anything inside the process. --- ## Security Requirements - Workers run outside the main JVM process - Worker authentication via scoped tokens or mTLS - Capability-based access control - Install-time permission prompt - Audit logging for worker registration and task execution - Rate limits and timeouts - Health checks and automatic disable on repeated failures - Secret access only through Synapse secret references, not raw secrets - Optional network restrictions for local deployments --- ## Developer Experience The worker runtime should later integrate with SDKs from #39. Possible SDK helpers: - worker registration - event subscription - task handler registration - typed request/response models - authentication helpers - local dev mode - test harness / simulator Example future usage: ```python worker = SynapseWorker("my-python-plugin") worker.capability("provider.embedding") worker.on_task("embedding.create", handle_embedding) worker.run() ``` --- ## Alternatives - Only support Java-native plugins - simpler, but limits the ecosystem - Load Python/Node.js directly into the backend process - risky and hard to isolate - Let every plugin define its own API style - flexible, but hard to govern and validate --- ## Priority High / Long-Term Architecture This should be considered before external runtimes become stable and before the public plugin ecosystem grows too much.