X Tutup
Skip to content

feat: Avatar Ready Player Me plugin + MCP Memory Cloud (Firebase)#172

Open
Phoenixai36 wants to merge 1 commit intoszczyglis-dev:masterfrom
Phoenixai36:feature/avatar-rpm-mcp-memory
Open

feat: Avatar Ready Player Me plugin + MCP Memory Cloud (Firebase)#172
Phoenixai36 wants to merge 1 commit intoszczyglis-dev:masterfrom
Phoenixai36:feature/avatar-rpm-mcp-memory

Conversation

@Phoenixai36
Copy link

Summary

This PR adds two new community plugins to PyGPT:


🎭 Plugin: Avatar Ready Player Me (avatar_rpm)

A floating 3D avatar window powered by Ready Player Me (free tier) and Three.js, rendered inside PyGPT's existing WebEngine — no additional renderer overhead.

Features:

  • 🔊 Lip-sync reactive to audio output chunks (via on_audio_output_chunk hook)
  • 😊 Emotion detection based on text sentiment (Spanish + English, TextBlob + keyword matching)
  • 🌀 Idle animations (breathing, subtle rotation)
  • 🪟 Floating window, always-on-top, fully configurable size/position
  • 🎨 One-click avatar creator via Ready Player Me web
  • Morph target-based expressions: happy, sad, surprised, angry, thinking, neutral

Overhead: ~15 MB RAM, <1% CPU idle, ~4% CPU when speaking.


🧠 Plugin: MCP Memory Cloud (mcp_memory)

Persistent cloud memory for PyGPT using Firebase Firestore (free Spark tier) exposed as an MCP server.

Features:

  • 💾 Auto-saves conversations to Firestore in realtime
  • 🔍 Semantic search with local embeddings (sentence-transformers/all-MiniLM-L6-v2) — no API key needed
  • 🌐 Multi-device sync — resume context on any machine
  • MCP tools: remember_fact, semantic_recall, summarize_context, sync_devices
  • Supports local / openai / gemini embedding backends
  • Standalone MCP server (mcp-memory-server/server.py) via stdio transport

Free tier Firebase limits (more than enough for personal use):

  • 1 GB storage, 50K reads/day, 20K writes/day

Overhead: ~50 MB RAM (with embeddings loaded), <1% CPU idle.


Files Added

src/pygpt_net/plugin/avatar_rpm/
  __init__.py          # Plugin entry + hooks
  avatar_window.py     # QWidget with WebEngine + Qt WebChannel bridge
  lip_sync.py          # Audio-reactive lip-sync controller
  emotions.py          # Sentiment-based emotion detector
  assets/
    avatar_viewer.html # Three.js viewer with RPM GLB loader

src/pygpt_net/plugin/mcp_memory/
  __init__.py
  plugin.py            # Plugin entry + MCP server lifecycle

mcp-memory-server/
  server.py            # MCP server (stdio transport)
  firebase/client.py   # Firestore CRUD wrapper
  tools/remember.py    # remember_fact tool
  tools/recall.py      # semantic_recall tool (embeddings)
  tools/summarize.py   # summarize_context tool
  tools/sync.py        # sync_devices tool
  requirements.txt
  config.example.json

install.sh             # One-command install script
README.md              # Full documentation

Setup Requirements

# MCP Memory Server
pip install firebase-admin sentence-transformers numpy textblob

# Avatar RPM
pip install textblob numpy
# (WebEngine and Three.js are already included in PyGPT)

Notes

  • Both plugins are opt-in and disabled by default — zero impact on existing users
  • The MCP Memory server runs as a subprocess, isolated from PyGPT core
  • Avatar uses PyGPT's existing QWebEngineView — no new Qt dependencies
  • Firebase free tier is sufficient for personal/single-user usage
  • Fully compatible with existing PyGPT plugin hooks (on_audio_output_start/chunk/end, on_ctx_begin/end)

Developed as a community enhancement. Feedback and improvements welcome!

- Plugin avatar_rpm: Ready Player Me 3D avatar with lip-sync, emotion detection and WebEngine viewer
- Plugin mcp_memory: Firebase Firestore memory cloud via MCP protocol with semantic search
- MCP standalone server with remember/recall/summarize/sync tools
- Install script and full documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

X Tutup