The MOSS AI research team, a Shanghai-based group with origins in the OpenMOSS research team at Fudan University and the Shanghai Innovation Institute, held an “Ask Us Anything” (AMA) session on Reddit to discuss its expanding ecosystem of multimodal models. The session, which took place on October 30, focused on the team’s diverse portfolio spanning speech, audio, image, and video technologies.

Multimodal Model Portfolio

The team highlighted several key developments within the MOSS model family. A significant portion of the discussion centered on MOSS-Image, a text-to-image model utilizing a 5B Diffusion Transformer (DiT) backbone. According to the team’s documentation, the model supports Chinese and English text rendering and features JSON layout control. The model’s ComfyUI version has reportedly reached 3,702,490 downloads, and the base and SFT weights are available under the Apache 2.0 license.

In the realm of audio and speech, the team detailed several specialized models:

  • MOSS-TTS and MOSS-TTSD: Focused on speech synthesis, voice cloning, and long-form dialogue or podcast generation.
  • MOSS-TTS-Realtime and MOSS-TTS-Nano: Designed for streaming speech generation and CPU-friendly inference.
  • MOSS-VoiceGenerator and MOSS-SoundEffect: Capable of generating voices from text descriptions and sound effects from text prompts.

The team also presented its audio understanding capabilities through MOSS-Audio and MOSS-Music, which are designed to reason about speech, environmental sounds, and musical structures, respectively.

Visual and Video Capabilities

The MOSS ecosystem extends into visual understanding and synchronized generation. The team noted that MOSS-VL is designed for understanding images and long-form video, while MOSS-VL-Realtime aims to process streaming video as it unfolds. Additionally, the MOVA model is intended to generate video and audio in synchronization.

The stated research objective of the MOSS team is to develop AI capable of real-time environmental awareness, specifically regarding speech, visual input, and temporal changes in a given situation.

Community Feedback and Technical Inquiries

During the Reddit AMA, users raised several technical questions regarding the practical application of these models. Some users expressed interest in the future of local AI on consumer-grade hardware, while others inquired about the feasibility of training the MOSS-TTS models for languages not currently supported by the team.

Technical discussions also touched upon the training of LoRA (Low-Rank Adaptation) for MOSS-Image, with some users reporting difficulties in achieving consistent results for character-specific training compared to other models. While some community members on the platform suggested that MOSS-Image could serve as a successor to older models like Stable Diffusion 1.5, such assessments remain unverified user opinions and lack independent technical confirmation.

The team’s research and tools are accessible via their official website and various open-source repositories on GitHub and Hugging Face.