Introducing PolyStrata: An Advanced Multi-Model Strata Engine Featuring Four Additional MoE Models
In the evolving landscape of AI deployment on consumer hardware, optimizing performance while maintaining flexibility remains paramount. Today, we delve into PolyStrata—a sophisticated extension of the open-source Strata engine—designed to seamlessly integrate multiple mixture-of-experts (MoE) models and deliver impressive inference speeds on commonplace hardware setups.
Overview of PolyStrata
PolyStrata builds upon the foundation established by the original Strata engine, which supports running models like Qwen3.8-Flash-Next efficiently on consumer-grade graphics cards. This new iteration introduces support for four additional MoE models, each accessed via their standard GGUF format, expanding the engine’s versatility.
The integrated models include:
– GLM 5.3 Flash
– Qwen3.6-35B-A3B
– Ornith 1.5
– Gemma 4 26B-A4B
These models not only read text but also process images, broadening the scope of applications.
Platform Support and Compatibility
A recent update marks significant strides in platform support:
– AMD Support (Beta): Thanks to contributions from iMilnb on GitHub, PolyStrata now offers beta support for AMD GPUs via HIP. Although currently limited to Linux, this adaptation has been tested on an RX 7600 8 GB card, achieving performance metrics of approximately 64 tokens per second for Qwen3.6 UD-Q2_K_XL and 35 tokens per second for Ornith Mini.
– Windows Enhancements: The latest version (v0.1.42-poly) eliminates the need for Visual Studio or CUDA toolkit installation for Windows users. An all-in-one installer now downloads prebuilt Strata engines tailored for NVIDIA RTX series cards (from RTX 20 to RTX 50), simplifying setup.
Performance Benchmarks
PolyStrata’s performance metrics are encouraging:
On a Lenovo Legion Laptop with an RTX 4070 8 GB & Intel i7-14700HX & 32 GB RAM:
– Qwen3.6-35B-A3B: 70-72 tokens (short chat) / 60-62 tokens (after a 31,000-token prompt) / 1,390 tokens read per second
– Gemma 4 26B-A4B: 70-71 tokens / 51-53 tokens / 1,907 tokens read/sec
– Ornith 1.5 APEX-I-Compact: 70 tokens / 59 tokens / 1,340 tokens read/sec
On a high-end workstation with an RTX 5090 32 GB & 120 GB RAM:
– Qwen3.6-35B-A3B: 332-334 tokens / 186-189 tokens / 5,520-5,580 tokens read/sec
– Gemma 4 26B-A4B: 306 tokens / 198-200 tokens / 7,170-7,210 tokens read/sec
– Ornith 1.5 APEX-I-Compact: 267-272 tokens / 145-149 tokens / 5,700-5,890 tokens read/sec
Notably, models like GLM 5.3 Flash, which demand over 79 GB of storage, require around 27-31 GB of RAM in their larger configurations. The engine intelligently manages experts, prioritizing the most-used to optimize VRAM utilization, with CPU handles for the remaining computations.
Technical Highlights
PolyStrata employs a unique inference technique:
– The GPU retains only the most frequently used experts, while the CPU computes the less active ones directly from the GGUF files.
– During inference, the engine pre-emptively predicts subsequent tokens, checking its drafts in a single pass, thus accelerating response times.
– This hybrid approach allows large models to operate efficiently on hardware with limited VRAM, without sacrificing output quality.
Ease of Use and Deployment
This release includes:
– A one-click installer compatible with Windows and Linux
– A web-based interface for interactive use
– An API compatible with OpenAI and Anthropic, hosted locally for safe testing and deployment
Limitations and Considerations
The additional MoE models are still experimental and require an NVIDIA GPU for optimal performance—tests have been conducted primarily on the specified hardware. AMD support is available in beta, contingent on Linux and HIP-compatible hardware.
Because of the complexity involved in building these models, users will need to have Visual Studio and CUDA toolkit installed on Windows during setup, although future releases aim to streamline this process further.
Getting Started
The project’s codebase is open-source and can be accessed via GitHub: https://github.com/VecSzn/PolyStrata. Whether you’re experimenting with AI on consumer hardware or deploying production-grade solutions, PolyStrata presents a flexible, high-performance toolkit for advanced AI inference.
Conclusion
PolyStrata signifies a meaningful step forward in democratizing access to large, multi-model AI solutions. Its ability to run demanding models efficiently on existing consumer hardware opens new horizons for AI developers, researchers, and enthusiasts alike.
For more details, performance metrics, and deployment instructions, visit the repository or follow the project’s updates.
