Janas: Run LLMs larger than RAM by streaming experts from SSD

Janas is a new open-source project designed to address the memory limitations often encountered when running large language models (LLMs) on consumer hardware. By implementing a technique that streams model experts directly from an SSD, Janas allows users to execute models that exceed their system's available RAM capacity. This approach significantly lowers the hardware barrier for local AI inference, enabling developers and enthusiasts to experiment with high-parameter models on machines with limited memory. The project leverages efficient data retrieval methods to minimize latency during the streaming process, ensuring that the performance remains viable for various applications. As LLMs continue to grow in size, tools like Janas provide a practical solution for local deployment, reducing the reliance on expensive high-end GPUs or massive amounts of system memory. The project is currently available on GitHub for developers interested in exploring memory-efficient AI execution.