How we complete cut-off LLM responses and why stitching them is harder than it seems

Developers of a service integrating multiple language models in a single chat encountered issues with responses cutting off during long list generations. When a model hits its token limit, it stops mid-word without explicit errors. The article details the process of completing these responses using the finish_reason parameter. The authors share their experience in overcoming technical challenges related to the 'seamless' stitching of text fragments. The focus is on three key traps engineers face when trying to combine response parts to avoid visual artifacts and context loss. A solution is presented in the form of a specialized function that correctly handles interrupted generations, ensuring a coherent and high-quality result for the user. This experience is valuable for developers working with LLM APIs who aim to improve the stability of their AI applications.