Langchain: Weighing the Realities of Implementing Open-Source LLMs in Production

Langchain: Weighing the Realities of Implementing Open-Source LLMs in Production

Nov 16, 2023·

4 min read

It's a fact that there are thousands of open-source frameworks for LLM right now, and the most prominent one is Langchain.

If you're reading this article right now, there's a pretty good chance you know about Langchain, and probably already had hands-on experience with it.

But for those who are still exploring: think of it as a starter kit, it gives you the basic tools and a sandbox to start creating something cool. But, and this is a big 'but', turning what you get from Langchain into something that can stand up to the demands of the real world – that's where the challenge lies.

Turning Langchain's foundation into something that doesn't just work, but works wonders in the real world, is a marathon, not a sprint. It's an exciting journey, no doubt, but one that needs more than just enthusiasm – it needs preparation, skill, and a whole lot of elbow grease.

By no means I'm bashing Langchain. I LOVE Langchain. Full disclaimer, Vext, the platform I'm building started from Langchain, and it was extremely beneficial since it got a lot of things figured out for us, but as we dive deeper, there are many challenges that we have to resolve in terms of technical stability and scalability.

Documentation Dilemmas: Langchain's Knowledge Gaps

From my experience, one of the key hurdles when working with Langchain, which can't be overstated, is the state of its documentation. Even with a robust and active community behind it, the platform falls short in providing documentation that's easy to navigate or thorough. For those who are venturing into the intricate world of LLM for the first time, this gap can be more than just a minor inconvenience; it can be a significant stumbling block.

If you're a startup or if you're tight on time and resources, commodity and agility are what matters. The teams in such environments are often running on tight schedules and have limited bandwidth to spare on deciphering cryptic documentation. They need resources that can help them move swiftly from concept to production, ensuring their journey with LLMs is smooth and efficient.

Addressing the Out-of-the-Box Limitations of Langchain

Here are some of the things to look out for if you're looking into implementing Langchain for your application. We've been there, done that, have the scar tissues, and survived to tell the tale.

Citations and Source Transparency: Where Did That Answer Come From?

Langchain allows quick prototyping and showcasing of AI capabilities, but it lacks built-in features for showing where responses come from. This is crucial when you need to explain the origin and logic behind an AI's response, especially in professional or critical settings.

Vext: we offer log/history for better visibility

This is also one of the reasons why we built Vext, to give users the ability to configure their LLM while having the visibility of what the sources the LLM is referring to (vs your imported data).

The Economy of Scale Challenge: Cost of DIY Solutions

Using Langchain means bearing the full cost of development, maintenance, and updates, a significant consideration against the backdrop of continuously evolving AI technologies and the economy of scale benefits offered by larger platforms like OpenAI's API.

Down the road you will have to think about how to auto-scale, what resources are required to maintain optimal experiences, etc.

Deployment Challenges: Taking Langchain to the Cloud

One of the primary hurdles is ensuring that the deployment aligns seamlessly with cloud infrastructure, which often requires a deep understanding of cloud services and their limitations. This is particularly tricky when dealing with Langchain, as it might demand specific configurations and resources to function optimally.

Additionally, the integration of Langchain with other cloud-based services and APIs can introduce compatibility issues, requiring meticulous coordination and testing to ensure that all components work harmoniously together. This integration is vital for achieving a smooth, efficient system that can handle real-world demands.

This is also one of the reasons why we started Vext, just offering a SaaS that figures out everything for users, not having to worry about cloud environment, deployment, and maintenance is a huge relief on top of millions of other tasks we developers have to worry about.

Final Note

Langchain serves as a great starting point and is highly valuable as a learning resource, but it's important to recognize that it's not tailored for deploying real-world applications into production.

The situation is akin to the difference between a hobbyist assembling a server from parts purchased at a store like Best Buy and a professional IT setup. While it can be enjoyable and educational to piece together your own Retrieval Augmented Generation (RAG) pipeline, this approach is not ideal for serious, production-level applications.

But we're hoping and contribute to the community, hoping it can truly become a production-ready, stable, and scalable framework for sophisticated LLM applications.