Docker, Inc. Makes Invoking LLMs Simpler for Application Developers
Docker Inc. this week added an ability to make it possible for application developers using its tools to build cloud-native applications to run large language models (LLMs) on their local machines.
Available in beta, Docker Model Runner achieves that by embedding an AI inference engine into Docker Desktop using an open source llama.cpp library to make the LLM available locally using the OpenAI application programming interface (API). As a result, no extra tools or setup is required.
In the future, Docker also plans to add Docker Model Runner to the Windows instance of Docker Desktop.
Nikhil Kaul, vice president of product marketing for Docker, Inc., said Docker Model Runner, in effect, brings those LLMs into the inner loop of an individual developer to eliminate the need to make calls to an LLM residing on a server or in the cloud. That ability also serves to reduce the total cost of application development by eliminating the need to rely on an external cloud service when rapidly building prototypes of applications, he added.
Despite the rise of integrated development environments (IDEs) in the cloud, most initial application development continues to be down on laptops, largely because developers prefer the level of interactivity provided. Calling out to external services when prototyping an application adds a lot of unnecessary friction, said Kaul.
There, of course, may be a day when that changes as network bandwidth improves and concerns about the security of software supply chains requires more application development work to be done in a more secure cloud computing environment.
It’s not clear how many developers are incorporating LLMs into applications, but it may now be only a matter of time before every application is infused with some type of AI capability. Most of those next generation applications will, at least for the immediate future, be constructed using containers.
Less clear is to what degree building those applications will require developers to attain additional levels of skills and expertise. Right now, the best thing most application developers should, at the very least, be doing is familiarizing themselves with LLMs to determine how steep that learning curve might be, said Kaul. The most important thing is for developers to get their hands dirty now, because going forward most organizations will be looking to hire developers that have a strong AI foundation, he added.
Ultimately, LLM will soon prove to be disposable in the sense that not only will developers make use of multiple ones in the same application; they will also swap them out as new capabilities are added to them. The overall goal, given the pace of AI innovation, should be to not get locked into a specific LLM.
Developers, of course, will also rely on LLMs to write code, run tests and even deploy their applications but ultimately, they are still responsible for any code they commit and, as a general rule, no developer should be committing any kind of code unless they truly understand how it functions.
