Ask HN: Where do I begin studying to run LLMs locally or in a private cloud?

by bulldog13 on 5/3/2023, 8:53:40 PM with 2 comments
Where do I begin studying if my goal is to run LLMs locally or in a private cloud with solid software engineering practices? I am unsure of where to even begin.

Let's say I want to:

0.) Find the locally run LLMs and identify which are applicable. 1.) Containerize the LLMs 2.) Use source control to capture changes to the LLM. Versioning output. 3.) Develop repeatable pipelines driven by APIs for sending data to it. 4.) Prompt engineering. 5.) Best ways to use langchain (or others) to make the system data-aware and agentic.

Any thoughts are appreciated.

  • by smoldesu on 5/3/2023, 9:02:07 PM

    A lot of it will revolve around Nvidia hardware that you either own or rent. I've built CPU-accelerated AI bots on free VPSes before, but it's slow and not a reflection of best-practices nor state-of-the-art inferencing. Right now, a lot of the meaningful "private cloud" AI stuff is built with extremely proprietary runtimes.

  • by ftxbro on 5/3/2023, 8:54:23 PM

    ask GPT-4