Ask HN: How will LLMs work on an entire codebase?

by gettodachoppa on 2/28/2024, 8:23:15 AM with 1 comments
Like many of you I use ChatGPT for specific questions, completing a function from comments, etc. But I'm reading that LLMs will soon become actual developers.

How can that be? Let's forget about quality, hallucinations, etc. The largest context window from an accessible/affordable LLM is 32k (Mixtral or GPT4). That's barely enough for a TODO app, let alone a real project. The smallest project I work on, a desktop app, has 60k LOC/6M characters/1.5M tokens.

So what changes are coming that would allow an LLM modify an existing codebase, e.g. to modify a feature and write its tests? (without having to spoonfeed it the perfect context the way we do now in ChatGPT)

  • by hiddencost on 2/28/2024, 8:30:37 AM

    Your question is posed as a hypothetical, but the problem is already solved...

    Add a dependency graph of different agents and tools. Use summarization (either selecting subsections or rewriting). Give it a scratch space. Use RAG.

    Why would it need to load the whole code base into memory? We can build very complex architectures on top of this task that mix LLMs with software.

    https://arxiv.org/abs/2402.09171

    This isn't hypothetical; all of these