by PaulHoule on 4/23/2025, 2:56:17 PM
If it's hard for you than it's hard for your customers and they have a reason to pay for your product.
After I left a job where I developed a neural search engine for patents (years before BERT) I talked with many of the vendors in the enterprise search and what I found was that few of them did systematic work to improve the relevance of their results [1] and few of them tried to sell their product based on the quality of the results.
What they all promoted was ease of integration with hundreds of data sources, security, privacy, scale, rapid sync, etc. Looking at the way these got sold, I'd say that all of that is the core work and the actual search engine is an afterthought.
[1] See https://trec.nist.gov/
by barrenko on 4/23/2025, 2:46:24 PM
I'd say this is normal? There may be some solutions popping up, but I haven't been drinking straight from X.com AI/ML firehose lately so I don't know of one unisolution at the moment.
I’ve been working on an AI agent system over the past year that connects to internal company tools like Slack, GitHub, Notion, etc, to help investigate production incidents. The agent needs context, so we built a system that ingests this data, processes it, and builds a structured knowledge graph (kind of a mix of RAG and GraphRAG).
What we didn’t expect was just how much infra work that would require.
We ended up:
- Using LlamaIndex's OS abstractions for chunking, embedding and retrieval.
- Adopting Chroma as the vector store.
- Writing custom integrations for Slack/GitHub/Notion. We used LlamaHub here for the actual querying, although some parts were a bit unmaintained and we had to fork + fix. We could’ve used Nango or Airbyte tbh but eventually didn't do that.
- Building an auto-refresh pipeline to sync data every few hours and do diffs based on timestamps. This was pretty hard as well.
- Handling security and privacy (most customers needed to keep data in their own environments).
- Handling scale - some orgs had hundreds of thousands of documents across different tools.
It became clear we were spending a lot more time on data infrastructure than on the actual agent logic. I think it might be ok for a company that interacts with customers' data, but definitely we felt like we were dealing with a lot of non-core work.
So I’m curious: for folks building LLM apps that connect to company systems, how are you approaching this? Are you building it all from scratch too? Using open-source tools? Is there something obvious we’re missing?
Would really appreciate hearing how others are tackling this part of the stack.