congrats on launching! i think my continuing struggle with looking at Ragas as a company/library rather than a very successful mental model is that the core of it is like 8 metrics (https://github.com/explodinggradients/ragas/tree/main/src/ra...) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever.

why is Ragas a library and a company, rather than an overall "standard" or philosophy (eg like Heroku's 12 Factor Apps) that could maybe be more universally adopted without using the library?

(just giving an opp to pitch some underappreciated benefits of using this library)

Based on our initial analysis with RAGAS a few months ago, it didn't provide the results that our team was expecting. Required a lot of customisation on top of it. Nevertheless a pretty solid library.

Also check out DeepEval... our team has been using it for a while, and it's been working well for us because we can evaluate any LLMs, something this library doesn't seem to support (https://github.com/confident-ai/deepeval).

This is nice, we've got more Open Source LLM Evaluation Libraries coming in more often.

We're using DeepEval (https://github.com/confident-ai/deepeval) currently. How is this different from that?

Great product and great progress.

The first step to build rage is always to evaluate.

Except all the current evaluations, cost and perf should also be part of evaluations

Congratulations on the launch! Personally would love to see a rough estimates of the expected number of requests and tokens required to run tasks like synthetic data generation for different amounts of data. Though this is likely highly variable, would like to have a loose idea of possible incurred costs and execution time.

Congratulations on the launch of Ragas! This looks like an incredibly valuable tool for the LLM community. As the library continues to evolve, it will be interesting to see how it adapts to handle the growing diversity of LLM architectures and use cases.

Congratulations on the launch! I was unable to use this library: I was trying to evaluate different non-OpenAI models and it consistently failed due to malformed JSONs coming from the model.

Any thoughts about using different models? Is this just a langchain limitation?

Phenomenal to see how Ragas has progressed. Congratulations on the launch