AI_dev Europe 2024

The pace of innovation for open source LLMs is exciting—new models are being released and fine-tuned on a daily basis. But when it comes to selecting the right LLM to solve real-world problems, the hype and open leaderboards can be misleading. Before you make significant investments in LLM infrastructure, building new GenAI pipelines, and fine-tuning, we'll walk you through the process of evaluating and selecting the best LLMs for your use case. Topics covered in the technical workshop will include: - How the open leaderboards work today: popular benchmarks and methodologies for NLP model evaluation - Dimensions to evaluate LLMs, and which measures are most important based on your use case - Why auto evaluators are not enough, and ultimately human supervision based on ground truth data is the best indicator of quality - How to efficiently apply human supervision to LLM evaluation: open source toolchain and process - Different approaches to curating test data for deterministic vs generative AI Attendees will walk away with actionable steps, referenced reports, and open source tools to evaluate LLMs for their business applications.