Introduction to iterating on LLMs
Weave set up - 3 magic lines
Integrations
Track LLM inputs & outputs
Track data flows and app metadata
Models
Datasets
Evaluation
Getting started with Evaluations
Example Filter & Output Comparison
User defined metrics: one vs many
Objects
Tracing
Feedback
Tutorial: Build an Evaluation pipeline
Tutorial: Model-Based Evaluation of RAG applications
Course assignment
Next steps
At the end of this course you will be able to:
Log and debug all LLM inputs, outputs, and traces so you can easily examine and debug failures and understand how data flows through your application.
Create a system of record for your LLM apps.
Build rigorous evaluations of LLM use cases to score any aspect of your app.
Organize information generated across the LLM workflow, from experimentation to evaluations to production.
Evaluate and analyze LLM performance and visually compare model results across different performance dimensions.
Use human feedback such as emojis and text notes to compile evaluation datasets, monitor performance, and collect examples for fine-tuning.
This course covers everything you need to get started with prompt engineering, from system prompts and structural techniques to model-specific strategies. Using a text-understanding use case, you will gain experience with code and industry knowledge.
Free
Explore the architecture, training techniques, and fine-tuning methods for creating powerful LLMs. Gain theory and hands-on experience from Jonathan Frankle (MosaicML), and other industry leaders, and learn cutting-edge techniques like LoRA and RLHF.
Free
Learn how to build LLM-powered applications using LLM APIs, Langchain and Weights & Biases LLM tooling. This course will guide you through the entire process of designing, experimenting, and evaluating LLM-based apps.
Free
