LLM Zoomcamp

We're talking about using tracking to get insights user tracing using Snowplow and Vercel AI

https://www.youtube.com/watch?v=A2nMgZWyza8

Watch now or later in recording

In the course we use the FAQ dataset as the main running example

In this article I describe how it's curated and how the FAQ assistant you see in Slack works

https://alexeyondata.substack.com/p/rebuilding-a-faq-system-for-datatalksclub

Also, if you'd rather watch me explain it than read the article, we'll have a live stream about it on YouTube

https://luma.com/fb91wje9

See you soon!

I hope you're having fun with your projects!

We have prepared the homework for the dlt workshop as well as the monitoring module

And they are both about monitoring and observability, and in both you will learn something new in addition to what we covered in the course:

- for the monitoring homework we'll learn about OTel and instrument the RAG assistant with OTel collectors
- for the dlt hub homework, we'll use Pydantic Logfire and see how we can use dlt to ingest the traces back from Logfire for our local analysis

Have fun!

It's time to start working on your final project.

Check out https://github.com/DataTalksClub/llm-zoomcamp/tree/main/07-project-example to see an example of a complete project as a reference.

It covers:

• Intro: Generating data, setting up the project, initial RAG flow
• Evaluating Retrieval: Ground truth data, Hit Rate, MRR, boosting
• Evaluating RAG: LLM-as-a-Judge, comparing models
• Interface and Ingestion: Flask API, ingestion pipeline, project structure
• Monitoring and Containerization: Docker Compose, PostgreSQL logging, Grafana
• Summary: Final result, cost analysis, tips for your project
• Chunking for Longer Texts: Different chunking strategies for articles, transcripts, slides

For more information about the project guidelines and peer review, read the https://github.com/DataTalksClub/llm-zoomcamp/blob/main/project.md.

You also asked us to postpone the deadline for https://github.com/DataTalksClub/llm-zoomcamp/blob/main/cohorts/2026/workshops/dlt.md. You'll have one more week to complete it.

We are hosting a live workshop on tracking and personalizing AI agents.

📅 Tuesday, July 21
🕟 16:30 CEST

Agent applications usually use the current conversation as context. But users also interact with the rest of the product. They open pages, change filters, compare options, and complete different steps before asking the agent a question.

In this workshop, we will show how to capture that behavioral data and pass it back into an AI agent.

We will use a travel chatbot as an example and instrument it across 3 layers:

• Client-side user activity
• Server-side application events
• Agent actions and decisions

You will learn how to:

• Set up a pipeline
• Validate events against schemas
• Track activity across the application
• Compute live session attributes
• Add behavioral context to the agent's system prompt

Register here: https://luma.com/vz9iv4f2

On Tuesday, July 21, we have a workshop on tracking and personalizing AI agents with Snowplow and the Vercel AI SDK.

Your agent can see the chat history but not what the user is doing in the product. If you want the agent to respond based on the user's current session, capture that context and make it available to the model.

This workshop shows how to do that with a Next.js travel chatbot.

The approach:

1. Track client behavior (page views, clicks, filters)
2. Track server-side events (purchases, bookings)
3. Track agent decisions (tool calls, outcomes)
4. Validate events against schemas in real time
5. Compute session attributes with Snowplow Signals
6. Inject those attributes into the system prompt

You'll set up the pipeline, add tracking at each layer, and use live session data to shape the agent's responses.

Tuesday, July 21
4:30 PM - 6:00 PM GMT+2
YouTube

Register: https://luma.com/vz9iv4f2

This week in LLM Zoomcamp: Module 5, Monitoring.

We build:

- A Streamlit chat app with RAG
- Metric capture for LLM calls and cost
- PostgreSQL storage for conversations
- User feedback with thumbs up and thumbs down
- Automatic relevance evaluation with a built-in judge
- Streamlit and Grafana dashboards
- A Docker Compose setup for running everything together

You’ll see what your LLM application is doing after it is deployed.

You need conversations, feedback, cost, relevance, and dashboards. Otherwise it is hard to know what to improve next.

Materials: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/05-monitoring

We will have office hours today with Will from Kestra at 18:00 CET

We will share the link here 5-10 minutes before the start

If you want to get a reminder, you can use this link https://luma.com/d19r4ko1

You can ask questions in advance using this link

https://app.sli.do/event/pH1w9PSq8dJZahTia24SKe

We're starting in a few minites

https://www.youtube.com/watch?v=rG2YW3YCq64

Join now or watch later in recording

We're starting in 3 minutes

https://www.youtube.com/watch?v=A0LmmZf-ggM

Join now or watch later in recording

It's time to learn from your peers!

If you submitted your project as attempt 2, you will find your peer review assignments here

https://courses.datatalks.club/llm-zoomcamp-2025/project/project2/eval

Remember that this step is mandatory for passing the project

Enjoy learning!

If you submitted your project for attempt 3 now it's time to evaluate your peers and learn from their submissions!

You will find your assignments here

https://courses.datatalks.club/llm-zoomcamp-2025/project/project3/eval

Have fun!

There was a case of self-plagiarism. Someone posted the same project they already submitted and got points for.

This person will lose all the points but we will still give them a certificate at the end. But in future cases, self-plagiarism will result in disqualification.

It's time to learn from your peers!

We have closed the form for submitting your attempt 1 and opened the form for submitting attempt 2

If you took part in the first attempt, you can find your peer review assignments here

https://courses.datatalks.club/llm-zoomcamp-2025/project/project1/eval

Remember that the evaluation part is mandatory for passing the project attempt

We've received multiple requests to extend the deadline for project attempt 2

We will not do it because it would delay everyone who's already submitted their project

However, we opened attempt 3: https://courses.datatalks.club/llm-zoomcamp-2025/project/project3

If you're behind and still want to take part in the course, this is your chance

We will generate certificates after attempt 2 is over, and then one more time after attempt 3 is over

We scored project attempt 1

Congratulations to 53 people who passed the project!

We will publish certificates in a couple of weeks when attempt 2 is over

In the meantime, please make sure your certificate name is correct here:

https://courses.datatalks.club/llm-zoomcamp-2025/enrollment

(You need to be logged in)

If you didn't pass, you still have a chance to work on your project and submit the improved version for attempt 2 or 3

Is everyone ready with project attempt 1? We will close the form tonight

If you haven't made the submission yet, you still have a few hours

It's time to move to our projects!

You can find more information about them here:


https://github.com/DataTalksClub/llm-zoomcamp/blob/main/project.md (check the videos!)

https://github.com/DataTalksClub/llm-zoomcamp/blob/main/cohorts/2025/project.md

Submit your work here:

https://courses.datatalks.club/llm-zoomcamp-2025/project/project1

Regarding the monitoring module: we won't have homework and once the remaining videos are published, we will let you know

How is it going with your projects? Everyone is on track?

How is it going? Did you start working on the projects?

Small update for the monitoring module: it's still work in progress, but we hope to finish it in the coming week

We have published 3 videos for the monitoring module:

https://github.com/DataTalksClub/llm-zoomcamp/tree/main/04-monitoring

It's still work in progress, so there will be more videos

A few updates:

- We updated the deadlines for evaluation and agents homework on the platform
- As we're preparing an update for the Monitoring module, you can check the content from the last year: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/cohorts/2024/04-monitoring (starting from 4.6)
- The form for submitting project 1 is already open. The deadline is August 25 EOD. You can read more about projects here: https://github.com/DataTalksClub/llm-zoomcamp/blob/main/project.md
- Here is a project walk-though from the last year: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/06-project-example
- You can see the projects from the last year here: https://courses.datatalks.club/llm-zoomcamp-2024/projects
- Also, check the best practices module: https://github.com/DataTalksClub/llm-zoomcamp/tree/main/05-best-practices. We haven't updated the videos, but the materials are still useful

Also found this video - https://www.loom.com/share/1dd375ec4b0d458fabdfc2b841089031
If you're struggling with the evaluation module, it may help!

Stream: https://www.youtube.com/watch?v=yS_hwnJusDk

We will start a part 2 of the Agents module in 30 minutes

Some useful links:

- https://github.com/alexeygrigorev/rag-agents-workshop
- https://github.com/alexeygrigorev/toyaikit

If you miss it, don't worry, you can watch it later in recording

See you soon

We have to postpone module 4 on monitoring - it should be ready by August 11. Apologies for the inconvenience

In the meantime you can

- watch the dlt workshop if you haven't
- do the agents and the evaluation modules
- check the best practices module. We may update it slightly but you can already watch the materials
- start working on your project - you can already do everything except the monitoring part