Introduction to Large Language Models

Table of Contents

Course Information

• Instructors:
  • Lecture: Frank Niu;
  • Practice Sessions:
    • Hassan Soliman;
    • Hovhannes Tamoyan.
• Moodle: https://moodle.informatik.tu-darmstadt.de/course/view.php?id=1875

When and Where

Schedule & Class Materials

• Textbooks:

  • [J&M] Jurafsky and Martin, Speech and Language Processing, 3rd edition (2025 draft).
    • Textbook Website.
    • Some chapters are not yet available for the third edition. In this case, we will use the corresponding chapters from the 2nd edition (available online here).

  This book is available online for free, so the physical copy is optional.

  • [Kracht] Markus Kracht, Introduction to Linguistics.
    • We will use the first few chapters of this book to cover some basic linguistics concepts.

Week 1. Introduction

• Date: 14.10.2025

• Readings:

  • [J&M, 2nd Edition] Chapter 1: Introduction.

• Lecture:

  • Lecture Slides: L1 - Introduction.
  • Deep learning refresher: Google Colab.
  • Recording: Video Link

• Practice Sessions:

  • Date: 16.10.2025
  • Materials:
    • Google Colab: Introduction (Python and ML Foundations) and its solutions.
    • Recordings: Video Link

Week 2. CL/NLP Basics

• Date: 21.10.2025
• Readings:
  • [Kracht] Chapter 1: Introduction
  • [J&M] Chapter 2: Words and Tokens
• Lecture:
  • Lecture Slides: L2 - CL/NLP Basics.
  • Recording: Video Link
• Practice Sessions:
  • Date: 28.10.2025
  • Materials:
    • Colab: Practice Session 2 - CL/NLP Basics.
    • Slides: Practice Session 2 - CL/NLP Basics.
    • Recording: Video Link

Week 3. Word Representation I: Basics

• Date: 23.10.2025
• Readings:
  • [J&M] Section 11.1: Information Retrieval
  • [J&M] Chapter 3: N-gram Language Models
• Lecture:
  • Lecture Slides: Word Representation I.
  • Recording: Video Link
• Practice Sessions:
  • Date: 30.10.2025
  • Materials:
    • Colab: Practice Session 3 - Word Representation I.
    • Slides: Practice Session 3 - Word Representation I.
    • Recording: Video Link

Week 4. Word Representation I: Basics (continued)

• Date: 04.11.2025
• Readings:
  • [J&M] Chapter 5: Embeddings
• Lecture:
  • Lecture Slides: Word Representation I
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Slides: Practice Session 4 - Word Representation I (continued)
    • Recording: Video Link

Week 5. Word Representation II: Statistical Methods and “Language Models”

• Date: 11.11.2025
• Readings:
  • [J&M] Chapter 5: Embeddings
• Lecture:
  • Lecture Slides: Word Representation II
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Slides: Practice Session 5 - Word Representation II
    • Recording: Video Link

Week 6. Transformers

• Date: 13.11.2025
• Readings:
  • [J&M] Chapter 8: Transformers
  • [J&M] Chapter 10: Masked Language Models
• Lecture:
  • Lecture Slides: Transformers
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Colab: Practice Session 6 - Transformers: Tokenizers.
    • Colab Solutions: Practice Session 6 - Transformers: Tokenizers (Solutions).
    • Recording: Video Link

Week 7. Natural Language Understanding

• Date: 25.11.2025
• Readings:
  • [J&M] Chapter 20: Information Extraction: Relations, Events, and Time
  • Jacob Devlin talks about BERT: Video Link
• Lecture:
  • Lecture Slides: Natural Language Understanding
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Colab: Transformers 1:.
    • Recording: Video Link

Week 8. Large Language Models

• Date: 02.12.2025
• Readings:
  • [J&M] Chapter 7: Large Language Models
• Lecture:
  • Lecture Slides: Large Language Models
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Colab: Transformers 2:.
    • Recording: Video Link

Week 9. Test Time Scaling I: Prompting

• Date: 09.12.2025
• Readings:
  • [J&M] Section 9.4: Test-time Compute
• Lecture:
  • Lecture Slides: Test Time Scaling I: Prompting
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Colab: Prompting Techniques.
    • Recording: Video Link

Week 10. Test Time Scaling II: Fine-tuning

• Date: 16.12.2025
• Readings:
  • [J&M] Section 9.1-3: Post-training: Instruction Tuning and Alignment
• Lecture:
  • Lecture Slides: Test Time Scaling II: Fine-tuning
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Colab: Interpretability.
    • Recording: Video Link

🎅 Winter Break 🎄

Week 11. Understanding Large Language Models

• Date: 13.01.2026
• Lecture:
  • Lecture Slides: L11 - Understanding Large Language Models
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Colab: Practice Session 11 - Understanding Large Language Models.
    • Recording: Video Link

Week 12. Question Answering

• Date: 20.01.2026
• Readings:
  • [J&M] Chapter 11: Information Retrieval and Retrieval-Augmented Generation
• Lecture:
  • Lecture Slides: L12 - Question Answering
  • Recording: Video Link
• Practice Sessions:
  • Date: 23.10.2025
  • Materials:
    • Slides: Practice Session 12 - Question Answering
    • Recording: Video Link
    • Colab: Practice Session 12 - Question Answering.

Week 13. RAG and Agents

• Date: 27.01.2026
• Readings:
  • [J&M 2nd Edition] Sections 23.2,5: Question Answering
  • [J&M] Sections 11.3-6: Retrieval-Augmented Generation
• Lecture:
  • Lecture Slides: L13 - RAG and Agents
  • Recording: Video Link
• Practice Sessions:
  • Date: 29.01.2025
  • Materials:
    • Slides: Practice Session 13 - RAG and Agents
    • Recording: Video Link
    • Colab: Practice Session 13 - RAG and Agents.

Week 14. Career in NLP & Exam Preparation

• Date: 03.02.2026
• Lecture:
  • Lecture Slides: TBA

Final Exam Information

• Date & Time: February 24, 15:00
• Duration: 100 minutes

Review Materials

You are expected to be familiar with all material covered in the course, in particular:

• The problem set (HW5)
• All homework assignments
• Practice session materials
• Lecture slides

Exam Materials

• You may bring one single-sided A4 cheat sheet to the exam.
• The cheat sheet will be collected at the end of the exam.
• If you would like to keep a copy for your own records, make a copy in advance.
• No calculator is allowed or required for the exam.

Please plan accordingly.

Resources

Corrections

• L2 Slides Page 14: the example sentence should be “Every student takes a course.”
  That sentence has two different interpretations:
  1. For every student, there exists a (potentially different) course that the student takes.
     Every student takes a course. Frank took three courses.
  2. There is a single course that every student takes.
     Every student takes a course. The course is Intro2LLM.

Instructions

• Homework submit steps.

Recordings