CSE 164 / EECS 264: Introduction to Intelligent Vehicles and Autonomous Driving
UC Merced · Prof. Ross Greer
A conjoined undergraduate + graduate course on the sensing, learning, planning, and human-centered design of self-driving systems — grounded in public datasets, simulation, and real research literature.
About this course
Introduces intelligent vehicles and autonomous driving systems, emphasizing multimodal perception, prediction, and motion planning under real-world constraints — both for the external driving scene and the in-cabin observation of drivers and passengers.
You'll work with camera, lidar, radar, and audio sensing; learning-based scene understanding; behavior and trajectory forecasting; the fundamentals of localization, mapping, and control; and in-cabin monitoring and interaction. Evaluation is grounded in public driving datasets and simulation, with a running theme of safety, human interaction, ethics, and responsible deployment.
Conjoined UG / graduate offering
CSE 164 and EECS 264 share the same lectures and topic structure. Where the versions differ:
- Undergraduate (CSE 164): paired paper presentations, team-based project reimplementing or adapting existing methods with clear experimental validation, mastery of established techniques.
- Graduate (EECS 264): individual paper presentations with critical evaluation of assumptions and limitations; individual or team project requiring a novel or extended technical contribution beyond existing baselines, culminating in a final presentation or demo.
Prerequisites
Required: CSE 100 and MATH 024.
Strongly recommended: linear algebra, probability/statistics, Python, and machine learning fundamentals. If you're rusty on any of these, that's okay — come to office hours and we'll point you to refreshers.
Course staff
How to reach us: use the course discussion forum for course questions so the whole class benefits. Email is for private matters (grading concerns, accommodations, personal circumstances).
Topic outline
Thirteen units span sensing through responsible deployment. Each unit pairs lecture material with a small set of foundational readings.
| Unit | Theme | Sample readings |
|---|---|---|
| 1 | Overview of Autonomous Driving and Safety Autonomy levels, ODD, autonomy stack, evaluation, human–machine interaction. |
Bengler et al. (2014); Trivedi et al. (2007); Urmson et al. (2008) |
| 2 | Sensors, Calibration, and Data Foundations Camera / lidar / radar / audio; frames; synchronization; calibration; data artifacts. |
KITTI (Geiger et al., 2012); nuScenes (Caesar et al., 2020); Argoverse 2 (Wilson et al., 2022) |
| 3 | Perception Preliminaries (Vision + Audio + Multimodal) Image formation; frequency and audio representations (STFT); multimodal features. |
Gonzalez & Woods, Digital Image Processing; Dubnov & Greer, Machine Learning in Music & Audio |
| 4 | Detection, Segmentation, Tracking 2D object detection, semantic segmentation, tracking; sensor fusion. |
PointPainting (Vora et al., 2020); HOTA (Luiten et al., 2021) |
| 5 | 3D Perception and BEV Representations 3D object detection and tracking; birds-eye-view maps; HD maps; BEV networks. |
CenterPoint (Yin et al., 2021); Zhu et al. (2023); BEVFormer (Li et al., 2024) |
| 6 | State Estimation and Mapping Ego-vehicle localization and state estimation. |
Thrun, Burgard & Fox, Probabilistic Robotics; Barfoot, State Estimation for Robotics; Reid et al. (2019) |
| 7 | Prediction and Behavior Modeling Motion of surrounding vehicles and pedestrians, including intent prediction. |
Cui et al. (2019); Deo & Trivedi (CVPRW 2018, T-IV 2018) |
| 8 | Planning and Control Motion planning, control inputs, feedback systems. |
Paden et al. (2016); Gerdes (2021); Liu et al. (2022) |
| 9 | Evaluation and Simulation Open- vs. closed-loop evaluation; datasets; practical validation. |
CARLA (Dosovitskiy et al., 2017); NAVSIM (Dauner et al., 2024); nuPlan (Caesar et al., 2021) |
| 10 | Vision-Language Models for Navigation and Driving VLMs, VLA models, vision-language navigation. |
CLIP (Radford et al., 2021); DriveLM (Huang et al., 2023); Alpamayo-R1 (Wang et al., 2025) |
| 11 | Human-Centered Autonomy In-cabin monitoring, driver attention, control handoff, takeover readiness, ethics. |
Ohn-Bar & Trivedi (2016); Martin et al. (2019); Deo et al. (2019) |
| 12 | Robustness, Safety, and Security Probabilistic vs. deterministic systems, generalization, active/passive safety, adversarial attacks. |
Amodei et al., Concrete Problems in AI Safety (2016); Sinha et al. (RSS 2024) |
| 13 | Ethics and Responsible Deployment Algorithmic and model bias; deployment history and case studies; accessibility, privacy, decision-making, explainability, and trust. |
Koopman & Wagner (2016); Selbst et al. (FAccT 2019); DeVries et al. (CVPRW 2019); Awad et al., Moral Machine (Nature, 2018) |
How you'll be graded
Grad students (EECS 264) are additionally required to give a presentation or demo of their project.
Grade scale
An A+ category, reserved for exceptional performance, is determined at the end of the semester.
Semester project
The heart of the course. Each team designs and executes an empirical study using real autonomous-driving data or simulation. The project runs the length of the semester — introduced in the early weeks, refined through a milestone check-in, and delivered as a final report.
What you'll do
- Hypothesis: submit a written proposal with a testable question (e.g., how a particular method or model choice affects perception or driving performance).
- Experimental design: pick datasets or simulators, define evaluation metrics, and justify controls and comparisons.
- Data & analysis: run models on public datasets (nuScenes, Argoverse, CARLA, nuPlan) or collect/annotate your own data; compute quantitative results (detection accuracy, trajectory error, failure rates).
- Interpretation: analyze uncertainty, domain shift, and failure cases.
- Revision: refine methods after the milestone review.
Undergraduate vs. graduate expectations
- Undergraduate (CSE 164): team-based project reimplementing or adapting existing methods, with correct frameworks, experimental validation, and clear documentation of task performance.
- Graduate (EECS 264): individual or team-based project requiring a novel or extended technical contribution beyond existing baselines — literature positioning, controlled experiments, and discussion of limitations. Suitable for continued research. Final presentation or demo required.
Ethical impact section
Every project must include a required Ethical Impact Section analyzing potential bias in your data or models, safety and failure modes, privacy and human-interaction implications, and consequences of real-world deployment.
Paper presentations
Each student (or pair, for undergraduates) presents one peer-reviewed paper from the reading list.
- State the research question or hypothesis, the experimental design, the evaluation methodology, and sources of uncertainty.
- Lead the class discussion — be prepared to defend or critique the paper's scientific validity.
- Graduate presenters present individually and are expected to include critical evaluation of assumptions, limitations, and research implications.
Exams
Midterm
- In-class, in-person.
- Focuses on interpreting experimental results, comparing models on evidence, and reasoning about uncertainty and robustness.
- Graduate students receive additional questions emphasizing interpretation, abstraction, and research-oriented reasoning.
Final exam
- In-person, no alternate times.
- If you have a known conflict (religious observance, official UC event), report it before the midterm.
Course learning outcomes
- Systems thinking. Explain the autonomous driving stack (sensing → perception → prediction → planning/control) and evaluate design tradeoffs across components.
- Implementation. Implement, assess, and present core perception, forecasting, or planning components using public datasets and standard evaluation metrics.
- Multimodal representations. Understand, apply, critically assess, and communicate multimodal learning concepts — fusion across vision, lidar, radar, and audio, and representation learning for spatiotemporal data.
- Classical + data-driven models. Understand, implement, and critically evaluate classical and data-driven models for autonomous driving tasks.
- Failure & safety analysis. Analyze failure modes, dataset bias, safety constraints, and human-interaction considerations relevant to intelligent vehicles.
Policies
Academic integrity
Academic honesty is taken very seriously at UC Merced. Talk to each other about concepts — that's how you learn. But when it's time to turn in work, write it yourself.
Not allowed:
- Sharing or receiving solutions before, during, or after the semester.
- Submitting a friend's work.
- Examining someone else's solution for "ideas."
- Submitting ChatGPT / GitHub Copilot output as your own solution.
- Posting course solutions or materials publicly.
- Code obfuscation to dodge detection.
Penalties: first offense → the negative of the assignment's maximum points (and it cannot be dropped). Second offense → automatic F in the course. All cases are reported to the Office of Student Rights and Responsibilities.
Participation
Participation (10%) rewards active engagement in the discussion forum, in-class discussion, and project check-ins — not just showing up. Good participation looks like:
- Public forum posts (private when posting unreleased solutions).
- Searching before posting.
- Asking specific, detailed questions.
- Reading and responding to peers.
Inclusion & classroom climate
You belong here. We welcome students from every background, identity, and path. If your name or pronouns differ from official records, let us know — we'll use what you prefer. If a religious holiday conflicts with an exam, tell us early and we'll work it out.
If something about the classroom environment isn't working for you, bring it to us directly or by email. We can't fix what we don't hear about.
Extenuating circumstances
Circumstances outside your control — health crises, family emergencies, technical disasters — can qualify for extra support. Tell us early. The more lead time we have, the more options we have.
Campus resources
- Counseling and Psychological Services — mental health support.
- Office for Prevention of Harassment and Discrimination — for survivors of sexual violence.
- 988 Suicide & Crisis Lifeline — call or text 988.
- Technology Resources Program — laptop and technology support.
- Dean of Students — (209) 228-3633.
- Student Accessibility Services — (209) 228-6996, access@ucmerced.edu.
Resources
No official textbook. The readings below and the per-unit paper lists in the topic outline provide everything you need.
Reference books
- Probabilistic Robotics — Thrun, Burgard & Fox
- State Estimation for Robotics — Barfoot
- Digital Image Processing — Gonzalez & Woods
- Deep and Shallow: Machine Learning in Music and Audio — Dubnov & Greer
Datasets & simulators
- KITTI, nuScenes, Argoverse 2
- CARLA, NAVSIM, nuPlan
Tools
- Python, PyTorch, Jupyter
- ROS 2, OpenCV
- librosa, torchaudio
FAQ
Click a question to expand.
Should I take CSE 164 or EECS 264?
CSE 164 is the undergraduate offering; EECS 264 is the conjoined graduate version. Same lectures, differentiated expectations on presentations, project, and exams. Graduate students are held to elevated standards in analytical depth, novelty of contribution, and scholarly communication.
What background do I need?
CSE 100 and MATH 024 are the formal prerequisites. Comfort with linear algebra, probability, and Python will help a lot. Prior ML exposure isn't required but is useful; we'll cover perception fundamentals before moving to learning-based models.
Do I need my own compute?
No — the course uses GPU-capable campus compute resources for deep learning experimentation. Dataset storage is also provided.
Can I use ChatGPT or Copilot?
Not as a solution generator. Pasting AI output as your submission is an academic integrity violation. Using AI to learn concepts or debug your own code is fine — the line is what you turn in.
What if I miss an exam?
Contact Prof. Greer as early as possible. Documented extenuating circumstances can qualify for accommodations; without them, missed exams cannot be made up.
What does the project look like?
A semester-long empirical study using real driving data or simulation, in three phases: proposal, milestone check-in, final report. Grad students extend this with a novel contribution and a final presentation/demo. Every project includes a required Ethical Impact Section.
Does this satisfy a GE requirement?
Yes — the course is designed to meet the Ethics and Scientific Method Intellectual Experience GE designations.
Is there an Honors version?
Yes. Honors students give an individual paper presentation, complete a project with an explicit novel research component, and are held to elevated standards in scholarly engagement, analytical depth, and project rigor — bridging the undergraduate and graduate versions of the course.
I need accommodations. What do I do?
Contact Student Accessibility Services ((209) 228-6996, access@ucmerced.edu) and forward your accommodation letter to Prof. Greer. Reach out as early as possible so we can make sure everything is in place.