Fall 2022 Assigments

Paper discussion and presentation

The key idea here is for students to read and present papers. While we will select papers for initial readings, we expect students to themselves select papers and get those approved. Students should also select dates for when they will present these papers.

 

What papers to select for readings when we do not provide papers

There are two sets of papers. The first set of papers are more established case studies in AI for social impact. These are from the AI4SI textbook readings. Students will need to sign up for these.

For the second set of papers students need to pick a paper on AI for Social Impact from calendar  years 2021/2022. The paper should have appeared at a top AI conference: AAAI, IJCAI, AAMAS, NeurIPS, ICML, UAI, etc, and should focus on AI for Social Impact in some way. 

This is just one example of a set of papers that would be of interest. We do not want to lock in the students to one set of papers because students will come with many diverse backgrounds. We will also point students to other venues for AI for social impact papers. These papers must have been rigorously reviewed and typically must have appeared at an AI conference.

Students should check with us once they have selected their papers. In order to preserve some coherence, we will stick to topics of public health, environment (climate change, conservation, agriculture), public safety, and disaster response.

 

Guidance for reading and presentation

Paper reading: While the first week of readings will be done individually, for later weeks, readings should be done by two or three people together. The optional readings are included only in case someone is very interested in the topic. They are purely optional.

We will have discussion forums on Canvas where we expect students to post a comment on the readings for each class. The comments submitted may be regarding something that stood out to you, a question that you have after reading the paper, a short review, or even a criticism; all these in line with the definition of AI4SI and the review rubric AAAI Special track on “AI for social impact”. Discussion submissions will count towards participation grade. We expect students to participate in at least one discussion per class period. Please submit your comments by 12pm same day of class. 

Paper presentation: This should be done in small groups. Presenters will be asked to present not only a summary, but also some evaluation of the paper. For evaluation, presenters should use the “AAAI rubric” discussed below, but there may be exceptions. At the end of the paper presentations, there will be a short discussion. Presenters should come prepared with at least one discussion question. We expect that others in the class have read some of the papers being presented.

For the first round of case studies, presentations will be 12 minutes long, with 5 minutes of discussion and questions afterwards. Please email your slides to Paula and Sonja by 2pm (15 minutes before class begins).

First round presentation evaluation rubric (out of 10 pts):
Summary (3 pts): Is the paper accurately summarized?
Review (4 pts): Does the presentation address the relevant sections of the review rubric?
Overall (3 pts): Is the presentation content delivered clearly, with a coherent narrative? Do all group members participate?  Does the group stay within the time limit? Did the group send over the slides beforehand?

The second round of presentations will be a maximum of 10 minutes long, with 5 additional minutes for discussion. Please email your slides to Paula and Sonja by 2pm (15 minutes before class begins). Our hope is for you to dive deep into the paper. You can assume that everyone has read the paper — you're welcome to give a quick summary to jog everyone's memory, but no need for a detailed explanation of the paper. We want you to delve into a more thorough criticism of the paper and/or an exploration of what is truly novel and exceptional about the paper. You can use the AI4SI criteria to guide your criticism (e.g. critiquing the "Justification of approach"), but we're not looking for scores but rather a deeper exploration of the paper and its strengths and weaknesses.

Second round presentation evaluation rubric (out of 10 pts):
Review (7 pts): Does the presentation give a deep, well-thought-out critique of one aspect of the paper? (Alternatively, does the presentation delve into the novelty of the paper?)
Overall (3 pts): Is the presentation content delivered clearly, with a coherent narrative? Do all group members participate?  Does the group stay within the time limit? Did the group send over the slides beforehand?

 

Review rubric taken from AAAI Special track on “AI for social impact”

 

Significance of the problem

4. The social impact problem considered by this paper is significant and has not been adequately addressed by the AI community.

3. This paper represents a new take on a significant social impact problem that has been considered in the AI community before

2. The social impact problem considered by this paper has some significance and this paper represents a new take on the problem

1. This paper’s contribution was elsewhere: it follows up on an existing problem formulation or introduces a new problem with limited immediate potential for social impact

Engagement with literature 4. Shows an excellent understanding of other literature on the problem, including that outside computer science

3. Shows a strong understanding of other literature on the problem, perhaps focusing on various subtopics or on the CS literature

2. Shows a moderate understanding of other literature on the topic, but does not engage in depth

1. Does not engage sufficiently with other literature on the topic
Novelty of approach 4. Introduces a new model, data gathering technique, algorithm, and/or data analysis technique

3. Substantially improves upon an existing model, data gathering technique, algorithm, and/or data analysis technique

2. Makes a moderate improvement to an existing model, data gathering technique, algorithm, and/or data analysis technique

1. This paper’s contribution was elsewhere: it employs existing models, data gathering techniques, algorithms, and/or data analysis technique (e.g., the paper presents a new experimental design and evaluation procedure).
Justification of approach 4. Thoroughly and convincingly justifies the approach taken, explaining strengths and weaknesses as compared to other alternatives

3. The justification of the approach is convincing overall, but could have been more thorough and/or alternatives could have been considered in more detail

2. The justification of the approach is relatively convincing, but has weaknesses

1. The justification of the approach is flawed and/or not convincing
Quality of evaluation 4. Evaluation was exemplary: data described the real world and was analyzed thoroughly

3. Evaluation was convincing: datasets were realistic; analysis was solid

2. Evaluation was adequate, but had significant flaws: datasets were unrealistic and/or analysis was insufficient

1. Evaluation was unconvincing
Facilitation of follow-up work 4. Excellent facilitation of follow-up work: open-source code; public datasets; and a very clear description of how to use these elements in practice

3. Strong facilitation of follow-up work: some elements are shared publicly (data, code, or a running system) and little effort would be required to replicate the results or apply them to a new domain

2. Adequate facilitation of follow-up work: moderate effort would be required to replicate the results or apply them to a new domain

1. Weak facilitation of follow-up work: considerable effort would be required to replicate the results or apply them to a new domain
Scope and promise for social impact 4. Likelihood of social impact is extremely high: the paper’s ideas are already being used in practice or could be immediately

3. Likelihood of social impact is high: relatively little effort would be required to put this paper’s ideas into practice, at least for a pilot study

2. Likelihood of social impact is moderate: this paper gets us closer to its goal, but considerably more work would be required before the paper’s ideas could be implemented in practice

1. Likelihood of social impact is low: the ideas proposed in this paper are unlikely to make a significant impact on the proposed problem

Projects

 Project related to use of AI and multiagent systems research for Social Impact, in any of the many application areas discussed in class.

 We will provide various resources to create these projects. Ideally, the projects could lead to publications in key AI venues. 

Project grades

  • Groups of 2-3 students, project proposal due by mid semester (10% of grade), final project presentation and final project writeup at the end of the semester (30% of grade). Our teaching staff will provide detailed feedback on the project proposal and that should be addressed in the final project presentation.
  • Main points of evaluation in the 30% grade will include: (i) Relevance to class topic; (ii) Illustration of use of concepts learned in class or from research papers in related areas; (iii) Survey of related research in the field to indicate awareness of this work; (iv) Novelty of ideas. There is considerable flexibility in the kind of project completed.
  • 10% of project grade will be awarded for broader impact statement of the project. We will discuss this during the embedded ethics lecture.

 

Project Proposals

The project proposal can be 0.5-1.5 pages long. Our goal in reading these proposals is to set you up for success by ensuring that your projects are properly scoped, feasible, and relevant. We encourage you to use the AAAI AI4SI review rubric as a guide in describing your problem and approach.

  • Motivation + Social Impact: What is the problem that you are addressing? Why is this problem important? How does your project relate to AI for Social Impact?

  • Methods: What techniques are you planning to apply or improve upon? (It is okay if you are not sure what methods you’ll be using, but in that case we’d like to see you lay out a set of possibilities)

  • Experiments: What experiments are you planning to run? How do you plan to evaluate your results?

We recommend pointing to one or several relevant datasets to concretize your proposal. It is okay if the dataset is synthetic as long as it is realistic. You’re also welcome to provide examples of related research on your selected topic. 

Proposal evaluation rubric:

  • Overall (5 pts): Does the proposal follow the above instructions? Has the project been well thought-out? Does the proposed plan address the stated research question?

 

Project Progress Presentations:

The project progress presentation is a short presentation in which you describe your problem statement, your approach, and give a quick overview of the progress you have made so far. We understand that these presentations are 13 days after the project proposals are due and that you may not have made huge strides on your projects. We largely want to see that you have made some progress and give you an opportunity to practice communicating about the project.

 

We want you to prepare two deliverables for class on Monday, 10/24. Please make a post that contains both your progress report and a link or attachment to your slides.

  • Post a half-page progress report on the Canvas discussion thread that covers:

    • Title of project

    • Names of team members

    • Problem statement

    • What progress you have made so far

  • Slides for a 7 minute presentation, covering the same material as the progress report. You may additionally include material that you want to receive peer feedback on. Please link or attach these slides in the same Canvas discussion thread with your progress report.

 

Presentation evaluation rubric (10 pts):

  • Project Overview (4 pts): Is the project statement clearly summarized?

  • Progress (3 pts): Has the group made progress? If not, have they outlined their blockers and what they’re doing to address them?

  • Overall (3 pts): Is the presentation content delivered clearly, with a coherent narrative? Do all group members participate? Does the group stay within the time limit? Did the group send over the slides beforehand?

 

Final Project Presentations:

The final project presentation is a 10 minute presentation, followed by 5 minutes of discussion. Please email your slides to Paula and Sonja by 2pm (15 min before class begins).

 

Presentation evaluation rubric (out of 18 pts):

  • Project Motivation (5 pts): Is the problem statement clearly motivated? Is it clear how this project would lead to positive social impact?

  • Project Innovation (5 pts): What is the novel contribution of this project? Why is this valuable?

  • Project Evaluation (5 pts): Does the group present convincing evidence for the efficacy of the proposed method?

  • Overall (3 pts): Is the presentation content delivered clearly, with a coherent narrative? Do all group members participate? Does the group stay within the time limit? Did the group send over the slides beforehand?

 

Final Project Reports:

Final project reports can be at most 8 pages long, with additional pages for references. They are due at 11:59pm EST December 9, 2022 via email — please send to Milind, Paula, and Sonja. 

 

Here is an example for the structure of your report:

  • Abstract

  • Introduction

  • Related Work

  • Dataset OR Problem Statement

  • Methods

  • Experiments/Results/Discussion

  • Conclusion

  • References

  • (optional Appendix)

 

Additional Requirements

  • Please include a “Contributions” section before the references where you outline what each team member contributed to the project

  • Please share the code you used for the project — you can include a link to a GitHub repo (either in the paper or in the email submission) or send a zip file containing the code. You do not need to share the data. 

 

Rubric:

  • Significance of the problem (4 pts): Does the paper make a strong case for the significance of the problem they are addressing?

  • Engagement with literature (4 pts): Does the paper demonstrate a thorough understanding of other literature on the problem, including that outside computer science?

  • Novelty of approach (4 pts): Does the paper introduce a new model, data gathering technique, algorithm, and/or data analysis technique?

  • Justification of approach (4 pts): Does the paper thoroughly and convincingly justify the approach taken, explaining strengths and weaknesses as compared to other alternatives?

  • Quality of evaluation (4 pts): Were the datasets realistic and the analysis thorough?

  • Scope and promise for social impact (4 pts): Does the paper show promise in terms of social impact, where relatively little effort would be required to put this paper’s ideas into practice, at least for a pilot study?

 

 

Project examples

Final projects for CS288 have covered a wide variety of applications such as environmental conservation, wildlife conservation, and public health. Here we share some outstanding final projects developed by our students in Spring 2021:

 

Grading Breakdown

  • Homework 1: Select papers and presentation partner (5%)
  • Paper presentation, discussion I (15%)
  • Midterm project ideas and discussion (10%)
  • Homework 2: Select papers and presentation partner (5%)
  • Paper presentation, discussion II (15%)
  • Project (30%)
  • Ethics and broader impact in project (10%)
  • Attendance and participation (10%)