Ava DeBartolomeis
← Back

Case Study Project

CodeJam

Create a streamlined, centralized grading experience that consolidates the workflow for undergrad Computer Science TAs

Duration
3.5 months
Role
Product Designer
  • Workflow Consolidation
  • Team-based Research
  • Prototyping
CodeJam

Overview

For my Human-Computer Interaction course, I was assigned a team with diverse backgrounds ranging from UX research to product management. We were tasked with using the human-centered design process to understand a user group of choice and create a solution that addressed the users' needs.

Problem

Cornell CS TAs graded across scattered tools: an external rubric, code in another window, screenshots messaged to other TAs, and grades entered separately in Canvas, with nothing keeping grading aligned.

My role

On a cross-disciplinary HCI team, I led design and prototyping of CodeJam: one grading screen with in-line help, live video chat, and cross-check references, tested from paper to interactive prototype.

Impact

Testing with 4 users pinpointed four redesign areas for the second iteration. Users said it was more convenient, aligned their grading, and made them more confident.

My Role

Although I was fully involved at every stage of the design process, I was primarily responsible for the Designing and Prototyping stages. I constantly communicated with my teammates, gave critical feedback throughout project checkpoints, and met tight deadlines.

Approach

The team followed a human-centered design process to ensure that we were tending to the users' needs. After completing the project, I realized that the team closely followed the Double Diamond model since we went through various cycles of diverging and converging our findings and iterations to ensure an impactful solution.

Discover

Finding the right user group

Each teammate brought five problem spaces, and we first landed on non-professionals who want to share knowledge, inspired by "YouTube University" and a teammate's mother's struggle to share educational content outside the classroom.

While writing the interview script, I spotted a red flag: our questions were framed to get the answers we wanted, for a solution we were already picturing.

We were trying to fit a solution into a problem rather than allowing an observed problem to inform a solution.

I initiated a group meeting to re-evaluate our approach, and we narrowed the user group:

First user group

Non-professionals who want to share knowledge

Final user group

Undergraduate Computer Science TAs

Frustrated by constant shifts in how courses were taught during COVID.

  • Too broad to identify members
  • Not feasible to recruit
  • We were solution-first, not problem-first

Contextual user interviews

With a more defined user group in mind, the team began understanding user needs and patterns by interviewing five Cornell students who fit into our identified user group.

Our goal was to identify motivations, behaviors, activities, and problems that CS TAs currently face. We did this by asking questions about their experience being a TA as well as asking them to walk us through some of their current activities.

Define

The team then converged our findings by creating an affinity diagram with common patterns and themes found during our user interviews. Using this, we identified and evaluated opportunity areas for our solution.

We decided to focus on the grading process since many of the user’s pain points revolved around this user activity.

Key Takeaways

The main problem that we found was that users did not have a consolidated work flow that provided them confidence and consistency in the grading process.

Users feel overwhelmed by the number of tools required for grading.

“I have at least five windows open at once and I get lost switching between them all.”

The grading process is time-consuming and fragmented.

“I have to wait for the rubric and code file to download and then I have to re-save the filled out rubric and upload it on a different platform.”

Users feel unsure of their grading decisions.

“A lot of times I’ll check in with another TA to make sure I’ve graded something fairly.”

Develop

Determining the design concept

After each teammate independently brainstormed 10 potential design solutions, we converged our ideas.

We decided that a grading web application that consolidated all necessary tools while maximizing grading alignment would best tend to our users’ needs.

Many users indicated they grade on the same device they use to do school work – their laptop – thus, a web application seemed most appropriate.

Investigating the solution space

The team then researched the current solutions tending to our user problem in order to identify the MVP feature requirements as well as the gaps that our solution could fill.

Although solutions currently targeted the user problem of a lack of consolidation of tools, there was a major gap: alignment of grading through resources and communication was lacking. The team aimed to address this gap in our solution.

Assumptions & Constraints

Assumptions

Users only grade code-based assignments, and already have a TA account in CodeJam.

Constraint: design vertically

CodeJam would span classes, assignments, and gradebooks, but project scope meant focusing on grading.

User tasks

Research shaped 4 key tasks, which we later used in usability testing. Each replaces a workaround TAs relied on:

1

Grade an assignment

Today: external rubric, code in another window, no in-line comments, rubric re-uploaded, grades entered in Canvas. CodeJam: all of it on one screen.

2

Request in-line help

Today: screenshot the code and ask another TA in a separate messaging app. CodeJam: in-line communication.

3

Video chat to discuss

TAs found live conversation fastest. CodeJam: real-time video chat with screen sharing.

4

Cross-check previous submissions

Today: past submissions buried in a separate drive. CodeJam: cross-check references one click away.

Storyboarding & Wireframing

Each teammate then made a storyboard for each of the tasks to better understand how our solution would fit into the user workflow and resolve our users’ goals.

We then sketched UI wireframes to accompany each storyboard. These would inform the paper prototype I created.

Paper prototyping

I created a paper prototype based on the tasks we created to gain preliminary feedback on the design. I user-tested the prototype and analyzed our findings to then advance to the prototyping stage.

I discovered that the rubric wasn’t structured in the way our user group used rubrics to grade. Furthermore, the user requested that rubric items be more integrated into the point deduction user flows. This finding, along with others, would directly influence our first prototype iteration.

Deliver

First prototype iteration

Justified on the basis of user feedback from the paper prototypes, usability and UX principles and common UI patterns, as well as the persona and requirements, I led the prototyping of the solution with the help of the team.

Evaluation

We tested with 4 users on the four tasks. From the screen recordings, we built a usability analysis spreadsheet of qualitative and quantitative data to prioritize redesigns by expected impact. It showed a steep learning curve: more clicks, errors, and confusion on the first task than the last.

Four areas to address in the redesign

Confusing, undiscoverable buttons

Uninformative progress bar

Grading UI prompting user hesitation

Lack of onboarding elements

Second prototype iteration

After evaluation of our findings, the team made informed design decisions. All of the features and functions of our solution are outlined below.

User testimonies

Overall, users were very pleased with our design solution. Their comments validated the team’s solution for the identified user problem. Here’s what users had to say:

Users felt that it was more convenient.

“I love that I don’t have to open a bunch of different windows.”

Users felt it aligned their grading methods.

“I really like that I’m able to see how others are grading. This would definitely help me grade more fairly.”

Users felt more confident in grading.

“Being able to easily communicate with TAs reassures me that I’m grading how I should be.”

What I Learned

Working with teammates from many backgrounds grew my design skills in a collaborative setting:

Effective communication

Blockers and disagreements pushed me to communicate more succinctly and listen better to teammates.

Putting the user first

When team preferences clashed with users’, we let findings decide, and I got comfortable letting go of my designs.

Good design takes time

Our first user group was too broad to be feasible. Patience kept us from jumping to a solution before understanding the problem.

Future opportunities

Beyond the course's 3.5-month timeline, I would:

Design horizontally

Prototype beyond the Assignments page to cover classes and gradebooks.

Offline notifications

Testing showed users want to see missed responses to their Help Requests.

All CS assignment types

Interviews showed TAs also grade open-ended assignments.

Light mode

Users preferred light mode and wanted to switch between the two.

Each would need more user testing, since the success so far came from our user-centric process.

More Projects

Back to top ↑