Week 8: Study materials manifest

UPDATE: I added one more paper to the Papers folder, "Authoring multil-stage code examples with editable code histories", from our own Bjoern Hartmann and his students. Skim the paper but in particular look at the Evaluation section for a good template of how an informal pilot evaluation of a software tool is run. If you're instead doing an in-class ("in vivo") study with no control group, I encourage you to check recent SIGCSE proceedings in the "Experience Papers" track to find examples of in-class studies of new tools/exercises.

As you prepare to run your pilot studies, it's a good time to gather a list of all materials and preparation required: from the moment a volunteer shows up (virtually or in person), what is the exact sequence of steps they will follow, and what materials or setup need to be in place for each?

You can either add this to your ongoing slide deck (easiest) or start a Google doc and link to it as your submission; either way submit a URL as your submission. Please list everything that has to happen/be ready for your study, and call out items where you need external support (e.g. setting up a PrairieLearn course or assessment, getting access to a room or lab, etc.) You don't need to include details about volunteer compensation.

Here's a list of suggestions to think about, though not exhaustive:

• Recruitment materials: where will you recruit and how will you announce? What are the qualifications for volunteers, i.e things volunteers should (or should not) already know, courses they need to have taken, etc.
• Place and time: will volunteers do tasks separately or synchronously as a group? Will it be done entirely online, in which case do you have the necessary Zoom privileges? Or in person, in which case will you need to schedule a room?
• Prep material: What background material will students be given? (Written notes? Textbook chapter? Video mini-lecture?) You'll need a timeline to prepare and debug these materials.
• Ceiling effects: how long will students have to complete the task? It may be useful to deliberately allow less time than needed, to avoid ceiling effects (especially if you're comparing to a control group not using your tool). You can get a time estimate by having an "expert" (eg GSI) do a couple of exercises, then multiply that by about 2x, perhaps a bit less if you want to constrain the time.
• Technical prereqs: do you need to create assessment(s) and/or question(s) in PrairieLearn? Do the experimenters and volunteers have access to those as needed? (Will they sign in directly to PL?) Have you tested the actual study exercise(s) on the campus server if you plan to use that? Or will students use a different server/laptop?
• Surveys/questionnaires: Have someone proofread any survey instruments you use. The NASA TLX questions may be a good starting point; in general, whenever you have the opportunity to use a standardized instrument, you should do so if possible. Some time spent searching for this will be well rewarded.

For any items where you need instructor or technical help, please let us know ASAP so it doesn't become a bottleneck.

You should also dry run the study materials, in particular:

• It is really hard to write clear survey questions. For example, "Did you find this process mentally challenging", when asked with respect to an interactive exercise, can be read at least two ways. One way means "Did this exercise the parts of your brain needed to solve this type of problem". In that case, you'd hope for a "yes" answer. But another way means "Did you have to work hard to understand what the problem was asking, or how to interact with the exercise, in such a way that it distracted you from the mental work of thinking about the problem?" In that case you'd hope the answer is "no". But how do you know which way the subject interprets the question? Asking a colleague to "think out loud" while answering each question is one way to clarify wording.
• If you have a pre/post test with a time limit, time how long it takes dry-run subjects to do it. In particular, if you care about "ceiling effects" (everyone gets max points as long as they're given long enough to do it), you should choose your pre/post-test time limit appropriately. Again, the best way to estimate it is to have a few friends who don't know your work perform a dry run.
• Similarly, if you have a particular subject population in mind—for example, only middling performers or only weak performers with respect to a certain topic—the pre/post assessment needs to be calibrated to identify those people, and/or to rule out people outside your target group. Without a dry run, you won't know until after the study whether the pre-test was effective in doing this. (Of course, you might be relying on something other than the pre-test for this, such as a survey asking the students to self-assess their own proficiency on certain topics, but that might be less reliable on its own than when paired with a pre-test.)

Great work on the projects so far!