A New Model for Course Placement with Qualtrics

By Anna Hubbard and Carter Griffith, University of Minnesota

Introduction

Having an efficient and accurate course placement tool saves time and resources for both students and instructors. A good placement tool should assist students early in their academic career so they can register for the appropriate course based on their current proficiency in the target language. For instructors, it should be easy to access, edit, and administer. Other desirable attributes include a test that adapts based on the test taker’s answers, has the ability to randomize questions and answers, has a built-in timer, and is accessible and adaptable to all users. 

In this article, we will describe how we created the Somali Entrance Placement Test (EPT), with all of the benefits above, using Qualtrics. We will talk about the motivation to create this placement tool, describe the tool and how the results are handled, what we learned from using Qualtrics, and where we plan on going in the future.

The Motivation

The primary purpose of the Somali EPT is to determine whether incoming University of Minnesota (UMN) students who are heritage speakers of Somali have the necessary Standard Somali literacy levels to be successful in the accelerated Somali course track. In the Twin Cities area, there is a large population of second and third generation Somali speakers. Advisors and Somali instructors saw a rising need to help these incoming Somali heritage speakers in determining their course placement in the language sequence. 

Not having a systematic course placement test created challenges for all stakeholders (students, instructors, and advisors). The original placement test was informal, on paper, and administered during the first week of class. Students were not getting placed in the appropriate language course early in their academic career and often enrolled in a higher than appropriate language course. As a student, it can be anxiety-inducing to find out that you are not in the appropriate class for your language level on the first day of class. Other consequences of not being placed in the appropriate class can include a large range of proficiency levels within a single course section, disengagement from the students in the course, and students simply withdrawing from the course. To address this need, we were approached to help create an online asynchronous placement test that students could take from any location, at any time, using any device (e.g., desktop, laptop, tablet, and smartphone).

Somali Entrance Placement Test Content

The components of the Somali EPT are the Background section, Section 1, and Reading section.

The Background section is designed to briefly explore the amount of exposure and usage of Standard Somali that students have experienced prior to enrolling at the UMN. The responses to these questions allow stakeholders (particularly instructors) to consider whether or not a student is a heritage speaker of Somali. There are three questions with five-point Likert scale responses designed to gauge the frequency and complexity of the test taker’s use of Somali in three modalities: speaking, writing, and reading. 

Section 1 addresses three topics: orthography, vocabulary, and grammar. The orthography subsection is designed to determine the student’s level of familiarity of the Standard Somali Latin-based alphabet. 

The vocabulary section has four subsections. The first vocabulary subsection asks learners to identify the correct word in Standard Somali that describes an image. The questions in the next subsection require a student to select the best single English word that summarizes the topic of a Standard Somali sentence. The final two subsections focus on vocabulary in context. One portion displays a Standard Somali sentence with one word in bold font. The learner is tasked with selecting the best English word with the same meaning. In the last portion, students are presented with fill in the blank questions. 

 

The grammar section targets three main topics: the past tense, the present tense, and the ability to identify correctly-formed sentences in Standard Somali. All of the questions in Section 1 are multiple choice with four possible responses.

The Reading section contains three texts with multiple choice comprehension questions. The texts are targeted for an intermediate-level Standard Somali learner. 

The Somali EPT was created through a collaboration between the University of Minnesota Language Center, Language Testing Program, Department of African American and African Studies, and Somali language instructors. Through this collaboration, we expanded the content of the original informal placement test and adapted it into a Qualtrics test.

Qualtrics Functionality for the Somali EPT

Now that we have given a sense of the individual components of the Somali EPT, let’s take a look at how they work together in Qualtrics.

The Background section score, which is automatically tabulated in real-time by Qualtrics, ranges from three to 15 points (each of the three questions is worth one to five points). Advisors and Somali instructors may consider the responses collectively or individually, and qualitatively or quantitatively. The background questions occur before the timed portion of the Somali EPT and have no bearing on the course placement of the student.

Early in the development process of the Somali EPT, it was determined that students must get two thirds of the Section 1 questions correct in order to proceed to the Reading section. The rationale was that if a student did not demonstrate the literacy levels to reach the cut-off score, there was no benefit to be had in them attempting the Reading section.

Each of the subsections in Section 1 are organized into “blocks” in the Qualtrics Builder. The block positions are fixed and the question sequence is randomized within each block. We opted not to display any question numbers or question names to the students. This randomization function in Qualtrics allows for a different test experience for each student. 

At the end of Section 1, we created a conditional branching logic function to determine whether or not a student continues to the Reading section of the EPT. To get a sense of branching logic, picture a “yes/no” question followed by an “if then” statement. The scoring system and conditional branching logic are essential to the automation of the Somali EPT. Qualtrics automatically compiles the scores in real-time as a student progresses through the EPT. The branching condition results in two main outcomes: 1) the student does not reach the cut-off score in Section 1 and they receive a message stating that they have completed the necessary portions of the EPT, and 2) the student reaches the cut-off score and they continue to the Reading section. Again, the scoring and branching occur instantly and the participants move seamlessly along the appropriate branch. As illustrated in the flowchart below, there are two divergent paths after a student completes Section 1.

The Reading section score is not directly related to a student’s course placement. Rather, the score is used as additional information for advisors and Somali instructors, similar to the responses of the background questions.

In the interest of test security, students must have a valid and active University of Minnesota student account to gain access to the test. This instrument is intended for UMN students only. Another security measure is the randomization of questions within each block or subsection in Qualtrics. The real security is related to the course placement itself. If a student cheated on the test, they would be placed in a course that is too advanced for them and would have a low chance of success in the course.

Student Experience

The student-facing experience begins with the student visiting our testing website, logging in with university credentials, and selecting the desired language for their course placement. The student reviews the test instructions and academic integrity policies, and then clicks on the test link that takes them to the Qualtrics test. 

When the student reaches Section 1, a 50-minute timer and a progress bar appear at the top of the test window. Students are allowed 50 minutes to complete both Section 1 and the Reading section. In the case that a student requests test accommodations, e.g., an additional 25 minutes on a test, the test time can be adjusted using a third-party method. 

Test Results

The overall test scores, in addition to the responses to each question, are stored in Qualtrics. You can easily export the data as a .csv, .tsv, or .xlsx file or straight to your Google Drive. Our method is to export the results as a .csv file and enter the results on a master spreadsheet. From the master spreadsheet, we input the results to a secure internal database that allows students and their advisor to log in and view the results. When a student or advisor views the results page, they see an overall pass or fail of the placement test, information on what course the student should register for, and the numerical score they received on the test.

What We Learned from Using Qualtrics

Qualtrics is a powerful online tool that allows you to build, distribute, and analyze tests, technically known as surveys, and responses in one convenient online location (and you do not have to know any coding). Neither of us had very much experience in Qualtrics before this project. Without the guidance and support from colleagues and the Qualtrics support team at the University of Minnesota, this placement test would not have been possible. 

One of the main reasons for choosing Qualtrics was for its automated branching and scoring capabilities. The ability to have a test that could automatically determine if a student should move on to the next section of the test or not, based on how they scored on the previous section, was important to us. We needed a test that did not rely on someone to grade it and could produce results in a timely manner. 

Another beneficial aspect of Qualtrics is that it is very editor-friendly. Question blocks and individual questions can easily be edited, added, moved, or removed. Cut-off scores and branching pathways can readily be adjusted as needed.    

Other selling points of Qualtrics include the numerous features it has and how flexible the tool is. Further test security features include invitation-only access, password protection, and limiting a test taker to a single attempt. Instructors could create an assessment tool to be used within the context of the curriculum or create a fully-functioning placement test. It has built-in accessibility functions such as alternative text to images (alt text), supports non-Roman scripts, and is compatible with mobile devices. Our current placement tests for French, German, and Spanish can only be taken on a laptop or desktop computer. Qualtrics allowed us to expand access to our tests to smartphones and tablets, which can increase the accessibility of the tests. You can set up email triggers to be notified when someone has completed the test. You can also create custom end-of-test messages and have automated email notification of results sent to the test taker. All the data is compiled in one place and is easy to export. Lastly, there is robust support from the online community. There are many walkthrough guides, forums, and training available.

If you are new to Qualtrics, here are some tips from us. We highly recommend watching tutorial videos that explain the general format and layout of the platform. When you are ready to start experimenting, try copying an existing paper test into Qualtrics. Always remember to “publish” your test to save any edits. Even though Qualtrics has a preview option for both desktop and mobile view, double check your work using a smartphone to make sure the layout and functionality of the test are working as intended for your audience. Another tip that we have found useful is to save a new copy if you are trying to add new features to an existing test. Doing so allows for more freedom for experimentation, more room for trial and error, without having to fix or undo any of the new additions you have made. A good way to troubleshoot both the function and content of the test is to conduct informal pilot sessions with friends and/or colleagues. Simply share the link of the current version, ask them to “take” the test and provide their feedback. Conducting this informal, low-stakes piloting with individuals using PCs and Macs, Android and iOS, is an easy way to double check work and prevent issues once the test is formally launched. 

Future Directions

Our ongoing work with Qualtrics has inspired numerous potential future projects.

For the Somali EPT, a Writing section has been developed and is available for future use. Using the “text entry” type of question in Qualtrics, as opposed to multiple choice, test takers are instructed to write four to five sentences in response to each of four writing prompts. The important difference between the Writing section and the rest of the Somali EPT sections is that the writing responses would require a person to assess them. The Writing section could be used in a more robust version of the placement test, targeting learners of higher language proficiency and placement into possible advanced-level courses. Another element that can be added to the Somali EPT is the embedding of audio and video files. Compatible file types in Qualtrics include .mp3 (audio), .wav (audio), and .mp4 (video) files. These additional file formats could, for example, be used to assess listen comprehension.

Another potential project focuses on the branching functions, rather than the automatic scoring, in Qualtrics. This would be a preliminary questionnaire designed to help students determine which language placement or proficiency test they should take as a new incoming, transfer, or current student. This could be a “can do” statement type of self-assessment. 

Qualtrics also has the capability to be integrated in Canvas courses or sites. This may prove to be an efficient way to compile and store test results and give students and instructors easy access to their results. Housing a Qualtrics survey in Canvas would be an efficient way to ensure ease of access for participants.

A placement test using Qualtrics can also be used beyond the local population of students. We are currently exploring methods of utilizing Qualtrics surveys with the universities we partner with on language exchange programs.

Conclusion

After one full calendar year and 130 participants, our placement test has proven to be stable and functional with Qualtrics. It addresses many accessible needs from providing alt text to images to expanding access beyond laptops and desktops. We can easily add and remove content as needed. Most importantly, the department that houses the Somali language program has informed us that the automated scoring and branching of the Qualtrics tool has saved a considerable amount of time and effort for instructors, staff, advising, and presumably students. Before the Qualtrics EPT, we were frequently contacted by students and advisors with questions and concerns about Somali course placement. These questions have disappeared now that students have a mechanism for course placement. 

Our confidence is confirmed in continuing to develop placement tools, either binding or advisory in nature. As the Language Center and Language Testing Program’s first placement tool using Qualtrics, the Somali placement test serves as a template, or starting point, for other projects moving forward. Shortly after launching Somali, we started working on Hindi and Urdu placement tests. These tests were completed and launched in early 2023. Based on the experience we have gained through creating these instruments, we have demonstrated the capability of supporting more departments and institutions with their course placement tools.

Acknowledgements

We would like to gratefully acknowledge Dan Soneson and Fuqiang Zhou. Dan Soneson, in his time as Director of the UMN Language Center, attended a presentation of Fuqiang Zhou’s at the 2021 International Association for Language Learning Technology (IALLT) conference entitled “Developing Online Foreign Language Placement Exams using Qualtrics”. Dan was inspired by the idea of automated branching as part of a language placement tool, and later shared the concept with us as we were in the very early stages of planning the development of the Somali EPT. Thank you as well to the Institute of Global Studies at the University of Minnesota for their support in the creation of the Somali EPT.