INTRODUCTION

This study reviews the importance, background, definitions, benefits, and challenges regarding Objective Structured Clinical Examinations (OSCEs) and explore the use of a novel method called Excel Randomized Generated OSCEs (ERGOs) for technique classes in a chiropractic college.

Importance, Background, Definitions

OSCEs are the most prevalent method of practical exams to assess clinical skills in health-related education, especially in medical (1) and chiropractic colleges. The definition of OSCE by Harden (1988) is “An approach to the assessment of clinical competence in which the components are assessed in a planned or structured way with attention being paid to the objectivity of the examination.”(2) The candidates taking an OSCE move through a series of tasks in a specified time-limit and are objectively observed. (3) These assessment methods play a crucial role in evaluating student understanding and knowledge and have many benefits and limitations.

Benefits of OSCEs

The benefits of OSCEs include the following: standardization, randomization, and clinical proficiency. Standardization has involved specifying the clinical expectations through written detailed rubrics prior to the OSCE. Randomization involves multiple testing scenarios. Testing cards are placed in front of a student, and the student chooses a scenario at random. (2) Clinical proficiency is tested as the students perform a series of timed tasks listed on the testing card. This performance is compared to the standardized rubric.

Challenges

The largest challenges of OSCEs are time consumption and structuring. Structuring includes making sure the tasks are paired between students without repetition and to the relevant content module; as content lessons are packaged in modules to be tested. Traditionally, OSCEs have relied on instructor-created scenarios that are time-consuming to structure. Research suggests that the challenge with this current method is that it requires a substantial amount of additional time. (4) Other studies show that OSCEs can be very time-consuming to prepare, set-up and run. (5) Therefore, we decided to test a novel solution to address OSCE preparation.

Solution Proposed

We have created and developed a new way to generate OSCEs that solves some of the challenges. We use tools readily available in higher-education establishments, specifically Microsoft EXCEL, to streamline the creation of OSCEs. The evolution of higher-education testing methods should evolve with advancements in technology, to save time and to use all available resources. Technology in education has opened the door to more varied and efficient assessment formats. Specifically, the use of technology as noted in literature exhibits the effort to use Microsoft EXCEL with OSCES, but only in the capacity of data collection and research analysis. (6–8) Excel Randomized, Generated OSCEs (ERGOs) is our unique innovation using Excel based computer-generated randomized tasks sheets instead of the traditional instructor created testing scenarios.

ERGOs Details

ERGOs offer a variety of benefits such as unique paired cards, randomization, large-batch printing and quick updates. The first benefit is ensuring each student has a unique set of tasks compared to those provided to their partner.  Traditionally, a number of cards are placed on a table, and chosen at random by the student. This sometimes results in paired students choosing similar tasks, thus possibly giving one student an advantage. Another benefit is creating multiple randomized, paired OSCE task cards for any number of 2-person groups that can be printed in seconds. Traditional OSCE card generation typically required the instructor to create and type up various testing cards. If the instructor wanted to make any changes on the cards, they would have to create and type up a whole new series of testing cards. The ability to change tasks on the ERGOS sheet is done in EXCEL and saved for the ease of making future changes quickly and efficiently.

This ERGOs solution involves in-block function as opposed to VBA coding. In-block function for example would use functions like “=randbetween(1,10)” or index/match which are placed into the spreadsheet blocks to create the cards based upon an instruction sheet that can be manipulated by the user. These functions are hidden by hiding the columns that the programmer would deem unnecessary for the instructor and student. In addition, the sheets are password protected to keep the instructor/user from deleting important function strings. An in-depth exploration of Excel in-block functions is beyond the scope of this paper. See Figure 1 for sample Excel tasks and paired cards created.

With the detailed creation and development of ERGOs, we wondered if this method delivered on its intended solution of time efficiency.

Research Question, Specific Aims and Hypotheses

In this study we intended to explore the effectiveness of ERGOs versus hand-generated, student-picked task sheets from an instructor’s perspective and the creator of ERGOs. Specifically, we sought to determine if this approach improved efficiency for the instructor, especially time of preparation. Also, we explore other qualitative benefits of the ERGOs approach as well as potential barriers. The overall research question of this study was, “How can we decrease the instructor burden of OSCE generation?” The specific aims of this study included:

  1. What are the benefits and limitations to using ERGOs for technique OSCE generation?"

    a. What benefits do ERGOs offer?

      i. Does ERGOs decrease initial time usage and subsequent time usage by instructor/user?

      ii. Other benefits

    b. What are the limitations of ERGOs?

The null hypothesis is that “ERGOs does not provide any benefits” The alternate hypothesis is that “ERGOs provides some benefits.” In this study we intended to explore the effectiveness of ERGOs versus hand generated, student-picked task sheets from a user’s perspective and the creator of ERGOs. Specifically, this study sought to determine if this approach improved efficiency for the instructor, especially time of preparation. The creator’s time was not as crucial since ERGOs might be able to be used with multiple contexts. Also, we explore other qualitative benefits of the ERGOs approach as well as potential barriers.

METHODS

Summary

These are the general steps that we employed in the methods. 1) The ERGOs student card generator was created. 2) The study was designed regarding our context. 3) Implemented ERGOs in course. 4) Survey performed after IRB approval. 5) Data was collected, charted and analyzed. 6) Literature review and comparison

1) ERGO’s creation

Tasks were entered within categories into a customized Excel sheet entitled “Tasks.” Another excel sheet was created entitled “Setup” that delineated which categories are used from the “Tasks” sheet related to the three OSCEs: Activator, Thompson, Flexion Distraction. The last Excel sheet entitled “Key(1)” randomized the tasks from the “Tasks” sheet according to the categories related to the OSCE in the “Setup” sheet. Within “Key(1)”, this generated a randomized list of tasks for student 1, and also a paired randomized list of tasks for student 2 that avoided duplication. Infographic is available in Figure 1.

Figure 1
Figure 1.ERGO’s creation.

Sample created in excel producing randomized task cards

2) Study Design

ERGOS was created to help with the limitations of OSCE generation. We wanted to capture the quantitative data (time) and qualitative data of this experience. This study was based on questionnaire surveys for the creator of ERGOs and for the instructor using ERGOs.

3) Implemented ERGOs in course

Implementation was performed in the United States, with a graduate-level chiropractic curriculum, specifically the Assisted Adjusting course. This process used 3 separate OSCEs on 3 different chiropractic techniques. The OSCEs for this course were single station with multiple tasks graded by rubric.

4) Survey after IRB designation

IRB designation was obtained for non-human subject research. The survey was created and subjects included the creator of the ERGOs and the instructor using ERGOs. The surveys were reviewed by one of our authors who was independent of the generation and/or use of ERGOs.

5) Data collection, charting and analysis

The questionnaire was designed by the creator of ERGOs and the instructor using ERGOs. It was determined by the user and creator that a full description of the coding by in-block functions was beyond the scope of the paper and would distract from discussion of the specific aims.

The qualitative data from the surveys provided by the creator and the user of ERGOs was analyzed by us. In addition, the third independent evaluator (SN) reviewed the questions and answers for any other questions that needed to be answered. The independent evaluator recommended no changes.

6) Literature Review

A focused literature review was performed, and a selective search strategy was used to identify related literature to “Objective Structured Clinical Exam” and “Excel Randomized Generated”. We modified this strategy to find other relevant sources. Pubmed was searched utilizing select MeSH terms along with searching private archives. The search strategy that gave us quality results was: “objective structured clinical examination” and “EXCEL”. We reviewed the abstracts from all search results for relevant articles. The articles had to contain “Objective Structured Clinical Exam” and/or EXCEL for us to include the article.

RESULTS

According to the survey, using ERGOs decreased the amount of preparation time for the instructor from 2 hours to 1 hour. The instructor was able to use ERGOs for 3 iterations that term and for subsequent terms with minimal additional time. The additional time included printing and cutting the papers in half on a paper cutter. Paired students did not receive similar tasks since they were randomized by ERGOS. Automatic generation of the tasks was created in large batches for every student section. It was easy to transition between assessments for each of the three OSCEs given for the class. There was versatility in editing the tasks without having to individually redo every card by hand.

Table 1.Comparison Table Traditional Method vs ERGO Method
Issue Traditional Initial set ERGO Initial Set Traditional Future sets ERGO Future sets
Instructor time 2 hrs 1 hr 1 hr 30 min
Creator time if changes needed 2 hrs 2 hrs 1 hr 0 hrs
Paired Card by card check for duplicates Automated Use old card template Automated
Randomized Card by card check for duplicates Automated Not as randomized Automated
Student randomization Picked by selection or selection of who goes first Automated Student selection of randomized sets Automated
Automation after changes Redo many if changes made Automated with updates Redo many if changes made Automated with updates

DISCUSSION

General Comparisons

Compared to other studies, Microsoft EXCEL with OSCES was only used in the capacity of data collection and research analysis(6) Other studies were focused on student learning and satisfaction and general information on OSCEs. (9) Our study focused on the instructor’s workload and time efficiency in creating quality OSCE materials such as skill cards.

Benefits

Summary benefits to the instructor included time efficiency, randomization, pairing, free source, bulk printing and editing with automated changed output. OSCEs are an integral part of testing in medical and chiropractic education. As teaching becomes more complex, developing new efficient methods for OSCE creation becomes imperative. If innovations in testing development using technology can positively affect instructors and students alike, the change becomes a necessity. Time and resources in an educational setting can be limited, so the search for more efficient use of time and resources is always ongoing. The use of ERGOs saved time for the instructor and did not add any additional costs. Our results showed that time was saved by reducing the time to create OSCEs by half on the initial use of ERGOs. Reducing 2 hours of creation time of OSCEs to 1 hour on the initial use. There was then a reduced time to 30 minutes for each consecutive OSCE using ERGO. Changing 6 hours of OSCE creation time for a trimester, involving 3 OSCEs, down to two hours of creation time for the instructor. Additionally, while our study received the expected benefits, we were pleasantly surprised with the realization that the adoption of this process used existing resources and did not require the purchase of additional software. Future iterations should continue to be reduced in time as the faculty becomes comfortable modifying the ERGO generated output.

Regarding the creator, this method took additional time, however, this method can be used in multiple contexts. Once ERGOs has been created, it can be used by other technique instructors and other contexts.

The results from our study Studies mentioning time concerns are probably in reference to large multi-station OSCEs following the clinical management routine with practical (history, exam, treatment) and cognitive thinking stations. Our context involved a single-station OSCE with multiple technique tasks. Although our context is different than previous studies, ERGOs decreased the time commitment for generating small scale OSCEs or practical exams for this instructor.

Challenges

A potential challenge with ERGOs is present if one is not proficient at using the Excel program. Any issues would have to be brought to the creator of ERGOs. A dependence on the creator by the instructor could possibly be a barrier for large-scale use. As mentioned, the creator of ERGOs found that an improvement to be made was limiting the instructor who was using ERGOs to not be able to edit the EXCEL sheet unless it is highlighted in yellow. If the ERGOs form is not password protected and can be edited, the instructor can accidentally change the format and alter the in-block coding, making it unusable. We also found that providing an instruction page on an EXCEL sheet assisted with the ease of using ERGOs. This instruction page also included how to alter the instructor’s name, course name and iteration (modules).

Limitations

Our study was limited by sample size which can be increased in future studies with additional instructors.

Future research

Other studies on ERGOs could include student experience, AI improvements, or narrative on in-block function details to create ERGOs. Use in other contexts could be studied such as other techniques (cervicothoracic class, lumbopelvic class), other fields (physical diagnosis, anatomy identification, physical exam).

CONCLUSION

As expected, the process of ERGOs was incredibly beneficial to the instructor through a reduction of external time spent on OSCE creation. As education continues to make a push towards adopting technological advances, utilization of existing resources should be encouraged to reduce both workload and unnecessary expansion of resources. Ultimately, the comparison of computer-generated randomized vs traditional OSCE generation will help us understand how different assessment formats affect instructor time efficiency. The findings of this study provided valuable insights for educators looking to enhance their OSCE generation strategies.  Our purpose was to streamline the practical exam creation and administration process for instructors. Many additional benefits of ERGOs were realized with minimal challenges.