Evaluation and Assessment
Dec 7th, 2015 by jungg
When designing a product, it is important to understand the user and develop an analysis for your design choices. The best way to evaluate a design and determine if it was effective in the areas you wanted it to be is through implementation, testing, and user feedback. For our product it is important to evaluate each student’s role during the game and what they took away from the activity. It is also important to measure mathematical comprehension and abilities before and after using the device. To do this we could implement an after game survey and short mathematical exercises.
By carefully designing a survey that is unbiased and simple, we can create a table of information based on the experiences the children had using our device. Not only would the survey allow us to understand how we can improve the game for future applications, but it would also allow us to see what the students believed they were learning from playing the game. If we receive many responses that show the students perceive a different goal than intended, this would tell us that we must make revisions or remove parts of the original game that might be taking away from the main objective.
The short mathematical exercises could be designed to test many different understandings; this could include small comprehension problems, questions that test mathematical procedure (PEMDAS), and even a timed section testing how quickly the students can answer a number of equations. These exercises would need to be implemented before the students played the game and again after interacting with the game over the course of some number of uses. If these exercises could not be added to the curriculum it would also be possible to compare normal math exam scores over the course of their school year. However, the continued use of the device over the course of the year would be necessary. The game also makes it very easy for an instructor to increase difficulty as needed or necessary for continued improvement.
These evaluations would deal with threats of validity and statistical analysis by being balanced, fair, and neutral. The survey would ask questions that are written in a way that does not lead on a particular answer and would be purely designed to achieve an accurate understanding of what the children learned and what they liked or did not like about their experience. A survey such as this would not be meant to make our device look perfect but instead, and ideally, it would be designed for us to achieve optimal feedback that would in fact improve a particular aspect or process of the game. Certain questions could also be asked twice in different context to make sure a student was not only providing random answers and would further test the validity of a student’s responses.
The mathematical exercises would avoid threats by being carefully designed to test a particular understanding or skill in each question, or set of questions. By providing these exercises before and after continued use, we can measure a student’s performance in several mathematical comprehensions and any effect the device has had. Ideally, the exercise before use and the exercise after use would need to be different, while still testing the same abilities and comprehension. Students would not be allowed to receive their first exercise back, further avoiding the possibility of memorization or preparation.
Practical problems carrying out these evaluations arise if students participate in the game as little as possible, or if the students choose to not be a part of the mathematical aspects of the game and instead become the “builders” and “shakers”. Those students that choose to be purely uninvolved in the mathematical portions may have different survey answers and results from the exercises. To help clarify and organize the information from our evaluations, it is important to ask what the particular student’s roles were during the game. It is also possible for students to forget certain mathematical understandings that they learned from the game.
We believe that the combination of a carefully designed survey and mathematical exercises would help us to best evaluate the validity, success, and overall design of our device. By understanding each student’s individual role during the game, perceived goals, and mathematical comprehension, we can make improvements or alterations to the design, or prove the success of our game.