1. The Inception
I’ve worked as a student and as a teacher for most of my life and throughout all of it I’ve been thinking about education, teaching, and learning. My main questions are quite simple:
What are all of these things: education, teaching/teachers, students, learning, school? Why doesn’t anyone know? And why are there no constant answers in academia?? Surely school would know what school is, educators would know what education is, teachers would know what teaching is, and learners would know what learning is! (And ideally some of these people would agree!)
Turns out, nope!
It seems that everyone is equally as confused as I am by all of these questions and education research is lacking in studies that describe simple teaching methods, like “what are the effects of asking questions and explaining answers.” You would think these questions would be foundational to education.
Luckily we can make things simpler. We can ask a simpler question that education researchers surely have the answer to: “what role does memory play in learning?”
Turns out no one knows the answer to this one either. Sure there’s lots of theorists with theories (some of which I agree with) but you’d think that somewhere in the thousands of education studies there would be replicated results to this question.
Luckily I can turn to the psychologists who surely must’ve figured out memory by now, after all, John Dewey claimed that psychology has been going on since the 1780s. Certainly that’s enough time to settle into a theory of memory.
You’d never guess that there’s barely a shared understanding of memory, let alone an agreed upon theory of it.
Dear god! How can I design a study that asks questions about memory and learning and teaching strategies if no one knows what those things are?
This is the conundrum I found myself in at the beginning of my third semester of college when my Psychology Professor said “You have two weeks to write your proposal to the IRB if you want to run a study this semester.” I had no time to do a literature review, I had to invent a study and then run it STAT. So what’s something I can measure? Ideally something related to memory and practical teaching tools.
The question I settled on was as follows: Teachers commonly use two methods to impart information in classrooms. The first is ‘explaining’ (lecturing, leading discussions, answering questions, etc.) the second is ‘questioning’ (asking students to try it themselves, ask questions, share what they think, etc.). What are the effects of these two approaches on retention of information? What is their effect on accuracy of students information? What is their effect on each other?
Surely studying this will be easy, simple, and yield interesting results and I will not run into any issues with the study design, the IRB, or scheduling a study!
*My friend Ike (Isaac Rubin) found a study that claimed that less than half a percent of all education research is a replication (meaning 99.5% or more of education research is unique and therefore has a much higher susespabtliiy to false positives) but he has refused to link this study to me in spite of me asking many times so if anyone could find the link I’d give them a heartfelt thanks and a pat on the head.
2. The Study Plan & The IRB
Step one to running a study is to turn my research question into a hypothesis, here’s mine: Teachers use two strategies to communicate information, explaining, where they further contextualize facts—going beyond teaching students facts by teaching students how to look at the facts—, and ‘questioning’ where teachers ask students to explain and contextualize ideas for themselves. The ‘explaining’ approach will increase the accuracy of student’s responses, whereas the ‘questioning’ approach will increase student’s retention of information.
To test this I designed a simple digital lesson from old projects lying around my computer. This lesson had four variants: facts, facts + explanation, facts + questions, and facts + explanation & questions. Look at my table!
|
|
No ‘questioning’ |
‘Questioning’ |
|
No ‘explanation’ |
Facts |
Questioning |
|
‘Explanation’ |
Explanation |
Both |
Participants of my study would be sorted into one of the following categories and then would be asked to complete a short assessment containing the open ended question “Describe the RGB color theory.” And then five free response questions with a clear specific answer.
Then I just needed to submit this to the IRB (Internal Review Board). Surely they will get back to me in a timely way and clear my study on the first submission.
3. Waiting.
Then I had to wait.
And wait.
And wait.
And wait.
And wait some more.
Still waiting.
I am no longer excited about this study and I’ve honestly kind of forgotten what it’s about.
I am still waiting.
Finally!! Something! The IRB asked me to make some edits and re-submit my proposal that way I get to wait some more.
Dang it.
Stillllll waiting
araogihenopnpsdfpasdmfpgaivwe
THIS IS A NIGHTMARE!
Oh god I have other classes and responsibilities now!
4. Finally, Work.
Finally, the IRB approves my study and it’s time for it to get run! I send out emails, I put up posters I tell friends, but my heart's just not in it like it was at the begining of the semester. With each passing day I grow increasingly unconvinced that my study will even yeild interesting results. If I collected all the data immediately I would’ve been stoked to have it, but because I had to wait so long and because my study didn't have a clear place in a theory of learning (meaning I couldn't easily figure out what I was trying to meassure at its inception) I grew uninterested with it even as I tried to rustle up responses.
The process of going through the IRB drained my creative spirit from this project. Which is not to say that you shouldn’t go through the IRB, nor that you shouldn’t run studies, but I doubt I will go through an IRB again any time soon. Heck, later in this semester I ran an informal study where I just emailed everyone on campus and collected some data then wrote up a report in a day or two and now I have meetings with the Dean of Faculty to discuss educational systems on campus. This is much more creatively and spiritually fulfilling than talking with the IRB.
That being said I still got the survey out there and began to get responses.
However I noticed something silly, it turned out that Qualtrics was incapable of tracking data I needed to know what group participants are in and I had to develop a somewhat bodged solution. For some reason I didn’t click publish on this solution and accidentally ran the rest of the study without collecting this crucial data that would tell me whether people were in the explaining condition or not. I also didn’t think about how giving people a quiz regardless of if they’re in the questioning condition or not might undermine the effect of the questioning condition.
In short, my study is incapable of measuring anything I designed it to measure. But I had fun (sort of) so here it is being published nonetheless.
5. The Study:
0. Abstract
The purpose of this study was to answer the question: Teachers commonly use two methods to impart information in classrooms. The first is explaining (lecturing, leading discussions, answering questions, etc.) the second is questioning (asking students to try it themselves, ask questions, share what they think, etc.). What are the effects of these two approaches on retention of information? What is their effect on accuracy of students information? What is their effect on each other?
However, due to my inability to use technology, the study only tracked the impact of questions on retention, and because people were immediately given a survey (full of questions) in both the control and questioning condition it’s unclear if my study would’ve been capable of detecting any effect even if there is an effect. This study has significant flaws that prevent it from providing usable data to respond to its hypothesis.
All this being said, this study does still contain two interesting pieces.
The first is a very successful digital lesson. There was almost no difference between people’s average scores on the initial skill assessment (average score 3.86/5) and on the skill assessment they took two weeks later (average score 4.00/5). And most people tested highly with relatively low levels of serious misconceptions.
The second is results that suggest a relationship between one’s employment and enrollment status (full time, part time while being in college, or just being in college) and retention of information. Full-time workers scored the highest on the followup assessment, followed by students, followed by part time workers who were also students (who scored significantly lower). However I do not trust my very low sample size. MORE SCIENCE IS NEEDED!
I. Methods
The original methods of the study were as follows (keep in mind that these methods yielded no usable data).
Participants are sorted into four groups: explanations, questions, none, or both. All groups are presented with the facts of the RGB color theory, one group is presented with explanations that contextualize the RGB color theory, another is presented with questions about the facts, another is just presented with the facts, and the final group is presented with the facts, questions, and explanations.
|
|
No ‘questioning’ |
‘Questioning’ |
|
No ‘explanation’ |
Facts |
Questioning |
|
‘Explanation’ |
Explanation |
Both |
All groups are then given an RGB color theory knowledge assessment and asked to provide their emails so that they may be emailed a link to the followup survey. This second survey, which is sent out two weeks later via scheduled email, contains a repeat of the RGB color theory knowledge assessment followed by some debrief and demographics questions (including “what do you think this study was about?” And “what is your employment status, check all that apply”).
A link to the preview version of both survey’s can be found here:
The knowledge assessment contained in both surveys consisted of six questions. The first was “In your own words please describe the RGB color theory.” (This is solely so that I may analyze people’s theories). The remaining five were open response questions that participants were graded on. The questions were:
- “What are the primary colors of the RGB color theory?”
- “How would you make red with RGB color sliders?”
- “What are the secondary colors of the RGB color theory?”
- “How would you make yellow using the RGB color theory?” and
- “Can the RGB color theory be used for paint mixing?”
I scored participants based on the number of questions they got right and I also marked whether their answers contained a misconception (eg. The primary colors are red, yellow, and blue).
Due to my inability to correctly use Qualtrics I was not able to track which group participants were in, as such I could only track what group they were in based on whether or not they answered any of the questions I asked them throughout the study. This allows me to sort participants by whether or not they were asked questions while filling out the initial survey, however, because all participants participated a skill assessment which asked them questions about the RGB color theory immediately after they completed the first part of the survey the effects of whether or not they were asked questions was significantly undermined.
II. Results
This study found no correlation between any of the variables it attempted to measure. The average first and second scores of participants who were asked questions in the survey was 4.17/5.00 and 4.30/5.00, the average first and second scores of participants who were not asked questions in the survey were 3.91/5.00 and 4.27/5.00. However, there are still two findings of note that may illuminate avenues for future research.
The first is that the vast majority of people who took this study understood the material, even when you control for whether or not they had any preexisting knowledge of the RGB color theory. The average second assessment score for participants (count = 38) was 4.32 (37% misconception). When selecting only participants who said they knew “almost nothing” or “some” about the RGB color (count = 22) theory this score becomes 4.00 (45% misconception).
The second is that people who were working part time and in college scores slightly lower and significantly more misconceptions than people who were just in college, and people who had a full time job. The table below displays these results (misc = misconception).
|
|
Full-Time (7 people) |
College (7 people) |
Part-Time + College (4 people) |
|
Assessment 1 |
3.86/5.00 (43% misc) |
4.25/5.00 (50% misc) |
3.25/5.00 (75% misc) |
|
Assessment 2 |
4.43/5.00 (29% misc) |
4.00/5.00 (25% misc) |
3.50/5.00 (75% misc) |
It is also interesting to note here that Part-Time workers who were also in college did not see their rate of misconceptions drop in the followup assessment. There may be something here about how being overworked or spread thin impacts your ability to change or think critically about ideas you formed. But it is likely just statistical anomaly. More science is surely needed!
A link to my results spreadsheet can be found here (warning: this was not designed to make sense to anyone except for me): https://docs.google.com/spreadsheets/d/1cl-dQ5641FdYK43tg4uplkm7EmdGILYEnHz7Xfqf92c/edit?usp=sharing
III. Conclusion
This study was seriously flawed from its inception and I only noticed how ineffective the design was near the end of my collection of data. As I run studies in the future I really ought to learn from this by running a pilot study first and putting more thought into the design of my systems and surveys.
However all is not lost! The efficacy with which this survey imparted long lasting information regarding the RGB color theory means that I have created a good digital lesson. By giving this lesson to various people and removing various parts of it I should be able to identify what components lead to higher assessment scores and I can likely use this as a proxy to identify specific strategies that are crucial for understanding the RGB color theory. And I may then be able to generalize my findings to education in general.
IV. Ideas For Further Research
- This was already discussed, but this study supports the idea of mental load, that there’s limits to how much humans can be doing or learning at a time. (Relevant to this is the fact that this survey was conducted during finals week). I don’t care about this line of research so I will not be pursuing it.
- A complete dissection of this study could be incredibly fun. Turning it into a “digital lesson” and breaking it down into various parts that could be removed or tuned up or down could provide insight to the teaching and learning process within a lesson framework. It is also very much in the spirit of the original design for the study. Furthermore, I suspect this could be paradigm generating or strengthening for specific teaching strategies. It could lead to insights into what makes a good class or lesson.
- One person’s response to the question “Does the RGB color theory assist with paint mixing?” was “I think so!” which turned into “Yes, not with light mixing though!” when they took the survey again. Another person’s response to “How would you make red on RGB sliders?” went from “Red at 255% G&B at 0%” to “Red 100% G & B 0%” In both of these cases the misconception held by the participate strengthened during the later assessment. I don’t know why this is! (maybe personality?) There was also a person who went from “255?” to “256?” when taking the followup. So maybe this has to do with doubt or unsureness in one’s answer and not with misconception strengthening (or maybe these things are one and the same). Maybe doubt leads to a strengthened misconception. Maybe if you iterate doubt on a mind it destroys true conceptions and strengthens misconceptions with each step. Example in table below:
|
Iteration of doubt |
Idea 1 |
Idea 2 |
Idea 3 |
|
0 |
The maximum value is 255 |
Values might be expressed as percents or numbers |
I think RGB color theory can be used for paint! |
|
1 |
The maximum value might be 255 |
Values might be expressed as percents |
RGB color theory can be used for paint |
|
2 |
The maximum value is 255? |
Values are expressed as percents? |
??? |
|
3 |
The maximum value is 256? |
Values are expressed as percents. |
|
- Iteration 2 of idea 3 is where doubt gets tricky. Does doubt lead to “RBG color theory can be used for paint?” (adding doubt to the misconception) or does doubt lead to “RGB color theory can definitely be used for paint” adding further doubt between the misconception and the truth. Unclear. I suspect the former is true but I want the latter to be true…
- A participant who claimed to have “a fair amount” of knowledge in the RGB color theory scored a 2/5 on the first test with a seeming misconception, however they scored a 5/5 on the second test with no misconception. I wonder if this gets at an idea of “activation.” When they had just taken the survey their mind was “activated” and therefore they were attempting to express and think about ideas beyond the scope of the question, whereas, when they were taking the followup survey they were not activated in the same way.*
- One person’s response to the question “how do you make yellow with RGB sliders?” was “Move the red and green sliders to a positive value and have the blue sliders set to zero” in the first one and “you would turn the red and green sliders all the way up and the blue slider all the way down” in the second one. This is more accurate. There was also a person who said that the primary colors were “red green and cyan” in the first answer, and “red green and blue” in the second. In the first answer they also said that “green and cyan?” made yellow but in the second answer they said “I don’t remember.” I’m not sure what this means but I’ve wanted to do research into mindsets and the power of “I don’t know” and “I don’t remember” as mindsets for a while. I suspect this has some overlap with the Zen idea of Beginner Mind.
- One person described the RGB color theory as “RGB color theory is how colors are mixed to display other colors. It is based on the idea that the three primary colors of red yellow and blue can be mixed to create not just six, but all the other colors in the visual spectrum. It is not enough to just mix the colors because the amount of light can also effect how the colors are viewed.” And then preceded to answer every question wrong in the first survey. When they got the second survey they said “RBG color theory is the idea that all colors can be displayed with a combination of magenta, cyan and yellow. By mixing these colors in different proportions/saturation you can create all the other colors available for digital displays.” And got all but one question correct? Including the question of primary colors? What is going on? I am baffled by this response! They also said they did not research the RGB color at all! This is wild!
6. Meta-Conclusion
I began this study with no idea to study but I figured two things: 1. Measuring stuff is more interesting than not, therefore I should measure something, and 2. I don’t know quite what I’m measuring so I should create as many open response fields as possible, so that I potentially glean some insight into what is going on in people’s minds while they are learning.
The result of this was a study that measured nothing and contained results full of interesting questions! I’ve written down my research ideas based on the studies specifics, now I am going to list things that I’d like to be able to define for myself and maybe for other teachers or psychologists:
-
Mind thoughts:
-
What is a feeling?
- theory 1 - a feeling is sensory input. This can either be an introceptive sense (like hunger) or any extroceptive sense (like sight).
- I assume most people think that ‘feeling’ is the word to describe an introceptive sense and ‘sense’ is a word exclusively used to describe an extroceptive sense.
-
What is an emotion?
- theory 1 - an emotion is a collection of feelings that contextualize (or lay the groundwork for) thoughts.
-
What is a thought?
- theory 1 - an internal imagining of language.
-
What is a language?
- theory 1 -a meaningful formal system (read Gödel Escher Bach).
-
What is a rationality?
- theory 1 - a specific type of thought which can be compared or rationed with other thoughts.
-
What is imagination?
- theory 1 - imagination is a) the process by which an external object (including an external thought or language) is converted into a thought and b) the capacity of an individual to convert various external objects into thoughts, or to translate thoughts into other thoughts.
-
What is a feeling?
- What is learning? Can you learn falsehoods? Can you learn truths?
- What is changing one’s mind? Is this the same thing as learning?
- What is knowing?
- What is memory?
- What distinguishes human minds from minds of other animals, of computers, of plants, and of mushrooms? What makes them similar? In essence, what is a mind?
I am also curious why people say some people are good teachers and others are bad. People also tell me that I am, or that I would be, a good teacher and I wonder what they mean by this. They must have rationalities for evaluating teachers that they believe I can share or can translate into my own rationalities, but I have rarely been able to pin down what they are.
Last night Clarity said to me that I have a lot of enthusiasm when I teach and that makes for an engaging lesson and I said “it’s the only trick I got.” She told me that was not true and I argued that it was the only trick I had when teaching anything and that I could therefore combine that trick with any knowledge I had in order to teach it.
This is a broad oversimplification! I also utilize specific teaching techniques such as asking questions, explaining my answers, and trading roles around (at various points in the lesson I was the teacher, Clarity was the teacher, and Julius was the teacher).
This role-trading stands in contrast to many other teachers I’ve had but I suspect it is common in indigenous education practices. This is due to my experiences with indigenous studies professors and a conversation I had with Noah Romero.
I am not sure what this role-trading means for education but I suspect it means that a theory of learning must be rooted in play (specifically in role-play). Books (from larp camp) once used the word Alibi to describe why, once everyone is in costume but before the larp starts, everyone is super talkative and chummy (much more so than usual) and I believe classes can create that same phenomena of Alibi.
I also reckon that the feeling of giddiness is a part of the Alibi phenomena, and so is the feeling of curiosity, and so is the feeling of openness. The importance of feelings and emotions in teaching is why teaching is a sensitive art*. Teachers must be sensitive to the emotional states of all of their students because thoughts are dependent on emotions
I have no idea how to research any of this stuff going forward, but this is the rough framework I’m using to interpret results I get as I teach classes and study education.