Does AI actually help students learn? A recent experiment in a high school provides a cautionary tale.
Researchers at the University of Pennsylvania found that Turkish high school students who had access to ChatGPT while doing practice math problems did worse on a math test compared with students who didn’t have access to ChatGPT. Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning.
A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. But on a test afterwards, these AI-tutored students did no better. Students who just did their practice problems the old fashioned way — on their own — matched their test scores.
Perhaps unsurprisingly. Any sort of “assistance” with answers will do that.
Students have to learn why things work the way they do, and they won’t be able to grasp it without going ahead and doing every piece manually.
Of all the students in the world, they pick ones from a “Turkish high school”. Any clear indication why there of all places when conducted by a US university?
If I had access to ChatGPT during my college years and it helped me parse things I didn’t fully understand from the texts or provided much-needed context for what I was studying, I would’ve done much better having integrated my learning. That’s one of the areas where ChatGPT shines. I only got there on my way out. But math problems? Ugh.
When you automate these processes you lose the experience. I wouldn’t be surprised if you couldn’t parse information as well as you can now, if you had access to chat GPT.
It’s had to get better at solving your problems if something else does it for you.
Also the reliability of these systems is poor, and they’re specifically trained to produce output that appears correct. Not actually is correct.
I quickly learned how ChatGPT works so I’m aware of its limitations. And since I’m talking about university students, I’m fairly sure those smart cookies can figure it out themselves. The thing is, studying the biological sciences requires you to understand other subjects you haven’t learned yet, and having someone explain how that fits into the overall picture puts you way ahead of the curve because you start integrating knowledge earlier. You only get that from retrospection once you’ve passed all your classes and have a panoramic view of the field, which, in my opinion, is too late for excellent grades. This is why I think having parents with degrees in a related field or personal tutors gives an incredibly unfair advantage to anyone in college. That’s what ChatGPT gives you for free. Your parents and the tutors will also make mistakes, but that doesn’t take away the value which is also true for the AIs.
And regarding the output that appears correct, some tools help mitigate that. I’ve used the Consensus plugin to some degree and think it’s fairly accurate for resolving some questions based on research. What’s more useful is that it’ll cite the paper directly so you can learn more instead of relying on ChatGPT alone. It’s a great tool I wish I had that would’ve saved me so much time to focus on other more important things instead of going down the list of fruitless search results with a million tabs open.
One thing I will agree with you is probably learning how to use Google Scholar and Google Books and
pirating booksusing the library to find the exact information as it appears in the textbooks to answer homework questions which I did meticulously down to the paragraph. But only I did that. Everybody else copied their homework, so at least in my university it was a personal choice how far you wanted to take those skills. So now instead of your peers giving you the answers, it’s ChatGPT. So my question is, are we really losing anything?Overall I think other skills need honing today, particularly verifying information, together with critical thinking which is always relevant. And the former is only hard because it’s tedious work, honestly.
I read that comment, and use it similarly, as more a super-dictionary/encyclopedia in the same way I’d watch supplementary YouTube videos to enhance my understanding. Rather than automating the understanding process.
More like having a tutor who you ask all the too-stupid and too-hard questions to, who never gets tired or fed up with you.
Exactly this! That is why I always have at least one instance of AI chatbot running when I am coding or better said analyse code for debugging.
It makes it possible to debug kernel stuff without much pre-knowledge, if you are proficient in prompting your questions. Well, it did work for me.
The names of the authors suggest there could be a cultural link somewhere.
Ah thanks, that does appear to be the case.
The study was done in Turkey, probably because students are for sale and have no rights.
It doesn’t matter though. They could pick any weird, tiny sample and do another meaningless study. It would still get hyped and they would still get funding.
The paper only says it’s a collaboration. It’s pretty large scale, so the opportunity might be rare. There’s a chance that (the same or other) researchers will follow up and experiment in more schools.
I’m guessing there was a previous connection with some of the study authors.
I skimmed the paper, and I didn’t see it mention language. I’d be more interested to know if they were using ChatGPT in English or Turkish, and how that would affect performance, since I assume the model is trained on significantly more English language data than Turkish.
GPTs are designed with translation in mind, so I could see it being extremely useful in providing me instruction on a topic in a non-English native language.
But they haven’t been around long enough for the novelty factor to wear off.
It’s like computers in the 1980s… people played Oregon Trail on them, but they didn’t really help much with general education.
Fast forward to today, and computers are the core of many facets of education, allowing students to learn knowledge and skills that they’d otherwise have no access to.
GPTs will eventually go the same way.
Would kids do better if the AI doesn’t hallucinate?
Would snails be happier if it kept raining? What can we do to make it rain forever and all time?
Paradoxically, they would probably do better if the AI hallucinated more. When you realize your tutor is capable of making mistakes, you can’t just blindly follow their process; you have to analyze and verify their work, which forces a more complete understanding of the concept, and some insight into what errors can occur and how they might affect outcomes.
I’m not entirely sold on the argument I lay out here, but this is where I would start were I to defend using chatGPT in school as they laid out in their experiment.
It’s a tool. Just like a calculator. If a kid learns and does all their homework with a calculator, then suddenly it’s taken away for a test, of course they will do poorly. Contrary to what we were warned about as kids though, each of us does carry a calculator around in our pocket at nearly all times.
We’re not far off from having an AI assistant with us 24/7 is feasible. Why not teach kids to use the tools they will have in their pocket for the rest of their lives?
It’s a tool. Just like a calculator.
lol my calculator never “hallucinated”.
Ask your calculator what 1-(1-1e-99) is and see if it never halucinates (confidently gives an incorrect answer) still.
Yeah it’s like if you had a calculator and 10% of the time it gave you the wrong answer. Would that be a good tool for learning? We should be careful when using these tools and understand their limitations. Gen AI may give you an answer that happens to be correct some of the time (maybe even most of the time!) but they do not have the ability to actually reason. This is why they give back answers that we understand intuitively are incorrect (like putting glue on pizza), but sometimes the mistakes can be less intuitive or subtle which is worse in my opinion.
I think here you also need to teach your kid not to trust unconditionally this tool and to question the quality of the tool. As well as teaching it how to write better prompts, this is the same like with Google, if you put shitty queries you will get subpar results.
And believe me I have seen plenty of tech people asking the most lame prompts.
I remember teachers telling us not to trust the calculators. What if we hit the wrong key? Lol
Some things never change.
Human error =/= machine unreliability
You’re right. The commenter who made the comparison to Wikipedia made a better point.
I remember the teachers telling us not to trust Wikipedia, but they had utmost faith in the shitty old books that were probably never verified by another human before being published.
i mean, usually wikipedia’s references ARE from those old books
Eh I find they’re usually from a more direct source. The schoolbooks are just information sourced from who knows where else.
I don’t know about your textbooks and what ages you’re referring to but I remember many of my technical textbooks had citations in the back.
Yep, students these days have no idea about the back of their books and how useful the index can be and the citations after that.
Even after repeatedly pointing it out, they still don’t make use of it. Despite the index being nearly a cheat code in itself.
As adults we are dubious of the results that AI gives us. We take the answers with a handful of salt and I feel like over the years we have built up a skillset for using search engines for answers and sifting through the results. Kids haven’t got years of experience of this and so they may take what is said to be true and not question the results.
As you say, the kids should be taught to use the tool properly, and verify the answers. AI is going to be forced onto us whether we like it or not, people should be empowered to use it and not accept what it puts out as gospel.
This is true for the whole internet, not only AI Chatbots. Kids need to get teached that there is BS around. In fact kids had to learn that even pre-internet. Every human has to learn that you can not blindly trust anything, that one has to think critically. This is nothing new. AI chatbots just show how flawed human education is these days.
No shit
While I get that, AI could be handy for some subjects, where you wont put your future on. However using it extinsively for everything is quite an exaggeration.
deleted by creator
You could always try reading the article
Which, in a fun bit of meta, is a decent description of artificial “intelligence” too.
Maybe the real ChatGPT was the children we tested along the way
In the study they said they used a modified version that acted as a tutor, that refused to give direct answers and gave hints to the solution instead.
So it’s still not surprising since ChatGPT doesn’t give you factual information. It just gives you what it statistically thinks you want to read.
That’s like cheating with extra steps.
Ain’t getting hints on your in class exam.
Did those using tutor AI spend less time on learning? That would have been worth measuring
Interesting thought, I would be curious about this too.
Shocked, I tell you!
I’ve found AI helpful in asking for it to explain stuff. Why is the problem solved like this, why did you use this and not that, could you put it in simpler terms and so on. Much like you might ask a teacher.
To an extent, but it’s often just wrong about stuff.
It’s been a good second step for things I have questions about that I can’t immediately find good search results for. I don’t wanna get off topic but I have major beef with Stack Overflow and posting questions there makes me anxious as hell because I’ll do so much diligence to make sure it is clear, reproducible, and not a duplicate only for my questions to still get closed. It’s a major fucking waste of my time. Why put all that effort in when it’s still going to get closed?? Anyways – ChatGPT never gets mad at me. Sure, it’s often wrong as hell but it never berates me or makes me feel stupid for asking a question. It generally gets me close enough on topics that I can search for other terms in search engines and get different results that are more helpful.
Yep. My first interaction with GPT pro lasted 36 hours and I nearly changed my religion.
AI is the best thing to come to learning, ever. If you are a curious person, this is bigger than Gutenberg, IMO.
That sounds like a manic episode
I think this works great if the student is interested in the subject, but if you’re just trying to work through a bunch of problems so you can stop working through a bunch of problems, it ain’t gonna help you.
I have personally learned so much from LLMs (although you can’t really take anything at face value and have to look things up independently, but it gives you a great starting place), but it comes from a genuine interest in the questions I’m asking and things I dig at.
I have personally learned so much from LLMs
No offense but that’s what the article is also highlighting, naming that students, even the good, believe they did learn. Once it’s time to pass a test designed to evaluate if they actually did, it’s not that positive.
I mean…
私と日本語で会話したいか 😅
At the end of the day, I feel like it’s how you use the tool. “if you’re just trying to work through a bunch of problems so you can stop working through a bunch of problems, it ain’t gonna help you.” How do you think a bunch of kids using this are going to be using it when it comes to school work that they’re required to finish, but not likely actually interested in?
IKR?
Cheaters who cheat rather than learn don’t learn. More on this shocking development at 11.
Using ChatGPT as a study aid is cheating how?
Because a huge part about learning is actually figuring out how to extract/summarise information from imperfect sources to solve related problems.
If you use CHATGPT as a crutch because you’re too lazy to read between the lines and infer meaning from text, then you’re not exercising that particular skill.
I don’t disagree, but thats like saying using a calculator will hurt you in understanding higher order math. It’s a tool, not a crutch. I’ve used it many times to help me understand concepts just out of reach. I don’t trust anything LLMs implicitly but it can and does help me.
Congrats but there’s a reason teachers ban calculators… And it’s not always for the pain.
Take a college physics test without a calculator if you wanna talk about pain. And I doubt you could find a single person who could calculate trig functions or logarithms long hand. At some point you move past the point to prove you can do arithmetic. It’s just not necessary.
The real interesting thing here is whether an LLM is useful as a study aid. It looks like there is more research necessary. But an LLM is not smart. It’s a complicated next word predictor and they have been known to go off the rails for sure. And this article suggests its not as useful and you might think for new learners.
Chem is a long forgotten memory, but trig… It’s a matter of precision to do by hand. Very far from impossible… I’m pretty sure you learn about precision before trig… maybe algebra I or ii. E.g. can you accept pi as 3.14? Or 3.14xxxxxxxxxxxxxxxxxxxxxxxxxx
Trig is just rad with pi.
it’s not always for the pain
Use the rod, beat the child!
There are many reasons for why some teachers do some things.
We should not forget that one of them is “because they’re useless cunts who have no idea what they’re doing and they’re just powertripping their way through some kids’ education until the next paycheck”.
Not knowing how to add 6 + 8 just because a calculator is always available isn’t okay.
I have friends in my DnD session who have to count the numbers together on their fingers. I feel bad for the person. Don’t blame a teacher for wanting you to be a smarter more efficient and productive person, for banning a calculator.
In some cases I’d argue, as an engineer, that having no calculator makes students better at advanced math and problem solving. It forces you to work with the variables and understand how to do the derivation. You learn a lot more manipulating the ideal gas formula as variables and then plugging in numbers at the end, versus adding numbers to start with. You start to implicitly understand the direct and inverse relationships with variables.
Plus, learning to directly use variables is very helpful for coding. And it makes problem solving much more of a focus. I once didn’t have enough time left in an exam to come to a final numerical answer, so I instead wrote out exactly what steps I would take to get the answer – which included doing some graphical solutions on a graphing calculator. I wrote how to use all the results, and I ended up with full credit for the question.
To me, that is the ultimate goal of math and problem solving education. The student should be able to describe how to solve the problem even without the tools to find the exact answer.
Yes.
ChatGPT hallucinations inspire me to search for real references. It teaches we cannot blindly trust on things that are said. Teachers will commonly reinforce they are correct.
I do honestly have a tendency to more thoroughly verify anything AI tells me.
Yeh because it’s just like having their dumb parents do homework for them
Kids who take shortcuts and don’t learn suck at recalling knowledge they never had…
Actually if you read the article ChatGPT is horrible at math a modified version where chatGPT was fed the correct answers with the problem didn’t make the kids stupider but it didn’t make them any better either because they mostly just asked it for the answers.
Good tl;dr
The only reason we’re trying to somehow compromise and allow or even incorporate cheating software into student education is because the tech-bros and singularity cultists have been hyping this technology like it’s the new, unstoppable force of nature that is going to wash over all things and bring about the new Golden Age of humanity as none of us have to work ever again.
Meanwhile, 80% of AI startups sink and something like 75% of the “new techs” like AI drive-thru orders and AI phone support go to call centers in India and Philippines. The only thing we seem to have gotten is the absolute rotting destruction of all content on the internet and children growing up thinking it’s normal to consume this watered-down, plagiarized, worthless content.
deleted by creator
I took German in high school and cheated by inventing my own runic script. I would draw elaborate fantasy/sci-fi drawings on the covers of my notebooks with the German verb declensions and whatnot written all over monoliths or knight’s armor or dueling spaceships, using my own script instead of regular characters, and then have these notebook sitting on my desk while taking the tests. I got 100% on every test and now the only German I can speak is the bullshit I remember Nightcrawler from the X-Men saying. Unglaublich!
I just wrote really small on a paper in my glasses case, or hidden data in the depths of my TI86.
We love Nightcrawler in this house.
Meanwhile the teacher was thinking, “interesting tactic you’ve got there, admiring your art in the middle of a test”
God knows what he would have done to me if he’d caught me. He once threw an eraser at my head for speaking German with a Texas accent. In his defense, he grew up in a post-war Yugoslavian concentration camp.
Must be same era, my elderly off the boat Italian teacher in 90s Brooklyn used to hit me with his cane.
This isn’t a new issue. Wolfram alpha has been around for 15 years and can easily handle high school level math problems.
Except wolfram alpha is able to correctly explain step by step solutions. Which was an aid in my education.
deleted by creator
Where did you think you were?
What do young idiots use?
ChatGPT apparently lol
I can’t remember, but my dad said before he retired he would just pirate Wolfram because he was too old to bother learning whatever they were using. He spent 25 years in academia teaching graduate chem-e before moving to the private sector. He very briefly worked with one of the Wolfram founders at UIUC.
Edit: I’m thinking of Mathematica, he didn’t want to mess with learning python.
Maybe, if the system taught more of HOW to think and not WHAT. Basically more critical thinking/deduction.
This same kinda topic came up back when I was in middle/highschool when search engines became wide spread.
However, LLM’s shouldn’t be trusted for factual anything, same as Joe blows blog on some random subject. Did they forget to teach cross referencing too? I’m sounding too bitter and old so I’ll stop.
However, LLM’s shouldn’t be trusted for factual anything, same as Joe blows blog on some random subject.
Podcasts are 100% reliable tho
