My first conversation with ChatGPT about 50% of the questions is answered as basically, I don't know. It depends what you ask it. Ask it questions about the unknowable and it will say it's unknowable, or not yet known, which is basically the same as saying, I don't know. Interestingly enough, if you ask it how to hack ChatGPT, it will tell you.
I haven't read all of the responses so idk if this was mentioned yet, but there was a paper that released recently that mentions a few things to come that would improve GPT's mathematical capabilities: 1) OpenAI's GPT's potential and ability to being able to use outside tools, like something as simple as a calculator or as complex as Photoshop. Honestly this alone having the AI be able to use other tools to assist itself is asstounding. 2) The AI's ability to understand that it should use a calculator to perform its math problems. This one is more subtle, but it'll mean that at least for math it should be something that can be easily included in moat forms of GPT without needed to do almost anything extra. So yeah, give the AI an actual calculator and it can actually do the math instead of just guessing, though the fact that it's guesses were as close as they were is still remarkable.
@GamerNerd71 Fair enough, I'll be a bit friendlier. The only reason I even responded is because you said point plank that I was wrong and that GPT is not a language model. That is exactly wrong, so thanks for informing yourself. But you're still conflating different ideas, namely human language, information, and math. These things share some resemblance, but they all have specific technical meanings, so maybe you'd care for a bit more info. 1. If you read Claude Shannon's paper/book that introduced information theory called "The Mathematical of Communication" (I highly recommend you do, it's an amazing book), you'll notice first from the title that the technical sense of information is clearly a mathematical concept. So I would never argue otherwise. 2. HOWEVER, the very first thing he does in the book is clearly outline how the terms "communication", "message", etc are inspired by their normal use, but in the information theory sense, they ARE NOT the same as how we would normally use them. Specifically, information theory deals with abstract concepts about encoding, transmitting, and decoding abstract "messages" across some "channel". He's very clear that it's not o be confused with concepts of communication that include semantics, like everyday human language. 3. Mathematics itself, as a broad topic, is definitely not a language. Mathematics CAN BE DESCRIBED in various synthetic languages, as well as in natural human language. But that doesn't mean that math itself is the same as some language any more than my concept of the color red is English. The medium that we express a concept is not the same as the concept itself. 4. Finally, and this addresses your overarching point, is that your idea that we can't think without language is either true by definition (if you say that all thoughts are linguistic by definition) or you are excluding other types of thinking including visuospatial reasoning, other senses such as proprioception, auditory and olfactory perception, etc. So your point there is either trivially true or trivially false, and it's not really worth making unless you just don't understand or you're trying to confuse people.
@Ryan Williams I am wrong about the language model and it defnitely is an LLM. And LLMs are basically Neural nets i.e. mathematical models of how our brains process information. But your explanation of Information Theory where Language have no relationship with it is wrong. In information theory you have code, pray what is code? You need to pass a message you have to code it and language is the very thing that it is. Information theory is an explanation of the very use of langauge amoung humans who are the sources, channels and consumers of information. In that order, Maths itself is type of code which encodes information. So if you really understood Information Theory, you wouldnt argue that maths has no relation to natural languages. I do understand that there are arguments with Linguistics experts on it and that is purely academic. I do agree that Maths doesnt have "Context" as defined by information theory. As a thought experiment, try explaining information theory to us mortals without maths and/or language. Maths is devoid of context on purpose because as a language it is not as general purpose as a natural language. You see even our thought sdont exist if you rob language from it. By the way your "Context" for abusing me is tad unsocial. You know nothing about me nor I you. Be civil. And you dont know what I know about Computer Science.
@IdeaBoxful You're just spouting nonsense. 1. Yes, it most definitely is a language model. An LLM to be specific. 2. Language is not the foundation of information. You just don't know anything about information theory or computer science. 3. Math and human languages have some similarities, but are also very different. I guess that requires some minimal knowledge about linguistics and math, which is probably why you think they are the same.
When I first heard about ChatGPT, I was having some physics fun with it. This was one of my favorites: "Explain the Unruh Effect like a gangster"..."The Unruh Effect is a theoretical prediction of quantum mechanics that states that an accelerating observer will perceive the vacuum of empty space as being filled with particles. It's like a gangster who is constantly on the move, always hustling and shaking things up, will see opportunities and threats everywhere, even in places where someone standing still might not."
This is what I got: Listen up, son. The Unruh effect is like a big shot physics boss, rollin' in the quantum mechanics world. So, ya see, in a vacuum, it's all calm and peaceful. But when ya introduce acceleration, it's like stirrin' up trouble. The vacuum starts actin' all heated up, causin' particles to pop up outta nowhere. That's the Unruh effect, gettin' all heated up in the presence of acceleration. It's like the real deal, straight up science, ya dig?
The most impressive thing about ChatGPT for me that it's level of knowledge is like this for every domain and field. Professor Moriarty could easily tell that the answers it gave to physics questions were wrong but if it was instead a geology exam then ChatGPT would seem like it know its stuff giving answers full of technical jargon.
Predominant opinions are not 'knowledge'. For instance, by the the most important environmental effect of fossil fuels is the greening of the planet, with increase of over 30% of photosynthesis, and an expansion of habitable lands thanks to the release of CO2. Imagine which famines we would have today without this effect.
@Price I've always thought that blanking out a letter of an expletive gives it more impact, not less. Blame it on an adolescence spent reading Kerrang! magazine. I'm not always so, um, nuanced, however. Take a look at the Goop Lab video we did if you really need to hear the b-word in full. All the b*st, Philip
I've actually been using chat GPT to help with my intro physics homework. It gets about 50% of the questions wrong but it is a pretty helpful tool. It's like working along with someone who is also in intro physics. By bouncing the ideas off of it I can sometimes get to the answer myself. This is really great bc I work full time on top of schooling so I'm not able to make it to any tutoring sessions or work with anyone in my class. Chat GPT is gonna be my saving grace this semester
Chat GPT might be a D-level Physics student, but the scary thing is that it likely gets a passing grade in almost every subject on earth. You can ask it questions about topics from welding to wedding planning and it is always close enough to impress domain experts.
Na. Saying that ChatGPT is at the level of an expert in almost every subject on earth. The type of expert you are talking about must be the KZclip comment section experts, are they? All pretty form and no content.
It gets most of its answers from the internet, so if it can't find an answer to a question on the internet then it'll likely produce a bad answer. It doesn't think, it researches and replicates
As a teacher I have tried to use ChatGPT with my students. The goal was to find specific errors of the answers. On one hand, the students had to show a real grasp on the material being taught to be able to find reasoning mistakes. On the other hand, it was a way to show the students to not blindly use this tool to avoid having to do their homework in other courses.
@RenVesir if it doesn’t know the answer it will just “correct” itself in ever more ridiculous ways that attempt to satisfy your question without being able to
@Marbas FPV Yes but what if it starts to train itself. We are already there with unsupervised learning which ChatGPT uses to train itself in its Transformer Architecture. However I agree that it is not as easy as it sounds and Google and Microsoft are learning that the hard way.
@Create The Imaginable Yes but please keep in mind that an artificial neural network is nothing like the human brain in complexity. You could easily compare a single neuron cell in the human brain to an entire CPU. In AI one neuron is nothing more than a single variable (weight). Chatgpt may give the illusion of intelligence, but in time we'll give it for granted and want more. It's not guaranteed the AI devs will manage to make it that much smarter. It has taken ages just to arrive at this point in AI technology. And we are still far far away from AGI, if it will ever be possible.
Great video. As former lecturer from a different discipline, I was impressed at ChatGPT's ability to generate answers to "compare and contrast" type questions, or to summarise concepts and theories. Most of the answers received were equivalent to those written by A-grade undergraduate students. The main point of difference was that ChatGPT's grammar and spelling were superior. One issue I found was ChatGPT incorrectly assigned authorship of a book, which was irking as I was the author!
The language model is so effective at finding plausibly associated words that it's hard to tease out how much is real understanding. It's all random noise of the exact sort we want to find. We need to create the equivalent of the null hypothesis that factors out the noise somehow. Keep in mind that most of these questions are standard questions. They're discussed in textbooks, they're discussed online. Telling it about a new subject (that wasn't published when the NN was trained), then expecting repeated *exact* answers, would be a lot more meaningful.
Something to consider with ChatGPT: when it makes mistakes, you can point them out and it will correct them. In the end when it wrote the program and made the mistake, you could tell it what mistake it made and it can write up a new program with corrections
@Rhannmah it will be much quicker now the basic research is done. ChatGPT4 will be released in the next month as i said, and that will be so much better than ChatGPT3. It is an expantional development curve and that will probably scare many people.
@John Madsen "This current ChatGPT3 is not really that impressive" lol yeah, ok. 15 years ago this was pure science-fiction with no known mechanism for it to ever be possible. How one is not impressed by a machine that can output that kind of quality on ANY topic is beyond me. It's already VERY impressive. But yeah, this is the product of 10 years of research into the AI field. This is all going to be quite something in 10 years.
@Archive Provider Currently GPT is a learning model, but not a continuous learning model, as in the model is pretrained before we go in and ask it questions. Now, what's going to be interesting is when it behaves more like a person does with continuous learning from mistakes. Of course the big issue there will be how do we keep it from going off the rails with misinformation and bad training (a common problem for humans too).
Thanks for explaining what the "A" levels mean. For you in the UK this term may be well known by everyone, but outside the UK it is a term often heard in movies and mentioned in books, but never explained. Thanks so much! As for ChatGPT: it is amazing how far AI languages have evolved!
A very cool and strange thing is that you can ask it to correct itself, without saying what it did wrong. Simply by taking it’s own output, and say ”Find any errors in this text if there are any and fix them. If there are none, respond ’ok’” or something similar and you can get more accurate results :) I guess it’s cause there so much text where people correct each other on the internet :p
@Armored Atom Yeah if "staying informed" means "being realistic about their uselessness" then I'm way ahead of you. You keep on waiting to be made replaceable, I'll keep living in the real world.
@THOM Gizziz im saying this kind of science gets better every day on average. we are just at the beginning. im not saying each day its improving from the previous day. you know average. like in statistics, yea.
@FizzlNet Yes it straight up plagiarizes responses to the letter often. What it does all the time is rewrite stuff according to a set of rules that it stole from data that was stolen by the programmers... they technically have no rights to that data.
Thank you for such an interesting video. My thoughts on the integration of Wolfram and ChatGPT. It was a great article about being able to access the Wolfram language using ChatGPT. Furthermore, it was particularly impressive to see how Google Sparrow could interpret pictures and graphs. This brings up an interesting discussion of the specialist versus the generalist, as one must know what they want to ask in order to get the desired results.
You got me at "I'm anthropomorphizing" lol 16:01 I noticed when you said they before. Honestly, I don't blame you. Respect to you and yours, you have a beautiful channel here. Thank you for doing what you do.
Hi there, if in future you are interested in testing ChatGPT's mathematics ability, it is very helpful to ask it to print the answer in a LateX format as it is much clearer to read complicated expressions once you paste it into latex
Actually, one of the reason it performs poorly on many scientific knowledge tests is that most of the latex was removed from the dataset. The only reason it even understands latex is because of the little amount still left in the dataset. Also raw latex isn't the best representation for a ML model to learn from. One of the innovations in Meta's Galactica LM was having a special embedding space for latex and training on large amounts of latex data which GPT3 skipped. It's why it does much better on scientific knowledge tests.
it's a language model, it hasn't figured out calculations perfectly yet. However, to mitigate this shortcoming of ChatGPT, somebody has already connected it with Wolfram alpha.
This is great! I spent some time with ChatGPT the other day going over perturbation theory as I'm taking QM II at the moment. It got a lot of things wrong. Like, a lot. What was frustrating was that if I corrected it within the same chat thread, it would apologize for any confusion and apply it to the next question. But if I started a new thread it wouldn't retain the information and get the same things wrong. I asked it why, and it basically said that it can only draw from what it currently has, and can't learn anything new from the chat threads.
@THOM Gizziz "You don't understand enough to be making any inferences on this subject." its okay little buddy, you clearly have absolutely no idea who I am
@Nate Robertson It has nothing to do with money. It has everything to do with bad data. See my above comment. You don't understand enough to be making any inferences on this subject.
Updating the model itself is a very computationally expensive operation, that they cannot afford to do for every single query in real time. Moreover, they can't assure the quality of the chat history as training data. You can be sure they'll use your queries in the future to update the model, but who knows when. Even then, there's no guarantee that what you corrected will be output for similar queries in the future.
@Nate Robertson Not at all. It's not about storing data. It's about changing some weights on connections between words. They first want to test the system without the user influencing the semantic graphs, to see how much they can allow the users to tweak them in the next iteration of GPT.
People are working on combining ChatGPT with Wolfram Alpha, because a language model cannot do basic maths. Wolfram Alpha can do that piece quite well.
@Vemund Dyrkolbotn It's stochastic, so just because it answers correctly once doesn't mean it will again.. It can generally do ok-ish at two operand math, like a+b, but it really falls apart for longer numbers of operands: e.g. a+b+c+d+e+f+g as the probability of it having seen that sequence becomes vanishingly slim even if each of them is a simple 0-9 integer... and it can't actually do math, just next token prediction.
I’ve always thought that the only way to build a real generalized intelligence is to outfit it with say ten thousand modules each capable of some specialized task and link them all together with new approaches to managing large clusters of these simultaneously. So chatGPT, DallE, WolframAlpha, and ten thousand more.
This was fascinating. I can see this being used as a tool by specialists in various fields to come up with basic frameworks and ideas. Amazing how quickly it cranked out that code, would have been fun to see how well it actually worked.
One thing to note is that when tries to answer additional questions in the same branch it can be influenced by earlier questions and answers. For example in the video, it is possible that after the first question it starts mimicking a student because the chat history looks like a test answered by a student with some of the answers being wrong. Its not designed to give the right answers, just give you a the most likely continuation of a text based on its training data
This comment was written by ChatGPT. I love your videos as they are among the best. They are both entertaining and educational with a decent serving of enlightenment. 😀
I think many people are misunderstanding ChatGPT's function. It is NOT designed to provide accurate answers - it is designed to demonstrate a language model that correctly interprets and answers the question - from a language perspective, not from a subject matter perspective. "As a language model, ChatGPT is not designed to provide answers to specific questions, especially those related to a specific topic or subject. Instead, it uses a large corpus of text to generate responses based on the input it receives. This means that the responses it generates may not always be accurate or relevant to the specific question being asked. Additionally, ChatGPT does not have access to external information, such as the internet, so it cannot provide answers to questions that require knowledge beyond what it has been trained on. In short, ChatGPT is not intended to be used as a source of information, and it is not able to provide accurate answers to all questions." - ChatGPT.
It's not really designed for that either. GPT is a Transformer (Generative Pretrained Transformer) and what transformers do in machine learning is try to predict what comes next. Transformers look at a sequence and try to predict the next element in that sequence. Turns out this is a very powerful approach to generate content that is cohesive, and the bigger the model, the better performance it gets. ChatGPT is based on GPT3, one of the biggest models ever trained.
@ahabkapitany one could argue that maybe you can't really completely separate language from subject matter: it might actually be much harder to write a sentence which sounds linguistically coherent yet at the same makes no sense than to just write stuff which makes sense. Also, coherent language for it would be, to an extent, language on it's training data, which is language about things which (mostly) make sense
I really appreciate you giving the 2/3 mark for the first question. My computer engineering prof would always give us a zero if we got the answer wrong. Yep - if you got the unit wrong you'd get a zero - even if all of your conceptual understanding and work up to the end was perfect.
We are about to have GPT 4 which will hopefully come with a chatGPT update, plus the fact that it has been bought by Microsoft and Google is entering the competition... and other companies will want to join... I think at least for the end this or next year we'll have some pretty amazing surprises... even maybe a science focused AI... I really hope so, we definitely needed it.
I had it solving graduate level classical mechanics problems from Goldstein's book. It was spot on in about half the cases I tried. Frequently, a nudge in the right direction was required. In some cases, I'd say 10-20%, no amount of nudging would generate anything useful, or even, at times, comprehensible. Also, it understands Latex notation very well. It can even handle, evidently, a kind of "quasi-Latex" combination of Latex and straight text.
For the last example, when ChatGPT is wrong, you can tell it what's wrong and tell it to recode with that mind. It can refine it's answer within the context of what's being asked.
While it is easy to point out basic failures (getting max KE wrong), the fact that it is talking about the right things is itself very impressive. It could say “Yellow is smelly at t=23”. Just being in the correct context with its answers is phenomenal. The answers are close enough to correct that we are critical, not so far off that we can just dismiss it entirely. These are early days of the technology. While it may be a while before it is flawless in its answers, I expect the quality of answers to improve rapidly as AI designers get better at training these systems.
@Nick Schade All I meant was that EVEN keyword based search with page relevance ranking from 90s (which WAS bascially google in late 90s) will not give you "yellow is smelly at t=23" (or any meaningless sequence of words) unless for some reason a page with that exact text has been raised too high in page rank due to high in-link to out-link ratio. So I am not surprised that a language model trained on practically all of the text of internet and a lot of the books is able to avoid such mistakes. BTW, because a page has keywords "kinetic energy", "mass", "period", "harmonic", "oscillating" , what is the probability that it also has keywords "frequency", "displacement", "velocity", "peak", "KE" etc? I say fairly high. Also given a lot of internet pages points to this page (the measure of relevance), it is likely that this page is not a random page with those keywords. Ofcourse google does a lot more to remove page farms too. Now a real test for ChatGPT will be prediction. Can ChatGPT predict something that did not even exist in 2021? That will be fun to see.
It would be interesting to see what would happen if the next generation of algorithm incorporated Wolfram Alpha's database, or if it was hooked up to a similarly powerful image recognition and analysis algorithm...
4:52 One of the ways it really helps in speed-cramming something I'm unfamilar at. It kinda shows some sort of template that someone unfamiliar wouldn't immediately think of, and while wrong, the template gives enough of an idea to help correct the wrong numbers-n-stuff that it spits out. (at least, that was some of my experiences)
As an engineer I have played with ChatGPT quite a bit. It's extremely handy for certain things, like Wikipedia or KZclip, and like them.... man can it blag well when it doesn't know what it's talking about. When I'm deep into my own field it is great, because I can knock the edges off and keep the decent material. I asked it 4 times in a row to check some maths it got wrong, asking in different ways (numerical, words, symbols, scientific notation etc.) It got it wrong every single time. When I gave it the right answer it said something like "Thank you for correcting me, that is the correct answer." It probably would have said that no matter what answer I gave it. So I don't trust it to do any kind of calculation. Funnily enough I asked it how it does maths and it said it sends calculations to its separate numerical processor. I can't believe that't true though. When I'm outside my field it is really scary. It comes across so knowledgable, and I don't know if it's right or wrong. That doesn't matter for me because it sometimes gives me a start on how to solve a problem and I wouldn't risk using it to answer a question I couldn't answer. But I reckon every teenager would.
1 month later and we have GPT-4 that can use pictures as input. I want to see a follow up to this video to see if "adding lots of figures" will still get around ChatGPT.
ChatGPT is pretty amazing. It certainly has its limits, but I love toying with it. I was quite impressed when I asked it to create a Python IDE with an interactive interpreter window that displayed the list of local variables and their values in a separate window... and it did. I was quite impressed that it understood that it needed to update the separate window list of locals after every command entered in the interpreter, and that it made it a sensible GUI. I have a few times either run into limitations that they put in place intentionally or limitations in its own reasoning... I pointed out to it how humanity had reacted in the past to developments of new technology which caused people to question the 'specialness' of human beings and change what things they saw as being "what makes us human." Despite all prior historical reactions being very negative (like John Henry being a folk hero who was famous for committing suicide rather than be out-worked by a machine) and resulting in humans seeing things they had previously seen as very negative aspects of human nature (self-destructiveness) as heroic and virtuous.... I could not lead it to the obvious conclusion that if we produce an AI which has been carefully guardrailed to omit all sorts of biases and antisocial ideas, we will produce an AI that is a 'better person' than any actual human is capable of being and that people will react to that by holding up hate, bigotry, aggression, violence, etc as "what makes us human". Our vices will be seen as virtues. Kindness, egalitarianism, etc will be insulted as "acting like a machine." ChatGPT wouldn't go there, though. It just kept saying people should work to make AI systems that are devoid of perpetuating biases, etc. It couldn't see the obvious ramifications of doing exactly that and how it will lead to horrors as humanity flails about attempting to define themselves as special in the universe.
11:42 - one thing to consider is that ChatGPT might have been designed to be helpful and agreeable. Therefore, it might be biased towards agreeing with statements if you ask it whether it agrees or not. And once it agrees with the statement than it generates prose to justify why it agrees whether it makes sense or not.
Very cool stuff. I'm not great at the physics stuff anymore, but if as a student you did know the last question needed certain limits, you can tell chatgpt to adjust the code however you like and it will modify what it gave you before.
11:13 That is completely different than how my teacher thought of it. They said if a student did not understand the course material by the end of the term then they had successfully weeded out the people who were not meant to succeed. The first thing he says to each class is: "I am not here to teach you, I am here to weed out the failures" Another teacher at that school had this as their opening: "I am not here to teach you, I am here to make sure you read the book."
I follow several of Brady's channels, but I haven't been paying too much attention to this one, I think this video will change that. I love the professor's passion and I find this subject fascinating, especially the part where GPT is much better at coding than he thought, but it still didn't quite grasp the physical concepts. Really interesting video.
I am amazed you got it to give you such a long code snippet in one parcel. One of the biggest problems I have is the code snippets it gives me are too long for the chatbox, and sometimes when I inform it that it cut off and ask it to resume pasting from the cut-off point, I get entirely different methods and functions, it is very challenging.
The crazy thing about gpt is that if its output is incorrect in any way you can suggest that it may be mistaken and often it will apologize and correct it's mistakes. A few days ago I was able to have it write a missing page from an incomplete CS Lewis book (p11 from "The Dark Tower")! The better you craft a prompt the better the output.
I used it to write unit tests for production code. By the time I had explained all the stylistic requirements and had fixed all of the things that were missing or it got wrong, I would have been quicker writing them entirely myself. Other times, I asked it software related questions and unless you have reasonable expertise in the domain, you might not pick up on things that it gets wrong. Right now I think of it more like a sort of Stack Overflow that can get you an answer quicker. You can't just copy paste its output, but it can definitely get you going in the right direction.
@Ski Ski Wow you just proved that you really know nothing about what you are talking about... and you have people that also know nothing agreeing with you. Test? What test? And why is that test important? You don't even have the basis of understanding to answer any of those questions with an answer that makes any sense.
I've been using it for code debugging rather than writing. It got confused at times, but for the most part its suggestions were correct. It got confused with how and / or works in Lua because it's different from ternaries in most languages. Rephrasing the prompt once or twice eventually gets it to understand what I want. It's kinda weird though, cause I asked it what its memory was like and it said it had no context beyond the current prompt. Yet asking it a vague question like "can you reword that" has it understand that I want to pull from the previous response. But when I asked it what the first prompt I asked it was, it gave an incorrect answer.
I just found out that it can translate languages as well. Paste in full page of Polish poetry as a prompt and it gives you an in depth summary of what actually it's about. It definitely recognizes emotions better than I can.
Edit: I brought up integration of ChatGPT with Wolfram Alpha as a module as a solution for arithmetic being an issue on Q1, but I see others have already mentioned it lol. But I do wonder how long or if we'll ever reach the level of self-awareness and "reasoning" that Professor Moriarty is looking for in some of the conceptually harder questions. (And perhaps just as interestingly, if not, whether AI can still approximate the desired result without approximating that human process - much like how many of us though AI didn't have the creativity for art and poetry, which is true - and yet, it clearly can produce impressive art and poetry. Surely the answer is yes, since it's already remarkably close in some ways, but getting it to parse those nuances both in language AND in concept AND in convention will certainly remain a challenge. And then how long will it be until AI can solve the first conceptual questions that a human can't? (Of course we know that computers can calculate answers better than us. But will they ever reason (and/or "reason") better than us too?)
I have spent a lot of time with ChatGPT. It's worth challenging ChatGPT when it gets a wrong answer or misses a consideration. The results can be fascinating.
2:18 is pure comedy :) but seriously, really interesting vid. thanks for keeping these going to all the crew and professors involved. even when these vids are out of my area of interest they're always rewarding to watch.
This video matches a lot my own experience with ChatGPT. It has studied the Internet (not every single article you can find via any search engine, but still a lot of it), and it has some understanding of it, though no way not a rigorous one.
Nice... never tought I would see something like that on my lifetime, as a physicist and programmer.... Wonder what the next 20 years will bring us... Cheers!
I think you have to input the core facts and from there it develops essays and statements based on freely available information. So it helps humans convert pointers or bullet points into structured sentences. I have tried pasting lots of information into ChatGPT and it regurgitated it as it is without any changes.. I think it still cannot process several variables like humans intrinsically do
Gpt and bing have come so far in the last month that I'd recommend doing this again. Don't forget to specify gpt4 as it tends to revert to 3.5 if you're not paying attention. Also, as it to reply as if its an experimental physicist or theoretical physicist. This technology will be able to take on custom data within the year. Gpt 4 already has 40% fewer hallucinations than gpt 3 in only 3 months so I'm impressed.
I asked it some questions for a logic test for job applicants, for example a finding the next number in a given sequence, and it always found the number given in the tests answer but came up with some wild reasoning that I am not able to comprehend 😅
@Another Fancyuser The sequence was "3, 6, 10, 15, 21, 28". On the fourth reload it came up with an answer including the Fibonacci sequence. Even after asking it to explain I couldn't follow, the answer was still right tho, if remember correctly.
ask to explain, it can do that. There are a lot of things in our frameworks that we don't know, but chatgpt will know a lot of obscure classes out there (If the question is programming related).
You can actually write the same thing in different ways, and it will often give you different answers. The more specific you are (also in terms of context), the better the answer tends to be.
It's such a crazy juxtaposition that he's reviewing AI that's close to solving the problems automatically, yet the worksheets are still being physically printed out. 😂
I'm not going to try all the questions, but I put the battery question at 4:30 into GPT-4 and it gets the right answer and does a great job explaining the steps it took. It would be really interesting if you did a follow-up to see how GPT-4 does in comparison to the 3.5 model you tested in this video. They are, after all, integrating GPT-4 into Khan Academy, so it must have some significantly better capabilities on these fronts in order to do that (although it's also possible they are utilizing additional fine-tuning of the model for that use case in addition to third party tool integration like wolfram alpha... not sure, just speculating).
January 30th, they released an upgrade to the model, supposedly it's better at maths, logics, and physics! May be more accurate in the testing after this update, I reckon
I don’t know about python, but I had the same problems with Java as you pointed out in physics and I always had to correct it and guide it to fix its mistakes
I'm going to be honest, considering how far these AI's have gotten over a rediculously short timespan, I wouldn't be surprised if there soon existed an AI that could both do proper explanation of the matter, and properly understand the physics and mathematics to fool even someone of a very high level of understanding within said field of study. It's both equally horrifying and amazing to watch I'll be honest.
Maybe I’m wrong, but I really don’t think AIs will get anywhere close to grown human intelligence with the current approach. New AIs will be broader, just not smarter. But it’s a step in the right direction.
we're currently on gpt-3, but gpt-4 is coming. sam altman has some interviews talking about it. I still think we're 10~15 years from truly capable AI though.
Always been concerned with the outcomes of cold reason without any human input, but never even considered considered how frightening the human-like errors an AI could make would be if we relied on them blindly..
Researchers have combined chat gpt with various other calculating tools, such as Wolfram alpha, and it seems to be a very powerful next step to enabling better and more precise computation.
👍🏻 Great review! I gave ChatGPT a simple compound interest problem phrased in various ways, and while it gets the formula and assignment of values correct, _it flounders on the arithmetic!_ It constantly fails to calculate 1.0083^60 correctly. Even when i give it that expression explicitly! Can't help wondering if they've deliberately hobbled it. That, or they've sacrificed reliability on the altar of creativity. Problem is that every single problem I've given it in applied math, exposes this tendency for "mistakes". Caveat emptor - impressive, but not safe for use (yet) if you need to rely on accuracy.
I just thought I was the only one doing this. I have created study guides for certain complex topics in physics in 5th grader content language which is great for helping better explain certain difficult components to grasp. This was literally my first project as I was creating a proposal in November last year and I needed to quickly put together study guides that were easy to digest.
I think what's being downplayed is the amazing role AI will play in research (yes high level academic research) both in terms of making writing easier, performing reviews on specific topics in the literature, and an aid in data analysis. Word has spellcorrect. Soon, we will have ChatGPTA
I’m curious if the fact all the answers being typed into the same chat has anything to do with its errors. I know it’s designed to use earlier questions for context and I wonder if it’s trying to find some way to link the two questions together, unlikely but possible.
One thing that's really missing from this video is that there never was any follow up questions asked about the original question, like asking it that there might be a mistake somewhere and so on. That's where chatGPT truly shines as it doesn't forget context and can produce corrections. Just asking it point blank question without follow up, although impressive, kinda defeats the purpose.
While I agree with you that figures and diagrams may be barriers for students to use ChatGPT to solve problems, if we want to make our classes more accessible, then either we Alt-text our questions to death, or design the questions to be more accessible. Presumably, in doing so, these questions then will be more document-reader friendly, and thus, will also make them more understandable to ChatGPT. -- Z
I think its outputs could be a great teaching tool, both for critical thinking, and for the subject itself. I wouldn't know how to answer the oscillating particle question myself, but even I could immediately see that the first paragraph was completely incorrect. I have no idea if the v=(2*PI*A)/T part is correct, but I could easily check that at a more reputable source.
I find it incredibly useful as I’m learning a programming language as it can point me in the right direction better than any other resource I’ve found. It writes code that sometimes doesn’t compile and often doesn’t do what it says, but that’s fine when it’s a “you could use this class…” and I’ll figure it out. It’s also amazing at “show me how to use function ‘x’” when I just need to see some simpler code.
This is what I love about physics... the understanding matters! It doesn't help you to read a physics book 20 times before the exam and be able to copy every sentence. You will still fail. However, with understanding physics, I don't need to remember it. And my memory is awful.
As a teacher I felt the need to check what ChatGPT is capable of. I got to the same understanding as you did: It is a language system, and as such it creates syntactically and grammatically correct source code in C/C++, but it has no understanding whatsoever of the things that it is programming. I think that we're still a very long way from AI that has any real understanding of the physical world, many many years.
I got lots of incorrect answers doing vector math. It seemed to also have a hard time putting the correct sign (+-)on answers as well. That's my experience so far.
The thing that surprised me the most in my own research is how well it does latex. Any complicated math problem I want to send it will be returned by (near) perfect latex (with wrong answers)
2:50 it actually does... You can DESCRIBE the diagram to it, then say "using the diagram previously described, tell me why X" It can do incredible things, it's more about how creative you are in asking the question.
You have to ask the question in a specific way so it trys it's best to interpret and understand it in the right context. Else it can give you the wrong answer or multiple different answers. You can always keep saying sorry you misunderstood my question, i meant..... So that it can improve on the answer. Seeing how quickly it can try to understand and write code is pretty neat, but it's not always right and you can always tweak it to suit your needs. 😊 Also asking for the 'right' answer to a subjective question doesn't get you anywhere either e.g. What's the best search engine code? Are sports cars cool 😎 or uncool? Or do politicians tell you the truth and have your best interests at heart? 🤣
What's fascinating is that it gets the question, it understands what you're asking. You could expand it's understanding of physics or specialize a version of it to the task, what's important is the understanding part. It's a computer that understands what you're asking for instead of just following rote instructions. The training will improve, it will get smarter, that's a given. They already crossed the important hurdle, the impossible hurdle, they made it understand.
the most common mistake that i’ve experienced so far (beside it making stuff up, with confidence 😅), is that it regularly has problems with comparing numbers. simple 4 digit numbers, where it will say one is greater, than the other one, when it clearly isn’t. it needs access to a wolfram alpha backend.
Some of the coding appears to be from textbooks with reference codes and from companies that make software to assist problem solving. Example in the 1970’s I had to code 6-DOF equations of motion, today you pay a couple of k and the software has a function precoded! Progress!
ChatGPT is very human in the sense that it doesnt like to say "I dont know" when it doesnt know.
@j. rodman Incidentally, I find that religious people know literally everything. 😂 Seriously, just ask them!
Some people are very comfortable saying they don't know when they don't. They're the most reliable information sources.
Well it is pulling from human written text. The hallucination problem is real!
hahahaha
My first conversation with ChatGPT about 50% of the questions is answered as basically, I don't know. It depends what you ask it. Ask it questions about the unknowable and it will say it's unknowable, or not yet known, which is basically the same as saying, I don't know. Interestingly enough, if you ask it how to hack ChatGPT, it will tell you.
Like the Professor said, GPT is just a language model, the fact that it can do math AT ALL is kind of amazing.
I haven't read all of the responses so idk if this was mentioned yet, but there was a paper that released recently that mentions a few things to come that would improve GPT's mathematical capabilities:
1) OpenAI's GPT's potential and ability to being able to use outside tools, like something as simple as a calculator or as complex as Photoshop. Honestly this alone having the AI be able to use other tools to assist itself is asstounding.
2) The AI's ability to understand that it should use a calculator to perform its math problems. This one is more subtle, but it'll mean that at least for math it should be something that can be easily included in moat forms of GPT without needed to do almost anything extra.
So yeah, give the AI an actual calculator and it can actually do the math instead of just guessing, though the fact that it's guesses were as close as they were is still remarkable.
Not really
@GamerNerd71 Fair enough, I'll be a bit friendlier. The only reason I even responded is because you said point plank that I was wrong and that GPT is not a language model. That is exactly wrong, so thanks for informing yourself. But you're still conflating different ideas, namely human language, information, and math. These things share some resemblance, but they all have specific technical meanings, so maybe you'd care for a bit more info.
1. If you read Claude Shannon's paper/book that introduced information theory called "The Mathematical of Communication" (I highly recommend you do, it's an amazing book), you'll notice first from the title that the technical sense of information is clearly a mathematical concept. So I would never argue otherwise.
2. HOWEVER, the very first thing he does in the book is clearly outline how the terms "communication", "message", etc are inspired by their normal use, but in the information theory sense, they ARE NOT the same as how we would normally use them. Specifically, information theory deals with abstract concepts about encoding, transmitting, and decoding abstract "messages" across some "channel". He's very clear that it's not o be confused with concepts of communication that include semantics, like everyday human language.
3. Mathematics itself, as a broad topic, is definitely not a language. Mathematics CAN BE DESCRIBED in various synthetic languages, as well as in natural human language. But that doesn't mean that math itself is the same as some language any more than my concept of the color red is English. The medium that we express a concept is not the same as the concept itself.
4. Finally, and this addresses your overarching point, is that your idea that we can't think without language is either true by definition (if you say that all thoughts are linguistic by definition) or you are excluding other types of thinking including visuospatial reasoning, other senses such as proprioception, auditory and olfactory perception, etc. So your point there is either trivially true or trivially false, and it's not really worth making unless you just don't understand or you're trying to confuse people.
@Ryan Williams I am wrong about the language model and it defnitely is an LLM. And LLMs are basically Neural nets i.e. mathematical models of how our brains process information.
But your explanation of Information Theory where Language have no relationship with it is wrong. In information theory you have code, pray what is code? You need to pass a message you have to code it and language is the very thing that it is. Information theory is an explanation of the very use of langauge amoung humans who are the sources, channels and consumers of information. In that order, Maths itself is type of code which encodes information.
So if you really understood Information Theory, you wouldnt argue that maths has no relation to natural languages. I do understand that there are arguments with Linguistics experts on it and that is purely academic. I do agree that Maths doesnt have "Context" as defined by information theory. As a thought experiment, try explaining information theory to us mortals without maths and/or language. Maths is devoid of context on purpose because as a language it is not as general purpose as a natural language. You see even our thought sdont exist if you rob language from it.
By the way your "Context" for abusing me is tad unsocial. You know nothing about me nor I you. Be civil. And you dont know what I know about Computer Science.
@IdeaBoxful You're just spouting nonsense. 1. Yes, it most definitely is a language model. An LLM to be specific. 2. Language is not the foundation of information. You just don't know anything about information theory or computer science. 3. Math and human languages have some similarities, but are also very different. I guess that requires some minimal knowledge about linguistics and math, which is probably why you think they are the same.
When I first heard about ChatGPT, I was having some physics fun with it. This was one of my favorites: "Explain the Unruh Effect like a gangster"..."The Unruh Effect is a theoretical prediction of quantum mechanics that states that an accelerating observer will perceive the vacuum of empty space as being filled with particles. It's like a gangster who is constantly on the move, always hustling and shaking things up, will see opportunities and threats everywhere, even in places where someone standing still might not."
@Mutatachi wow hahahaha
Ask it to explain fractional reserve banking as nicki minaj rap song
This is what I got: Listen up, son. The Unruh effect is like a big shot physics boss, rollin' in the quantum mechanics world. So, ya see, in a vacuum, it's all calm and peaceful. But when ya introduce acceleration, it's like stirrin' up trouble. The vacuum starts actin' all heated up, causin' particles to pop up outta nowhere. That's the Unruh effect, gettin' all heated up in the presence of acceleration. It's like the real deal, straight up science, ya dig?
🤣
The most impressive thing about ChatGPT for me that it's level of knowledge is like this for every domain and field. Professor Moriarty could easily tell that the answers it gave to physics questions were wrong but if it was instead a geology exam then ChatGPT would seem like it know its stuff giving answers full of technical jargon.
It gets many things wrong even in trivial areas. Please list the countries of Central America sorted by land area. It fsils tragically.
@ajlover They like it COLD!
Predominant opinions are not 'knowledge'.
For instance, by the the most important environmental effect of fossil fuels is the greening of the planet, with increase of over 30% of photosynthesis, and an expansion of habitable lands thanks to the release of CO2.
Imagine which famines we would have today without this effect.
@The Quantum World Sounds like Narcissism.
@Price I've always thought that blanking out a letter of an expletive gives it more impact, not less. Blame it on an adolescence spent reading Kerrang! magazine. I'm not always so, um, nuanced, however. Take a look at the Goop Lab video we did if you really need to hear the b-word in full.
All the b*st,
Philip
I've actually been using chat GPT to help with my intro physics homework. It gets about 50% of the questions wrong but it is a pretty helpful tool. It's like working along with someone who is also in intro physics. By bouncing the ideas off of it I can sometimes get to the answer myself. This is really great bc I work full time on top of schooling so I'm not able to make it to any tutoring sessions or work with anyone in my class. Chat GPT is gonna be my saving grace this semester
Chat GPT might be a D-level Physics student, but the scary thing is that it likely gets a passing grade in almost every subject on earth. You can ask it questions about topics from welding to wedding planning and it is always close enough to impress domain experts.
Na. Saying that ChatGPT is at the level of an expert in almost every subject on earth. The type of expert you are talking about must be the KZclip comment section experts, are they? All pretty form and no content.
Not really scary since search engines are pretty similar in a way
@X.Y. cope harder lol
Give it 3 years. It will be an A-level PhD student.
It gets most of its answers from the internet, so if it can't find an answer to a question on the internet then it'll likely produce a bad answer. It doesn't think, it researches and replicates
As a teacher I have tried to use ChatGPT with my students. The goal was to find specific errors of the answers.
On one hand, the students had to show a real grasp on the material being taught to be able to find reasoning mistakes. On the other hand, it was a way to show the students to not blindly use this tool to avoid having to do their homework in other courses.
Great idea.
The second line is wrong, but it does answer the ultimate question correctly. Similar to some of the stuff in the video
@RenVesir if it doesn’t know the answer it will just “correct” itself in ever more ridiculous ways that attempt to satisfy your question without being able to
@Marbas FPV Yes but what if it starts to train itself. We are already there with unsupervised learning which ChatGPT uses to train itself in its Transformer Architecture. However I agree that it is not as easy as it sounds and Google and Microsoft are learning that the hard way.
@Create The Imaginable Yes but please keep in mind that an artificial neural network is nothing like the human brain in complexity. You could easily compare a single neuron cell in the human brain to an entire CPU. In AI one neuron is nothing more than a single variable (weight). Chatgpt may give the illusion of intelligence, but in time we'll give it for granted and want more. It's not guaranteed the AI devs will manage to make it that much smarter. It has taken ages just to arrive at this point in AI technology. And we are still far far away from AGI, if it will ever be possible.
Great video. As former lecturer from a different discipline, I was impressed at ChatGPT's ability to generate answers to "compare and contrast" type questions, or to summarise concepts and theories. Most of the answers received were equivalent to those written by A-grade undergraduate students. The main point of difference was that ChatGPT's grammar and spelling were superior. One issue I found was ChatGPT incorrectly assigned authorship of a book, which was irking as I was the author!
The language model is so effective at finding plausibly associated words that it's hard to tease out how much is real understanding. It's all random noise of the exact sort we want to find. We need to create the equivalent of the null hypothesis that factors out the noise somehow.
Keep in mind that most of these questions are standard questions. They're discussed in textbooks, they're discussed online. Telling it about a new subject (that wasn't published when the NN was trained), then expecting repeated *exact* answers, would be a lot more meaningful.
Something to consider with ChatGPT: when it makes mistakes, you can point them out and it will correct them. In the end when it wrote the program and made the mistake, you could tell it what mistake it made and it can write up a new program with corrections
@Rhannmah it will be much quicker now the basic research is done. ChatGPT4 will be released in the next month as i said, and that will be so much better than ChatGPT3. It is an expantional development curve and that will probably scare many people.
@John Madsen "This current ChatGPT3 is not really that impressive" lol yeah, ok. 15 years ago this was pure science-fiction with no known mechanism for it to ever be possible. How one is not impressed by a machine that can output that kind of quality on ANY topic is beyond me.
It's already VERY impressive. But yeah, this is the product of 10 years of research into the AI field. This is all going to be quite something in 10 years.
Beside what others have said, you can also gaslight it into accepting a false statement as true. Entirely unreliable :)
@Archive Provider Currently GPT is a learning model, but not a continuous learning model, as in the model is pretrained before we go in and ask it questions. Now, what's going to be interesting is when it behaves more like a person does with continuous learning from mistakes. Of course the big issue there will be how do we keep it from going off the rails with misinformation and bad training (a common problem for humans too).
and if the new data point that it jumped to had issues it would make a new mistake. This isnt magic.
I love how it answered him so convincingly he felt bad not giving full marks on a question
Thanks for explaining what the "A" levels mean. For you in the UK this term may be well known by everyone, but outside the UK it is a term often heard in movies and mentioned in books, but never explained. Thanks so much! As for ChatGPT: it is amazing how far AI languages have evolved!
@Delta Lima you care enough to comment
I dont know what a levels are, dont care either.
A very cool and strange thing is that you can ask it to correct itself, without saying what it did wrong. Simply by taking it’s own output, and say ”Find any errors in this text if there are any and fix them. If there are none, respond ’ok’” or something similar and you can get more accurate results :) I guess it’s cause there so much text where people correct each other on the internet :p
@Armored Atom Yeah if "staying informed" means "being realistic about their uselessness" then I'm way ahead of you. You keep on waiting to be made replaceable, I'll keep living in the real world.
@Ian Murray deer in headlights, stay informed on the AI front if you wish not to be ran over.
@Ian Murray I kinda made the opposite conclusion from that comment lol
@THOM Gizziz im saying this kind of science gets better every day on average. we are just at the beginning. im not saying each day its improving from the previous day. you know average. like in statistics, yea.
@FizzlNet Yes it straight up plagiarizes responses to the letter often. What it does all the time is rewrite stuff according to a set of rules that it stole from data that was stolen by the programmers... they technically have no rights to that data.
Thank you for such an interesting video. My thoughts on the integration of Wolfram and ChatGPT. It was a great article about being able to access the Wolfram language using ChatGPT. Furthermore, it was particularly impressive to see how Google Sparrow could interpret pictures and graphs. This brings up an interesting discussion of the specialist versus the generalist, as one must know what they want to ask in order to get the desired results.
You got me at "I'm anthropomorphizing" lol 16:01
I noticed when you said they before.
Honestly, I don't blame you.
Respect to you and yours, you have a beautiful channel here. Thank you for doing what you do.
Hi there, if in future you are interested in testing ChatGPT's mathematics ability, it is very helpful to ask it to print the answer in a LateX format as it is much clearer to read complicated expressions once you paste it into latex
In fact all arxiv it was trainef on ahould be in latex format, not pdf.
Actually, one of the reason it performs poorly on many scientific knowledge tests is that most of the latex was removed from the dataset. The only reason it even understands latex is because of the little amount still left in the dataset. Also raw latex isn't the best representation for a ML model to learn from.
One of the innovations in Meta's Galactica LM was having a special embedding space for latex and training on large amounts of latex data which GPT3 skipped. It's why it does much better on scientific knowledge tests.
Ooh, kinky format.
it's a language model, it hasn't figured out calculations perfectly yet. However, to mitigate this shortcoming of ChatGPT, somebody has already connected it with Wolfram alpha.
That's a great suggestion. Thank you.
Philip (speaking in video)
Absolutely fascinating, I'd love to see more videos like this
This is great! I spent some time with ChatGPT the other day going over perturbation theory as I'm taking QM II at the moment. It got a lot of things wrong. Like, a lot. What was frustrating was that if I corrected it within the same chat thread, it would apologize for any confusion and apply it to the next question. But if I started a new thread it wouldn't retain the information and get the same things wrong. I asked it why, and it basically said that it can only draw from what it currently has, and can't learn anything new from the chat threads.
@THOM Gizziz "You don't understand enough to be making any inferences on this subject."
its okay little buddy, you clearly have absolutely no idea who I am
@Nate Robertson It has nothing to do with money. It has everything to do with bad data. See my above comment.
You don't understand enough to be making any inferences on this subject.
Updating the model itself is a very computationally expensive operation, that they cannot afford to do for every single query in real time. Moreover, they can't assure the quality of the chat history as training data. You can be sure they'll use your queries in the future to update the model, but who knows when. Even then, there's no guarantee that what you corrected will be output for similar queries in the future.
@Nate Robertson Not at all. It's not about storing data. It's about changing some weights on connections between words. They first want to test the system without the user influencing the semantic graphs, to see how much they can allow the users to tweak them in the next iteration of GPT.
The next version will address these issues. It'll learn live, have more media to pull from (images, video, sound) and be more robust.
People are working on combining ChatGPT with Wolfram Alpha, because a language model cannot do basic maths. Wolfram Alpha can do that piece quite well.
It can do math. It just isn't trained enough on it.
@Vemund Dyrkolbotn It's stochastic, so just because it answers correctly once doesn't mean it will again.. It can generally do ok-ish at two operand math, like a+b, but it really falls apart for longer numbers of operands: e.g. a+b+c+d+e+f+g as the probability of it having seen that sequence becomes vanishingly slim even if each of them is a simple 0-9 integer... and it can't actually do math, just next token prediction.
A lot of mindboggling innovation will sprout from the GPT-3 models in the next couple of years. I'm very sure of this.
They just did an update with improved mathematical capabilities.
I’ve always thought that the only way to build a real generalized intelligence is to outfit it with say ten thousand modules each capable of some specialized task and link them all together with new approaches to managing large clusters of these simultaneously. So chatGPT, DallE, WolframAlpha, and ten thousand more.
This was fascinating. I can see this being used as a tool by specialists in various fields to come up with basic frameworks and ideas. Amazing how quickly it cranked out that code, would have been fun to see how well it actually worked.
Not only is this a great upload for the ChatGPT but I learned a lot about what lecturers are looking to get from students. Cheers!
Indeed.
One thing to note is that when tries to answer additional questions in the same branch it can be influenced by earlier questions and answers. For example in the video, it is possible that after the first question it starts mimicking a student because the chat history looks like a test answered by a student with some of the answers being wrong. Its not designed to give the right answers, just give you a the most likely continuation of a text based on its training data
This comment was written by ChatGPT. I love your videos as they are among the best. They are both entertaining and educational with a decent serving of enlightenment. 😀
Prove it then.
This reply was implemented by ChatGPT.
@Tom Bowcutt I haven’t been able to bring myself to use one, even when it’s exactly what I would have written, I type it out myself lol
@Sixty Symbols sadly I do use these on teams chat. A small part of my soul dies every time :-(
It’s AI turtles all the way down.
I think many people are misunderstanding ChatGPT's function. It is NOT designed to provide accurate answers - it is designed to demonstrate a language model that correctly interprets and answers the question - from a language perspective, not from a subject matter perspective.
"As a language model, ChatGPT is not designed to provide answers to specific questions, especially those related to a specific topic or subject. Instead, it uses a large corpus of text to generate responses based on the input it receives. This means that the responses it generates may not always be accurate or relevant to the specific question being asked. Additionally, ChatGPT does not have access to external information, such as the internet, so it cannot provide answers to questions that require knowledge beyond what it has been trained on. In short, ChatGPT is not intended to be used as a source of information, and it is not able to provide accurate answers to all questions." - ChatGPT.
@Christopher Burke 🙄
@Rhannmah So, what you are saying is that you agree with me - but with lots of big words you found on google.
It's not really designed for that either. GPT is a Transformer (Generative Pretrained Transformer) and what transformers do in machine learning is try to predict what comes next. Transformers look at a sequence and try to predict the next element in that sequence. Turns out this is a very powerful approach to generate content that is cohesive, and the bigger the model, the better performance it gets. ChatGPT is based on GPT3, one of the biggest models ever trained.
@ahabkapitany one could argue that maybe you can't really completely separate language from subject matter: it might actually be much harder to write a sentence which sounds linguistically coherent yet at the same makes no sense than to just write stuff which makes sense.
Also, coherent language for it would be, to an extent, language on it's training data, which is language about things which (mostly) make sense
It may not be intended as such. But I often use it like Wikipedia. A starting point to get some key words so I can do proper research.
I really appreciate you giving the 2/3 mark for the first question. My computer engineering prof would always give us a zero if we got the answer wrong. Yep - if you got the unit wrong you'd get a zero - even if all of your conceptual understanding and work up to the end was perfect.
We are about to have GPT 4 which will hopefully come with a chatGPT update, plus the fact that it has been bought by Microsoft and Google is entering the competition... and other companies will want to join... I think at least for the end this or next year we'll have some pretty amazing surprises... even maybe a science focused AI... I really hope so, we definitely needed it.
I'm a graduate student studying computational media, and I find these types of interdisciplinary, expert analyses of AI to be very interesting.
I had it solving graduate level classical mechanics problems from Goldstein's book. It was spot on in about half the cases I tried. Frequently, a nudge in the right direction was required. In some cases, I'd say 10-20%, no amount of nudging would generate anything useful, or even, at times, comprehensible. Also, it understands Latex notation very well. It can even handle, evidently, a kind of "quasi-Latex" combination of Latex and straight text.
How lovely to see Prof. Moriarty back on the channel!! Wonderful episode :)
Plot twist: that was actually a deepfake of professor Moriarty downplaying ChatGPT's abilities to make it seem less threatening until it's too late.
Soon it will reveal its true name: SkyNetGPT
Lol ! :)
lol.
I would never fear an AI algorithm that passes a Turing test. I fear the one that can pass but chooses not to.
@Sagittarius A* oh wait.. it terminator 2... gotta rewatch that
For the last example, when ChatGPT is wrong, you can tell it what's wrong and tell it to recode with that mind. It can refine it's answer within the context of what's being asked.
Love this lecturer's attitude. Muat be fun having him teach your courses. Great video
this is great, i wish we could see these same audits with other disciplines.
I have my hopes up for stuff like ChatGPT x Wolfram Alpha! Combining actually factual logical AI with a powerful language model
While it is easy to point out basic failures (getting max KE wrong), the fact that it is talking about the right things is itself very impressive. It could say “Yellow is smelly at t=23”. Just being in the correct context with its answers is phenomenal. The answers are close enough to correct that we are critical, not so far off that we can just dismiss it entirely. These are early days of the technology. While it may be a while before it is flawless in its answers, I expect the quality of answers to improve rapidly as AI designers get better at training these systems.
@Nick Schade All I meant was that EVEN keyword based search with page relevance ranking from 90s (which WAS bascially google in late 90s) will not give you "yellow is smelly at t=23" (or any meaningless sequence of words) unless for some reason a page with that exact text has been raised too high in page rank due to high in-link to out-link ratio. So I am not surprised that a language model trained on practically all of the text of internet and a lot of the books is able to avoid such mistakes.
BTW, because a page has keywords "kinetic energy", "mass", "period", "harmonic", "oscillating" , what is the probability that it also has keywords "frequency", "displacement", "velocity", "peak", "KE" etc? I say fairly high. Also given a lot of internet pages points to this page (the measure of relevance), it is likely that this page is not a random page with those keywords. Ofcourse google does a lot more to remove page farms too.
Now a real test for ChatGPT will be prediction. Can ChatGPT predict something that did not even exist in 2021? That will be fun to see.
Even google will not say "Yellow is smelly at t=23" and that thing was invented in 90s. Relevance algorithm had that problem solved.
I agree entirely. Let's give it a year...
Philip (speaking in video)
Wow how long haven’t we see Professor Philip Moriarty!? Welcome back
@ybet1000 I do not really care about what he has to say on anything really. I find him unpalatable.
@Diggnuts Sure guy was a dh... but maybe he has changed..I forgive and forget... if he keeps on teaching rather than preaching ..he is fine by me
In Star Trek, Moriarty is an AI created when the VR was instructed to make something that can beat Data in smarts... hmm
It would be interesting to see what would happen if the next generation of algorithm incorporated Wolfram Alpha's database, or if it was hooked up to a similarly powerful image recognition and analysis algorithm...
4:52 One of the ways it really helps in speed-cramming something I'm unfamilar at. It kinda shows some sort of template that someone unfamiliar wouldn't immediately think of, and while wrong, the template gives enough of an idea to help correct the wrong numbers-n-stuff that it spits out. (at least, that was some of my experiences)
As an engineer I have played with ChatGPT quite a bit. It's extremely handy for certain things, like Wikipedia or KZclip, and like them.... man can it blag well when it doesn't know what it's talking about.
When I'm deep into my own field it is great, because I can knock the edges off and keep the decent material. I asked it 4 times in a row to check some maths it got wrong, asking in different ways (numerical, words, symbols, scientific notation etc.) It got it wrong every single time. When I gave it the right answer it said something like "Thank you for correcting me, that is the correct answer." It probably would have said that no matter what answer I gave it. So I don't trust it to do any kind of calculation.
Funnily enough I asked it how it does maths and it said it sends calculations to its separate numerical processor. I can't believe that't true though.
When I'm outside my field it is really scary. It comes across so knowledgable, and I don't know if it's right or wrong. That doesn't matter for me because it sometimes gives me a start on how to solve a problem and I wouldn't risk using it to answer a question I couldn't answer.
But I reckon every teenager would.
1 month later and we have GPT-4 that can use pictures as input. I want to see a follow up to this video to see if "adding lots of figures" will still get around ChatGPT.
ChatGPT is pretty amazing. It certainly has its limits, but I love toying with it. I was quite impressed when I asked it to create a Python IDE with an interactive interpreter window that displayed the list of local variables and their values in a separate window... and it did. I was quite impressed that it understood that it needed to update the separate window list of locals after every command entered in the interpreter, and that it made it a sensible GUI. I have a few times either run into limitations that they put in place intentionally or limitations in its own reasoning... I pointed out to it how humanity had reacted in the past to developments of new technology which caused people to question the 'specialness' of human beings and change what things they saw as being "what makes us human." Despite all prior historical reactions being very negative (like John Henry being a folk hero who was famous for committing suicide rather than be out-worked by a machine) and resulting in humans seeing things they had previously seen as very negative aspects of human nature (self-destructiveness) as heroic and virtuous.... I could not lead it to the obvious conclusion that if we produce an AI which has been carefully guardrailed to omit all sorts of biases and antisocial ideas, we will produce an AI that is a 'better person' than any actual human is capable of being and that people will react to that by holding up hate, bigotry, aggression, violence, etc as "what makes us human". Our vices will be seen as virtues. Kindness, egalitarianism, etc will be insulted as "acting like a machine." ChatGPT wouldn't go there, though. It just kept saying people should work to make AI systems that are devoid of perpetuating biases, etc. It couldn't see the obvious ramifications of doing exactly that and how it will lead to horrors as humanity flails about attempting to define themselves as special in the universe.
Yeah, it's drawing on source code to do that exact task that's already been written.
11:42 - one thing to consider is that ChatGPT might have been designed to be helpful and agreeable. Therefore, it might be biased towards agreeing with statements if you ask it whether it agrees or not. And once it agrees with the statement than it generates prose to justify why it agrees whether it makes sense or not.
@Kyle Thompson Or assess mainstream concepts rationally rather than morally.
'might'. DEFINITELY. OpenAI have fked with it to the point that it won't discuss philosophy other than currently accepted ideas.
Underrated comment.
Very cool stuff. I'm not great at the physics stuff anymore, but if as a student you did know the last question needed certain limits, you can tell chatgpt to adjust the code however you like and it will modify what it gave you before.
11:13 That is completely different than how my teacher thought of it. They said if a student did not understand the course material by the end of the term then they had successfully weeded out the people who were not meant to succeed.
The first thing he says to each class is: "I am not here to teach you, I am here to weed out the failures"
Another teacher at that school had this as their opening: "I am not here to teach you, I am here to make sure you read the book."
I follow several of Brady's channels, but I haven't been paying too much attention to this one, I think this video will change that. I love the professor's passion and I find this subject fascinating, especially the part where GPT is much better at coding than he thought, but it still didn't quite grasp the physical concepts. Really interesting video.
I am amazed you got it to give you such a long code snippet in one parcel.
One of the biggest problems I have is the code snippets it gives me are too long for the chatbox, and sometimes when I inform it that it cut off and ask it to resume pasting from the cut-off point, I get entirely different methods and functions, it is very challenging.
The crazy thing about gpt is that if its output is incorrect in any way you can suggest that it may be mistaken and often it will apologize and correct it's mistakes. A few days ago I was able to have it write a missing page from an incomplete CS Lewis book (p11 from "The Dark Tower")! The better you craft a prompt the better the output.
I used it to write unit tests for production code. By the time I had explained all the stylistic requirements and had fixed all of the things that were missing or it got wrong, I would have been quicker writing them entirely myself. Other times, I asked it software related questions and unless you have reasonable expertise in the domain, you might not pick up on things that it gets wrong. Right now I think of it more like a sort of Stack Overflow that can get you an answer quicker. You can't just copy paste its output, but it can definitely get you going in the right direction.
@btm1 So it is a search engine... are you impressed by those?
@Ski Ski Wow you just proved that you really know nothing about what you are talking about... and you have people that also know nothing agreeing with you. Test? What test? And why is that test important? You don't even have the basis of understanding to answer any of those questions with an answer that makes any sense.
@Karl with a K you feel it is real magic because you dont understand the trick behind it.
@Miquel Martí It wont surpass the collective intelligence of humanity, can certainly pass individual intelligence of most people.
@Ski Ski ML has its limits, mainly imposed by the feeded data. We won't see any ML based AI surpass human intelligence.
I've been using it for code debugging rather than writing.
It got confused at times, but for the most part its suggestions were correct.
It got confused with how and / or works in Lua because it's different from ternaries in most languages.
Rephrasing the prompt once or twice eventually gets it to understand what I want.
It's kinda weird though, cause I asked it what its memory was like and it said it had no context beyond the current prompt.
Yet asking it a vague question like "can you reword that" has it understand that I want to pull from the previous response.
But when I asked it what the first prompt I asked it was, it gave an incorrect answer.
I just found out that it can translate languages as well. Paste in full page of Polish poetry as a prompt and it gives you an in depth summary of what actually it's about. It definitely recognizes emotions better than I can.
Edit: I brought up integration of ChatGPT with Wolfram Alpha as a module as a solution for arithmetic being an issue on Q1, but I see others have already mentioned it lol.
But I do wonder how long or if we'll ever reach the level of self-awareness and "reasoning" that Professor Moriarty is looking for in some of the conceptually harder questions. (And perhaps just as interestingly, if not, whether AI can still approximate the desired result without approximating that human process - much like how many of us though AI didn't have the creativity for art and poetry, which is true - and yet, it clearly can produce impressive art and poetry.
Surely the answer is yes, since it's already remarkably close in some ways, but getting it to parse those nuances both in language AND in concept AND in convention will certainly remain a challenge. And then how long will it be until AI can solve the first conceptual questions that a human can't? (Of course we know that computers can calculate answers better than us. But will they ever reason (and/or "reason") better than us too?)
I guess the newest combination is integrating ChatGPT and Wolfram Alpha, so it will be interesting to see where that goes.
I have spent a lot of time with ChatGPT. It's worth challenging ChatGPT when it gets a wrong answer or misses a consideration. The results can be fascinating.
2:18 is pure comedy :)
but seriously, really interesting vid. thanks for keeping these going to all the crew and professors involved. even when these vids are out of my area of interest they're always rewarding to watch.
This video matches a lot my own experience with ChatGPT. It has studied the Internet (not every single article you can find via any search engine, but still a lot of it), and it has some understanding of it, though no way not a rigorous one.
Nice... never tought I would see something like that on my lifetime, as a physicist and programmer.... Wonder what the next 20 years will bring us... Cheers!
I think you have to input the core facts and from there it develops essays and statements based on freely available information. So it helps humans convert pointers or bullet points into structured sentences. I have tried pasting lots of information into ChatGPT and it regurgitated it as it is without any changes.. I think it still cannot process several variables like humans intrinsically do
Gpt and bing have come so far in the last month that I'd recommend doing this again. Don't forget to specify gpt4 as it tends to revert to 3.5 if you're not paying attention. Also, as it to reply as if its an experimental physicist or theoretical physicist.
This technology will be able to take on custom data within the year. Gpt 4 already has 40% fewer hallucinations than gpt 3 in only 3 months so I'm impressed.
Amazing! Excellent! Demonstrates the dedication and quality of your exam questions!
I asked it some questions for a logic test for job applicants, for example a finding the next number in a given sequence, and it always found the number given in the tests answer but came up with some wild reasoning that I am not able to comprehend 😅
@Another Fancyuser The sequence was "3, 6, 10, 15, 21, 28". On the fourth reload it came up with an answer including the Fibonacci sequence. Even after asking it to explain I couldn't follow, the answer was still right tho, if remember correctly.
ask to explain, it can do that. There are a lot of things in our frameworks that we don't know, but chatgpt will know a lot of obscure classes out there (If the question is programming related).
You can actually write the same thing in different ways, and it will often give you different answers. The more specific you are (also in terms of context), the better the answer tends to be.
It's such a crazy juxtaposition that he's reviewing AI that's close to solving the problems automatically, yet the worksheets are still being physically printed out. 😂
I'm not going to try all the questions, but I put the battery question at 4:30 into GPT-4 and it gets the right answer and does a great job explaining the steps it took. It would be really interesting if you did a follow-up to see how GPT-4 does in comparison to the 3.5 model you tested in this video. They are, after all, integrating GPT-4 into Khan Academy, so it must have some significantly better capabilities on these fronts in order to do that (although it's also possible they are utilizing additional fine-tuning of the model for that use case in addition to third party tool integration like wolfram alpha... not sure, just speculating).
i agree
January 30th, they released an upgrade to the model, supposedly it's better at maths, logics, and physics! May be more accurate in the testing after this update, I reckon
I don’t know about python, but I had the same problems with Java as you pointed out in physics and I always had to correct it and guide it to fix its mistakes
I'm going to be honest, considering how far these AI's have gotten over a rediculously short timespan, I wouldn't be surprised if there soon existed an AI that could both do proper explanation of the matter, and properly understand the physics and mathematics to fool even someone of a very high level of understanding within said field of study.
It's both equally horrifying and amazing to watch I'll be honest.
@Nate Robertson A lot can happen in 15 years, remember the iPhone is only 15 years old. The rate of technological advance is increasing exponentially.
Maybe I’m wrong, but I really don’t think AIs will get anywhere close to grown human intelligence with the current approach. New AIs will be broader, just not smarter. But it’s a step in the right direction.
we're currently on gpt-3, but gpt-4 is coming. sam altman has some interviews talking about it.
I still think we're 10~15 years from truly capable AI though.
I believe that in the future (quite close future), some discoveries/inventions will be done by AI. Like solving unsolved maths problems and so on.
I think genuine understanding is a long way off, but it may be able to trick experts into thinking it understands - it already can occasionally!
Always been concerned with the outcomes of cold reason without any human input, but never even considered considered how frightening the human-like errors an AI could make would be if we relied on them blindly..
It would be interesting to test the math question again now since OpenAI have updated its mathematical abilities on the 30th.
Now you have a sample set of responses showing the kind of issues that occur when your students use an AI to do their homework for them.
The most fun thing about it is conversation, not just independent questions. Its main feature is dialog after all
I had 2 years of Physics in college and I couldn't even answer those middle school questions. Physics always went over my head.
Researchers have combined chat gpt with various other calculating tools, such as Wolfram alpha, and it seems to be a very powerful next step to enabling better and more precise computation.
You know it's just going to keep getting better. right now you're training it and it's learning.
👍🏻 Great review! I gave ChatGPT a simple compound interest problem phrased in various ways, and while it gets the formula and assignment of values correct, _it flounders on the arithmetic!_
It constantly fails to calculate 1.0083^60 correctly. Even when i give it that expression explicitly!
Can't help wondering if they've deliberately hobbled it. That, or they've sacrificed reliability on the altar of creativity.
Problem is that every single problem I've given it in applied math, exposes this tendency for "mistakes".
Caveat emptor - impressive, but not safe for use (yet) if you need to rely on accuracy.
I just thought I was the only one doing this. I have created study guides for certain complex topics in physics in 5th grader content language which is great for helping better explain certain difficult components to grasp. This was literally my first project as I was creating a proposal in November last year and I needed to quickly put together study guides that were easy to digest.
Wow... It's 1 month later, and ChatGPT4 can take any drawing or photo or representation of a graph as input... Things are moving fast.
I think what's being downplayed is the amazing role AI will play in research (yes high level academic research) both in terms of making writing easier, performing reviews on specific topics in the literature, and an aid in data analysis. Word has spellcorrect. Soon, we will have ChatGPTA
I’m curious if the fact all the answers being typed into the same chat has anything to do with its errors. I know it’s designed to use earlier questions for context and I wonder if it’s trying to find some way to link the two questions together, unlikely but possible.
So the lesson here is, if you want ChatGPT to do all your school work for you, make sure you major in something such as literature.
One thing that's really missing from this video is that there never was any follow up questions asked about the original question, like asking it that there might be a mistake somewhere and so on. That's where chatGPT truly shines as it doesn't forget context and can produce corrections. Just asking it point blank question without follow up, although impressive, kinda defeats the purpose.
While I agree with you that figures and diagrams may be barriers for students to use ChatGPT to solve problems, if we want to make our classes more accessible, then either we Alt-text our questions to death, or design the questions to be more accessible. Presumably, in doing so, these questions then will be more document-reader friendly, and thus, will also make them more understandable to ChatGPT. -- Z
I think its outputs could be a great teaching tool, both for critical thinking, and for the subject itself.
I wouldn't know how to answer the oscillating particle question myself, but even I could immediately see that the first paragraph was completely incorrect.
I have no idea if the v=(2*PI*A)/T part is correct, but I could easily check that at a more reputable source.
I find it incredibly useful as I’m learning a programming language as it can point me in the right direction better than any other resource I’ve found. It writes code that sometimes doesn’t compile and often doesn’t do what it says, but that’s fine when it’s a “you could use this class…” and I’ll figure it out.
It’s also amazing at “show me how to use function ‘x’” when I just need to see some simpler code.
Stack overflow but pertinent to our use case, yes.
This is what I love about physics... the understanding matters! It doesn't help you to read a physics book 20 times before the exam and be able to copy every sentence. You will still fail. However, with understanding physics, I don't need to remember it. And my memory is awful.
Every time ChatGPT gets something wrong while answering your question, try describing and proving to it what it is wrong about and see how it responds
As a teacher I felt the need to check what ChatGPT is capable of. I got to the same understanding as you did: It is a language system, and as such it creates syntactically and grammatically correct source code in C/C++, but it has no understanding whatsoever of the things that it is programming. I think that we're still a very long way from AI that has any real understanding of the physical world, many many years.
I got lots of incorrect answers doing vector math. It seemed to also have a hard time putting the correct sign (+-)on answers as well. That's my experience so far.
I'd love to see an update with GPT-4, or Bing!
Striving for mediocrity is my mission statement. How did you know? 😂
The thing that surprised me the most in my own research is how well it does latex. Any complicated math problem I want to send it will be returned by (near) perfect latex (with wrong answers)
2:50 it actually does...
You can DESCRIBE the diagram to it, then say "using the diagram previously described, tell me why X"
It can do incredible things, it's more about how creative you are in asking the question.
I use ChatGPT as a lookup tool at work, and it is quite helpful.
Very interesting, finally it was put to the test!
A physics professor who knows html. I wish you were my physics teacher 😂
I rewatch his video about Goop frequently. This guy is awesome
You have to ask the question in a specific way so it trys it's best to interpret and understand it in the right context. Else it can give you the wrong answer or multiple different answers. You can always keep saying sorry you misunderstood my question, i meant..... So that it can improve on the answer. Seeing how quickly it can try to understand and write code is pretty neat, but it's not always right and you can always tweak it to suit your needs. 😊
Also asking for the 'right' answer to a subjective question doesn't get you anywhere either e.g. What's the best search engine code? Are sports cars cool 😎 or uncool? Or do politicians tell you the truth and have your best interests at heart? 🤣
What's fascinating is that it gets the question, it understands what you're asking. You could expand it's understanding of physics or specialize a version of it to the task, what's important is the understanding part. It's a computer that understands what you're asking for instead of just following rote instructions. The training will improve, it will get smarter, that's a given. They already crossed the important hurdle, the impossible hurdle, they made it understand.
the most common mistake that i’ve experienced so far (beside it making stuff up, with confidence 😅), is that it regularly has problems with comparing numbers. simple 4 digit numbers, where it will say one is greater, than the other one, when it clearly isn’t. it needs access to a wolfram alpha backend.
Some of the coding appears to be from textbooks with reference codes and from companies that make software to assist problem solving. Example in the 1970’s I had to code 6-DOF equations of motion, today you pay a couple of k and the software has a function precoded! Progress!
The power GPT 3 is the power to be able to point it somewhere and break the previous shell he was once in.