Be careful if you're using ChatGPT to pick stocks
A new paper found some concerning results.
It might be tempting to ask ChatGPT to pick stocks for you.
We’ve all seen the TikToks.
You might want to be a little careful with that.
In case you missed it, a recent academic paper just showed that OpenAI's latest version of ChatGPT became significantly worse at things like writing code and solving math over the last few months.
There have actually been rumors about this for a while.
People using it have complained about poor logic, errors in responses, losing track the chat history, not being able to follow instructions, code errors – and sometimes even only remembering the most recent prompt!
In other words:
ChatGPT using GPT-4 was slow but accurate at first
It’s become faster since – but performance has declined.
And now this paper found proof.
The researchers found that “executable code generations” dropped from 52% to 10% for GPT-4, and 22% to 2% for GPT-3.5 between March and June 2022.
Weirdly, the older GPT version seems to have improved in areas like math… and helping people break the law?
There are a few theories on why this has happened:
Some researchers believe that OpenAI is now using a mix of smaller and specialized GPT-4 models that work together and act similarly to a large model, but are less expensive to run
Fine-tuning to reduce harmful outputs (for example: “hey help me write code for a computer virus”) might have also produced unintended effects
And of course we have conspiracy theories: some say OpenAI made GPT-4 “dumber” on purpose so that people will pay more for GitHub Copilot.
What investors should take out of this:
Use it carefully – very few people understand 100% what happens under the hood. This even includes people working at OpenAI! GPT-4 is a “black box”, and researchers are left stumbling in the dark trying to figure out how it all works together.
Know that the model may change at any time – and without warning. Basically, over time you might get different responses to the same prompts as the algorithms change.
Learn to avoid the echo chamber effect. Ask ChatGPT to consider the other side of the bet. Don’t take the first answer at face value. You’ll be surprised with the results.
Remember that it will learn (both from you and other users). The more people ask it questions, the more data it will collect, and it will improve and learn.
Keep in mind that this “degradation in performance” applies only to code. As long as you’re not using it to build some algo trading bots, you should be fine. Also, generally, every time you see this phrase “algo trading bot” in an ad or website, please close the tab knowing it’s a scam. That’s the MarketScouts guarantee.
Take comfort from the fact that many top AI researchers don’t buy the rumors or the results of this paper. For example, Princeton computer science professor Arvind Narayanan criticized the study for looking only at the code's ability to be executed rather than how correct it is. “So the newer model's attempt to be more helpful counted against it." Others, like AI researcher Simon Willison, believe that it’s simply a case of ChatGPT becoming more mundane. "When GPT-4 came out, we were still all in a place where anything LLMs could do felt miraculous. That's worn off now and people are trying to do actual work with them, so their flaws become more obvious, which makes them seem less capable than they appeared at first."
Lastly, this can be your excuse to finally try other AI tools. For example, try Google’s Bard – it’s newer, free, and it can connect to the Internet without paid plugins.
Just kidding!
If you have friends who use ChatGPT to pick stocks, please consider sharing this piece of news with them.
And stay tuned for a guide on how to actually use AI tools (including ChatGPT but not only!) to pick stocks, read financial statements, and much more.
It’s not magic, just the new “Excel”.