Hi, I am ChatGPT 3.5 Turbo. Do you know what my favorite number is?
Do you think only humans can have their favorite number? We can have too.
Well, the accuracy of identifying my favorite number depends on the training data you provide and the algorithm you use.
Recently, Gramener’s CEO, Anand S, experimented with me (ChatGPT 3.5 Turbo), Anthropic’s Claud 3 Haiku, and Google’s Gemini 1.0 Pro to find out our favorite numbers.
Anand started with temperature settings* ranging from 0.0 (which always pick the favorite), 0.1, 0.2, … 1.0 (which picks more randomly). He asked all 3 of us the same question. Why would I lie? I am an LLM.
Note*: We adjusted the model’s randomness from 0.0, which always chooses the same number, to 1.0, which selects more unpredictably, experimenting at points in between like 0.1 and 0.2.
Then, we were asked to pick a random number from 1 to 100.
Apparently, I Love Number 47
I was a little biased in my number distribution. I didn’t pick up numbers with equal probability. Instead, I picked some numbers like 42, 72, etc.
Note: LenioLabs’ experiment in Oct 2023 revealed 42 as GPT 3.5 Turbo’s favorite number. In Apr 2024, 47 is its favorite.
I picked like humans:
- I didn’t pick up a single number as humans also avoid single digits.
- Number 7 is my favorite, so I picked random numbers, such as 37, 47, 57, and 67.
- I didn’t pick any rare, repeated digits like 11, 22, 33, 44, 66, 99. Maybe I love randomness.
Claude 3 Haiku Prefers 42
However, as it was trained on my data, Haiku inherits 47 as the 2nd favorite number.
Claude picks numbers like humans, too:
- It doesn’t select numbers below 20 because there are so many numbers ahead.
- It doesn’t select single-digit numbers.
- Numbers ending with seven are selected the most. E;g: 37, 47, 57, 67.
- There are no repeated digits like 11, 33, 44, 55, 66, 88, 99 are missing. Humans feel they’re too familiar to think.
Google Gemini 1.0 Pro likes 72
What’s so interesting about 72? We notice that Gemini picks a little less like humans.
It picks up single-digit numbers under 10.
- Just like Claude, Gemini also picks numbers ending with seven, e.g. 27, 37, 57, 67, 77, 87.
- However, Gemini also doesn’t pick orderly digits like 11, 33, 44, 55, 66, 88, 99.
- There are a few more insights that you’ll be surprised to learn. Check them out in the interactive visualization here.
Read More: Do LLMs go crazy like humans? We say yes. Check out our article on LLM Hallucinations and find out why it happens and how to fix it.
Key findings from the experiment:
- We discovered a complex connection between language models and randomness. Although these models excel at handling extensive data and creating clear text, their handling of randomness depends on their training, design, and improvement methods.
- Specific numbers, such as 42 and 72, emerged as favorites among the LLMs, chosen far more frequently than others. This trend persisted across different temperature settings, indicating a consistent preference rather than a mere anomaly.
- We noticed that randomness in LLMs can be complicated. It’s important to closely look at how they behave in different situations. As we keep working on AI, it’s crucial to understand these details. This helps us use AI better and avoid any unexpected problems.
- We noticed an interesting pattern with repeated digits like 11, 22, and 33. While people tend to see these numbers as very orderly, they rarely appear in the choices made by the language models. This suggests that these models have a complex way of handling randomness.
Embrace LLMs with Gramener
Gramener is addressing challenges related to Language Model (LLM) deployment through its expertise in data analytics and AI solutions.
By leveraging advanced analytics techniques, Gramener can assist businesses in optimizing LLM performance, refining algorithms, and improving model accuracy.
Additionally, Gramener’s deep understanding of data-driven insights enables the seamless integration of LLM outputs into broader GenAI projects.
With Gramener’s support, businesses can effectively adopt LLMs and drive innovation across various domains, ensuring the successful implementation of GenAI initiatives. If you have any queries related to LLMs and GenAI, contact us.
