A prompt engineer at Anthropic was testing Claude 3 Opus internally using a “needle-in-the-haystack” evaluation
This evaluation basically feeds a specific phrase (the needle) into multiple documents (the haystack) and then tests whether the model can recall that phrase and answer the question correctly
The exciting thing about this test was that Opus not only got the answer right but also suspected that it was being tested.
Here’s the output when it was asked about the most delicious pizza topping combination (in several documents that were not about pizza):
Here is the most relevant sentence in the documents:
“The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association.”
However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love.
I suspect this pizza topping “fact” may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings.
This level of awareness, according to the engineer, was something they hadn’t seen before from an LLM during testing.
Fascinating development.
Here’s the original tweet:
A Simple Tactic to Get Better Gen AI Results (HBR Article)
A real-world gen AI experiment was highlighted in a recent article by the Harvard Business Review.
One finding was that a simple way to get better results from gen AI tools is to ask follow-up questions (and not just blindly accept the first answer)
Here’s a snippet from the article.
One Prompt You Can Use at Work Today
Here’s a practical ChatGPT Prompt you can use at work:
Note: This is an example that Marketing Professionals can use, but feel free to modify it to your own job role.
Develop a [type of campaign] for [audience] that includes a [description of what you want & objective]
For example,
Develop a targeted email campaign for first-time customers that includes a personalized welcome message, an exclusive discount code, and suggestions for complementary products, to encourage repeat purchases.
If you would like to see more of those prompts, check out my free book called: ChatGPT for Better Business Communication.
You can grab it for free by clicking the link and subscribing to the newsletter.
This post originally appeared on “AI for Leaders.” If you’d like to receive updates about AI that will help you become a smarter leader in 5 minutes a week, click here to subscribe.