The LLMs aren’t non compos mentis…

P/C Dall-E

… they don’t even have a mentis to be unsound!

For example, if I ask Bard what languages it speaks (a question you might think would be pretty straightforward), you get this response:

Sounds great!

Bard claims English, French, Spanish, German, Italian, Arabic… etc.

So, if I ask a question in, say, Spanish, I’d expect a response. Instead, I get this:

Okay, that’s really strangely broken. From what I know, Bard WAS trained on multiple languages (including that entire list), but despite claiming that it knows all these languages, it actually can’t do anything in any of them. There’s a big disconnect here.

I’m sure this will change in the future, but the deeper point is that Bard doesn’t know its own capabilities.

Since the response to the previous question was to “refer to the Bard Help Center,” I asked the obvious question next:

I’m not too surprised–I’m pretty sure that the Bard help center document wasn’t part of the training set… but you see my point: the ability of an LLM to talk about what it knows is just as subject to fabrication as anything else.

Interestingly, when I ask ChatGPT3 the same question:

Does ChatGPT3 really use those languages? Let’s try the same question I gave to Bard earlier:

Which is, according to translator, a reasonable answer.

How about another language–let’s try Arabic? Here I ask ChatGP3 “what is the capital of France?”

The Arabic answer is right: “باريس (Paris باللغة الفرنسية)” That is, “The capital of France is Paris (Paris in French).”

Clearly, ChatGPT has multiple languages under its virtual belt and can woo us in many tongues. I have no doubt that Bard will too–but the deeper point is that you can’t believe what an LLM tells you, even about itself. (There’s another point here that Bard is–in its current form–a fairly limited version. Sundar says that more capability will be unleashed soon… but why hold it back in such a way? What’s the upside here?)

For the record, the Bard Help Center page is https://support.google.com/bard/ — but I had to use regular Google search to find that–Bard didn’t know.

But does ChatGPT?

I don’t know if it’s intentional, but ChatGPT3 seems to be throwing a bit of shade on Bard when I asked it about the Bard Help Center:

I had to smile at that.

But when I tried ChatGPT4 (in Google’s Edge browser), I found that the update knows a lot more about the world:

It’s ironically great that ChatGPT4 knows more about Bard’s Help Center than Bard itself.

Keep searching… (but always follow-up and verify what you think you’ve learned)…