The real point is that we should never use LLMs for knowledge. They are only language models, not knowledge models.

This mis-use of LLMs to answer our questions is all part of the misdirection of the LLM hype and lies.
LLMs are not capable of being factual or reliable and are a dead end technology whilst based on transformers.

This concludes this basic introduction to transformers which aspired to be mathematically precise and to provide intuitions behind the design decisions.We have not talked about loss functions or training in any detail, but this is because rather standard deep learning approaches are used for these. Briefly,transformers are typically trained using the Adam optimiser. They are often slow to train compared to other architectures and typically get more unstable as training progresses. Gradient clipping, decaying learning rate schedules, and increasing batch sizes through training help to mitigate these instabilities, but often they still persist. 

oops

Thanks for enlightening the world about the fact that transformers are not knowledge retrieval tools. For that you use eg RAG pipelines. That guy you’re quoting needs to get a life- or a major update.

Thanks for enlightening the world about the fact that transformers are not knowledge retrieval tools. For that you use eg RAG pipelines. That guy you’re quoting needs to get a life- or a major update.

I'm on outside looking in.  Some people swear by Transformers.

I looked up your TLA yet another buzzword.
"Retrieval augmented generation (RAG)"

Pretty sure you’re not looking in. You seem to be playing a game of telephone, trying to repeat what somebody (who may be looking in) is saying.

"Some people swear by Transformers." 
They are preferred architectures for big LLMs for all sorts of good reasons. I remember years ago, when I wanted to make fun of some folks here in the forum, I trained LSTM on their posts and it worked pretty well. I definitely wouldn't enroll a transformer for such a small dataset.

Thanks for enlightening the world about the fact that transformers are not knowledge retrieval tools. For that you use eg RAG pipelines. That guy you’re quoting needs to get a life- or a major update.

I'm on outside looking in.  Some people swear by Transformers.

I looked up your TLA yet another buzzword.
"Retrieval augmented generation (RAG)"

It's not a bee. It's an LLM peeping into the database for up-to-date information.

Cuchulainn, here are the details of the training methods including DeepSeek approach. If you read this, you’ll probably realise the weaknesses too.
https://rlhfbook.com/book.pdf

Pretty sure you’re not looking in. You seem to be playing a game of telephone, trying to repeat what somebody (who may be looking in) is saying.

When I worked in  industry on a daily basis, one of my roles was thaf of requirements analyst (besides designer and language). Same techniques.
So, knowing what's behind the "AI curtain" is important.

Many things stop looking "revolutionary" when you explain them in detail. Who cares?

"No matter if it is a white cat or a black cat; as long as it can catch mice, it is a good cat."

An awkward feelng is those involved have no long term memory. They are making the mathematical mistakes that were solved donkeys' years ago.

Good Heavens! For more than forty years I have been speaking prose without knowing it.

 Moliere

@kats, thanks for the link.

Many things stop looking "revolutionary" when you explain them in detail. Who cares?

"No matter if it is a white cat or a black cat; as long as it can catch mice, it is a good cat."

An awkward feelng is those involved have no long term memory. They are making the mathematical mistakes that were solved donkeys' years ago.

Good Heavens! For more than forty years I have been speaking prose without knowing it.

 Moliere

@kats, thanks for the link.

We have to be humble when talking about ML modellers making mathematical mistakes, not because maths is irrelevant, but because it is important to understand precisely under what conditions the mathematical results you think are being ignored or violated apply, and what they are actually saying.

Maths is particularly good/mandatory at flagging implicit assumptions.
In CS/Engineering we assume a solution exists and then proceed to use heuristics, tweaks and extensive experimentation to find it, by hook or by crook.. In which space of functions does a solution exist?

A bit like Crank Nicolson for pricing (the issues were already known by 1975). 

// A recent poll among data scientists  showed Marvin Minsky was the inventor of gradient descent method.

with so many parameters in LLM you don’t need to worry about the existence of solutions! And if you bother to analyse the loss surface you can see the majority are good minima. Actually the bigger the model the smoother and better it gets, they say.

The problem of optimising those models is so intricate that there is little insightful discussion published - what works for my model/data, may be not good for yours.

You’re joking about Minsky I hope?

Both ChatGPT and DeepSeek put it to an 1847 paper by Cauchy. But, strictly speaking, knowing how to use it is more important than knowing who invented it.

Both ChatGPT and DeepSeek put it to an 1847 paper by Cauchy. But, strictly speaking, knowing how to use it is more important than knowing who invented it.

I am talking about people; ChatGPT is a name dropper.
You are not even wrong. 
A plumber is not a fluid dynamicist.

Just look at the rigmarole on SGD,

You’re joking about Minsky I hope?

I don't do joking. It was a LI poll.
Even worse, 45% said Newton invented GD.