LLMs are compressing information in a wildly different way than I understand. If we compare a couple open source LLMs to Wikipedia, they are all 20%-25% smaller than the compressed version of English wikipedia. And yet you can ask questions about the LLM, they can – in a sense – reason about things, and they know how to code.
| NAME | SIZE | |
| gemma:7b | 5.2 GB | |
| llava:latest | 4.7 GB | |
| mistral:7b | 4.1 GB | |
| zephyr:latest | 4.1 GB |
Contrast that to the the size of English wikipedia – 22gb. That's without media or images.
Shannon Entropy is a measure of information desitity, and whatever happens in training LLMs gets a lot closer to the limit than our current way of sharing information.
Previously
fragments5 year old hacking chatgpt
2024-02-29Next
howtoPOSSE rss to mastodon keep it local and then share
2024-03-04I need a trigger warning
30thOh javascript
18thDiscovering idagio
26thThings I love about my phone
24thMy physical relationship to the internet
14thWhy are LLMs so small?
1st5 year old hacking chatgpt
29thFifteen or twenty thousand years
27thpolitical implications
1sta good death
23rdlocations in the magicians
22ndeverything is equally evolved
5thtimezones and dialects
16thShould Robots Have Rites or Rights
11thbad analogies
9ththreads is a mess
7ththings that are neat about the rivian
6thhallucinations are bullshit
5thA Taxonomy of AI Panic Facilitators
4thrivian trusts the driver
2ndModern Improvements
30th