March 1, 2024 9:42 am

Why are LLMs so small?

LLMs are compressing information in a wildly different way than I understand. If we compare a couple open source LLMs to Wikipedia, they are all 20%-25% smaller than the compressed version of English wikipedia. And yet you can ask questions about the LLM, they can – in a sense – reason about things, and they know how to code.

NAMESIZE
gemma:7b5.2 GB
llava:latest4.7 GB
mistral:7b4.1 GB
zephyr:latest4.1 GB

Contrast that to the the size of English wikipedia – 22gb. That's without media or images.

Shannon Entropy is a measure of information desitity, and whatever happens in training LLMs gets a lot closer to the limit than our current way of sharing information.

Previously

5 year old hacking chatgpt

Next

My physical relationship to the internet

Previously

5 year old hacking chatgpt

2024-02-29

Next

POSSE rss to mastodon keep it local and then share

2024-03-04

July 2024

Vacation Book Reading

8th

June

Four freedoms

15th

Adapting to new mediums

9th

April

I need a trigger warning

30th

Oh javascript

18th

March

Discovering idagio

26th

Things I love about my phone

24th

My physical relationship to the internet

14th

Why are LLMs so small?

1st

February

5 year old hacking chatgpt

29th

Fifteen or twenty thousand years

27th

2023

October

political implications

1st

August

a good death

23rd

locations in the magicians

22nd

everything is equally evolved

5th

July

timezones and dialects

16th

Should Robots Have Rites or Rights

11th

bad analogies

9th

threads is a mess

7th

things that are neat about the rivian

6th

hallucinations are bullshit

5th

A Taxonomy of AI Panic Facilitators

4th

rivian trusts the driver

2nd

June

Modern Improvements

30th