Horace He Calls Cached Token Counts Incredibly Dumb
Researchers note that visible dashboard numbers lead users to include cached inputs when reporting token usage.
Horace He posted on X that people discussing LLM token usage usually include cached input tokens and called the practice incredibly dumb. Hieu Pham replied that users read the most visible dashboard number rather than seeking a breakdown. Lucas Beyer joked that a "kv tokens" metric multiplying by layer count would make the numbers even bigger. Horace He followed up suggesting counting every decode step load of cached tokens. The posts discuss reported LLM token counts that include cached inputs.
I've realized when people are talking about how many tokens they use, they're usually including cached input tokens... which is incredibly dumb...
Combined views
56.9K
Horace He Calls Cached Token Counts Incredibly Dumb
Researchers note that visible dashboard numbers lead users to include cached inputs when reporting token usage.
Horace He posted on X that people discussing LLM token usage usually include cached input tokens and called the practice incredibly dumb. Hieu Pham replied that users read the most visible dashboard number rather than seeking a breakdown. Lucas Beyer joked that a "kv tokens" metric multiplying by layer count would make the numbers even bigger. Horace He followed up suggesting counting every decode step load of cached tokens. The posts discuss reported LLM token counts that include cached inputs.
Reactions
3 postsHP 
Hieu Pham@hyhieu226@cHHillee Not entirely them to blame. I think most people would read the number from their dashboard that is most visible to them.
- likes: 52
- replies: 0
- bookmarks: 0
- reposts: 0
LB Lucas Beyer (bl16)@giffmana@cHHillee @YouJiacheng I'm just waiting for "kv tokens" to catch on. It would be what you say times number of layers. Big number now bigger
- likes: 48
- replies: 2
- bookmarks: 4
- reposts: 0
HH Horace He@cHHillee@giffmana @YouJiacheng Why stop there? Every single decode step you're loading all of your cached tokens! We can just count those in tokens used :)
- likes: 41
- replies: 2
- bookmarks: 1
- reposts: 0