That's the strange thing about caffeine; besides the almost complete lack of tolerance
Oh, there is tolerance. From what I've read, a person who has caffeine regularly will have the same wakefulness/alertness while on caffeine as someone who doesn't consume caffeine. Because: caffeine blocks adenosine receptors. So in the presence of caffeine our bodies construct more adenosine receptors so that adenosine will still have an effect. We end up with enough receptors to accommodate the amount of caffeine we consume
and the normal function of adenosine.
Adenosine is believed to make us feel sleepy as the day goes on, btw. So you can see why blocking the action of adenosine would give you the familiar caffeine alertness. However, in those of us with caffeine tolerance, the adenosine still functions even when we drink coffee. But without caffeine, we have more free receptors for adenosine than someone who doesn't drink coffee. Effectively we should be more drowsy than a non-caffeine user at those times because of increased adenosine action.
Way too much info, I know.

Sorry. The only reason I know this is because I had to argue with my Pharmacology instructor who insisted that there is no physical withdrawal from caffeine.