The difference, as far as I could understand writing at nearly 4:00 AM CST while doped up on codeine cough syrup and antibiotics, is that of the scale of appreciation. It seems to me that being a "fan" of something is as much about the fan as it is the object/person subject to the desires and expectations of the fan. Vergil wrote this great unfinished poem nearly two thousand years before I was born. The world in which he wrote and personages he addresses in (mostly) oblique form had sometimes very drastic differences in social/personal values than the ones that influence us today. Yet what he wrote influenced countless other writers, even though Vergil has always been overshadowed by Homer, at least the minds of the hoi polloi. Since it is difficult to identify the Vergilian source, seeing as it has been filtered down through Dante, Ariosto, Camões, and others, the relationship becomes a more generalized yet somehow deeper one between reader and text (or perhaps the text as metastasized by centuries of attributions and alterations of motif and wording to fit other media of storytelling) than that between an extant author and his/her "fans." "I sing of arms and of the man..." – there are echoes of this that still reverberate around us even today. Much as someone might like the work of a current writer, chances are that the relationship revolves more around how that "fan" can craft his/her own relationships with the author than with the text itself.
This is not to say that "fans" cannot greatly appreciate what an author has produced or means as a person him/herself, but rather that it is very possible to appreciate something quite deeply without engaging in the behaviors (including perceptions of close ties existing between fan/object of fan's passion) commonly associated with "fandom." Related to this is the concept of "geeking out." I find this term to be loathsome, at least in the context of it being applied to things that otherwise would fall outside the parameters of what is considered to be "geekish" actions (intense, sometimes overly so, connections with a created object being a prime example). In coming to appreciate what a Vergil or a Dante has produced (to continue with the epic poetic references), for myself at least, there is no obsessing over what these poets could have meant in certain passages. Yes, studying deeply their writings for themes certainly would be lauded, but such studies are generally more considered, reflective responses than the perceived overly passionate responses of those who enthuse over a subject. Historians often are taken with the fields that they study, but their writings reflect a more reflective tone than what generally is produced when the so-called "history buffs" wax eloquent over their chosen historical era of interest.
Noting these differences is not to praise one uncritically and condemn the other, but rather is just a musing over why, for myself at least, that it is baffling when some want to conflate the two. I enjoy what I do. I consider what I have read and appreciate it. But I do not feel an intense, passionate feeling toward the creators of such things nor do I believe that the measured reactions that I typically have when reading/watching these things is akin to the enthusiastic rhapsodies that some people apparently produce when they "geek out" over something. But perhaps there are things that I am forgetting here that others can discuss?