In his 1994 book Science Fiction in the 20th Century, Edward James cites Farah Mendelsohn as suggesting that science fiction could be viewed as a form of elite fiction, written for a technocratic elite rather than the literary one that a certain strain of high-brow fiction aims for.
I can see where she was coming from. Most sci-fi is written for a particular audience, one that’s often tech-savvy and which has the experience to decode science fiction’s tropes and assumptions. What makes a thrilling story to an experienced sci-fi reader, with the skills to wrap their head around sci-fi writing, would be bewildering to some other readers.
But I don’t think that elite is the right description for this group. I think it’s better viewed as a community, a selection of people with shared tastes and interests. Some do regard themselves as an elite, made superior by their grasp of the genre, but that doesn’t make them better sci-fi fans, it just makes them snobs.
Sci-fi might aspire to be for an elite because that’s how literary fiction presents itself. But again, I call bullshit on that. The ability to appreciate literary fiction means you have a particular set of skills that most people don’t. So does the ability to build a dry stone wall, plumb a boiler, or play the violin. Those skills are awesome but they don’t make you better than other people, as the word elite implies.
Sci-fi is for a particular community, one that can choose to elitist or inclusive. I know which I prefer.