Many people use the word "art" to mean anything that is done well so if you say "That isn't art" they think you are saying that it isn't done well. In the visual world there are many images, objects, spaces and experiences that are of high quality but are not "Art".
If you are a student in most art schools, for example, and enjoy doing "fantasy art" they will give you a hard time because it isn't "Art". These students need to find a school of Design where illustration is taught. Fantasy Art is not accepted as "Art" in most galleries and museums of art so they have their own museums and publications. Fantasy Art is part of the Visual Culture world of popular images for mass audiences and is part of the Design world where illustrations are made for publications rather than hanging on the wall. Many examples of Fantasy Art are recognized as works of very high quality in Visual Culture and Design.
Spectrum (left) is an annual publication compiling examples of excellent Fantasy Art. They are currently collecting submissions for the 18th edition. According to their announcement Spectrum was created as a direct reaction to the lack of attention — the lack of respect — afforded to the creators of fantastic art. The goal of Spectrum is to provide a venue that acknowledges fantasy artists and calls attention to the importance (and popularity) of fantastic art to the culture as a whole. In addition to the print publication, they have exhibits at the Museum of American Illustration which draws large crowds.
A complete visual literacy program should include Visual Culture, Design and Visual Communication as well as Art and help students understand the differences among them in terms of goals, criteria, techniques, expectations, preparation and exemplars.