It depends on the individual with their Sun in Virgo, and on the individual with their Sun in Aries.
In astrology attraction occurs when one person has their Sun or Moon or Venus or Mars within 8 degrees of the other person's Ascendant or Descendant (which depends on a birthtime that is exact within 4 minutes of the actual time of birth).
Without this type of interaction between the two unique birthchart, either no relationship is likely, or if one does start, it will usually die out within a year or two
Other synastry aspects can indicate attraction: Mostly the conjunction aspects involving Sun, Moon, Venus and any planet in the other person's birthchart. Sextiles and trines are also positive, but are very weak and may not trigger enough attraction for a relationship.
Annoyance usually involved discordant synastry aspects/interactions between the two charts, usually involving either Moon (our emotional responses) or Mars (how we take action, so it can include not handling annoyance well, or becoming aggressive/angry).
As for this guy, he doesn't have much capacity to care for anyone. As evidenced by the fact that he was snobby towards you at the start. A person who CAN care about others will care about all people, and never be snobby at all.
Also, he has poor emotional control (as evidenced by his initial irritation with you), but the temporary infatuation phase has temporarily dulled his irritation with you. Once the infatuation phase has died out and reality comes back, he will be even more irritated with you and will hurt your feelings.
ALL synastry comparisons between birthcharts will show SOME areas of conflict and discord. Just as all relationships, in actual life, have problems once the honeymoon dies out.
Relationships that last and work well require about 20 years of working with OUR emotional responses. And those relationships always start out as a strong sense of mutual friendship, and without a crush or infatuation .. and they build slowly. AND the problems do NOT show up until the couple has lived together a few years.
YOUR relationship with this guy does not fit that model .. it started out with irritation on his part, and lack of interest ... so multiply that and THAT is how the relationship will end up. And end.
Enjoy it while it lasts .. it will be fun. But don't hang on too hard when the excitement starts to die out and the hurting starts. Because this relationship started out rocky, it will end even rockier.