Some activities were popular with all classes. Music and dancing, for instance, were enjoyed by all classes. Dances would vary in size and setting of course, from the grand balls of the aristocracy to a simple dancehall or a village dance in a hall or barn for instance. Every household that could afford it had a piano, and the daughters of the household would learn to play and sing, to entertain themselves and the family and company. This would apply to upper and middle class families, and even some of the more prosperous working class families might have an upright piano in the front room.
Reading was another pastime that was enjoyed by all classes. Of course, the upper class and upper middle class had the most money to spend on books, newspapers, and magazines, but there were plenty of cheap publications, for instance the 'penny dreadfuls' and 'shilling shockers', lurid stories of murder, crimes, and supernatural happenings, which were extremely popular with many of the lower classes. And the Victorian era was when the first public libraries opened, putting books within the reach of the poor to a greater extent.
Card games were enjoyed by all classes, though among the upper classes it was probably taken most seriously. Large sums of money might be played for, and to be caught cheating at cards was considered extremely scandalous, a man caught cheating might be ostracised by society.
Various kinds of theatrical entertainment was enjoyed. the upper and middle classes would enjoy going to the theatre, opera, and concerts. The music hall was very popular with the working class, music halls showed a variety of different turns - singers, comedians, jugglers, acrobats etc.
There were other forms of public entertainment, like circuses, fairs, waxwork shows etc. Some of these might not be regarded as quite respectable by the upper and middle classes, but they were enjoyed by many people.
Boating was all the rage in the late Victorian era, and people of all classes enjoyed taking a boat out on the Thames. With cheap railway fares, even a poor family might be able to hire a boat for the day. The wealthy often stayed on the river in houseboats, which were more like floating houses, and some of them had steam launches. See Jerome K. Jerome's book Three Men In A Boat for example, which is an account of a boating trip taken by three middle class young men from Kingston up to Oxford.
The seaside was very fashionable in the Victorian era. A middle class family might spend a holiday at a boarding house or hotel, a working class family might have to make do with a day trip. An upper class family might stay at a fashionable seaside resort, like Torquay in Devon for example, or even stay on the French riviera, Cannes and Nice were very popular resorts for the upper class English.
Sports were played by all classes, but again there would be differences. For example, county cricket teams were normally made up of 'amateurs' (men who could afford to play simply for fun, and didn't have to worry about taking time off work etc), and 'professionals' who played for a living and were usually from working class backgrounds.
Tennis, archery, and croquet were all popular games with the upper and middle classes, as they could be played on a lawn by both sexes, and so gave opportunities for socialising.
Horseriding was a popular activity with the upper classes, as was hunting. Upper class gentlemen enjoyed going shooting, ladies did not normally shoot, but often followed the shooting party to watch the men, and they would all have lunch together.