The number of buttons and the occasion where it was worn. Don't laugh about the buttons, I have an article from 1926 decrying modern men wearing the wrong number of buttons to the office!
Let's go back to the twenties when these distinctions of dress were quite important for every well dressed man to know. Blazers were originally emblazoned with a badge or patch from a private school or club. They were usually navy blue, or the official colour of the club school whatever. Now it means a solid colored jacket, three buttons in two rows (double breasted) worn with non-matching pants. A sport coat was distinguished from a business suit or formal suit by having three or more buttons in the front and a looser fit, There would be pocket flaps, extra pockets, and top-stitched details. The sport jacket was also made in colours such as white, beige, stripes, light blues, pale greys and browns, and others. It was also made from less formal fabrics such as tweeds, linens silks etc back when business and formal suits were always dark wools in either black or dark grey. It was considered informal and for everyday wear, such as boating, golfing, walking, afternoons, but not evening or business. It wasn't required to match the pants. Matching pants and sport jackets were known as "dittos" back in the twenties. Today's sport jackets are dressy jackets, looser cut than a business suit jacket, with or without pocket flaps worked in various fabrics and colours. They may or may not have elbow patches. Business suits are made of fine fabrics, fitted, tailored, lined, and the pants and jacket are made from the same fabric. Business suits usually have two or three buttons, some modern cuts have four, but only two or three are fastened.
In modern times the designations have all blended into one. A solid color blazer is more formal than a tweed sport coat. Solids are always consiered more formal in men's wear. If the jacket coordinates with the pants, then it's suitable for daytime or office wear. If your problem is a nice jacket with worn out pants, then keep the jacket and buy new pants that coordinate.