First, if you go to Port Authority, then you can't catch a train a to CT. The trains to CT leave from Grand Central, Penn Station, and various other places, but not Port Authority. Port Authority is mostly for bus, but some NYC subway trains stop there (including the A, C, and E); however, these trains don't go to CT.
The least expensive way to get anywhere is to walk.
The easiest and second least expensive way to get from LGA to a train to CT is to take the city bus from LGA to the intersection of 125 St. and Park Avenue and then take the MetroNorth train from the the intersection of 125 St. and Park Avenue to CT.
However, the intersection of 125 St. and Park Avenue is in a neighborhood that you might want to avoid. Another option is to take a bus from LGA to the NYC subway and then take the subway to Grand Central or Penn Station.
If you really have your heart set going from Port Authority, then you're going to have to take a bus from Port Authority to CT, which is a bad idea, because the bus is likely to get caught in traffic, and is likely to cost more than the train.