It's from the musical play Guys and Dolls, based on the much-beloved characters and stories of Damon Runyon. It's sung by the character Nicely-Nicely Johnson (he was called that because whenever anyone asked how he was doing he'd answer 'Nicely nicely'). The character was played by Stubby Kaye on Broadway and he got to reprise his role in the movie.
I dreamed last night I was on the boat to heaven
And by some chance I had brought my dice along
And there I stood, and I hollered, "Someone fade me"
But the passengers they knew right from wrong
And the people all said 'Sit down!
Sit down you're rockin' the boat"
The people all said 'Sit down.'
Sit down you're rockin' the boat
And the devil will drag you under
By the sharp lapel of your checkered coat
Sit down, sit down, sit down, sit down
Sit down you're rocking the boat"