Anything with any moisture in it will dissolve the sponge toffee. That includes mousse, even though most of the liquid is not flowing. Think of it like the way crackers will go soggy eventually if you have a moist spread on them.
The key factor is time. If you put the mousse and the sponge toffee together right before you serve it, there will still be some crispness to it. There might be a soft layer on the outside of the toffee but it will blend in with the mousse so it won't be too noticeable. If you are leaving the toffee in a slab, that will leave the most crunchy part. If you break it into bits, make sure that they are fairly large chunks and don't ovemix it.
If you want to make it in advance, you might think about dipping the chunks of toffee into a coating of chocolate. That will form a barrier that will prevent the mousse form dissolving it.
It sounds yummy!
A great love of Cadbury Crunchie and Violet Crunch bars.