Their economy won't collapse without EU membership. Also, Norway is closely associated with the EU as it is a member of the European Economic Area and is a founding member of the European Free Trade Association.
Every time Norway's been invited to join the EU, it's put it to a vote among its citizens, and its citizens have always voted no. Last time, it was 53% against and 47% for. The main reason voters cite for not wanting to join is sovereignty. Norwegians feel that joining the EU will too much compromise Norway's sovereignty and its right to self-rule. They foresee the EU becoming eventually like the United States where a strong central government forms, its member nations become more like our states than independent countries, and its member nations won't be permitted to leave and reestablish their sovereignty, just like states in the United States aren't allowed to secede (as evidenced by the Civil War).
Are the Norwegians' fears justified? Maybe. Maybe not. It's somewhat irrelevant. Whether or not Norwegians are right in their thinking, those are the reasons, the reasons of Norway's populace and supported by the populace's repeated voting down of every referendum put before them to join the EU.
By the way, Norway and Norwegians aren't ignorant of the downside of not being a member of the EU. When Brexit was being voted on, Norwegians far and wide quickly piped up to tell Brits that not being a member of the EU is hard when so much of Europe is, definitely no walk in the park. Still, Norwegians must think its a price worth paying to not incur the risk they perceive joining would present to their ongoing sovereignty.