It's going to take time to recover. There's a rule of thumb: For each week you take off, it will take twice that number of weeks to recover to your previous condition. You take 3 weeks off, for example, you will need about 6 more weeks to get back to the condition you were before the layoff. However, this rule of thumb breaks down at the short end and the long. At the short end, if you have been training regularly for a while, you can take a week off without significant loss of conditioning. At the long end, once you take 4 weeks off, you've lost most of your conditioning: It can take several months to get back into shape.
Talk to your coach. And, yes, do workouts with the team during the off season. To get better than you were before, you might need a longer period of training than you were in before. But, don't overdo it. Getting overtraining syndrome will be counterproductive.