The part of the scene you can see with the 20mm is plenty of light and dark. You can see details easily.
When you put in the 4mm lens, the part you can see is only 1/25th the area of before. You just happened to be pointing at a spot that was dark colored. If you had moved the scope a bit to the side, you might have seen a light colored area. Also, you have to focus the telescope slightly.
As well, the image will be only 1/25th as bright as before. So you should look only at brightly lit objects with the 4mm lens. At night, that would be the moon, maybe a planet like Jupiter if you can find it in the view. That is very difficult to do without an accurately aligned finder scope, and center the view perfectly first with the 20mm lens. And the telescope mounting must be motorized to track a planet, and the moon too.
Sixty-one years of amateur astronomy and telescope building and use.