The question is very badly phrased. After a while I realized that what is MEANT is that the small cone lies entirely inside the larger one. But your idiotic teacher never said so (and so there is no "maximum" volume, except that the smaller cone must not exceed the size of the larger one.) Oh well, I'll address the problem that was intended...
The "base" of the smaller cone is in contact with the inner surface of the large cone. Put a coordinate system with (0,0) at the tip of the small cone, and a slant height of the large cone along the line y = 15 - (5/3)x.
Volume of the small cone is
(1/3)pi*r^2*[15 - (5/3)r]
= (pi/3)*[15r^2 - (5/3)r^3].
Obviously as r->0 or r->9, the volume of the small cone approaches 0. Somewhere in between, there will be a maximum.
dV/dr = (pi/3)*[30r - 5r^2],
and this will be 0 when r = 6.
(a) r = 6, h = 5.
(b) V = (pi/3)*(180) = 60*pi.
(c) For a test of reasonableness, consider the volume of the large cone, which is 405*pi. It is reasonable that the smaller cone be a good deal smaller than the larger one, and that the smaller cone be "flatter" than the large one, as the radius is squared in the volume while the height appears only as a first power.