Ugh...I haven't done one of these in many years. Used to be good at it, though...
The required expression is Y = AB(A+B). This uses AND and OR gates which according to the answers we aren't allowed to use. But what if we transform the whole thing using DeMorgan's law?
Y = A' + B' + (A'B')
Negating flips the values of all the variables, changes every AND to an OR and every OR to an AND.
So count the operations, now. to get Y we need...
a) An inverter to produce A' (a 2-input NAND or NOR gate can do this; just connect A to both inputs)
b) An inverter to produce B' (again, a 2-input NAND or NOR can do this)
c) A 3-input OR gate (they won't give us OR gates, but we can use a NOR gate and then invert the output with a NOR or NAND)
d) An AND gate to produce the A'B'. (we can't have AND gates, but we can use a NAND and then invert the output with a NOR or NAND).
So counting up elements from the above list, Y requires
a) 1 (NAND or NOR)
b) 1 (NAND or NOR)
c) 1 NOR + 1 (NAND or NOR)
d) 1 NAND + 1(NAND or NOR)
Total:1 NOR, 1 NAND, 4 of whichever.
So I believe (5) is correct; any combination of six gates with at least one NAND and at least one NOR is fine.
(As mentioned above, this expression is stupidly easy if you simplify it. AB(A+B) = AAB + ABB = AB + AB = AB...but they aren't letting us do that.)