The formula you wrote is correct. So you can do the calculation with this formula.

Think it this way:

For a polygon, the number of its sides are equal to the number of its vertices. And for each vertices, it cannot form diagonal with itself and the two neighbor vertices, so the number of the left vertices that could form diagonals with it are (n-3).

So if there are n total vertices, one would assume the number of the total diagonals are n(n-3). But remember two vertices form one diagonals, so in the formula above every diagonals are double-counted. Therefore the real number of diagonals should be n(n-3)/2.

And for Question a), I think the answer should be the polygon has to have at least 4 sides to have diagonals.