When do you use chain rule and when do you use product rule?

For this problem, you would probably be expected to use a technique called "implicit differentiation" which is just another way of looking at the chain rule. But for the xy^2 term, you'd need to use the product rule. So the answer to your question is that you'd use both here.

You differentiate through both sides of the equation, using the chain rule when encountering functions of y (like y^2)

So for this one you'd have 2x + 2xy*y' + y^2 = 0

Then you solve for y' = (-2x - y^2) / 2xy

Now in order to close the form, you solve for y in the original statement of the problem and plug that result into our y' expression.

Another way to solve this problem would be to solve for y from the start and employ the chain rule on the square root of a function of x, the function itself requiring you to use the quotient rule.

You would use both.

By the product rule:
d/dx (xy^2) = d/dx(x) y^2 + x d/dx(y^2)
By the chain rule:
d/dx(y^2) = 2y dy/dx

Put those together with the rest of the problem (which I hope you can do on your own):
2x + y^2 + 2xy dy/dx = 6

Chain rule is used for composition of functions, product rule for product of functions.

In your case, the term xy^2 is a product of x and y^2, two different functions, so you must use the product rule here. But when differentiating y^2 with respect to x, it is a composition of y (regarded as a function of x) and x^2 so for that part you must use the chain rule.

So, differentiating both sides of your equation with respect to x one gets

2x + 1*y^2 + x(2yy') = 0

Then you can solve for y':  y'=(-2x-y^2)/(2xy).