find values of A and B so that the points (a,b,3), (2,0,-1), and (1,-1,3) are collinear

points are collinear therefore area of triangle = 0

area for points (a,b,3) & (2,0,-1) = 0 {these are also collinear}

                                                     0   = a2+b2-4a+20

                                                4a-20 =a2+b2 -------(1)

area for points (a,b,3) & (1,-1,3) =0{collinear}

                             0                          =a2+1-2a+b2+1+2b+0

                         4a-20                     =2+2a-2b

                                           a            =11-b  

now put the value of "a" in eqn (1) you will get  a & b.