Make the denominators equal by taking their LCM. Since 7 and 11 are co-primes their LCM is their product. Thus LCM is 7x11 = 77.
Now 5/7 = (5x11)/(7x11) i.e. 55/77 and 9/11 = (9x7)/(11x7) = 63/77. Thus we have to find 3 irrational numbers between 55/77 and 63/77. They are: (1) square root of 56/77, (2) square root of 59/77 and (3) sqaure root of 61/77.
In fact you can write any integer between 55 and 63 in the numerator and take square root since it has to be irrational number.
If the question was to find rational numbers do not find the square roots i.e. 57/77, 58/77, 59/77 are 3 rational numbers between 5/7 and 9/11