Vtyjkyuk

I tried to solve this inequality by taking the square outside the floor function $[y]$ (greatest integer less than $y$)but it was wrong since if $x=2.5$ then $[x]= 2$ and $x^2=4$ while $[x^2]=[6.25]=6$.

We can you Calvin's suggested method here (see his comment) by solving the equation as we would with the quadratic equation.
You solve this equation as following, let us take $[x]$ another variable say $n$, and $[x^2]$ as $n^2 + c$. Now your equation becomes :

$$n^2 + c + 5n + 6 = 2$$
$$n (n + 5) = -4 - c$$

Now, we know from above equation that,
n varies from , $$ n = -4 -c$$
to $$ n = -9 - c$$

here, for our required equation to be true, the equation should be strictly greater than $2$. Hence, the least possible value for $x$ in greatest integer function should be greater than $-4$.

I hope this clears your doubt.

Define $f(x)=operatornamefloorleft(x^2right)+5operatornamefloorleft(xright)+6 = [x^2] + 5[x] + 6$.

I'll present some graphs that indicate the solution set is more complicated than $xge -4$.

(Desmos link)

We see the solution is made up of 3 different regions. As you can see Desmos is not certain about the accuracy of its plot so I found a second opinion that indeed indicates that there is a small hole in the solution regions a small bit to the left of $-5$ (W|A link),

I don't understand where $-4$ came from. Firstly, it does not come from the other potential interpretation of $[cdot]$ as the ceiling function, as these calculations in W|A show. Secondly, under the interpretation $[x] = operatornamefloor(x)$, we have $f(-2)=0$.

Clearly, even without a graph, there are at least two distinct regions that satisfy the inequality, since if $x>0$, then

$$ fleft(xright)=[x^2] + 5[x]+6 > 6 > 2$$
and if $x < -6 $ then
$$ [x^2] + 5[x]+6 ≥ (x^2-1) + 5(x-1) + 6 = underbracex(x+5)_>6 > 2 $$
so the solution region must contain
$$ x < -6 cup x > 0$$
as a subset. The fact that there is a third region is not so obvious and requires more careful study of the difference between $[x^2]$ and $x^2$.

I think the answer should be: $xle-sqrt27qquad $ or $qquad -5lexle-sqrt22qquad $
or
$qquad 0lexqquad $
Here is a sketch:

For $$-4lexle-1$$ it is $$x^2+5x+4le0$$ or $$x^2+5xle-4$$, but also
$$ [x^2]+5[x]lex^2+5x$$
so there is no solution in $[-4, -1]$.

For $$-5lexlt-4$$ it is $qquad [x]=-5qquad $ and so $qquad 5[x]=-25qquad $ therefor $qquad [x^2]-25gt-4qquad $ $iff$ $qquad [x^2]geqslant22qquad $ $iff$ $qquad x^2geqslant22qquad $ $implies$ $qquad xle-sqrt22qquad $