Excellent question. It's tough. In fact, it can be close to impossible. If you're checking for LINEAR factors with integer coefficients of the form (ax - b), which is what you do most of the time, you can use the rational roots theorem to do an exhaustive search. If b/a is a root of the polynomial (in other worsd, if (ax-b) is a factor), then the numerator b must divide the constant evently, and the denominator a must divide the leading coefficient evently.
So, for x^3 - 2x^2 + 5x + 6, you only need to check +/-(1,2,3,6}. If you check those 8 numbers and find no roots, it's not factorable over integers. It may have an irrational factor, like x^2 - 2 has the factor (x - SQRT(2)), and it may have an imaginary factor, like x^2 + 4 has the factor (x - 2i). But it has NO nice linear factors if those 8 values never return 0.
That said, you can make nasty quintics that can be factored into quadratics. Let's say we take x^2 + 4x - 16, which has no factors since there are no two integers that have a product of -16 and a sum of -4. Let's multiply it by x^2 + 1, which again doesn't factor over real integers. We get:
f(x) = x^4 + 4x^3 - 15x^2 + 4x - 16
Obviously, that can be factored, but it's next to IMPOSSIBLE to do so. If you were given that, you'd just check +/- {1, 2, 4, 8, 16}, and see that there are no linear factors. I guess you could find the quadratics by multiplying two generic quadratics (ax^2 + bx + c)(dx^2 + ex + f) together and equating coefficient, but that's WAY beyond most algebra classes.
|