Luciferase, the enzyme involved in created light from ATP, didn't appear into existence suddenly. It is actually related to a highly preserved enzyme used in the overall metabolic pathway, known as
fatty acyl-CoA synthetase, which is responsible for fatty acid biosynthesis. So when scientists approached this question, your question, they started off decoding the protein, luciferase, by comparing its gene to other genes found in the species and related species to find any relatives to it.
As mentioned, fatty acyl-CoA synthetase is an enzyme that plays a key role in fat metabolism, creating an intermediary that allows cells to add more carbons onto growing chains of fatty acids. Not surprisingly, luciferase nearly does the same thing. It catalyzes a very similar reaction, but works with a specialized chemical that produces a lot of light. However, if you put fatty acyl-CoA synthetase in with the specialized chemical that makes fireflies glow, nothing happens. This raised the question of how the ability to produce light began to be selected for in the first place; if there's no light, there's nothing to select.
The author of this paper, which I suggest you read --
http://www.pnas.org/content/111/12/4443 -- reasoned that the process must have started with a mundane, non-glowing fatty acyl-CoA synthetase. If that was identical, then it probably meant that the glow came from a less specialized chemical. So they began testing relatives of the firefly chemical, using the fatty acyl-CoA synthetase from the fruit fly Drosophila. They quickly found a chemical that, when given to Drosophila cells,
caused them to emit a dull, red glow. If they put the fly version of fatty acyl-CoA synthetase into human cells, giving them the same chemical would cause them to glow, as well.
The results suggest that a random accident - getting the right chemical in cells with a particular enzyme - provided enough glow for evolution to start selecting for it. And, with enough time, both the chemical and the enzyme became specialized, producing a brighter, more intense glow.