It would be okay in a fantasy setting ("it's magic", and that's it), but not in a sci-fi setting.
Someone tell Halo, Mass Effect, Star Wars, Star Trek, Starcraft, Stargate, Warhammer 40K, Babylon 5, and literally every pop non-superhero sci-fi space opera and more general sci-fi setting under the sun that get's away with violating fundamental laws and properties of biology, chemistry, and physics and are still loved, beloved, and make oodles of money--wink, wink, nudge, nudge, X-Com very much included.
For the record, soft sci-fi is a thing, and it is the significantly more prevalent form of the genre in virtually everything you could even hope to be a fan in nowadays.
And, also, for the record, 'it's magic' can be just as unacceptable as 'it's science,' and neither should be excuses to break far more fundamental and foundational laws of logic. So, in otherwords, neither should be used to justify something so extremely illogical and irrational as time travel.