Full Idea
There seem to be no actual infinites in the physical realm. Given the correctness of atomism, there are no infinitely small things, no infinite divisibility. And General Relativity says that the universe is only finitely large.
Gist of Idea
Given atomism at one end, and a finite universe at the other, there are no physical infinities
Source
James Robert Brown (Philosophy of Mathematics [1999], Ch. 5)
Book Reference
Brown,James Robert: 'Philosophy of Mathematics' [Routledge 2002], p.66
A Reaction
If time was infinite, you could travel round in a circle forever. An atom has size, so it has a left, middle and right to it. Etc. They seem to be physical, so we will count those too.