Wait, the speed of time can be measured or at least ascertained through mathematics?

It's more that, in the relativistic treatment of space and time, you can describe a particle's motion through spacetime with a four-dimensional quantity called the four-vector. When you take the magnitude of this four-vector, the magnitude is a constant — the speed of light. This means that your total speed through the three spatial dimensions plus one time dimension of spacetime must sum to the speed of light. Thus, any particle moving at light speed does not experience time progressing, and anything at rest in space must be moving through the time dimension at the speed of light. It also means that, if there are any particles moving faster than light, they would experience time backwards (like Merlin in The Once and Future King).