The concept of a vector in physics is similar to that of mathematics: a geometric object with both magnitude and direction. The magnitude is in units that may be any physical units. The direction is in angular units such as radians or degrees. These are called geometric vectors (also known as Euclidian vectors).
Note that the units for the direction are the same for all vectors. Whether a vector represents force or momentum or current density, the angular units are the same. The directional units do not depend on the units of magnitude. If an observable has magnitude and direction, the units of direction are the same as every other physical vector. What kind of physical vector it is depends on the units of magnitude, not the units of direction.
If the magnitude represents duration in a particular direction, is this a temporal magnitude with a spatial direction? No, like every other physical vector the kind of physical vector it is depends only on the units of magnitude, not on the units of direction, which are the same for all physical vectors. So a vector of duration in a particular direction is a vector of time. A vector of directional lengths or distances is a vector of space.
This is where the different senses of the word “space” can confuse us. There is space as an abstract mathematical concept, space as a directional or orientational concept, and space as a length concept. It is the sense of length or distance that distinguishes space from time in physics, with or without a direction.
There is an underlying geometry that relates to all observables and determines the meaning of “direction” in a geometric vector. In non-relativistic physics, this is an Euclidean geometry. In relativistic physics, the underlying geometry is non-Euclidean.