It kind of sounds like you are looking for a Platonic form.
As used in every day language numbers are an abstraction and an inference preserving metaphor. As abstractions go while you can find physical instantiations of something that shares the properties of the metaphor, but there isn’t instance of the pure abstraction. Trying to treat the abstraction as something concrete is called
reification. Note that this is also a classification of
logical fallacies when used in logical arguments.
Numbers are useful for constructing logical isomorphism of many systems. But how the numbers map to the system may vary. Operations performed on numbers (multiplication, division, subtraction, addition, and others) often (but not always) can be mapped to physical actions. Whether or not a mapping can be made will depend on the system.
For example, when dealing with sets the + in the expression 1+1=2 would represent the combining of sets and the digits represent set sizes. But when dealing with boolean logic the expression in the expression 1+1=1 the digits represent a bivalent truth value(true/false, yes/no, or many others; the metaphor is multi-layered) and the + operator represents something different.