Sfdwhf

I used to think because that, if objects are treated as waves, then using operators is the necessary thing to do in order to "retrieve" the observable from a given wavefunction.

For example, in position-space and a plane wave,

$$hat p = - ihbar frac{partial}{partial x} $$

$$psi(p) = frac{1}{2 pi hbar} e^{frac{i}{hbar} p x}$$

$$implies hat p left(frac{1}{2 pi hbar} e^{frac{i}{hbar} p x}right) = p psi(p) $$

But this only works for plane waves. Most wavefunctions I can think of are not eigenfunctions of the momentum operator, so I clearly have it wrong.

In mathematics, in order to describe objects that: do not commute, can be summed, can be multiplied, can be rescaled, and can be "complex conjugated" (the last mostly for convenience, admittedly); one needs to use a so-called noncommutative *-algebra.

If in addition one endows such objects with a notion of "magnitude" (mathematically a norm, physically the maximal magnitude of the possible observable values), then the natural mathematical concept is that of a noncommutative C*-algebra.

Noncommutative C*-algebras can always be concretely realized as noncommuting operators on some Hilbert space (this result is called Gelfand-Naimark Theorem).

Therefore, the latter is indeed the natural mathematical way of representing non-commutative objects that have the properties of quantum observables.

Any introductory quantum mechanics text book will cover this. The basic idea is that physical quantities that could in principle be observed, such as position, momentum, angular momentum, energy, are associated with operators. We say that each such quantity is represented by its associated operator. This is a mapping between a physical thing (e.g. momentum) and a mathematical thing (the momentum operator $hat{bf p}$).

These operators connect to physical predictions in mainly two ways. First, they may enter into the Hamiltonian operator which governs evolution over time through Schrodinger's equation. Second, each operator has a set of states associated with it---the eigenstates---and their corresponding eigenvalues. But as you have noted, the system does not have to be in one of those special states. However, whatever state $psi$ the system happens to be in can always be written as a superposition of eigenstates $u_n$ of whatever (Hermitian) operator you might take an interest in:
$$
psi({bf x},t) = sum_n a_n(t) , u_n({bf x})
$$
where $a_n$ are complex coefficients. In this way you can determine how much momentum (or whatever) the system has at any given time.
It typically won't have a single well-defined momentum (for example) but a spread of values corresponding to whichever $a_n$ are nonzero. A measurement of that property would give the $n$'th eigenvalue with probability $|a_n|^2$.

I hope this answer is helpful, but I must admit that I don't think you will really be able to understand this answer until you have explored quantum mechanics a bit more fully.

Let's try a somewhat more example-driven answer.

Take an electron in the state $psileft(xright)=frac{1}{sqrt{2pi}}e^{ifrac{p}{hbar}x}$ (assume $p>0$). As you said, the momentum operator $hat{p}=-ihbarfrac{partial}{partial x}$ retrieves the momentum by a simple relation

$$hat{p}psileft(xright)=ppsileft(xright)$$

So this is an electron moving to the right with momentum $p$. All good.

Now take a look at another perfectly valid state, $psileft(xright)=frac{1}{sqrt{pi}}cosleft(frac{p}{hbar}xright)$. Applying the momentum operator this time gives

$$hat{p}psileft(xright)=-ihbarfrac{partial}{partial x}frac{1}{sqrt{pi}}cosleft(frac{p}{hbar}xright)=ipfrac{1}{sqrt{pi}}sinleft(frac{p}{hbar}xright)$$

Yikes. This is in no way $left({rm some:constant}right)cdotpsileft(xright)$. So we can ask ourselves: what is the momentum of this state? To answer this question you need to understand that we are not reading the information correctly, or maybe even expecting something wrong. Let us rewrite our state differently

$$psileft(xright)=frac{1}{sqrt{pi}}cosleft(frac{p}{hbar}xright)=frac{1}{sqrt{2}}color{red}{frac{1}{sqrt{2pi}}e^{ifrac{p}{hbar}x}}+frac{1}{sqrt{2}}color{blue}{frac{1}{sqrt{2pi}}e^{-ifrac{p}{hbar}x}}$$

Now look what happened. This state has in fact two components - one with momentum $color{red}{p}$ and one with momentum $color{blue}{-p}$. It is as simple as that, our state simply doesn't have a single value of momentum, but rather two. An electron in this state moves to the right and to the left at the same time.

So $hat{p}$ knows how to retrieve the momentum of its eigenstates. Want to find to momentum of a general $psi$? Just write it as a sum of eigenstates of $hat{p}$. Let's call them $psi_{p}$, and write

$$psi=sum_{p}c_{p}psi_{p}$$

Each $psi_{p}$ have definite momentum $p$, and your $psi$ has a component of fraction $left|c_{p}right|^{2}$ with momentum $p$.