Ytukyg

Say we have a beam hinged to a wall. The beam is length $l$, and of mass $2m_1$. The other end of the beam is supported by a rope which goes over a pulley and is connected to a weight of mass $m_2$. We have three forces $vec{H}$ being the force on the hinge, $vec{M}$ being the force of weight of the beam and $vec{R}$ being the tension of the rope. Below is a diagram showing the situation on the left, and the forces on the right.

We take the origin $o$ to be from the hinge on the wall. We specify the axis with positive $x$ right along the beam and $y$ up from this. The rope forms an angle $theta$ with the beam. This system is in equilibrium. We're now asked to find $theta$ in terms of the parameters given. I have resolved the vectors as follows:

$$vec{H}=-m_1gcot{theta}(vec{x})+m_1g(vec{y})$$
$$vec{R}=m_1gcot{theta}(vec{x})+m_1g(vec{y})$$
$$vec{M}=-2m_1g(vec{y})$$

How do I resolve this into an equation for $theta$?

You've set up your equations wrong. Let's start with the weight of the beam. I think you call it $vec M_1$ in your figure, but $vec M$ in the equations. So let's stick with the equation. That's correct. Now, since the beam is in equilibrium, the force on the hinge, the weight of the beam, and the force in the rope add to $0$:
$$vec H+vec M+vec R=0$$
Once you calculate $vec R$, you can use this equation to calculate the force on the hinge. But that's not what they ask.

I have no idea where the equation for $vec R$ is coming from. Since it is a tension in an inextensible string, it means the magnitude is constant in the string. The other end, where you suspend mass $m_2$, the string is in equilibrium, so $|vec R|=m_2 g$

We need now one more equation. For that, note that the beam is not rotating around the hinge, so the total torque is zero. Note that only forces perpendicular to the beam have torques. So it will be $vec M$ in the negative $y$ direction and the $y$ component of $vec R$
$$-|vec M|frac l2+|vec R|sintheta l=0$$
Putting it all together, $g$ and $l$ cancel, so you have $$-frac{2m_1}2+m_2sintheta=0$$ or $$sintheta=frac{m_1}{m_2}$$