Were Joseph and Mary poor? How do you know?
Luke reported that their sacrificial animals were two pigeons or turtledoves.
And to offer a sacrifice according to that which is said in the law of the Lord, A pair of turtledoves, or two young pigeons. (Luke 2: 24)
Two pigeons or turtledoves were sacrificial animals determined by Torah for the poors who could not afford to provide a lamb.
And if she be not able to bring a lamb, then she shall bring two turtles, or two young pigeons; the one for the burnt offering, and the other for a sin offering: and the priest shall make an atonement for her, and she shall be clean. (Leviticus 12: 8)