Previously during an argument, my bf and I were arguing about how he was 2 hours late to pick me up because he went to a girl's house to help her fix her internet. He had never met this girl before, but she is his friend's friend. And his friend asked him to go help her out.

I was unhappy about this, and I admit that I made a bigger deal than I should have by saying, "Well I guess other girls are more important than your girlfriend, seeing as you are taking care of them first."

The argument got uglier.

At which point my boyfriend said to me, "Wow, I'm glad I'm not your husband, or else I would feel worthless. Thank god you're not my wife."

This was obviously very hurtful to me, and it shocked me so that I was speechless.

It is roughly 4 months later now, and while having dinner with my bf and his friend, his friend lets loose in the conversation that my bf has been telling some friends that he hopes to marry me this year. His friend teasingly asks if I would marry my bf this year.
Despite being slightly happy that my bf is planning on marrying me, I can't help but remember what he said to me in our previous argument.
So I laughed and said no very jokingly to round up the atmosphere.

Later on, my boyfriend casually asks me, "...You wouldn't marry me this year if I asked you?"
And I told him how I feel about marrying him when I can't forget what he said to me a few months ago.

My boyfriend claims that it was how he felt at the time regarding that situation and argument, not how he feels towards me in general.

But I feel so hurt by the words and what they might represent that I'm too scared to believe him now.

What should I do? Do you guys I should just let it go as an argument?