knock knock

Who's there???  Who knows.  I don't answer my door.  Now that it is nice out, there are more people knocking.  Sometimes I even yell through the door to go away, I'm not answering.  If it was someone I knew, they would call or yell or something so I would know it was them. 
Today I had someone knock on the door.  I didn't even look to see who it was.  Now, I want to take my bike out and get it on my car and head up to a park to ride and I feel trapped.  I don't want someone to come up while I'm doing all this.  I'm paranoid about it.  Does this feeling ever go away?

And for the record - if my HOA stopped allowing people to solicit at our doors I would probably pay the dues. It would be worth it.