spatially joining points within same polygons

I have point file "A" that I want to spatially join to point file "B" in order to get distance from the A points to the closest B point, but only when the A and B points at within the same polygon feature (e.g., county). What are the ways I can do this?

I do not want to select each polygon one-at-a-time.