Hardly and ever can be separate adverbs, but it is better to consider hardly ever a frequency adverb.