Legend not to interfere with swipe on tablets to scroll the screen

In a tablet, when I try to swipe the screen to scroll down/up, I have to avoid touching the legends. Otherwise, the screen won't scroll, instead, one of the legends would receive the touch event. Is it possible to make a change such that swipe on the legends can scroll the screen, instead of toggling the legends?