More than half of people believe MP’s absence is significantly damaging parliament’s reputation, poll shows