I work as a catsitter, and I'm against. Maybe if it was one particular room which the cats could be shut out of, and you already had had the same builders in before and knew they could be trusted to be careful.
I've seen it quite a few times. While all the cats have been OK and tend to act sensibly, it definitely does upset them, and the mess and people coming in and out is not good. Lets see.... one household has the cuddliest cat ever, but has had the builders in several times. Mostly OK, (besides mess all over the floors which the cat is going to lick off himself, and his bed being covered with junk), but there was one time he mysteriously became very scared and started running from the builders. Don't know what happened, I presume something got dropped near him or something. Another time I was supposed to be warned before builders (sent by landlord) came round so I could shut the emotional inside cat in the bedroom. I arrived to find the door wide open... luckily she'd had the sense to hide in the bedroom, but she was upset enough to bite me on the leg and draw blood (leg biting not unusual with her, but how hard was a measure of how upset she was!) Today I arrived at a sit to find a very nice guy drilling holes in the ceiling. He had apparently made friends with one of the rather shy cats, so points there (they were in hiding but fairly relaxed when I came in). On the other hand there was plaster dust all over the cats' food and water bowls...
There have been other times when the cats involved have seemed fairly relaxed: both times just one room was involved. But you can bet the cat wil lead you to the pile of builders equipment as if asking what the heck is going on. I don't think any cat likes having the builders in.
Personally I wouldn't want builders in without me there anyway. But if they are going to be crawling all over the place with two elderly/timid/dim cats in the way I really wouldn't do it. It doesn't sound like a good mix. Not really fair on your neighbour to ask them to cope with any problems either, whether with the cats or the builders.