It depends on what your goals are. As a swimmer myself who has done a few marathons I can understand where you are coming from.
If you are just aiming to get around the marathon you will be fine. I did all my marathons on three runs a week, similar to how you describe yours. My first one I completed in 5hr 25 seconds (damn those 25 seconds) without needing to walk at all and maintaining a consistent pace. To me that was a success and I prefered to take a bit longer and do a steady run than to get a faster time.
My fastest marathon was about 4 hr 37 min, done on the same training but a year later.
For me, the biggest challenge was with the mental training - convincing myself that I was capable of doing it and pushing myself through the tough times. You can train your brain during cross training sessions as well as during running. Try doing some of your running and cross training sessions back to back while maintaining the effort and keeping a positive 'can do' attitude