From what little research I've seen on the subject, if it's not loud enough to annoy you, it's not bothering the fish. The ocean is noisy place, so it also stands to reason. Even the loud whine of a boat passing overhead does not seem to cause fish to take notice.