Confession: I’m Sleeping With My Husband’s Brother

I need to get something off my chest.
I can’t talk to my friends or family about this but I’m between a rock and a hard place. I’ve been with my husband for 10 years and we’ve been married for 7 years in September. We have 2 boys, 5 and 2, and I would give my life for my family.
My husband and I have been having problems for the last 2 years because he’s been going out of town for work a lot and I don’t trust that he’s all business and no play while he’s away. I didn’t like the idea that he would be gone to different cities for two and three weeks at time and when he came home he would be too tired to give me any attention or even help with the kids.

After having my youngest son, I felt alone and was always at home with the kids by myself. While he was on the road, my husband’s brother who is also married with older kids would come by and pick up my oldest son and take him to get a haircut or do other guy stuff. He was basically doing all the manly things my husband doesn’t do. Him and my husband are like night and day. My husband being a corporate man and his brother owns a landscaping business.
My husband is very studious, straightlaced and matter of fact but his brother is sort of street smart, self taught works with his hands kind of man. For the longest time his brother and I didn’t get along and we constantly bickered back and forth but the more my husband traveled the closer we became. Back in April we had a party at our house and his brother was there talking about how much my son has come along with dribbling a basketball and my husband got jealous and made a scene saying that he had kids of his own to raise and that he needed no help raising ours.
His jealousy went through the roof when I backed his brother and told him that I appreciated that he makes the time to do these kind of things with our sons. The next week when my husband left town his brother called and asked if it was okay to come and get my sons to take them out and asked if I wanted to come along and I did. We had a good time and we came back to my house and I cooked dinner for all of us, the kids ate and fell asleep.
I had a few drinks and he had a few beers and the next thing I know we’re on the kitchen floor having the best sex I’ve had in years. I felt horrible about it and thought it was a one time ordeal and was scared to face my husband when he came home. For about a month I avoided all his brother’s phone calls and didn’t want anything to do with him. Within a few weeks my husband had made arrangements for him to make use of some of our garden equipment that we don’t use and he came to pick it up.
I opened the garage door so he could get what he needed and leave. Then he came in the house to let me know he was leaving and I said okay thanks but he closed the door behind him and starting kissing me and once again we had sex. I can’t resist him and we’ve slept together 4 times since then. I can’t go on this way. I’m constantly paranoid that they might argue and his brother will go off and tell him that we’ve been sleeping together.
Every time my husband calls my heart races thinking that he knows and he’s caught wind of it. I don’t know what to do because I don’t want to leave him but I’ve developed feelings for his brother.
Please tell me what to do.