Allegations of domestic violence, which Madonna has since denied, were in the news during and after the couple's rocky marriage from 1985 to 1989. However, Penn said he first heard the particular story about the bat over a morning cigarette.
A woman he was dating confronted him about it after "a lovely evening" together, he told The New York Times in an interview that published Saturday.
"She's looking at me like I killed her dog," the "Mystic River" star said.
"I didn't know what the hell she was talking about," the Oscar winner added. "Now I think it's fair to say that I'm not the biggest guy in the world. But if I hit Mike Tyson in the head with a baseball bat, he's going to the hospital."
The "Mystic River" star was accused of domestic violence during their turbulent marriage from 1985 to 1989 and, despite successfully settling a 2015 defamation case in which Madonna defended him, continues to meet people who haven't heard the news.
The Oscar winner was specifically alleged to have beaten Madonna with a baseball bat in 1987, a reported rumor he purportedly only discovered over a morning cigarette -- when a woman he was dating confronted him about it after "a lovely evening" together.
The rumor that Penn hit the singer with a bat in 1987 fit in with the narrative around the pair's relationship that flared up when Madonna filed an assault complaint against Penn in December 1988. While she reportedly withdrew the complaint a week later, police flooded the couple's Malibu, California, home and ordered Penn outside with his hands up.
"I had a freaking SWAT team come into my house," the 63-year-old actor told the Times. "I said: 'I'm not coming out. I'm going to finish my breakfast.' The next thing I knew, windows were being broken all around the house and they came in."
The allegations of abuse drew attention again in 2015 when director Lee Daniels defended his "Empire" star Terrence Howard by comparing the actor's own admitted domestic violence to Penn's alleged actions with Madonna.
Penn, who was also rumored in the tabloids to have tied Madonna up and left her "trussed up like a turkey" for nine hours, sued Daniels for defamation in the sum of $10 million in damages. Madonna provided an affidavit in support of Penn.
"While we certainly had more than one heated argument during our marriage, Sean has never struck me, 'tied me up,' or physically assaulted me, and any report to the contrary is completely outrageous, malicious, reckless and false," she said in her statement at the time.
