A bipartisan group of Senate negotiators is looking to replace a proposal to provide $40 billion in new funding for the Internal Revenue Service, which is projected to net $100 billion in new revenues through tougher tax enforcement, because the idea is drawing heat from conservatives.

Members of the group and sources briefed on the discussion say that negotiators are looking for an alternative to the so-called IRS pay-for in the bipartisan infrastructure bill, which is seen as a major source of revenue. But the problem is there aren’t any obvious options for finding $100 billion in revenue to replace the IRS provision.

Asked how the stalemate over IRS funding could get resolved, Sen. Jon Tester (D-Mont.), a member of the group, said: “Just got to find a pay-for, that’s it.”

Sen. Mitt Romney (R-Utah), another member of the group, said the proposal to beef up IRS enforcement as a strategy to raise more revenues could come out of the bill.

“We’re looking at different pay-fors. That’s been an area of concern for a number of us and whether that’s retained in our bill or not is uncertain,” he said.

That was confirmed by a person briefed on the negotiations who said a proposal discussed earlier in the week to put “guardrails” on the IRS as a condition for the $40 billion in additional enforcement funding has been “taken off the table.”

But the source said the problem with dropping the IRS provision altogether is that it leaves a big “hole” in the strategy for paying for the package, which is estimated to cost $1.2 trillion over eight years.

“The problem is things that were on the table before are getting removed and when they get removed you have a hole and you’ve got to fill the hole,” said the source.

Republicans have become increasingly hostile to the idea of adding more money for the IRS and a coalition of conservative groups circulated a letter among GOP lawmakers last week warning them against the idea. 

Sen. Ted Cruz (R-Texas) warned colleagues this week that sending more money to the IRS is a "foolish and dangerous idea." 

"Rather than giving tens of billions of dollars to the IRS to harass and persecute American taxpayers, I think we should abolish the IRS and instead adopt a simple flat tax," Cruz said.  

Sen. Rob Portman (Ohio), the lead Republican negotiator on the bill, said after a more than two-hour meeting with Senate colleagues and senior White House officials that the group is still working on ways to pay for the proposal.

“We’re working on the pay-fors still and we’ll end up with a good package we’ll have a chance to talk to you about,” he said. “We’re working through it.”

Three senior White House officials including Biden adviser Steve Ricchetti, National Economic Council Director Brian Deese and White House legislative affairs director Louisa Terrell walked into the bipartisan senators’ meeting shortly after the two-hour mark, a sign the negotiators are bouncing new ideas off the administration. 

Complicating the situation is that Senate Majority Leader Charles Schumer (D-N.Y.) has announced he will schedule a vote on Wednesday to proceed to the legislative vehicle that lawmakers hope to use to pass the bipartisan infrastructure package.

That gives negotiators only a few days to work out their differences.

“The good news/bad news is we have a pretty tight time frame. Schumer has indicated he wants to try to bring this up for cloture next Wednesday. That’s pretty aggressive. That means we have a lot of work to do,” said Sen. Lisa Murkowski (R-Alaska) after meeting with senior White House officials.

“There are details that we have to resolve and so those details involve things like pay-fors as well as some of the issues we are working on in other accounts,” she said.

Romney on Thursday threatened to vote against the motion to proceed to the bipartisan infrastructure bill before its text is finalized.

“I think it would be a dereliction of duty to vote for a bill that hasn’t been drafted yet,” he said.

Yet Romney sounded an optimistic tone before leaving the Capitol for the day, telling reporters that the bipartisan coalition is holding together.

Portman, when asked about Schumer’s plan to vote on the bipartisan framework as soon as Wednesday, called it an “arbitrary timeline.”

“The prospects remain good because infrastructure is so darn popular,” Portman said after Thursday’s meeting. “We’re going to get it done."

“I don’t know if we’ll make anybody’s arbitrary timeline but that’s not the point, the point is to get it right. We’re moving as fast as we possibly can,” he added. “We want to get this right. It’s a big bill. This is an unprecedented amount of spending for our nation’s infrastructure.”

Updated at 5:46 p.m.