Immigration officials must improve how they deal with people in detention so they are not held for longer than necessary, say two watchdogs.
The inspectors of prisons and immigration said most decisions to detain were justified.
But they called for an independent panel to examine cases where people are held for lengthy periods.
A spokesman for the UK Border Agency said it was already addressing issues raised by the report.
The report into UK immigration detention in the UK said that on any given day in the first quarter of 2012, there were about 3,500 people being held and expecting to be removed from the UK.
Immigration officials have broad powers to detain people with no authorisation to be in the UK, but the inspectors said that they had found 40 cases where people had been held for more than two years, rather than being removed.
Detention can cost up to £40,000 a year - and about a third of those held during the period were ultimately released.
The inspectors said that they found that in most cases the decision to detain had been "defensible and properly evidenced" and managed correctly, but they added that there was insufficient progress in a quarter of cases they had reviewed.
Torture victim
Many of the detainees were former prisoners and the inspectors said more could be done to resolve their cases before the end of their prison sentences.
Delays included problems with case files and even in persuading someone's home country to provide a passport.
The watchdogs said that they found instances where people had been detained even though they were victims of human trafficking. In one case, a victim of torture had been detained without evidence of exceptional circumstances to justify the decision.
Nick Hardwick, chief inspector of prisons, and John Vine, chief inspector of immigration, said: "Despite much effort at improving the system, it is questionable whether the length of detention in some cases was necessary or proportionate to the legitimate aim of maintaining immigration control.
"Although all immigration detainees have a right to apply for bail, there is no routine independent system of reviewing cases.
"We recommend that an independent panel should be established to review the cases of all individuals who are held for lengthy periods. We believe this would motivate change in the system."
A UK Border Agency spokesman said: "We are pleased that this report notes the progress which has been made.
"Prior to its publication, we had already identified several of the issues raised and begun to address them."
BBC.com
