The Langkær upper secondary school outside the city of Aarhus said its first-year students had been divided into seven different classes, out of which three classes had a 50 percent limit on the number of ethnic minority students.
The remaining four classes consisted only of students from an immigrant background.
The school's headmaster, Yago Bundgaard, denied allegations that the practice amounted to discrimination and said that the aim was to encourage integration by preventing a dwindling number of ethnic Danes from leaving the school.
"For real integration to take place in a class there has to be sufficient numbers from both groups for it to happen," he told public broadcaster DR.
The school had seen the number of ethnic minority students rise from 25 percent in 2007 to 80 percent of this year's first-year students.
Describing it as "the least bad solution", Bundgaard said that the ethnic minority students had been picked based on whether they had "a Danish-sounding name", but admitted that it was a "fluid" distinction.
Turkish-born commentator and former lawmaker Özlem Cekic said she would report the school to Denmark's Board of Equal Treatment (Ligebehandlingsnævnet).
"When a headmaster isolates the brown children from the white in an upper secondary school, he is part of sending a signal that the whites must be protected from the brown," she wrote on Facebook.
Human rights lawyer Nanna Krusaa also told broadcaster TV 2 that "placing students solely based on race or ethnicity is in my clear view illegal".
Danish Education Minister Ellen Trane Nørby said that she had requested a report from the school to ensure that the law was being upheld, but that she was also looking at introducing legislation to make upper secondary schools in Denmark more ethnically mixed.
"The fundamental problem is that we in Denmark have... schools with a too high ratio of students with a different ethnic background than Danish," she wrote on Facebook.