"There is not enough shale gas in the north Jutland underground at Dybvad for commercial extraction," the Danish energy ministry said in a statement.
Energy Minister Lars Christian Lilleholt said that while Total's move was "regrettable" for government finances, he was "glad" that local residents in Dybvad now had "clarity regarding Total's decision to discontinue its work in the area."
Danish state oil and gas company Nordsofonden held a 20 percent stake in the licence.
Environmental activists had sought to ban the so-called fracking process to extract the gas amid fears that ground water could be contaminated by drilling fluids.
Fracking, or hydraulic fracturing, is a process used to extract gas or oil from shale rock by blasting a high pressure mixture of water, sand and chemicals deep underground to release hydrocarbons trapped between layers of the rock.
"Total's exit from Denmark is good news for the climate, the environment, the ground water and public health," a spokesman for Greenpeace in Denmark, Tarjei Haaland, said in a statement.
Total abandoned another shale gas project in eastern Denmark last year, also citing poor exploration results.
Denmark's former centre-left government temporarily stopped awarding shale gas licences in 2012, and Lilleholt said previous exploration drilling projects had to be evaluated before new permits could be issued.