Updated

An experimental cancer vaccine from GlaxoSmithKline failed to help melanoma patients in a closely watched late-stage clinical trial, dealing a blow to the high-risk, high-reward project.

Britain's biggest drugmaker said on Thursday that it would continue with the Phase III trial of its MAGE-A3 therapeutic vaccine to see if it benefited a subset of patients with a particular genetic signature.

The trial has two joint targets, known as co-primary endpoints, looking at both the broad population and a patient subset, so it is possible the treatment will yet prove effective for a small number of people with the deadly skin cancer.

It is also being tested in lung cancer.

The vaccine is one of two particularly high-risk developmental GSK products that industry analysts believe could, in theory, become multi-billion dollar sellers. The other is a heart drug called darapladib, designed to fight clogged arteries in a different way to statins.

In both cases, however, analysts have been wary of modeling firm sales forecasts, given the above-average risk of failure, and as a result GSK stock fell only 1 percent by 1410 GMT.

Shares in its U.S. partner Agenus, however, slumped 33 percent, reflecting the high hopes riding on the project at the small biotech firm. Agenus provides one of the components in GSK's vaccine.

Citigroup analyst Andrew Baum said that already low consensus expectations for the product are now likely to fall further. He currently forecasts sales of only $233 million in 2018 for the MAGE-A3 vaccine in both melanoma and lung cancer. That figure is about half the industry consensus.

Unlike traditional preventative vaccines, the MAGE-A3 treatment is designed for people with established disease, helping their immune systems to prevent the return of disease after surgery.

Other companies, including Bristol-Myers Squibb, Roche and Merck & Co, have had some recent notable successes in clinical trials of innovative drugs to boost the immune system, but GSK is still pushing the scientific boundaries with its MAGE-A3 vaccine.

Difficult area

Cancer vaccines have proved difficult to develop over the years. A different vaccine for lung cancer from Germany's Merck KGaA also failed in a clinical test last December.

GSK's vaccine was developed for use in patients with tumors that express the cancer-related MAGE-A3 gene, which occurs in about 65 percent of Stage III melanomas.

It failed to extend disease-free survival significantly when compared with placebo in this population in the Phase III study.

The second endpoint of the trial is to discover whether disease-free survival is improved in a subset of MAGE-A3-positive patients. Results from this analysis are expected in 2015.

GSK added that no safety issues had emerged in the trial.

"We remain committed to identifying a patient sub-population who may benefit from this investigational treatment," said Vincent Brichard, head of immunotherapeutics at GSK Vaccines.

A spokesman said it was not yet possible to say what proportion of patients might fall into this subset.

GSK is also continuing to study MAGE-A3 in another Phase III study against non-small cell lung cancer. The first data from this trial are expected in the first half of 2014.

Alistair Campbell, of Berenberg Bank, described the chances of a success in lung cancer as "slim".

"This asset was widely regarded as a 'wild card' by the market. While there had been some hope that this approach might work, the result is not a surprise," he said in a research note.

Roth Captial Partners analyst Joseph Pantginis, who follows Agenus, said he had reduced the projected chance of success for MAGE-A3 in melanoma to 5 percent from 50 percent, with a 15 percent chance of success in lung cancer from 50 percent.

Although the news on melanoma was disappointing, 2013 has still been a strong year so far for GSK research, with new drug approvals in HIV, cancer and respiratory disease.

GSK is now waiting to hear if its latest lung drug, Anoro, will be approved. A U.S. Food and Drug Administration advisory panel is set to assess the medicine on September 10.