By Holly Yan, Nick Valencia and Barbara Starr
HAILEY, Idaho (CNN) -- No one here knew exactly where he was or what he was going through, but this tiny city in Idaho never forgot about Sgt. Bowe Bergdahl.
Yellow balloons and American flags line the sidewalks of Hailey, population 8,000. Residents planted one tree for each of the five years Bergdahl was held captive in Afghanistan.
Now, uncertainty has given way to elation as the Army sergeant has been released from his captors -- in exchange for freeing five Taliban members held at Guantanamo Bay.
"There were times where we wondered, but (parents) Jani and Bob Bergdahl never once gave up faith that their son was coming home to them," family friend Stefanie O'Neill said.
Bergdahl's parents said the battle is far from over.
"The recovery and reintegration of Bowe Bergdahl is a work in progress," Bob Bergdahl told reporters Sunday, one day after his son was freed.
"It isn't over for us. In many ways, it's just beginning for Jani and I, and our family. There's a long process here."
So what's next for Sgt. Bowe Bergdahl and the Obama administration, which has come under criticism from some lawmakers for the secretive nature of the swap?
WHAT'S NEXT FOR BERGDAHL
Recovery in Germany
Bergdahl is recovering at Landstuhl Regional Medical Center in Germany, a U.S. Defense official said Sunday. He is also expected to be debriefed at Landstuhl.
"He had lost ... a good bit of weight," National Security Adviser Susan Rice told CNN's "State of the Union With Candy Crowley."
Believing that his health was deteriorating, the U.S. acted quickly to save Bergdahl's life. U.S. special operations forces recovered Bergdahl without incident Saturday at a pickup point in eastern Afghanistan, near the border with Pakistan.
American officials said the government of Qatar brokered the deal.
"We were very concerned that time was not ... something we could play with -- that we needed to act when we had the opportunity," Rice said.
There was no immediate word on Bergdahl's condition after his release. A senior Defense official said Bergdahl is having trouble speaking English but the reasons for that were not clear, given the trauma he's been through.
After Germany, Bergdahl will likely go to Brooke Army Medical Center in San Antonio, Texas. It's unclear whether his parents will see him first in Germany or Texas, a Defense official said.
Speaking to CNN, White House spokesman Jay Carney declined to discuss Bergdahl's condition, saying he should be allowed to recover privately.
"His process of repatriation has begun, and reintegration into society," Carney said. "I'm sure it will not be an easy one, but we're overjoyed on behalf of his parents and his friends and family that he's returning home."
Transitioning to normal life
Bergdahl was believed to be held by operatives from the Haqqani network, which is affiliated with the Taliban and al Qaeda.
An Afghan Taliban commander not authorized to speak to the media confirmed to CNN that Bergdahl was captured by insurgents with links to the Haqqani network in Pakistan.
Over the years, the captive was transferred back and forth between Afghanistan and Pakistan, the Taliban commander said.
David Rhode, a former New York Times journalist who was held captive by the Taliban for seven months, said Bergdahl faces a long road ahead.
"Psychologically, there will be a long debrief," said Rhode, who was taken in Afghanistan and held in Pakistan until he managed to escape. "They're actually going to want intelligence (about) who held him."
But there's a chance Bergdahl might not know much about Taliban commanders.
"I was kept with very young guards, most of the time, and I didn't really meet very senior commanders that often," Rhode said.
One question that remains unanswered is how Bergdahl disappeared in the first place.
Several veterans and soldiers call him a deserter who walked off his base on June 30, 2009. At least six soldiers were killed in subsequent searches for him.
"Bowe Bergdahl deserted during a time of war and his fellow Americans lost their lives searching for him," said former Sgt. Matt Vierkant, a member of Bergdahl's platoon.
Vierkant said Bergdahl needs to not only acknowledge his actions publicly but face a military trial for desertion under the Uniform Code of Military Justice.
When asked Sunday whether Bergdahl had left his post without permission or deserted, Defense Secretary Chuck Hagel didn't answer directly.
"Our first priority is assuring his well-being and his health and getting him reunited with his family," Hagel said. "Other circumstances that may develop and questions, those will be dealt with later."
A senior Defense official said Bergdahl will not likely face any punishment.
"Five years is enough," he told CNN on condition of anonymity.
Carney said questions about the circumstances of Bergdahl's disappearance didn't play into efforts to retrieve him.
"Here's what matters. He was a prisoner in an armed conflict, a member of the military, and in that situation the United States does not leave its men and women behind," Carney told CNN.
WHAT'S NEXT FOR THE ADMINISTRATION
Defending the swap
Some lawmakers are openly wondering whether Bergdahl's release sets a dangerous precedent for future kidnappings of Americans.
"The methodology in what we used is very troublesome," said Rep. Mike Rogers, chairman of the U.S. House Intelligence Committee.
"We have a changing footprint in Afghanistan, which would put our soldiers at risk for this notion that if I can get one, I can get five Taliban released."
Rice defended the Obama administration's efforts.
"If we got into a situation where we said, 'Because of who has captured an American soldier on the battlefield we will leave that person behind,' we would be in a whole new ear for the safety of our personnel and for the nature of our commitment to our men and women in uniform," she said on CNN's "State of the Union."
"Because it was the Taliban that had him did not mean that we had any less of an obligation to bring him back."
Some members of Congress are trying to understand why they didn't know what was happening.
Rep. Buck McKeon, chairman of the Armed Services Committee, said he still hasn't been briefed by administration officials on the detainee swap.
The California Republican said he plans to hold hearings about the Bergdahl exchange, he told CNN Chief Congressional Correspondent Dana Bash on Sunday.
"We're getting our information -- other than a very brief notification -- we're getting most of our information now from the media," he said in an exclusive interview on "Newsroom."
McKeon said the Defense Department notified his staff on Saturday, after the exchange took place. But he pointed to a law that requires the administration to notify Congress 30 days before detainees are released from the facilities at Guantanamo Bay.
Rice said Sunday that the "acute urgency" of Bergdahl's declining health justified the administration's decision to execute the swap without telling Congress.
"It was determined that it was necessary and appropriate not to adhere to the 30-day notification requirement, because it would have potentially meant that the opportunity to get Sgt. Bergdahl would have been lost," she said.
CNN's Nick Valencia contributed to this report from Hailey, Idaho. CNN's Alexandra Field, Steve Almasy, Jake Tapper, Sophia Saifi, Victoria Eastwood and Ashley Killough and journalist Zahir Shah Sherazi also contributed to this report.
™ & © 2014 Cable News Network, Inc., a Time Warner Company. All rights reserved.