During my stay in Bujumbura I stayed in the splendid hotel Burundi Palace where I only paid 10 US dollars a night in a single room. It was clean, centrically located, and the staffs nice and helpful.
They can arrange you excursions around the country and tickets by boat through the Lake Tanganika (the second deepest in the world after Baikal Lake in Russia) to Tanzania, Republic Democratic of Congo or Zambia.
From Kigoma in Tanzania to Bujumbura in Burundi the boat journey will take you 12 hours. Usually it leaves during the night.
From Bujumbura to Kigali (capital of Rwanda) there are daily services which journey takes 12 hours.