Hotels in the Old City and surroundings:
The area marked in a red frame is the old city. The blue area is the fountain square, Nizami street and surrounding streets where you'll see only old and architectural buildings of late 19th - early 20th centuries. The boulevard (green frame) and Maiden Tower (yellow frame) are marked for orientation only. I can't recommend which hotel exactly to choose in this area, because as a local I never had to stay in any of them. has "reviews" option which may help you make the right decision.

Route of the bus number 65 around Old City walls:
Route marked in red; blue dots are the bus stops. Trip fair for all buses in downtown is 0.20 AZN. 
Location of the currency exchange point: 
Here they accept more than just regular (more than just USD, EUR, GBP) currencies. Blue dot is the location.
Grocery store which is open until late hours:
It's in a 5 minute walking distance from fountain square. You will see a "Romashka" sign on it. It's the red dot on the map.
Drugstores located nearby the fountain square:
Find the red dots in case you need to buy medicine or pills.
Police Station nearby the fountain square:
In case you get into trouble or something, find the blue dot.
Three Avenues to witness early Soviet Architecture:
The staring point is Hyatt Hotel nearby Bakhikanov bridge. From there you can take a cab (5-10 AZN at most for the whole ride back to downtown) or the bus number 95 (makes a small turn between Inshaatchilar and H.Javid avenues / you'll waste about 5 minutes). If you decide to walk, the whole path will take about 30 minutes to reach the end of Parliament avenue (mountainous park). If you do take a walk, you can return back to downtown from the ending point of Parliament avenue via bus number 95 or 18 (bus stop mark - yellow dot) or take a funicular ride in the mountainous park (yellow dot again / funicular ride is still free). If you decide to walk from there to downtown, just walk downwards by the road after a turn to the left (you'll see a stone plated road soon). It leads right to the downtown (15 minute walk distance).