Exceptionally high latency in Azure Web App

We are currently hosting our website ourselves, but we have had a few outages outside of our control and are considering moving it to Azure. This is an ASP.NET website using Umbraco as a CMS.

Yesterday I signed up for an Azure trial, copied a copy of our database to an Azure SQL Server instance, deployed a new web app, and used Web Deploy to download the app. This was my first experience with Azure, and I was pleasantly surprised how easy it was. There were several problems connecting my new application to my new database, but overall it was a simple process.

But the performance is terrible. The database is standard S2 and I originally created a web application at a free tier. I experienced both poor download speed and latency. The first thing I tried was to scale up the web application, so I took it to a standard environment. This seems to fix the download speed, but the latency is still impressive.

I am using google chrome web bar to test speed. Here I am downloading an image from our server:

Obviously this will be fast as it happens on our local network, but it at least shows that the application is not a problem.

Here's how I get a standard S2 hosted in Australia East:

The download speed is not that bad right away, but with TTFB 41.92s it's crazy! This is consistent, sometimes I only achieve 8s, but this is still not acceptable.

I don't have this problem when visiting other sites, so my internet is not a problem. I've tried using Small S2 and Large S2 without any change in the results.

Am I doing something wrong? I find it hard to believe that every Azure customer experiences this level of performance.

EDIT: Here's what we've learned in the comments so far:

Always On setting doesn't help.

Using Azure CDN is just as slow.