Azure .NET web apps connecting to an AWS SQL database kept throwing up this error sporadically, seemingly without any pattern.
The full message was this:
An unhandled exception occurred while processing the request. Win32Exception: An existing connection was forcibly closed by the remote host. Unknown location.
SqlException: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 0 – An existing connection was forcibly closed by the remote host.)
I tried many things.
- Added resilience using EnableRetryOnFailure() to the DB connection.
- Adding Trusted_Connection=False
- Checked the TLS_DHE bug https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/apps-forcibly-closed-tls-connection-errors
- Checked logs on both servers, but nothing seemed to be recorded.
So what was it?
In Azure Relay/Hyrbid connections, under the connection in question there were TWO listeners, when there should only be one. Each Hybrid Connection Manager you install and connect shows up there as a separate listener.
So where was the second listener? Nowhere. It seemed to be a hanging orphan link from a previously deleted connection.
The only way to delete the phantom listener was to
- uninstall HCM on the database server
- remove the connection from all azure apps using it
- delete the hybrid connection completely in azure
- recreate the connection in azure afresh
- reconnect the apps
- reinstall HCM on the database server
- connect HCM to the new hybrid connection
After this we showed one listener under the connection in Azure, and things worked immediately.
When you have two listeners the data is load balanced between them, so in my case half the time the data was being routed to a non-existent listener and failing. This is why no logs appeared on the database server – it wasn’t getting there at all!
This is a summary of my me talking to myself on StackOverflow about this, as no one came up with any answers! There are truly hundreds of ‘solutions’ to this problem, and I read every single one of them.
Comments