I am going to be more than happier ever for writing this blog as a predecessor to this one, as my for the failure seems more or less accurate. Ok, let's have the solution before explaining it, for the busy reader.
When invoking java pass the following, or set it in your properties file.
Now let's discuss why this works in detail.
The Environment & the Precise Failure reason
My host machine runs Windows7 x64, and hosts the Postgresql client application (host). I have a CentOS x64 VM running in VirtualBox, where my postgresql server is running (guest). I have forwarded the port 5432 from the guest to the host. The host and the guest are setup as a VirtualBox Host Only Network. Till the point i connect to my Office VPN, everything works, even when i am connected to internet using a different Network Adapter. But the moment i connect to office VPN, it fails. (Note : In my previous post, the failure i mentioned is when connected to VPN. But even then my sample python implementation passes there)
The Debugging part
I understood that everything works if i disconnect the VPN, but why. To understand this i took the source code of Postgresql JDBC Driver, built it and added it as the dependency to the java source code i posted in Part 1, and debugged it. As i expected, the connection failed when establishing the underlying socket connection. (The exception information has the entry on Socket Connection failure). The Postgresql JDBC driver uses the java socket implementation, to establish a communication with the respective PostgreSql Server port (e.x 5432).
I looked into the connect0 method (which is on top of the call stack , during failure) which is for the greatest disappointment a JNI call :-(, so we can't find much from the code. But there is one thing that we could check. i.e in the source code of the call stack hierarchy, which part could have changed that made the same program to work, when invoking with the preferipv4stack=true option. So i checked that. This happen in PlainSocketImpl.java which checks for the property preferIPv4Stack and decides which implementation to use.
So to give a little background here, Java has two implementation for sockets from the perspective of the Windows Network layer. They are TwoStackImplementation (which specifically handles IPv4, IPv6 separately) and DualStackImplementation (which leverages the capabilities of Windows to implement sockets using native dual stack configuration). Based on the value mentioned for preferIPv4Stack, one of them is used. It's clear that the TwoStack implementation works irrespective of whether VPN is connected or not. But the DualStackImplementation is the one that fails with VPN. Since this is clear, i kept this working part away from my focus (TwoStackImplementation) and focused on the part that is unclear which of course is the VPN related failure.
Now let's discuss the cases that works and the ones that doesn't
- No VPN, No preferIPv4Stack = works, uses DualStack network configuration
- No VPN, preferIPv4Stack = works, uses TwoStack Socket network configuration
- Vpn, preferIPv4Stack = works, uses TwoStack network configuration
- Vpn, no preferIPv4Stack = doesn't work, uses DualStack network configuration
So the issue is clear. What next. See the code flow for cases (3) and (4) and see where the code diverges. For that we need to step through JDK :-). The only way i could isolate this issue from JDK implementation is i need to step through the code and see what happens. The challenge of course is that i cannot watch variables in JDK, for that i have to create a JDK build with more debug information. But for sure stepping through the source is possible (check this post), which i did. And for my greatest surprise the native api connect0 succeeds for case (3) and fails for case (4), apart from it the states are the same. So what next, i have to look into the windows implementation to find the real issue.
I believed we would have solved it by now, but i guess this only get's more interesting and lengthy. I will post more about my findings in next post (hopefully the final one)
Update : Here it is, Part 3 a.k.a final part
Have a good time :-)