To ensure your customers have timely access to your website, website testing and monitoring is essential for your site maintenance plan. However, implementing this plan in China may not be as straightforward as elsewhere. Normally businesses implement site maintenance via VPN connection, but this is a problem due to China’s Great Firewall. That is because China has cracked down on VPN services, so only few options remain.
A VPN, or virtual private network, is an encrypted connection to an Internet service provider (ISP) done through a separate device from your own computer. It secures data transmission and prevents unwanted eavesdropping when conducting remote work. It secures network connections by encrypting data transfers while concealing your IP address. Concealment is achieved by bouncing network activity through a secured server chain from different locations. This strategy helps VPNs bypass geo-blocks, or in this case, China's Great Firewall.
Comparing VPN testing with manual testing
Traditionally, manual testing is the best for site maintenance, but given the status quo of China’s Internet, foreign businesses also have the option of VPN testing. To find out which is better for website maintenance, our team decided to compare VPN testing versus manual testing for three different business websites located in Canada.
Using three different VPN services and an Internet service in Beijing, we’ve identified and compared VPN testing and manual testing results for the following websites:
- Websites tested - Air Canada, Fairmont, Four Seasons
- VPN services used - HMA (Canada to Beijing), CyberGhost (Canada to Shenzhen), Astrill (Canada to non-specific node in China)
Manual testing shows more promising results than VPN testing
Each website was tested for three consecutive days, and one test per day was completed during the identified peak Internet usage period. To help you understand the comparisons, here are some of the terms we’ve used to measure the performances of the manual test and VPN test:
- Average - If the average values of the VPN and manual tests are close, this indicates there’s little difference in user experience between the two tests.
- Coefficient of Variation (CV) - The higher the value, the more unstable the test data. If the numerical value of the CV is less than 10% —indicating data concentration — which means data comes from a consistent source. Values over 50% — indicating divergent data results — which means data comes from inconsistent sources.
- Resource - The higher the resource rate, the more files are being downloaded during website loading.
- Load time - The higher the load time, the longer it takes to fully load a website.
A VPN is not a reliable tool for long-term site monitoring and testing
Our test results show the overall data of the three VPNs are not stable enough. The difference in the test data may be due to the differences in the path and bandwidth of each connection, which means VPN tests are unstable. Therefore, we do not recommend using VPN as a tool for long-term website monitoring and testing.
Manual testing is more stable and isn't easily affected by outside interference
The manual test data shows the connection route is the same. Additionally, the resource access differential to the terminal server is minimal, while the network connection status is slightly different. This indicates uncertainty with the daily network status. It is possible to improve the website performance by adjusting our strategy and observing different loading statuses under different network speeds. For example, the company can reduce the number of “Resources” to enable users with different Internet speeds to have faster website loading speeds and better user experiences.
Therefore, we recommend conducting manual testing in the user’s region to understand the actual performance of the website.
VPN paths are unknown thus casting doubts about the reliability of site performance results
After comparing the results of the three VPN’s website performance, Astrill’s results were closest to the manual tests. However, when we take a close look at other indicators (CV), the data suggests that there’s a huge difference between Astrill’s VPN tests and the manual tests. As for HMA and CyberGhost VPN test results, the gap between the VPN and manual test data in all metrics are quite large. This indicates the real location for the web server is not based on actual user experience.
The connection speed also is shown to be better than the manual test data since VPN vendors provide a better connection tunnel. While the speed looks good, this could be problematic if used as a simulation for real users since the location of the IP addresses can’t be verified. Therefore, we suggest not using VPNs as a tool for long-term site monitoring and testing.
Manual testing is more suitable for long-term site performance monitoring
The following graph shows that manual testing is more stable than VPN testing. While resource and load times are less than 40% for manual testing, it is more than 50% for VPN testing. We also discovered that despite using the same source city for a VPN and manual testing for the test websites, the actual server these networks connect to is different. This indicates a risk of having unreliable results if you hope to use a VPN to simulate location-based, real-time user experience. Therefore, we recommend local manual testing for long-term website maintenance in China.
Manual testing is not the easiest method, but it is the best method for long-term website performance monitoring
Based on our study, VPN isn’t a recommended long-term method for site monitoring and testing. Its data source and stability usually are inconsistent, so its reliability could be questionable. This would make it difficult for IT teams to pinpoint problems and observe trends that can occur over concentrated sources of website access.
Testing over a VPN is in no way equal to testing within the country. With VPN, there are issues such as network latency, GFoC censorship, and bandwidth concerns, whereas within China people have high speed Internet connections and reliable mobile devices, which aren’t supposed to impede Internet usage. For example, when you are testing with a VPN, you are probably introducing more issues in your application than when you’re using it without a VPN. Our data here proves just how unreliable VPN testing can be.
Generally, most VPN services cannot specify the location and ISP. However, GoClick China’s experience has found that the geographical location of the Internet and the ISP creates a substantial difference in network connection quality in China. Meanwhile manual testing makes it possible to control most variables and reduce uncertainty.
At GoClick China, we have the necessary expertise for China-specific web solutions. If your business organization wants to have long-term stable website performance, we would suggest conducting manual testing to give you more accurate test results. With manual testing, GoClick China can help you improve your website and web application performance and provide you with stable and reliable website services in China.