While each market has its own strengths and weaknesses, the Chinese market is unique when compared to Europe, the United States, or anywhere else in the world. Possibly the most relevant factor is that the Internet infrastructure in China is controlled in one way or another by the state. The Great Firewall of China (GFW) introduces variables not present in other markets, even those with strong censorship. However, this is not the only aspect to highlight.
Other factors to consider when doing business in China are the sheer number of local browsers and the considerable differences in speed and latency in different locations in China. For this reason, the way application’s performance is measured in China requires a strategy as unique as the market itself. Much of this strategy consists of using both synthetic tests and manual tests.
However, since manual testing is not as widely used in other markets, many developers underestimate the importance and benefits of this type of test. In that sense, this article explores the importance of manual testing and how it can complement the results obtained with synthetic tests to provide your team with valuable information on optimizing the user experience of your application or website in China.
Synthetic vs. manual testing, which is better?
The fact that manual tests are so necessary for China does not mean that they cannot bring benefits in any other market. In fact, after you check the pros and cons of each type of test, we expect that you will find that none is better or worse; they are simply different. It is similar to when you visit a physician; each test provides helpful information to diagnose your condition.
To that end, the following sections intend to give you an overview of the best use cases for each type of test, illustrating the pros and cons of each, so you have a better understanding of why balancing the two is essential.
Synthetic website testing: invincible?
Synthetic testing, also known as synthetic monitoring, aims to improve the performance of your application or website by proactively detecting problems that the user might experience while using it. To do this, these tests simulate user behavior by performing different actions and measuring the results.
You can think of these tests as a type of frontend application monitoring handy for early detection of bottlenecks, code bugs, performance issues, and more. Synthetic tests can be deployed either manually or automatically at predefined time intervals. Additionally, engineers can run these tests on remote servers located at strategic points in China to measure application performance in different areas. Synthetic tests can be classified into browser tests and API tests.
Synthetic browser testings have strong functions in simulating different actions such as password entry during account signup, sales cart checkout, TOTP usage, navigation menus, as well as other critical components and transactions performed with the browser. They allow identifying problems even before the release of new features proactively, they also promote the use of CI/CD pipelines during development, and allow verifying the functionality of the application on-demand.
On the other hand, API tests allow constant monitoring of application status (uptime) and page load time which is helpful in detecting outages and slowdowns, they also are useful to ensure that users can securely access the application by validating SSL certificates, and make life easier to detect problems caused by third-party scripts and API endpoints, and setting alerts when configuration errors or attacks occur.
Despite synthetic tests having such strong functions, they have their downside:
- Any change in the user interface, no matter how small, can break the script and force a time-consuming reprogramming of the script.
- Even if the tests show no problems, this does not guarantee that users will not experience functionality issues not verified by the script.
- Since they are synthetic, the application is not being tested by real users in real-life conditions.
- They are challenging to maintain at scale
- Cross-browser testing is limited
- They can be very complex to configure since they often require programming knowledge.
- Prone to false positives
- Do not take into account problems in China's infrastructure such as DNS servers performance, CDNs, GFW, poor ISP connection, and others.
Synthetic tests takeaways
Although synthetic tests have come a long way in recent years and are now easier to implement, they still suffer from the same fundamental problem; they are artificial tests by nature. Advances in AI, ML, and NLP still don't outweigh the insights that specialist engineers can provide by doing manual testing in the field. With this in mind, it's time to review the pros and cons of manual testing and how they can overcome most of the pain points of synthetic testing.
Manual website testing: more flexible and customized
In simple terms, manual testing consists of real users interacting with your application or website and taking note of all the problems they encounter. These problems can range from functionality errors to user experience issues caused by poor performance. Since these tests provide valuable insights that are impossible to obtain with synthetic testing, manual testing has proven to be a fundamental part of web application performance monitoring.
Furthermore, in markets with unique challenges such as China, having these tests ensures that critical issues affecting end users are not overlooked. These challenges include GFW censorship mechanisms and their impact on user experience, the wide diversity of local browsers, the performance difference of different ISPs in mainland China, infrastructure reliability in some regions, compatibility issues with applications, scripts, and plugins, and websites suffering from high latency, among other issues.
Benefits of manual testing over synthetic testing
Because of their nature, manual tests offer the flexibility needed for several use cases. This makes them ideal for overcoming some of the shortcomings inherent in synthetic testing.
- Great Firewall of China (GFW). As mentioned, the impact of the GFW on businesses in China is enormous. Not only can your application or website be blocked as part of the collateral damage caused by the GFW, but you can also be harmed by poor performance caused by the GFW's actions. In this sense, manual testing is vital to verify that your website or application is accessible and performs as expected from different locations inside and outside China.
- Last-mile tests. Synthetic tests can be performed from different locations on servers running scripts; however, only manual tests allow to measure the quality of service actually received by the end-user at those locations. The information from these tests is invaluable for detecting areas where your application is lagging or where high latency is ruining the user experience.
- Usability testing. Without a doubt, this is one of the great advantages of manual tests. While synthetic tests verify the usability and functionality of your application or website, they are limited to what is included in the script. On the other hand, manual tests are performed by real people who may have different behaviors or execute sequences of actions not intended by your script. Because of this, insights from manual usability tests are often more valuable than synthetic ones. In fact, insights from manual tests can even help include new behaviors in synthetic tests.
- Compatibility testing. A key point in the Chinese market is compatibility testing. Given its complexity, the synthetic tests are limited to a small group of web browsers. This does not happen with manual tests, which can be carried out in all types of local browsers, both in mobile and desktop versions. For this reason, when checking the compatibility of an application or a new feature in China, these types of tests are vital to detect problems with scripts, plugins, or features.
- Ad-hoc tests. One of the great disadvantages of synthetic tests is their complexity since they must be previously programmed to perform certain actions or test specific features. In this sense, manual tests offer unique flexibility since they can be organized in a short time to test a new feature, compatibility, or performance.
- Competition benchmarking. You can think of these tests as a combination of all of the above. After performing manual tests on your application or website, you can repeat this procedure on your competition's websites in order to compare the strengths and weaknesses of each one in different scenarios. Needless to say, the results of this type of comparison are valuable in keeping your business one step ahead of the competition.
Manual testing challenges
Manual tests certainly offer valuable insights as they take into account the point of view of real users doing tests on real devices using the most popular browsers in China. However, this does not mean that manual testing is not without its challenges.
- Limited monitoring capacity. Manual tests are useful for detecting problems while they are in progress. However, it is not feasible to keep real users testing your website 24/7/365. To that end, synthetic tests provide an easy way to keep an eye on the uptime and performance of your applications.
- They require a great understanding of the Chinese market. At first glance, it may seem that organizing this type of test is simple. Nothing could be further from the truth. The success of manual tests lies in the way they are carried out and the understanding of the behavior and particularities of the Chinese market. Moreover, this type of testing only makes sense when performed in key locations in China where there are often more problems with services such as video streaming for example. In this regard, GoClick China is a market leader, allowing you to take full advantage of manual testing and gain valuable reports to optimize your application or website.
Leveraging the benefits of manual and synthetic tests to achieve optimal results
At this point, you have probably realized that synthetic tests and manual tests are not mutually exclusive. Rather, they complement each other perfectly to provide a unique picture of your app's behavior in China. Manual tests allow you to identify unique problems in the user experience, while synthetic tests allow you to constantly monitor key locations in the Chinese territory. Given the quirks of China and its infrastructure, it's not uncommon for your website or app to work fine in one location and be blocked from accessing it in another. The same can be said for local browsers, which can suddenly cause your application to stop working.
For this reason, GoClick China stands out as it offers you a comprehensive solution that includes real-time monitoring and effective manual testing that achieves a 360-degree view of your application's behavior throughout China.
Conclusion
Obtaining optimal results has a lot to do with the way tests are carried out, both manual and synthetic. Only GoClick China can provide you with a balanced solution with the best of both worlds, which will allow your business to be sure that your application or website operates at maximum efficiency from any point in the vast Chinese geography. Moreover, only GoClick China has real quality assurance engineers in different cities in China managing the manual tests to ensure the best possible results. Contact us today, and one of our specialists will gladly answer all your questions.
Please click here to check out our synthetic testing solutions.
Please click here to check out our manual testing solutions.
Please click here to check out our comprehensive China testing and consulting solutions.
To learn more about synthetic testing and manual testing, please check out our article 6 things that manual testing can do but synthetic testing can't in China., A short guide to China website testing in 7 minutes.
To learn more about succeeding in the China, check out our other articles.
