Proper functioning of a website or an application is very important today, not only for search engines but also to live up to users’ expectations. If your page load speed is unbearably slow, even the best content is of no use if users don't stay long enough to make use of it. For this reason, web performance tests and monitoring are necessary, particularly for web properties serving the China market.
This piece will review each of these tests, explore their strengths and weaknesses, and the right approach to this complex issue.
User side testing
While server-side testing and optimizations are crucial, end-user experience should always get priority. In that sense, it is crucial to understand the different user-side tests you can perform, the advantages and disadvantages of each, and how to strike a balance between the two to ensure that your website or service is thriving in China.
We begin by explaining the easiest test to understand: the manual one. As the name suggests, this test consists of real people testing your website using the most popular browsers in China on PCs and cell phones. In other words, these users will test your web application simulating your target audience.
The manual tests will check your website or web application for possible problems and potential optimizations. To that end, a variety of tests are carried out, including:
- Last-mile tests: In telecommunications, the term "last mile" refers to the last leg of the network, the one that actually connects to the homes of retail end-users. To get a clear idea of the importance of this test, think of the water pressure at the tap in your home vs. the one at the source (the water pump). From an end-user perspective, what really matters is the water pressure in the house. The same is true for your target audience. What determines their experience is the speed with which they navigate your website or application from their phones and PCs. GoClick China has a complete team of manual testers distributed throughout China running last-mile connectivity tests using both LTE and WiFi networks. The importance of this type of testing is that the local Internet Service Providers (ISPs) that operate in different regions of China offer different speeds, so the monitoring provided by last-mile tests allows you to get a clear picture of the actual speed the users experience in each region. This in turn allows you to formulate specific strategies for each region, if necessary.
- Ad-hoc tests: There are also times when you need to perform tests quickly that can't wait. Some scenarios include testing before the release of new features, confirming that a critical bug has been resolved, verifying that content hosted offshore has successfully bypassed the Great Firewall of China, and testing a specific part of your site for bugs. In all of these cases and many others, ad-hoc testing is the solution. This type of manual testing can be quickly scheduled to test what you need when you need it.
- Usability testing: This type of testing revolves around checking numerous elements of your website/application, such as menus, outgoing links, internal links, buttons, general navigation, and more. These tests provide valuable insights in terms of Quality Assurance (QA) and user experience (UX) derived from the interaction of real local users with the graphical interface of your website or web application.
- Compatibility tests: If one thing characterizes the Chinese market, it is the wide variety of browsers available. For this reason, cross-browser testing is essential for your business in China. To that end, compatibility testing consists of a group of users testing different mobile and desktop browsers to check the performance of your application or website.
- Competition benchmarking: One advantage of manual testing is that you can run the same tests on your competitors' websites and applications to make a detailed comparison of their strengths and weaknesses. This information is of great interest to GAP analysis.
Manual testing is thus of utmost importance as it offers an effective way to gather valuable first-hand information about user experience, compatibility of your website and applications in China, as well as to detect bugs in the user interface. However, to ensure the optimal performance of your application, these results should be complemented with those obtained from synthetic and RUM tests, which are discussed below.
For more information on this subject, we invite you to read our article, Web application monitoring - landscape in China.
Real User Monitoring (RUM)
When it comes to solving frontend performance issues, Real User Monitoring (RUM) is your best ally. You can think of RUM as a type of testing that collects metrics related to the response time and performance of your website using real devices and browsers. These metrics in turn can be used to determine the root cause of problems that impact the user experience when using your web pages and applications.
The name RUM (Real User Monitoring) should not be confused with the Manual Testing discussed in the previous section. RUM automatically captures user data from your website or application using special software inside the browser/application, while manual testing does not require any software modification. Both tests use real devices running the most popular browsers in China, and both tests aim, among other things, to help improve the user experience, but the results provided by each test serve different purposes. Moreover, since the RUM test collects data automatically, the volume of information it provides is much higher than manual testing.
If you want to know more about mobile testing, our articles: Why you need device testing for smartphones in China, and Why you need to optimize for native mobile web browsers in China may interest you.
Synthetic testing and monitoring (STM)
STM or synthetic monitoring consists of the use of scripts that emulate the behavior and interactions of users with your website or applications from different locations. These scripts send different performance metrics and detect errors to your dashboard so that you have a clear notion of the performance of your website.
Synthetic monitoring offers a number of benefits, such as the ability to run 24/7 from key locations for your business within China, compare your website’s performance with that of your competitors, help your team make optimizations, and provide detailed diagnostic data which is invaluable for early detection of failures. Basically, thanks to STM, your organization can ensure that your website is performing as expected and if not, be informed of it immediately to take corrective measures.
Before moving on to server-side tests, it is important to understand the differences and similarities among various user-side tests. For this purpose, we have prepared the following table with the most relevant aspects discussed so far.
|Data source||Trained team of people using real devices||Users’ browsers||Pre-recorded scripts|
|Captures real UX||Yes||Yes||No|
|Locations||Selected locations throughout China||Location of the user (whether in China or elsewhere)||Strategic locations throughout China|
|Duration of the tests||Limited timeframe||24/7, if required||24/7, if required|
|Data volume collected||Low||High||High|
|Requires software modification||No||Yes||No|
|UX and QA insights||Thorough||Few||None|
If you are a developer who wants to enter the Chinese market, then we recommend that you read: How to distribute your apps to Chinese mobile users.
User-side tests are valuable in many ways. However, for a consistent experience for your audience, ensure that your infrastructure keeps up with the demand you receive. Thus, server-side testing is irreplaceable.
System monitoring & code tests
You can think of these tests as an advanced alerting system that constantly monitors your website/application availability (uptime), performance, and functionality. Thanks to this monitoring, your organization can detect critical failures before they impact your business or your users. Moreover, since these tests are done directly on your software, you can fine-tune them for a granular view of the subsystems of most interest, such as databases, application servers, load balancers, etc. Additionally, these tests can include automatic checks on your CI/CD pipeline before any code changes are pushed to the production environment.
The main difference between these tests and those described in the previous section is that instead of monitoring the software, the system-resource monitoring focuses on the physical infrastructure (hardware). This is how these tests yield relevant metrics, such as memory availability, storage, computing power, real-time system load, and more. Given the current trend of use of hybrid clouds and multi-CDNs, this information is vital while making decisions such as resource scaling at peak hours. Another benefit of this type of monitoring is that it allows detecting bottlenecks (or preventing them), which has a direct impact on user experience.
Manual vs. automated/synthetic testing, which is better?
The truth is that both manual and synthetic tests are necessary to get an overall picture of the performance of your website or application. This is because each and every test has "blind spots" or limitations. In other words, manual testing and synthetic testing are not mutually exclusive; on the contrary, they complement each other. To better illustrate this idea, we have summarized the pros and cons of each test in the following table.
|Manual testing||- Collects experiences from real users using real devices|
- The data collected considers the last-mile factor
- Valuable UX and QA insights are obtained as well as detailed diagnostic data
|- Pre-planning required|
- Offers low test data volume
- Can be performed for a limited time
|Real user monitoring||- Captures user experience regardless of location|
- Helps find the root cause of problems by reviewing individual user cases
- Provides full control of your website monitoring
- Provides a large amount of test data
- Can run 24/7
|- Does not consider the last-mile factor|
- Offers few insights
- Requires software modification
- Offers little diagnostic information
- To find a problem, the user must first experience it
|Synthetic testing||- Can detect errors before the user experiences them|
- Monitors the performance and reliability of your website 24/7
- Provides a large volume of test data
- No software modification required
- Easy to benchmark the competition
- Provides detailed diagnostic information
|- The information does not come from real users but from scripts.|
- Does not consider the last-mile
- May present problems when running in certain browsers
- Does not offer user insights
- Cannot perform complex testing procedures, such as Flash, etc.
|System monitoring/code tests||- Quickly detects software faults|
- Allows granular monitoring of software components
- Allows code testing
|- Requires significant effort for both implementation and maintenance|
|System-resource monitoring||- Helps optimize resources and plan for the future|
- Allows to detect bottlenecks that may affect the UX
- Quickly detects infrastructure problems
|- Requires significant effort for both implementation and maintenance|
In this article we have explored each of the different tests that help determine the performance of your application or website. Likewise, we have presented how these tests complement each other to offer you a comprehensive monitoring solution for your website. That said, you might wonder what would be the best combination of manual testing and synthetic testing. That is a difficult question to answer simply as it has a lot to do with your use case. An e-commerce business does not have the same needs or fault tolerance as a video streaming service.
Fortunately, GoClick China provides a total solution that includes both manual testing and automated testing. Our team has extensive experience in the field of website performance testing, so we can help your business establish itself in China by providing an unmatched user experience. Furthermore, at GoClick China, we don't just provide you with the test data, but also help you understand it so that you can make the necessary adjustments and outperform your competition.
Click on GoClick China to find out how we can make your website get seen by your ideal customers and perform just as well as their local counterparts.