Every year web applications grow more sophisticated. New development frameworks arrive every week and advances in database management shift paradigms of storage and retrieval. Even the very infrastructure we run platforms on has been transformed into a scalable service. As a result, the web is not a simple place anymore.
These new tools have decreased time to market significantly and increased the productivity of software engineers. For example, what used to take hundreds of man-hours can now be accomplished in a one-day hackathon.
However, these new tools have also added complexity. Modern web applications can break down, perform slowly, or fail to meet their business objectives in an almost endless amount of ways. The tools needed to monitor, test, and validate web applications have grown more numerous and complicated. Software companies no longer have a single testing tool. Instead, they employ a stack of them.
You might also be interested in: Why you need device testing for smartphones in China
The internet is simply the world’s largest computer network. In technical terms the infrastructure used to connect phones, computers, IoT and other devices to the internet are the same in most countries.
You might also be interested in: Optimizing for native mobile browsers in China
China is an exception to the rule. The Chinese internet has very notable differences to the way web applications are delivered and consumed:
- The “Great Firewall of China” has the ability to monitor, censor and block software or information it deems counter to its national interests.
- ICP Licenses must be granted for serving any content within Mainland China.
- Networking connectivity is cumbersome and expensive, especially for International providers.
- Many “Global” internet leaders have no presence in China. Google, for instance does not operate its search service within Mainland China. It does not host YouTube, or its Android App store. Many of its development tools are blocked, or intermittently offline for Chinese users.
- China has many home-grown Mobile Phone brands and internet browsers. Both of which ignore international standards. This can break things such as library compatibility, font rendering and video playback.
Traditional monitoring tools fail to account for many of these factors. For example, RUM-based monitoring would fail to show networking or censorship issues. Synthetic-based monitoring would fail to show poor video quality. Server-based monitoring would not explain that a “Call to Action” renders incorrectly on Chinese handsets. It’s difficult to detect what you cannot see.
The solution to these monitoring blind spots is deceptively simple. Real people, on real phones and PC’s – testing your web application. Or as we call it, “Manual testing”. Manual testing provides insights into 3 valuable categories.
Is my application functional? Does it load quickly enough?
Are there bugs or compatibility issues?
What is the overall experience?
At GoClick China, we have a team of manual testers spread throughout the country. This is important to account for regional variations of internet delivery, infrastructure and peering with the Global internet.
Each manual test performed includes a video recording. It also comes with diagnostic files such a console logs and HTTP Archive (HAR) files for detailed analysis. Specific meta-data such as IP Address, Phone or PC Model, Operating system etc. is viewable from the GoClick China web portal.
Manually testing has unique benefits. To name a few:
- Easily share videos with stakeholders. Share successful tests or problems with commercial partners, suppliers, product manager and technical resources.
- Flexible test methods. Changing test plans and steps is as simple as writing instructions. We can also test with specific phones, browsers or operating systems.
- Gain qualitative data for complex applications i.e. streaming video, real-time financial products or interactive games.
- Perform tests on your competitors and compare results for product strategy and GAP Analysis.
- Compare different last mile connectivity such as LTE v.s. Wifi.
With the growing complexity of web applications and market needs, Technical teams are deploying multiple systems to ensure business goals are met. Here’s how manual testing fits in with other methods.
| Test Type | Description | Strength | Weakness | |
|---|---|---|---|---|
| user side | Manual Testing | Real users on real devices. |
- Capture real user experience - Flexible - UX and QA Insigths - Tailored to Market - No software modification - Easily benchmark competitors - Detailed Diagnostic Data |
-Infrequent. Low volume of test data |
| Real User Monitoring (RUM) | Browser based metrics, often via Javascript / software include. |
- High Volume of test data - High frequency - Real user data |
- Fails to detect major failures / last mile - Few insights - Requires software modification - Little diagnostic data |
|
| Synthetic Testing | Scripted tests run by user agents or servers in predictable locations. |
- No software modification needed - Easily benchmark competitors - Detailed Diagnostic Data - Quickly detect failures |
- Not real user data. - Does not factor in last-mile performance. |
|
| server side | System Monitoring / Code tests |
Software availability, load, execution time etc.. |
- Quickly detect failures - Granular monitoring of software subsystems |
- Requires significant development resources to build and maintain - Few business insights |
| System Resource Monitoring | Computing resource availability such as storage, memory, computing power and load. |
- Quickly detect failures or bottlenecks - Important for capacity planning and design | - Few business insights - Requires significant development resources to build and maintain |
As applications grow more complex, specific systems are needed to monitor and test for specific use cases. A single monitoring system is rarely enough in today’s complex internet ecosystems. To get a full view of web performance and functionality, multiple systems must be used.
China is a market that has unique use cases and challenges. To succeed in this market, different software, hardware and regulatory policies require attention and fine tuning.
Manual testing is an important component of a well-rounded monitoring approach, especially for those who wish to compete in Mainland China.
If you wish to learn more, do not hesitate to Contact us now; or sign-up for a free trial of GoClick China’s manual testing service. We have years of experience providing guidance and insights for brands wishing to grow in the Mainland Chinese market.