A Guide – Email Testing For Web Applications
This article provides some good software testing practices and guidelines that I use diligently at my work. It provides an insight into web applications using email systems, such as mail server, rendering aspects in mail clients. This article does not focus on testing any existing or preexisting email service providers.
In this article, I have also suggested a few questions that will guide you on the simple yet highly effective ways that you need to take to attain and maintain high-level testing when dealing with emails.
Testing is a continuous process and what works now may not work in the future due to the involving changes in the technology and our email needs.
Most testers run a few tests and rely on the test results and report to solve all the problems. Testers who test effectively to achieve best results in every email testing and open to make changes in their testing strategy make a difference in any organization.
Before commencing my testing activity, I do the profiling of different elements that are associated with emails which helps me get test ideas. You can use any freeware for mind-mapping, but I use XMind. It is a handy tool to create nodes and to use the built-in features, such as, emoticons, color codes. It normally takes 30 minutes to create the draft version. However, you can update it over time by adding new nodes or editing nodes based on the new test requirements or enhancements to the test plan.
After the mind map is completed, take a print of it and stick it to the wall near your workstation, or also set the mind-map as “Background / Wallpaper” on the desktop. You can share it with the team, if working in a team, to get more test ideas.
Often we ignore testing the email templates and just skim through an email to check the functional aspects of it.
Organizations or businesses must understand the importance of email campaign and email content they send to their customers.
As a tester, you can test email templates for its content, purpose. Here are few emails that we often receive:
- Email account activation
- Forgot password
- Change password
- Delete account email confirmation
- Newsletter (un)subscription confirmation
- Alerts or notification
- And many more email/notification based on the business type and application type
It is very important to select the audience for A/B testing for emails. A few of the questions that you have to ask are:
- What is the type of application?
- Who are the target audience?
- What kind of problems do our target audiences might face while using these emails?
Test data and data coverage
To create the test data, here are the two effective ways that I follow:
One of the major components to create the test data is to create email addresses on public email service provider(s). The challenge here is to create thousands of email addresses with details, such as first name, last name and other generic details manually. iMacros is an add-on you can use for recording and replaying the repetitious work. Using this add-on, you can record the first name, last name, date of birth and other details that are generic. In the script generated by the add-on, change the email address and CAPTCHA and run the script.
To create email addresses for invoking the send function of the SMTP server to check the performance and reliability, you can use Microsoft Excel. Create three columns- username, domain name and email address respectively.
- In the first column, enter usernames with an increment number in two rows of the column. For example, SanthoshTuppad001” and “SanthoshTuppad002” and then drag them to create as many usernames you want.
- In the second column, enter a domain name in the first row of the column. For example, “EmailServiceProvider.com” and then drag it to create add it to the usernames that you have created in the first column.
- In the third column, use the Excel formula to create the email address using the values of the first and second column. For example, SanthoshTuppad001@ EmailServiceProvider.com, SanthoshTuppad002@EmailServiceProvider.com and so on.
- Then use a database script to dump the data in the respective tables.
Data coverage points
- Different character set
- Mix of character set
- Combinatorial testing
In the projects that I have worked, I have seen great challenge in implementing an end-to-end solution for check automation for email testing. There are many companies, who provide frameworks for automating checks, such as:
- Rendering aspects across cross browser, multi-clients and public e-mail service providers
- Code analysis
- Link and image validation
- Spell check
- Character conversion
It is possible to develop your own framework for automation check. In my next article, I will share my ideas about “Check automation for e-mail testing”, which you can use to implement to create a framework.
Reporting and Analyzing
Reporting bugs is a critical phase of the email testing. There will be a chance for the bugs you have reported to be rejected by the developers. They also may be consider them as low-priority bugs. This demotivates you for reporting bugs. I suggest that you must report the bugs without hesitation, even if the bugs that you have raised are rejected, because, on the other hand, you will learn more about the product and technology.
As software testers we have to ask ourselves several questions, as ignored information would be critical for the success of the business. As James Bach says, “Testing is questioning a product in order to evaluate it”. Here are a few questions that you can ask yourself before you commence testing, which you can also use as test ideas.
- Can email addresses be case-sensitive?
- What is the maximum length e-mail text field that has to be set on the client-side, server-side and database?
- What happens when an SMTP server is down?
- What could be the consequences or threats of using HTML templates when not developed properly?
- Is there a size limit for cached content or delivery failure notifications or any kind of data on the SMTP server?
- What are the consequences when the content reaches the limit? (E-mail size)
- Does the email system send emails with attachments (Auto-generated)? If yes:
- What is the permissible file size of the attachment?
- Is it a zip file or raw file format?
- If it is a zip file, does the user need to download an external application to unzip it? Or the default uncompressing application of the operating system can be used?
- If you are using multiple server and you are using some other server name in the from e-mail address then it would be delivery failure as it is kind of impersonation of some other sender. This helps in finding the threats during design phase of your e-mail system or SMTP architecture for your application.
Errors and Exceptions
- What happens when an email is bounced for any unknown reasons?
- What are the exceptions that can be handled when an SMTP server is down?
- Is there a plan to maintain the queue of bounced e-mails and re-sending them?
- When the SMTP server is down and there are features that invoke the email system; for example, Forgot Password Tokenized URL, a message must be provided, “We are facing some technical challenges, we will be back soon and you can use this feature”.
Stakeholders / Product Owner
- Is that email important or can I afford to ignore it?
- Should the emails be sent to all or only selected people who have opted for it?
- Should I consider Single and double opt-in?
- Do I use DKIM (Domain Key Identified Mail) to avoid phishing attempts, or is there any kind of counter-measures I have?
Reliability / Backup
- Can I have multiple SMTP servers as back-up or alternative, where emails can be routed if bulk emails are sent via the server?
- Do I know how an e-mail system works?
- Do I know SMTP, POP3, IMAP and PST? Is it important for me to understand this to do a good and an effective testing?
- What are the different characters that email service providers allow and do I have consistency in our email address field sanitization? (Blacklist and white-list for characters).
- Do I understand “SPAM” and “SPAM FILTERING”?
- Do I measure CTR (Click-Through Rate) and Conversion Rate of e-mail campaigns?
- If there is no better CTR, how can I improve it?
- Do I understand “Opt-in Marketing”?
- Am I aware of Phishing Attempts via email?
- Am I aware of “Backscatter spam”?
- Have I tested on widely-used free email service providers and also on different domain names; such as top level and country level (Sampling)?
- The HTML formatting must be compliant with e-mail service provider anti-spam rules.
- Have we tested enough if e-mails land in inbox, bulk, junk or spam folder with widely used free e-mail service providers; such as Gmail, Yahoo, Hotmail, Rediffmail
- Is there a plan for getting back the customers who have abandoned the website? For example, sending emails or alerts about discounts, or offers to encourage revisiting the website.
- Are there any creative concepts for email templates design?
- What is best time send or trigger specific emails?
Example: Let us say, you do a business which is for people who are techies and work from 0900 hours to 1800 hours. It is possible that they get lots of e-mails. Do you want your e-mail to be positioned in the first when they open their inbox? Then, deciding on when to send e-mail campaign or specific e-mails is one of the ways.
- If there is not, check with the stakeholders or product owner
- Different countries can have different policies or compliances of e-mail spam. Do you know them?
POSSIBLE REASONS TO ABANDON eMAILS
- Not engaging
- Not pleasing
- Not appealing content
- Usage of colors
- The way the email is started
- The way email is addressed to the receipt
- The frequency in which emails are sent
- Sources of getting same information
- Preference to seek information and applications pushing the information
- Lack of poor targeting
- Hexawise (Free & commercial) – http://www.hexawise.com/
- Data Generate (Free) – http://www.generatedata.com/
- Litmus (Commercial) – http://litmus.com/email-testing
- iMacros – https://chrome.google.com/webstore/detail/imacros-for-chrome/cplklnmnlbnpmjogncfgfijoopmnlemp
REFERENCES / BOOKS