06 Assertions Reporting

Imagine you have hired a world-class security guard to watch over your house while you go on vacation. You give them keys, a uniform, and a checklist. You return two weeks later, and the guard smiles and says, “I walked around the house every day.”

You ask, “Did anyone try to break in?” The guard shrugs. “I walked around the house.” You ask, “Was the back door locked?” The guard smiles. “I walked around the house.”

This guard is useless. They performed the actions (walking), but they didn’t verify anything (checking locks, spotting intruders) and they didn’t report back to you effectively.

In software testing, clicking buttons and typing text is just “walking around the house.” Assertions are the moment the guard checks the lock. Reporting is the detailed logbook they hand you when you return. Without these two, your automation is just code taking a walk.

In this module, we are going to turn your scripts into Sherlock Holmes—observant, critical, and excellent at explaining exactly what happened at the scene of the crime.

6.1 Assertion Libraries: The Art of the Truth

If a test performs an action but doesn’t check the result, it is not a test; it is just a tour. Assertions are the checkpoints where your code stops and asks: “Is the reality of the application matching my expectation?”

6.1.1 The Two Philosophies: Hard vs. Soft Assertions

When a detective finds a clue that doesn’t fit, they have two choices. They can immediately stop the investigation and scream “MURDER!” (Hard Assertion), or they can take a note in their notebook, continue looking for more clues to build a full picture, and then present the findings (Soft Assertion).

• Hard Assertions (The Traffic Cop): Imagine a traffic cop. If you run a red light, they stop you immediately. You don’t get to drive to the next intersection to see if you run that one too. The journey ends right there.
  • Technical Concept: In most testing frameworks (like JUnit, TestNG, or Pytest), a standard assertion throws an exception immediately upon failure. The test execution halts instantly.
  • When to use: Use this for “Showstoppers.” If the login page fails to load, there is no point in checking if the username field is visible. The test cannot proceed physically.
  • Code Metaphor: Assert.assertEquals(actual, expected) is your red light.
• Soft Assertions (The Food Critic): Imagine a food critic at a restaurant. If the soup is cold, they don’t flip the table and storm out immediately. They note it down (“Soup: Cold”), then they taste the main course (“Steak: Overcooked”), and finally the dessert. At the end of the meal, they publish a review listing all the failures.
  • Technical Concept: Soft assertions (often provided by libraries like AssertJ or TestNG’s SoftAssert) catch the error, store it in a list of failures, and allow the code to move to the next line. At the very end of the test method, you must call assertAll() to “publish the review” and fail the test if any errors were found.
  • When to use: Use this for validating forms or complex screens. If you are checking a “User Profile” page, and the “First Name” is wrong, you still want to know if the “Last Name” and “Email” are correct. Don’t hide five bugs behind the shadow of the first one.

6.1.2 Matchers: From Simple Math to DNA Matching

Assertions are not just about A == B. Sometimes we need to check if a suspect looks like the perpetrator, or if they were in the crowd.

• Standard Matchers (The “Exact Twins” Check): This is the most basic level. Is the number of items in the cart exactly 5? Is the user’s name exactly “Angela”?
  • Tools: assertEquals, assertTrue, assertFalse.
  • The Trap: Beware of “flaky” exact matches. If you assert message == "Welcome User!" but the dev changes it to “Welcome, User!”, your test fails. This is brittle.
• Complex Matchers (The “Fuzzy Logic” Check): Real life is messy, and so is data. Sometimes you don’t need an exact match; you need a pattern match.
  • Contains: You don’t care if the error says “Error 505: Server Down” or “Critical Error: Server Down”. You just want to ensure it contains “Server Down”.
  • Collections: Imagine checking a shopping list. You don’t care about the order of the items (Milk, Eggs, Bread vs. Bread, Milk, Eggs). You just care that the list has those items. Complex matchers allow you to check “List contains these items in any order.”
  • Regex (Regular Expressions): This is the DNA test. You want to verify a generated Order ID. You don’t know what the ID will be (it changes every time), but you know the format: “ORD-1234”. You use Regex to assert: “Does this string start with ‘ORD-‘ followed by 4 digits?”

6.2 Evidence Collection: The Crime Scene Photographer

You are sleeping. Your automated tests run at 3:00 AM. One fails. You wake up at 9:00 AM and see a red “X” on your dashboard.

Without evidence, you have to guess what happened. Did the page not load? Did the button move? Was there a popup covering the button? You are now a detective arriving at a crime scene that has already been cleaned up.

We need to freeze the crime scene in time.

6.2.1 Taking Screenshots on Failure (The Polaroid Moment)

A picture is worth a thousand log lines. When a test fails, the browser is looking at the error. We need to snap a photo right before the browser closes.

• The Mechanism (Listeners/Hooks): We don’t write “take screenshot” inside every test method. That’s repetitive (DRY principle violation). Instead, we use a Test Listener or a Hook (like AfterMethod in TestNG or AfterStep in Cucumber).
• The Metaphor: Think of a Listener as a “Black Box” flight recorder. It sits silently in the background observing the flight (the test). It does nothing while things are going well. But the moment the plane crashes (Test Failure Exception), the Black Box triggers automatically and saves the state of the instruments.
• Implementation Strategy:
  1. Create a class that “listens” for the onTestFailure event.
  2. Inside that event, cast the driver to a TakesScreenshot interface.
  3. Save the file with a timestamp and the test name (e.g., LoginTest_Failure_20231027.png).
  4. Crucial Step: Embed this image directly into your report (more on that later). Do not just save it to a folder on a server no one can access!

6.2.2 Logs and Network Dumps (The Wiretap)

A screenshot shows you what the user saw (the UI). But it doesn’t show you what the application felt (the internal logic).

• Browser Logs: Sometimes the UI looks fine, but the button doesn’t work because of a silent JavaScript crash in the background. Capturing the Browser Console Logs is like reading the suspect’s diary. It reveals the inner turmoil hidden behind a calm face.
• Network Dumps (HAR files): This is the wiretap. You clicked “Submit,” and the UI spun around for 10 seconds and failed. Why? A screenshot just shows a spinner. A Network Dump shows that the request went to /api/login and the server replied with 500 Internal Server Error.
  • The Lesson: If you only look at the UI, you are testing the skin. If you look at logs and network traffic, you are testing the organs.

6.3 Reporting Frameworks: The Storyteller

You have run 500 tests. 495 passed. 5 failed. If you send your manager a text file with 5,000 lines of console output saying System.out.println("Test passed"), they will fire you. Or at least, they will ignore you.

Stakeholders (Managers, Developers, PMs) do not speak “Code.” They speak “Charts,” “Trends,” and “summaries.” Your report is the translation layer between your technical brilliance and their business decisions.

6.3.1 Configuring Rich Reports (The Dashboard)

We are moving away from plain text logs to HTML dashboards. Tools like Allure, Extent Reports, or the HTML Publisher in Jenkins/CI tools transform dry data into a visual story.

• The “At a Glance” Metaphor: Think of your car dashboard. It doesn’t show you the exact RPM of every piston or the temperature of the exhaust manifold in raw numbers. It gives you a Green/Red check engine light and a fuel gauge.
• What a good report needs:
  1. Pie Chart: A quick visual of Green (Pass) vs. Red (Fail) vs. Yellow (Skipped). This is for the Manager who has 5 seconds to decide if we release the software.
  2. Trend History: “Are we getting better or worse?” If yesterday we had 10 failures and today we have 20, the build is unstable. If today we have 5, we are improving. Context is king.
  3. The “drill-down”: The report should let you click a red bar to see the specific test class, click the class to see the method, and click the method to see the…

6.3.2 Reading the “Stack Trace” (The Autopsy)

The Stack Trace is the most intimidating and most useful part of a failure. It is the autopsy report. It tells you exactly where the code died.

• Anatomy of a Stack Trace: It looks like a wall of gibberish, but it’s actually a map, read from top to bottom.
  • Top Line (The Cause of Death): ElementNotFoundException: Unable to locate element with ID 'submit-btn'. This tells you what happened.
  • The Middle Lines (The Breadcrumbs): at com.mycompany.pages.LoginPage.clickLogin(LoginPage.java:45). This is the gold. It tells you the exact file (LoginPage.java) and the exact line number (45) where the failure occurred.
• How to teach this: Don’t just stare at the error. Click the blue link in your IDE (IntelliJ/Eclipse/VS Code) that corresponds to your code. Ignore the lines that say java.lang.reflect or org.testng.internal. That is the framework’s plumbing. You are looking for your pipes.

6.3.3 Tagging: The Filing Cabinet

As your suite grows to 1,000 tests, you will rarely run all of them at once locally. You need a way to filter.

• The Metaphor: Imagine a library. You don’t just pile books on the floor. You tag them: “Sci-Fi,” “History,” “Romance.”
• Regression: The full library. Everything we have ever written. Takes hours to run. We run this before a major release.
• Smoke Test: The “Is the building on fire?” check. We pick the critical 5-10 tests (Login, Checkout, Search). If these fail, there is no point in checking the font size on the “About Us” page. We run this on every single code commit.
• Sanity Test: A quick check of a specific new feature. “We just fixed the ‘Forgot Password’ bug. Run only the tests tagged @ForgotPassword.”
• Filtering in Reports: A good reporting framework allows you to view results by tag. A developer might say, “I don’t care about the Payment module, I only touched the Search module. Show me the @Search test results.”

Summary: The “Definition of Done”

By the end of this module, you won’t just be writing scripts that click things. You will be architecting a Quality Safety Net.

1. Your Assertions will catch bugs with precision, using Soft Assertions to gather full context and Hard Assertions to stop catastrophic failures.
2. Your Evidence will be automatic. No more “I can’t reproduce it on my machine.” You will have a screenshot and a log for every crime scene.
3. Your Reporting will be beautiful, categorized, and readable by humans, not just robots.

When you hand over your test report, you are handing over confidence. You are saying, “I have checked the locks, I have walked the perimeter, and here is the photo of the open window I found.”

Let’s get coding.