Are end-to-end tests better than unit tests in situations where they are applicable?

Let me first define what I mean by unit tests and end-to-end tests. Let's say you have a program with a bunch of Java classes: A calls B, which calls C, etc.

A unit test is a test for A for bullying B and a separate test for B for bullying C, etc.

An end-to-end test is a test for A that tests A and, transitively, B and C.

For simplicity, and to keep the focus on the topic under discussion, rather than distracting secondary details, let's say the system as a whole is stateless: you call the top level (A) with input and you get the result. This input has exactly one valid output.

To be clear, I am not including external systems here such as RPC to other servers, databases, external states such as the file system, any type of user interface ("assert that clicking the Delete button programmatically deletes the current document") etc. We're just talking about a collection of classes within a single process.

Now you can choose two approaches:

  • Write end-to-end tests that try to cover all possible inputs and states. Write unit tests only when needed, such as if a particular class is not sufficiently validated as a result of an end-to-end test, or if an end-to-end test fails and you find it useful to write a unit test to isolate the error.But in general, the goal is to have comprehensive end-to-end tests.

  • Write unit tests that will exhaustively test every class or component. Write down the end-to-end test as an afterthought, or perhaps not at all. Even if you write it, don't try to exhaustively test every possible input.

I prefer (1) because if end-to-end tests pass and are exhaustive, I know the system as a whole works for all the cases I've tested. If each class or component is working correctly, there could still be errors at the points of integration between them, where I read most of the errors (sorry, I don't have a link right now).

So which one is best for you - rigorous end-to-end tests or rigorous unit tests? What for? Please provide specific reasons so that I and other readers can evaluate the answers themselves.

If this question is better suited for programers.stackexchange.com please move it there (moderators).

+3


source to share


3 answers


I would consider future changes and complexity to decide which one to go:

  • If the behavior is likely to be fixed (says it is an API) while B and C are likely to be replaced / changed in the near future (says they are internal components using an external library that is next to the new date release), then I would go with end-to-end testing A.
  • If end-to-end testing A leads to a very comprehensive list of test cases, or if there are too many test cases that are too difficult to build, I would go with unit testing A, B, and C.


I often find myself using a mix of both end-to-end and unit tests. End-to-end tests cover the critical cases (which is about 60-70% of all participating classes) and unit tests cover the rest (exceptions, very rare / deep paths), or I add when I want to be more confident with my logic.

+2


source


While it is impossible to give a general answer to a question like this, generally, you should consider a test pyramid :

  • Several system tests
  • Some integration tests
  • Lots of numerical tests


The reason for this is as outlined by JB Rainsberger , but the bottom line is that for any sufficiently complex application, the combination of an explosion covering all possible actions prevents effective coverage with anything other than unit tests. You will need to write tens of thousands or hundreds of thousands of integration tests to find out if your system is working or not.

+4


source


There is no correct answer (which is why Ive voted to close, as mostly opinion based). The best approach will be mostly situational and to some extent how you decide to define the unit. From your description of the problem domain, you have class A, calls class B, calls class C. You treat all classes as ones. This is one approach, but not the only one . You can view the module as a whole. If class A is part of the public interface for a module and classes B + C arent, then you have no reason to necessarily have tests specifically targeting classes B + C, as the available functionality would be tested through other classes that form the public interface. ...

The main problem with your first approach is its scalability. I'll demonstrate with very simple code (it's in C #, but I'm sure you can translate).

public class C {
    public int Method3(int cParam) {
        if (cParam > 0) {
            return cParam + 1;
        }
        else {
            return cParam - 1;
        }
    }
}

public class B {
    C _cInstance = new C();
    public int Method2(int bParam, int cParam) {
        if (bParam > 0) {
            return bParam + 1 * _cInstance.Method3(cParam);
        }
        else {
            return bParam - 1 * _cInstance.Method3(cParam);
        }
    }
}

public class A {
    B _bInstance = new B();
    public int Method1(int aParam, int bParam, int cParam) {
        if (aParam > 0) {
            return aParam + 1 * _bInstance.Method2(bParam, cParam);
        }
        else {
            return aParam - 1 * _bInstance.Method2(bParam, cParam);
        }
    }
}

      

To test all branches with class based unit testing, at least you have 2 branches to test in each class, so you have 2 + 2 + 2 = 6 test cases. If you go with Approach 1, testing all classes exhaustively, through class A, then instead of adding branches from each operation, you need to multiply it, so instead you have 2 * 2 * 2 = 8. Not much differences in this case, but if you did an extra check like MAX / MIN int for parameters and 0, you would test 5 scenarios for each value, which would give you a close to 15 versus 125 scenarios. The more differences / values ​​you have to test at your high level, the faster your test scripts multiply. Now consider the situation if an operation in class C is slow (say1 second). With option 2 it affects 2 tests, with option 1 it affects ALL 8 tests, the tests start to get painful. What happens if some of the requirements and class D change? It also uses class B and then class C. With option 1, you need to reassign all the different values ​​again to use class B + C. Youre essentially repeating tests for the same. Testing with option 1 is really starting to hurt now, running tests is very slow and without even considering hitting other external systems. With option 2, you knock out B and only test functionality D.and then class C. With option 1, you need to reassign all the different values ​​again to use class B + C. Youre essentially repeating tests for the same. Testing with option 1 is really starting to hurt now, running tests is very slow and without even considering hitting other external systems. With option 2, you knock out B and only test functionality D.and then class C. With option 1, you need to reassign all the different values ​​again to use class B + C. Youre essentially repeating tests for the same. Testing with option 1 is really starting to hurt now, running tests is very slow and without even considering hitting other external systems. With option 2, you knock out B and only test functionality D.

In my experience, if you go down route 1, you will probably create too many unnecessary tests that will make your test suite slow and / or you will start skipping scripts.

I need to rewrite option 2 as

Write unit tests that fully test each class or component. Write appropriate end-to-end tests to validate data flows through the system as expected. Do not try to completely check all possible inputs.

As Ive said, it might be wise for your specific scenario to consider classes A, B and C as a single unit / component. But if this is not the case, then testing individual classes individually helps control the number of scenarios that need to be tested, and therefore the amount of effort required to test thoroughly. It also helps to reduce the problem area that the person writing the tests has to do in their head. Its much easier to look at the branches in the class under test than it is to try to remember all possible branches in all the code that the class you are testing calls.

The reason Ive modified option 2 is because unit testing as a major issue does not mean you automatically reduce end-to-end testing to second grade, but they have different assignments. If you've been writing to a database, for example, you might have unit tests to verify that you are not trying to write strings longer than a certain value, and then do an integration test to verify that you can write strings that take that long. They work together, not in isolation.

+2


source







All Articles