Testify Project

Easy to Understand and Easy to Use Java Testing Framework

Overview

Testify is a Java implementation of Semantic Testing specification. For users, Testify provides a common set of primatives that can be learned once and used everywhere to write test cases. For developers, Testify provides primatives and extension points to enable their libraries and frameworks to be testable.

Benefits

Testify makes it painless to write unit, integration and system tests. Write simple and isolated test cases to verify your code works as expected without worrying about managing test state.
Build reusable and composable test components that manage their own state.
We don’t make assumptions about your technology stack, so you can develop and add new features to Testify without rewriting existing code.

Features

Uniform Annotations for writing Unit, Integration and System Tests
Managed Test Case Configuration, Isolation, and Execution
JUnit 4 Testing Framework Support
JUnit 5 Testing Framework Support
Pluggable Mocking SPI (Mockito and EasyMock supported)
Pluggable Dependency Injection Framework SPI (Spring, HK2 and Guice supported)
Pluggable Application Framework SPI (SpringBoot, Jersey 2, gRPC, and Spring Web MVC supported)
Pluggable Local Resource SPI (HSQL, ElasticSearch, ZooKeeper, etc supported)
Pluggable Virtual Resource SPI (Docker supported)
Pluggable Remote Resource SPI
Pluggable Server SPI (Undertow Supported)
Pluggable Client SPI (JAX-RS client supported)
Pluggable Test Configuration and Wring Validation SPI
Pluggable Test Inspection SPI
Pluggable Test Reification SPI

Supported Frameworks

Spring DI Integration Testing
SpringBoot System Testing
HK2 DI Integration Testing
Jersey 2 System Testing
Google Guice DI Integration Testing
gRPC System Testing

Getting Started

Configuration Checklist

Latest release version is 1.0.5
Take a look at the change log
Install JDK version 8 and insure JAVA_HOME environmental variable is set
Install Maven version 3.1.1 or above
Install Git version 2.9.3
(Optional) Install Docker 1.11.2 - 17.05.0~ce
Insure formal parameter names of constructors and methods are added to compiler generated class files:

1
2
3
4
5
6
7
8
9
10
11
<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
        <compilerArguments>
            <!-- Enable runtime discovery of parameter names -->
            <parameters />
        </compilerArguments>
    </configuration>
</plugin>

Insure Testify Java Agent is used by the surefire or failsafe plugins:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
<dependencies>
    <dependency>
        <groupId>org.testifyproject</groupId>
        <artifactId>core</artifactId>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <!-- 
                Enables the referencing of dependencies as properties. For example:
                ${org.testifyproject:core:jar}
            -->
            <artifactId>maven-dependency-plugin</artifactId>
            <version>${plugin.dependency}</version>
            <executions>
                <execution>
                    <id>properties</id>
                    <goals>
                        <goal>properties</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <argLine>javaagent:${org.testifyproject:core:jar}</argLine>
            </configuration>
        </plugin>
        <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <configuration>
                <argLine>javaagent:${org.testifyproject:core:jar}</argLine>
            </configuration>
        </plugin>
    </plugins>
</build>

JUnit 4

I find the best way to learn is through experiential learning. A number of examples are presented below to help you get acclimated with Testify and how to write effective unit, integration and systems tests. Please note that the examples are simple and intended to present the reader with the basic concepts of Testify. For complete examples refer to the links at the end of each section.

Unit Testing

Create Unit Test Project from Archetype

The easiest way to get started with unit testing with Testify is to create a new project using Testify’s unit test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-unittest-archetype

Example Unit Test

Given a CreateGreeting class with Map and RandomUUIDSupplier as its collaborators:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class CreateGreeting {

    private final Map<UUID, GreetingModel> store;
    private final RandomUuidSupplier randomUuidSupplier;

    CreateGreeting(Map<UUID, GreetingModel> store, RandomUuidSupplier randomUuidSupplier) {
        this.store = store;
        this.randomUuidSupplier = randomUuidSupplier;
    }

    /**
     * Create the given greeting.
     *
     * @param model the greeting model
     */
    public void createGreeting(GreetingModel model) {
        UUID id = randomUuidSupplier.get();
        store.put(id, model);
    }
}

The unit test for the CreateGreeting class would look:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
@RunWith(UnitTest.class)
public class CreateGreetingTest {

    @Sut
    CreateGreeting sut;

    @Fake
    Map<UUID, GreetingModel> datastore;

    @Fake
    RandomUuidSupplier randomUuidSupplier;

    @Test
    public void givenMapStoreConstructorShouldNotDoWork() {
        //Act
        CreateGreeting result = new CreateGreeting(datastore, randomUuidSupplier);

        //Assert
        assertThat(result).isNotNull();
        verifyZeroInteractions(datastore, randomUuidSupplier);
    }

    @Test
    public void givenNullCreateGreetingShouldReturn() {
        //Arrange
        GreetingModel model = null;

        //Act
        //Note that since we are using a fake Map to store values an NPE will
        //not be thrown but some Map implementations do not allow null values
        sut.createGreeting(model);
    }

    @Test
    public void givenExistingGreetingUpdateGreetingShouldUpdate() {
        //Arrange
        GreetingModel model = mock(GreetingModel.class);
        UUID id = UUID.fromString("aa216415-1b8e-4ab9-8531-fcbd25d5966f");
        given(randomUuidSupplier.get()).willReturn(id);

        //Act
        sut.createGreeting(model);

        //Assert
        verify(randomUuidSupplier).get();
        verify(datastore).put(id, model);
    }

}

@RunWith? @Sut? @Fake? Okaaay!

The first thing you will notice in the above unit test class is that it is annotated with @RunWith(UnitTest.class). UnitTest is a custom Testify JUnit Runner implementation that configures, wires, verifies and executes the test class.

The next thing you will notice are two annotations, @Sut and @Fake. The @Sutannotation denotes the System Under Test (SUT) and @Fake denotes the desire to create mock instances of the CreateGreeting class’s Map<UUID, GreetingEntity> store and RandomUuidSupplier randomUuidSupplier collaborators. In the example above the name of the collaborator field in the test and system under test are not important as Testify performs type based matching and falls back to type and name based matching in case there is ambiguity as to which collaborator we wish to be fake and initialize.

If you’re wondering what is going on under the hood, well, @Sut and @Fake annotations provide hints to Testify and behind the scene Testify inspects the test class fields as well as SUT class fields and then:

determines which test field corresponds to which system under test field
creates a fake instance of Map and RandomUuidSupplier
creates an instance of CreateGreeting class and sets its Map<UUID, GreetingModel> store and RandomUuidSupplier randomUuidSupplier collaborators to the fake instance
initializes the test class sut, store and randomUuidSuppler fields.

One other key feature of Testify of note is the fact that new instances of CreateGreeting and its collaborators are created for each test case. This means your tests run in complete isolation and you do not have to worry about managing state between test runs. Everything is taken care of for you so you can focus on writing your test cases and production code not boilerplate code to manage test state.

For complete unit test examples please take a look at: JUnit Unit Test Examples.

Spring Integration Testing

Create Spring Integration Project from Archetype

To get started with Spring integration testing with Testify create a new project using Testify’s JUnit Spring integration test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-spring-integrationtest-archetype

Example Spring Integration Test

Given a GetGreeting service with an GreetingRepository collaborator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@Service
public class GetGreeting {

    private final GreetingRepository greetingRepository;

    @Autowired
    GetGreeting(GreetingRepository greetingRepository) {
        this.greetingRepository = greetingRepository;
    }

    /**
     * Get a greeting with the given id.
     *
     * @param id the greeting id
     * @return the optional containing the greeting, empty otherwise
     */
    public Optional<GreetingEntity> getGreeting(UUID id) {
        return greetingRepository.findById(id);
    }

}

and Spring Data GreetingRepository and GreetingEntity:

1
2
3
4
@Repository
public interface GreetingRepository extends PagingAndSortingRepository<GreetingEntity, UUID> {

}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
@Entity
@Table(name = "Greetings")
@ToString
@EqualsAndHashCode
public class GreetingEntity implements Serializable {

    private UUID id;
    private String phrase;

    public GreetingEntity() {
    }

    public GreetingEntity(String phrase) {
        this.phrase = phrase;
    }

    public GreetingEntity(UUID id, String phrase) {
        this.id = id;
        this.phrase = phrase;
    }

    @Id
    @GeneratedValue(generator = "greetingIdGenerator")
    @GenericGenerator(name = "greetingIdGenerator", strategy = "uuid2")
    @Column(name = "greeting_id", updatable = false, insertable = false)
    public UUID getId() {
        return id;
    }

    public void setId(UUID id) {
        this.id = id;
    }

    @Column
    public String getPhrase() {
        return phrase;
    }

    public void setPhrase(String phrase) {
        this.phrase = phrase;
    }

}

and a Spring Java Config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
@ComponentScan
@Configuration
@EnableJpaRepositories
@EnableTransactionManagement
public class GreetingConfig {

    /**
     * An in-memory H2 database data source.
     *
     * @return the data source
     */
    @Bean
    DataSource productionDataSource() {
        PGSimpleDataSource dataSource = new PGSimpleDataSource();
        dataSource.setServerName("production.acme.com");
        dataSource.setPortNumber(5432);
        dataSource.setDatabaseName("postgres");
        dataSource.setUser("postgres");
        dataSource.setPassword("mysecretpassword");

        return dataSource;
    }

    @Bean
    LocalContainerEntityManagerFactoryBean entityManagerFactory(DataSource dataSource) {
        LocalContainerEntityManagerFactoryBean bean =
                new LocalContainerEntityManagerFactoryBean();
        bean.setDataSource(dataSource);
        bean.setPersistenceUnitName("example.greetings");

        return bean;
    }

    /**
     * Provides JPA based Spring transaction manager.
     *
     * @param entityManagerFactory the entity manager factory
     * @return jpa transaction manager
     */
    @Bean
    PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {
        JpaTransactionManager transactionManager =
                new JpaTransactionManager(entityManagerFactory);

        return transactionManager;
    }

}

The Spring integration test for the GetGreeting service would look:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
@Module(GreetingConfig.class)
@LocalResource(InMemoryHSQLResource.class)
@RunWith(IntegrationTest.class)
public class GetGreetingIT {

    @Sut
    GetGreeting sut;

    @Real
    GreetingRepository greetingRepository;

    @Test(expected = InvalidDataAccessApiUsageException.class)
    public void givenNullGetGreetingShouldThrowException() {
        //Arrange
        UUID id = null;

        //Act
        sut.getGreeting(id);
    }

    @Test
    public void givenNoneExistentKeyGetGreetingShouldReturnAnEmptyOptional() {
        //Arrange
        UUID id = UUID.fromString("aa216415-1b8e-4ab9-8531-fcbd25d5966f");

        //Act
        Optional<GreetingEntity> result = sut.getGreeting(id);

        //Assert
        assertThat(result).isEmpty();
    }

    @Test
    public void givenExistentKeyGetGreetingShouldReturnGreetingEntity() {
        //Arrange
        UUID id = UUID.fromString("0d216415-1b8e-4ab9-8531-fcbd25d5966f");

        //Act
        Optional<GreetingEntity> result = sut.getGreeting(id);

        //Assert
        assertThat(result).isPresent();
    }

}

IntegrationTest? @Module? @LocalResource? @Real? Whaaah?!

IntegrationTest is similar to UnitTest test runner in that it configures, wires, verifies and executes the test class.

Since this is an integration test we want to verify proper wiring and integration between services and their collaborators. Usually this involves loading a module that defines services we want to test and ideally working with real services.

In above GetGreetingTest test class the GetGreeting service and its GreetingRepository collaborator are discovered by the Spring Java Config class GreetingConfig using @ComponentScan annotation. We let Testify know this by annotating the test class with @Module(GreetingConfig.class). Behind the scenes Testify creates a new Spring ApplicationContext, register the GreetingConfigconfiguration class and insures that only the service we are testing and its collaborators are initialized.

Since our GetGreeting service has GreetingRepository, a Spring Data repository, as a collaborator we need a SQL database to test the service. This is where @LocalResource annotation can help. We simply annotate the test class with @LocalResource(InMemoryHSQLResource.class) and Testify takes care of creating an in-memory HSQL database and making it available for testing. Note that all InMemoryHSQLResource does is provide an in-memory java.sql.DataSource and java.sql.Connection which are used to replace all references to DataSource and Connection in the Spring application context.

We have already seen @Sut annotation in action in the unit test example. In the context of a Spring integration test it serves a similar purpose, to let Testify know the field represents the system under test. In this example it happens to be the GetGreeting service.

As mentioned earlier we typically want to work with real instance of collaborators when writing integration tests and so we annotate the greetingRepository field with @Real to let the framework know that we want the real instance of the GreetingRepository collaborator managed by Spring. Of course, there are times when you do not want to use real instance of collaborators (i.e. credit card processing service), and in those instances you can annotate the collaborator with @Fake to work with a fake instance (a mock instance) of the collaborator.

What if you want to use the real instance of GetGreeting's collaborators but want to mock or verify certain methods of the collaborator? Well, you can do that too. You just need to annotate the greetingRepository field with @Virtual annotation to work with a delegated mock instance of the GreetingRepository service that will allow you to mock certain methods and delegate others to the real instance of the collaborator in the Spring application context.

If you are curious about what is going on behind the scenes, Testify:

inspects the test and system under test fields
creates a Spring application context and loads the GreetingConfig module
configures and starts test resources
retrieves an instance of the GetGreeting services from the application context
initializes the test class’s sut and greetingRepository fields

As with unit tests you do not have to worry about managing test state. Every Spring integration test case runs in complete isolation and Testify takes care of managing the Spring application context and all scaffolding.

For complete Spring integration test examples please take a look at: Spring JUnit Integration Test Examples

Spring Boot System Testing

Please note that this example requires that you have Docker installed and Docker Remote API enabled. See the Virtual Resources section for details on how to install and configure Docker.

Create Spring Boot Project from Archetype

To get started with Spring Boot system testing create a new project using Testify’s Spring Boot system test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-springboot-systemtest-archetype

Example Spring Boot System Test

Given the following Spring Boot Application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
@SpringBootApplication
public class GreetingApplication {

    public static void main(String[] args) throws Exception {
        GreetingApplication application = new GreetingApplication();

        application.run(args);
    }

    public void run(String[] args) {
        SpringApplication.run(GreetingApplication.class, args);
    }

    @Bean
    DataSource productionDataSource() {
        PGSimpleDataSource dataSource = new PGSimpleDataSource();
        dataSource.setServerName("production.acme.com");
        dataSource.setPortNumber(5432);
        dataSource.setDatabaseName("postgres");
        dataSource.setUser("postgres");
        dataSource.setPassword("mysecretpassword");

        return dataSource;
    }

    @Bean
    LocalContainerEntityManagerFactoryBean entityManagerFactory(
            EntityManagerFactoryBuilder builder, DataSource dataSource) {
        Map<String, Object> properties = new HashMap<>();
        properties.put(DATASOURCE, dataSource);

        return builder.dataSource(dataSource)
                .persistenceUnit("example.greeter")
                .properties(properties)
                .build();
    }

    @Bean
    ModelMapper modelMapper() {
        ModelMapper mapper = new ModelMapper();

        Configuration configuration = mapper.getConfiguration();
        configuration.setMatchingStrategy(MatchingStrategies.STRICT);
        configuration.setFieldAccessLevel(Configuration.AccessLevel.PUBLIC);
        configuration.setMethodAccessLevel(Configuration.AccessLevel.PUBLIC);
        configuration.setAmbiguityIgnored(false);
        configuration.setDestinationNamingConvention(NamingConventions.JAVABEANS_MUTATOR);
        configuration.setSourceNamingConvention(NamingConventions.JAVABEANS_ACCESSOR);

        return mapper;
    }
}

and Spring Data GreetingRepository and GreetingEntity:

1
2
3
4
@Repository
public interface GreetingRepository extends PagingAndSortingRepository<GreetingEntity, UUID> {

}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
@Entity
@Table(name = "Greetings")
@ToString
@EqualsAndHashCode
public class GreetingEntity implements Serializable {

    private UUID id;
    private String phrase;

    public GreetingEntity() {
    }

    public GreetingEntity(String phrase) {
        this.phrase = phrase;
    }

    @Id
    @GeneratedValue(generator = "greetingIdGenerator")
    @GenericGenerator(name = "greetingIdGenerator", strategy = "uuid2")
    @Column(name = "greeting_id", updatable = false, insertable = false)
    public UUID getId() {
        return id;
    }

    @JsonProperty
    public void setId(UUID id) {
        this.id = id;
    }

    @SafeHtml
    @NotNull
    @Column
    public String getPhrase() {
        return phrase;
    }

    public void setPhrase(String phrase) {
        this.phrase = phrase;
    }

}

and a Spring REST controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
@RestController
public class ListGreetingsResource {

    private final GreetingRepository greetingRepository;

    @Autowired
    ListGreetingsResource(GreetingRepository greetingRepository) {
        this.greetingRepository = greetingRepository;
    }

    @RequestMapping(
            path = "/greetings/list",
            method = RequestMethod.GET,
            produces = MediaType.APPLICATION_JSON_VALUE)
    @ResponseStatus(HttpStatus.OK)
    public Iterable<GreetingEntity> listGreetings() {
        return greetingRepository.findAll();
    }
}

The Spring Boot system test for the ListGreetingsResource above would look:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@Application(GreetingApplication.class)
@Module(TestModule.class)
@VirtualResource(value = "postgres", version = "9.4")
@RunWith(SystemTest.class)
public class ListGreetingsResourceST {

    @Sut
    ClientInstance<WebTarget, Client> sut;

    @ConfigHandler
    void configureClient(ClientBuilder clientBuilder) {
        clientBuilder.register(JacksonFeature.class);
    }

    @Test
    public void callToListGreetingsShouldReturnGreetings() {
        //Act
        Response response = sut.getClient().getValue()
                .path("greetings")
                .path("list")
                .request()
                .get();

        //Assert
        assertThat(response.getStatus()).isEqualTo(OK.getStatusCode());
        GenericType<List<GreetingEntity>> genericType =
                new GenericType<List<GreetingEntity>>() {
        };
        List<GreetingEntity> result = response.readEntity(genericType);
        assertThat(result).hasSize(1);
    }

}

SystemTest? @Application? @VirtualResource? ClientInstance? @ConfigHandler? Oh my!

Once again you will note that we are running the test with SystemTest test runner. It is similar to UnitTest and IntegrationTest that we saw earlier in that it also configures, wires, verifies and executes the test class.

Since this is a system test we want to verify our application works from the client perspective. This means we load and start the Spring Boot application and start communicating with the application via HTTP. In the above code we are using Jersey Client implementation of JAX-RS to call /greetings/list endpoint. In ListGreetingsResourceST example above we load the the application by annotating the test class with @Application(GreetingApplication.class). Behind the scenes Testify starts the Spring Boot application and then creates a client instance that is aware of the base URI and port of the servlet container the application is deployed to. Finally, it injects an instance of ClientInstance into the test class so that it can be used to call the application’s endpoints from the client’s perspective.

Our ListGreetingsResource has GreetingRepository, a Spring data repository, as a collaborator. This means we need a SQL database, preferably our production database, to test the ListGreetingsResource. This is where @VirtualResource annotation and virtual resources shines. We simply annotate the test class with @VirtualResource(value = "postgres", version = "9.4") and Testify takes care of pulling and starting postgres resource (in this instance from Docker registry) and making it available for testing. To start testing with virtual resources you will need to:

install and configure Docker
create the TestModule and annotate your test class with @Module(TestModule.class):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
@Configuration
public class TestModule {

    /**
     * Create a datasource that takes precedence (@Primary) over the production datasource that
     * points to the postgres in the container resource.
     *
     * @param inetAddress the container address.
     * @return the test data source
     */
    @Primary
    @Bean
    DataSource testDataSource(
            @Qualifier("resource:/postgres:9.4/resource") InetAddress inetAddress) {
        PGSimpleDataSource dataSource = new PGSimpleDataSource();
        dataSource.setServerName(inetAddress.getHostAddress());
        dataSource.setPortNumber(5432);

        //Default postgres image database name, user and postword
        dataSource.setDatabaseName("postgres");
        dataSource.setUser("postgres");
        dataSource.setPassword("mysecretpassword");

        return dataSource;
    }

    /**
     * Create and configure a test entity manager bean factory.
     *
     * @param builder the entity manager builder
     * @param dataSource the test data source
     * @param applicationContext the application context
     * @return an entity manager bean factory
     */
    @Primary
    @Bean
    LocalContainerEntityManagerFactoryBean testEntityManagerFactory(
            EntityManagerFactoryBuilder builder,
            DataSource dataSource,
            ApplicationContext applicationContext) {
        Map<String, Object> properties = new HashMap<>();
        properties.put(DATASOURCE, dataSource);
        properties.put("hibernate.ejb.entitymanager_factory_name", applicationContext.getId());

        return builder.dataSource(dataSource)
                .persistenceUnit("test.example.greeter")
                .properties(properties)
                .build();
    }

}

Note that the above TestModule simply creates a new DataSource based on the virtual resource our test requires via @VirtualResource annotation. In our ListGreetingsResourceST test class we requested a postgres database image and so Testify creates a InetAddress whose name is derived from the virtual resource name and makes it available for injection. We can further configure the underlying client used by the ClientInstance by adding a method annotated with @ConfigHandler to the test class which takes client configuration object as a parameter. In this example we happen to be using the Jersey 2 JAX-RS client implementation and so we pass in a ClientBuilder instance. If you are curious about what is going on behind the scenes, Testify:

creates a proxy sub-class of GreetingApplication
intercepts and stores the application’s Spring application context so that all of its beans are eligible for injection
intercepts and alters the application’s port to point to a random port
intercepts and stores the base URL of the server
configures and starts the virtual resource and adds InetAddress with the virtual resource’s name
creates a ClientInstace instance that points to the server and add it to the application’s Spring application context
initializes the test class fields annotated with @Sut

As with the other testing levels you do not have to worry about managing test state. Testify will configure, start and inject a client instance before each test and manage the clean up process.

One other feature of note is that Testify supports “In-Container” system testing for Spring. This means you can inject any Spring managed beans in the deployed application into your test class and execute systems much like integration tests. Please note that the use of this feature is highly discouraged due to the fact that you are not performing system testing from the client’s perspective.

For complete Spring Boot system test examples please take a look at Spring Boot JUnit System Test Examples repository.

Spring MVC system test examples can be found in Spring MVC JUnit System Test Examples repository.

HK2 Integration Testing

Create HK2 Project from Archetype

Testify also supports integration testing of HK2 modules and services. To get started create a new project using Testify’s HK2 integration test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-hk2-integrationtest-archetype

Example HK2 Integration Tests

From integration testing perspective, testing HK2 modules is similar to integration testing Spring modules. For examples please take a look at HK2 JUnit Integration Test Examples repository.

Jersey 2 System Testing

Create Jersey 2 Project from Archetype

Jersey 2 applications system testing is also supported. To get started create a new project using Testify’s Jersey 2 system test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-jersey-systemtest-archetype

Example Jersey 2 System Tests

Testing Jersey 2 application is eerily similar to system testing Spring Boot applications. For examples please take a look at Jersey 2 JUnit System Test Examples repository.

Google Guice Integration Testing

Create Google Guice Project from Archetype

Testify also supports integration testing of Google Guice modules. To get started create a new project using Testify’s Guice integration test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-guice-integrationtest-archetype

Example Google Guice Integration Tests

From integration testing perspective, testing Guice modules is similar to integration testing Spring and HK2 modules. For examples please take a look at Guice JUnit Integration Test Examples repository.

gRPC System Testing

Create gRPC Project from Archetype

gRPC applications system testing is also supported. To get started create a new project using Testify’s gRPC system test archetype:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-grpc-systemtest-archetype

Example gRPC System Tests

Testing gRPC application is eerily similar to system testing Spring Boot applications. For examples please take a look at gRPC JUnit System Test Examples repository.

Local Resources

A local resource is an asset that is managed locally and similar to a deployment environment resource which be drawn on to effectively test assumptions. The easiest way to get started with local resources is to create a local resource provider implementation. In this example we will create an in-memory HSQLDB local resource. Using Testify’s resource provider archetype create a new project:

1
2
3
mvn archetype:generate \
 -DarchetypeGroupId=org.testifyproject.archetypes \
 -DarchetypeArtifactId=junit-resourceprovider-archetype

Example HSQL LocalResourceProvider

To create a hsql resource provider that can be used in your test cases simply implement the LocalResourceProvider SPI contract:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
public class InMemoryHSQLResource implements
        LocalResourceProvider<JDBCDataSource, DataSource, Connection> {

    private JDBCDataSource server;
    private Connection client;

    @Override
    public JDBCDataSource configure(TestContext testContext,
            LocalResource localResource,
            PropertiesReader configReader) {
        JDBCDataSource dataSource = new JDBCDataSource();
        String url = format("jdbc:hsqldb:mem:%s?default_schema=public", testContext.getName());
        dataSource.setUrl(url);
        dataSource.setUser("sa");
        dataSource.setPassword("");

        return dataSource;
    }

    @Override
    public LocalResourceInstance<DataSource, Connection> start(TestContext testContext,
            LocalResource localResource,
            JDBCDataSource dataSource)
            throws Exception {
        server = dataSource;
        client = dataSource.getConnection();

        return LocalResourceInstanceBuilder.builder()
                .resource(server, DataSource.class)
                .client(client, Connection.class)
                .build("hsql", localResource);
    }

    @Override
    public void stop(TestContext testContext,
            LocalResource localResource,
            LocalResourceInstance<DataSource, Connection> instance)
            throws Exception {
        server.getConnection()
                .createStatement()
                .executeQuery("SHUTDOWN");
        client.close();
    }

}

LocalResourceProvider? Hmmm…

In the above implementation of LocalResourceProvider contract we are creating a reusable HSQL in-memory database resource provider. Notice that our implementation:

Specifies 3 type parameters of LocalResourceProvider contract, JDBCDataSource (resource configuration), DataSource (the resource), and Connection (resource client)
Does not declare a constructor
Implements three methods defined by the contract (configure, start, and stop)
Has state (server and client instances)

Our example is pretty simple because our configuration object and resource object are the same but this may not be the case for a more complex implementations. Regardless of the complexity of an implementation the ultimate goal is to provide a managed, reusable, and disposable resource for our integration and system tests and to enable us to replace production code that relies on the resource (DataSource) and client (Connection) with the ones provided by our LocalResourceProvider implementation.

As you might guess the configure method is responsible for creating a pre-configured configuration object that can be refined further via @ConfigHander prior to starting the resource and execution of our test case. The start method is responsible for starting the resource and returning a LocalResourceInstance that encapsulates a resource component and a client component which is configured to communicate with the resource. In this example our resource instance happens to comprises of an in-memory HSQL JDBC DataSource and Connection. Finally, the stop method is responsible for gracefully stopping the resource and closing the client.

Now that we have a resource provider implementation we can start using it to provide a database resource for our integration and system tests by annotating our test class with @LocalResource(InMemoryHSQLResource.class):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
@Module(GreetingConfig.class)
@LocalResource(InMemoryHSQLResource.class)
@RunWith(SpringIntegrationTest.class)
public class GetGreetingIT {

    @Sut
    GetGreeting cut;

    @Real
    GreetingRepository greetingRepository;

    @Real
    ResourceInstance<DataSource, Connection> resourceInstance;

    @Real
    DataSource dataSource;

    @Real
    Connection connection;

    @ConfigHandler
    void configure(JDBCDataSource dataSource) {
        //XXX: You can refine the configuration object of the
        //InMemoryHSQLResource implementation
    }

    @Test
    public void givenExistentKeyGetGreetingShouldReturnGreetingEntity() {
        //Arrange
        UUID id = UUID.fromString("0d216415-1b8e-4ab9-8531-fcbd25d5966f");

        //Act
        Optional<GreetingEntity> result = cut.getGreeting(id);

        //Assert
        assertThat(result).isPresent();
    }

}

Notice that we are able to inject the resource instance, data source and connection created in our InMemoryHSQLResource resource provider implementation which can be useful if you wish to directly interact with the resource and the client. Please note that the above example is kitchen-sink example and may not reflect a typical use-case.

For complete LocalResourceProvider implementations and examples take a look at: Example JUnit Resource Provider Local Resource Implementations

Virtual Resources

A virtual resource is disposable asset similar to a deployment environment resource that can be drawn on to effectively test assumptions.

Docker Virtual Resource

Testify provides support for virtual resources based on Docker Containers. Before you can use virtual resources in your integration and system tests you will need to install and configure Docker on your system.

Install Docker

To install Docker please follow the Docker instalation instructions for your OS platform.

Configuring Docker

Testify uses Docker Remote API to pull images and manage Docker containers. By default Docker Remote API is not enabled. To enable Docker Remote API follow the bellow instructions for your OS platform.

Ubuntu 14.04 / Mint 17

After you install add yourself to the docker group and reboot your system:

1
2
3
4
sudo -s -- <<EOC
usermod -a -G docker $USER
reboot
EOC

Backup any existing docker daemon.json configuration file and create a new one that enables local and remote docker API hosts:

1
2
3
4
5
6
7
8
9
10
11
12
13
sudo -s -- <<EOC
mkdir -p /etc/docker
mv /etc/docker/daemon.json "/etc/docker/daemon.json.$(date -d "today" +"%Y%m%d%H%M")" \
 2>/dev/null
tee -a /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "hosts": [
    "unix:///var/run/docker.sock",
    "tcp://127.0.0.1:2375"
  ]
}
EOF
EOC

Since we are using a daemon.json file to configure the docker daemon we need to insure that daemon is not configured by our system docker service file:

1
sudo sed -i 's/ExecStart=.*/ExecStart=\/usr\/bin\/dockerd/g' /lib/systemd/system/docker.service

Restart the docker service:

1
sudo service docker restart

Test that the remote api is working:

1
docker -H tcp://127.0.0.1:2375 ps

Ubuntut 16.04 / Mint 18 / Debian 8

After you install add yourself to the docker group and reboot your system:

1
2
3
4
sudo -s -- <<EOC
usermod -a -G docker $USER
reboot
EOC

Backup any existing docker daemon.json configuration file and create a new one that enables local and remote docker API hosts:

1
2
3
4
5
6
7
8
9
10
11
12
13
sudo -s -- <<EOC
mkdir -p /etc/docker
mv /etc/docker/daemon.json "/etc/docker/daemon.json.$(date -d "today" +"%Y%m%d%H%M")" \
 2>/dev/null
tee -a /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "hosts": [
    "fd://",
    "tcp://127.0.0.1:2375"
  ]
}
EOF
EOC

Since we are using a daemon.json file to configure the docker daemon we need to insure that daemon is not configured by the system docker service file:

1
sudo sed -i 's/ExecStart=.*/ExecStart=\/usr\/bin\/dockerd/g' /lib/systemd/system/docker.service

Restart the docker service:

1
2
3
4
sudo -s -- <<EOC
systemctl daemon-reload
systemctl restart docker
EOC

Test that the remote api is working:

1
docker -H tcp://127.0.0.1:2375 ps

Fedora 24 / CentOS 7 / RHEL 7 / Oracle Linux 7

After you install add yourself to the docker group and reboot your system:

1
2
3
4
sudo -s -- <<EOC
usermod -a -G docker $USER
reboot
EOC

Backup any existing docker daemon.json configuration file and create a new one that enables local and remote docker API hosts:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
sudo -s -- <<EOC
mkdir -p /etc/docker
mv /etc/docker/daemon.json "/etc/docker/daemon.json.$(date -d "today" +"%Y%m%d%H%M")" \
 2>/dev/null
tee -a /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "storage-driver": "devicemapper",
  "hosts": [
    "unix:///var/run/docker.sock",
    "tcp://127.0.0.1:2375"
  ]
}
EOF
EOC

Since we are using a daemon.json file to configure the docker daemon we need to insure that daemon is not configured by our system docker service file:

1
sudo sed -i 's/ExecStart=.*/ExecStart=\/usr\/bin\/dockerd/g' /lib/systemd/system/docker.service

Restart the docker service:

1
2
3
4
sudo -s -- <<EOC
systemctl daemon-reload
systemctl restart docker
EOC

Test that the remote api is working:

1
docker -H tcp://127.0.0.1:2375 ps

Windows

Follow Docker installation and configuration instructions on Microsoft Windows Containers site.
Insure c:\ProgramData\docker\config\daemon.json file defines hosts value:

1
2
3
{
    "hosts": ["tcp://127.0.0.1:2375"]
}

Restart Docker:

1
Restart-Service docker

Mac OS X

TODO

Docker Remote API and Testify

By default Testify communicates with Docker through Docker Remote API on non-secure http://127.0.0.1:2375 URL endpoint. If you wish to use an endpoint with a different IP address and port or are using a Docker-Machine, or want to use secure-communication then you will need to explictly configure the Docker Client used by Testify using @ConfigHandler (not recommended):

1
2
3
4
5
@Config
public void configure(DefaultDockerClient.Builder builder) {
    builder.uri("http://192.168.99.100:2375");
    //add additional configuration such as registry configuration here
}

Glossary

Annotations

@Sut

An annotation used on single test class field to denote the field type as the system under test (SUT).

@Fake

An annotation that can be placed on unit, integration and system test class field to denote the field as a fake collaborator. A fake collaborator is a mock instance of the collaborator that allows us to mock functionality and verify interaction between the system under test and the fake collaborator in isolation. Note that if the value of the test class fake field is already initialized with:

a mock instance of the collaborator then this mock instance will be used and injected into the system under test.
the real instance of the collaborator then a mock instance that delegates to the real instance will be created and injected into the system under test.

@Real

An annotation that can be placed on integration and system test class fields to denote the field as a real collaborator of the system under test.

@Virtual

An annotation that can be placed on unit, integration and system test class field to denote the field as a virtual collaborator. A virtual collaborator is a mock instances that delegate to a real instance of the collaborator of the system under test and are useful if you wish to mock certain functionality and delegate others to the real collaborator instance.

@CollaboratorProvider

An annotation used to specifying a provider of the system under test’s collaborators. This annotation can be placed on a test class or method. This is useful for configuring a system under test whose collaborator(s) can not be faked or virtualized (i.e. a java.net.URL collaborator which is a final class). Note that if this annotation is:

placed on a test class method then this method will be called to provide the system under test's collaborators.
placed on the test class and `CollaboratorProvider#value()` is specified then a method within `CollaboratorProvider#value()` class will be called to provide collaborators for the system under test.

@ConfigHandler

An annotation that can be placed on a test class or test class method to configure various functions before each integration and system test run (i.e. HK2 ServiceLocator, Spring Application Context, required resources, etc). Note that if the annotation is placed on:

a test class method then the method will be called to perform pre-test run configuration.
the test class and `ConfigHandler#value()` is specified then the the appropriate configuration method in the `ConfigHandler#value()` class will be called to perform pre-test run configuration.

@Module

An annotation that can be placed on integration and system tests to load a module that contains services before each test run (i.e. Spring’s Java Config, HK2’s AbstractBinder, or Guice’s AbstractModule).

@Scan

An annotation that can be placed on integration and system tests to load a resources that contains services before each test run (i.e. Spring service fully qualified package name, HK2 service locator descriptor classpath).

@Application

An annotation that can be placed on system tests to specify an application (i.e. Jersey 2, Spring Boot, Spring MVC, etc) that should be loaded, configured, started, and stopped before and after each test run.

@LocalResource

An annotation that can be placed on integration and system tests to specify local resources that must be loaded, configured, started, stopped before and after each test case (i.e. an in-memory database). Note that a typical local resource consists of a resource component and resource client component (optional). For example, if a test class requires an in-memory database then the database javax.sql.DataSource can be thought of as the resource component and the java.sql.Connection to the DataSource as the resource client component.

@VirtualResource

An annotation that can be placed on integration and system tests to specify virtual resources that should be loaded, configured, started, stopped before and after each test run. This is useful when performing integration and system tests using real production environment (i.e. a real PostgresSQL database or Cassandra cluster).

@RemoteResource

An annotation that can be placed on integration and system tests to specify remote resources that should be loaded, configured, started, stopped before and after each test run. This is useful when performing integration and system tests using real cloud and third party production or sandbox services (i.e. a AWS SQS, GitHub API, Stripe API, etc).

@Fixture

An annotation that can be placed on test field to denote them as test fixture. When placed on a test class field then the value of the field can be initialized or destroyed using Fixture#init() and Fixture#destroy() attribute before and after each test run.

@Bundle

A meta-annotation that identifies an annotation as test group. A test group annotation provide the ability to define, group, and use one or more Testify annotations in a reusable manner in your test classes and avoid annotation bloat.

@Discoverable

An annotation that can be placed on service implementation classes to add entries for the service to the META-INF/services directory and allow the service to be discoverable through the JDK’s java.util.ServiceLoader service-provider loading facility.

@Name

An annotation that can be placed on field, method, or method parameter to give it a unique name to distinguish it from similar fields, methods or parameters.

@Property

An annotation that can be placed on test class field to inject runtime/configuration properties into a test class.

@Hint

An annotation that can be placed a test class to provide hints to the test class. This is useful when being explicit is necessary due to the presence of multiple implementations of discoverable contracts in the classpath.

FAQ

Why do we need yet another Java Testing Framework?

The intent of this project is not to add yet another testing framework to the Java eco-system or simply re-invent the wheel but to fill a void. There are numerous testing framework out there but none that allow you to write Unit, Integration and System tests quickly and intuitively using the same framework. Testify is built on testing best practices and we feel confident that it will help as you write production and test code. Ultimately we hope you find Testify enables you to be more productive by saving you time.

Who started the project?

Testify Project was started by Sharmarke Aden late 2015 because he was not satisfied with the landscape of Jave Testing Framework. Having written HK2 Testing Framework, contributed to Cargo, and used various testing frameworks (Spring Testing Framework, Arquillian, Selenium, etc), and seen countless of ineffective “unit”, “integration”, and “system” tests Sharmarke decided there has to be a better way.

Do you intend to support my favorite XYZ Framework?

Testify is designed to be modular and extensible. Adding support for a new dependency injection or application framework or any feature for that matter is a fairly straight forward process. Having said that, work has to be prioritized. We will seriously consider every request and if your request is aligned with Testify’s mission and vision we will add it. We am also very open to and would love to have contributions from the community. Please submit a pull request for features and fixes you would like to see in Testify.

Can my Test Class have multiple Resources?

Absolutely! @LocalResource, @VirtualResource, @RemoteResource annotations are repeatable and your test class can require as many resources as it needs ;)

Can my Test Class have multiple Modules

Yes. @Module annotation is repeatable. Having said that, it is recommended that your test class only import one single module to limit the scope of the test class and maintain modularity. If you find that you need many modules you should think about refactoring.

Isn’t what you call fake really a mock? Why not call it mock?

It is true that there is a distinction between a fake and a mock. Testify indeed conflates the two as described by Martin Flowers. Testify considered and in fact did use @Mock at inception but a decision was made to change it to @Fake for the following reason:

Testify has adopted Semantic Testing and takes systems thinking approach to thinking about what we call collaborators.
Many testing frameworks (i.e. Mockito, Powermock, etc) already have @Mock annotations and developers will have to make a conscious decision as to which annotation to import. This can lead to a number of issues that can slow down development.
@Fake is business people and novice friendly and we want to be as friendly as possible to a wider audience.
@Fake fits nicely with @Real and @Virtual lexicon. Collaborators are either fake, real, or virtual (delegated mock) and we believe that these concepts are easier to explain, understand and reason about.
Using @Fake is in-line with Testify’s mission of being an easy to understand and easy to use testing framework.

Docker Container based testing is eating my disk space? How do I reclaim disk space?

Testify tries to delete containers after each test but sometimes may not be able (i.ie the JVM is terminated in the middle of a test case). You can manually clean up the exited containers using the following command:

1
docker rm -v $(docker ps -a -q -f status=exited)

Docker keeps all images it pulls from the Docker registry on disk. If you are still low on disk space you can manually delete dangling Docker images (file system layer not being referenced) to create disk room. It is a good idea to use fixed version Docker images but if you want to always use the latest version of a Docker image then you may want to periodically clean up dangling images by executing the following command:

1
docker rmi $(docker images -f "dangling=true" -q)

Docker also keeps volumes of pulled images on disk. If you are still low on disk space and are using Docker v1.19.x and above you can delete dangling volumes using the following command:

1
docker volume rm $(docker volume ls -qf dangling=true)

I’m having issues, can you help?

If you come across an issue please check if it is a known issue
File a Bug
Gitter
Users Mailing List ([email protected]s.com)
Developers Mailing List ([email protected])