We need to make sure that integration tests with databases can be run again.
In the Camunda Jenkins pipeline it seems that they had Jenkins workers which provided these databases. This is not an option in GitHub Actions, also we would like to enable all developers to run the integration tests, and not be dependent on a non-open CI system.
Our first approach was to start up the database in a Docke Container with the fabric8 Maven plugin. While this is providing a database for the tests, it introduces two issues:
Tests can only be run with mvn, not from the IDE
The tests need to executed with failsafe instead of surefire, because the “Unit Test” phase does not provide hooks to set up and tear down test einvironments safely.
After looking at these two drawbacks, we brought up an idea to rebuild the integration tests so we can use Testcontainers instead. With Testcontainers, the lifecycle of the test infrastructure is managed by the Test Framework and not the build tool. This means the tests can be run anywhere (if Docker is installed on the machine) and can be executed by Maven, or the IDE or any other way and will still work. The downside is that this requires more effort to set up and change.
Any opionions on this? I’m leaning towards embracing Testcontainers, because the result will (imho) be more robust and more flexible.
I am really in favor of using Testcontainers. Right now, executing these integration tests is quite cumbersome. I know, this is how integration tests used to work in the past (i.e. using Cargo etc.), but Docker and Testcontainers significantly shifted the way integration tests are implemented nowadays.
Executing Testcontainers from the IDE is a huge benefit. Furthermore, it will potentially allow a higher level of parallelization, since different test executions do not block the same port.
Whatever helps us to write and execute integration tests easier and execute them faster is welcome.
We have made only good experiences with testcontainers. I can only recommend them. It’s worth mentioning that test execution time increases because db containers start up in every Test class again
I have made similar great experience with Testcontainers, I think it’s out of the question if our usecases could be covered by it. There will be some issues, like DB2 and Oracle not working smoothly on ARM / Apple Chips, but others have beaten those odds, already
I think that we all agree that adding Testcontainers to the Integration Tests would be beneficial and worth the cost until now?
I did a little digging, and we did not realize, yet, that the Testcontainers integration was actually done by Camunda. It is just a little hidden, because an important setting was hidden in the documentation to be put into the settings.xml file on the developers local machine.
A command enabling Testcontainers tests of the process engine is:
mvn clean test -Ppostgresql,testcontainers -Ddatabase.url='jdbc:tc:${database.tc.url}://localhost:${database.port}${database.tc.params}'
The testcontainers proflie activates the integration, and the JDBC url beginning with the jdbc:tc Prefix starts a Testcontainers instance of the specified database magically. All container providers apart from Postgres are referencing Camunda specific images in a private repository, so we will have to work around that.
My proposal is to test the scope of the unit- and integration tests which already support Testcontainers and then discuss making the testcontainers profile the default – I don’t see any reason to use a different test setup, if the Testcontainers setup already works.
I have executed the mentioned Maven command. I’m getting
IllegalState Mapped port can only be obtained after the container is started
for tests in the engine module.
Just for confirmation: You get the same?
And how do we want to handle the tests? IMO when this is executing fine for postgres, this should be part of the Integration Tests pipeline. I.e. when the Nightly Build is successful, the integration with Postgres would be tested.
Do you see any testcontainers logs, are the Docker containers even starting?
For me the suite runs sucessfully for a long time and then the ArchUnit test complains. I don’t think the ArchUnit Test is related, I sometimes get this issue randomly. Not at the computer right now, can run another test later in the evening
No, just that the exception is raised by Testcontainers itself. I have just executed this on a second machine. Stack of the tests is always:
[ERROR] org.operaton.bpm.engine.test.standalone.task.TaskDecoratorTest.testDecorateNameFromVariable -- Time elapsed: 0.003 s <<< ERROR!
java.lang.IllegalStateException: Mapped port can only be obtained after the container is started
at org.testcontainers.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:174)
at org.testcontainers.containers.ContainerState.getMappedPort(ContainerState.java:142)
at org.testcontainers.containers.PostgreSQLContainer.getJdbcUrl(PostgreSQLContainer.java:86)
at org.testcontainers.containers.JdbcDatabaseContainer.constructUrlForConnection(JdbcDatabaseContainer.java:241)
at org.testcontainers.containers.JdbcDatabaseContainer.createConnection(JdbcDatabaseContainer.java:207)
at org.testcontainers.jdbc.ContainerDatabaseDriver.connect(ContainerDatabaseDriver.java:124)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:683)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:191)
at org.apache.ibatis.datasource.unpooled.UnpooledDataSource.doGetConnection(UnpooledDataSource.java:228)
at org.apache.ibatis.datasource.unpooled.UnpooledDataSource.doGetConnection(UnpooledDataSource.java:223)
at org.apache.ibatis.datasource.unpooled.UnpooledDataSource.getConnection(UnpooledDataSource.java:96)
at org.apache.ibatis.datasource.pooled.PooledDataSource.popConnection(PooledDataSource.java:452)
at org.apache.ibatis.datasource.pooled.PooledDataSource.getConnection(PooledDataSource.java:100)
at org.operaton.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.getCurrentTransactionIsolationLevel(ProcessEngineConfigurationImpl.java:1714)
at org.operaton.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.checkTransactionIsolationLevel(ProcessEngineConfigurationImpl.java:1702)
at org.operaton.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.initDataSource(ProcessEngineConfigurationImpl.java:1693)
at org.operaton.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.init(ProcessEngineConfigurationImpl.java:1152)
at org.operaton.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl.buildProcessEngine(ProcessEngineConfigurationImpl.java:1123)
at org.operaton.bpm.engine.test.util.ProvidedProcessEngineRule.getOrInitializeCachedProcessEngine(ProvidedProcessEngineRule.java:72)
at org.operaton.bpm.engine.test.util.ProvidedProcessEngineRule.<init>(ProvidedProcessEngineRule.java:41)
at org.operaton.bpm.engine.test.util.PluggableProcessEngineTest.<init>(PluggableProcessEngineTest.java:50)
at org.operaton.bpm.engine.test.standalone.task.TaskDecoratorTest.<init>(TaskDecoratorTest.java:46)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
Good news, I just have successfully tested the engine with Postgres and Testcontainers for the first time – almost
I had to exclude one single test, ConnectionPersistenceExceptionTest temporarily because the return values for failing Connections are not as expected. Looking into that now, if the values in the drivers changed, or if there is indeed an issue, the rest is running just fine.
Managed to reach the same state for MariaDB now, our OperatonMariaDBContainerProvider locally on my machine now starts MariaDB with the correct transaction isolation, making the previous custom built images by camunda obsolete.
Locally, I am able to run the engine integration tests with Postgres, MariaDB in Testcontainers, and I managed to get Tomcat working. For this I had to add the Testcontainers dependencies to Tomcat, so the jdbc:tc:... JDBC URL manages to start a container. GitHub Actions don’t yet work with Tomcat/Wildfly and Postgres/MariaDB, I’m still seeing some deploy errors there and need to find a way for better logging.
One issue with the tests is, that the Arquillian Container manages the Testcontainer, not the test itself. Every time a test is done, the Arquillian context is torn down, the JVM it ran on terminates and the testcontainer which belongs to it is torn down, too. This is not too painful with MariaDB or Postgres, but will become a problem with DB2 and Oracle, which both have loooooong start times for their containers.
In the Camunda testsuite, Tomcat and Wildfly Integration tests only worked with H2 and Postgres, and I think I can see some sense in this. For fast tests, H2 is sufficient, and with Postgres it can be tested that a “proper” JDBC Connection to a real service is also functional. If we execute the engine tests (without Arquillian containers) against all supported database vendors, with properly reused Testcontainers images, this might be enough to test the functionality and we might not need to test both Tomcat and Wildfly with all 6 supported vendors (H2, Postgres, MariaDB, MySQL, Oracle DB, MSSQL, DB2), too.
I’m on your side here. I was also thinking if it is really necessary to test all databases with all containers. It should be enough to do so with h2/Postgres on Tomcat/Wildfly by default.
Further, it should be possible to execute the integration test against other databases as well. So great to hear that you made good progress on MariaDB.
It just came to my mind that we might run the integration tests on a nightly basis with h2 and a rolling “proper” database. Then even these DBs with slow start times might be tested frequently as well.
Yesterday I was able to talk to a developer from the Hibernate team. It seems they have solved our MS SQL problems already, will try their approach asap:
