# Job-execution-rejected **URL:** \ **Category:** Development\ **Created:** [November 14, 2024, 5:09pm UTC](https://forum.operaton.org/t/job-execution-rejected/68 "2024-11-14T17:09:16Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![Julian](https://forum.operaton.org/user_avatar/forum.operaton.org/julian/32/30_2.png) [@Julian](https://forum.operaton.org/u/Julian)\ **Post date:** [November 23, 2024, 7:41am UTC](https://forum.operaton.org/t/job-execution-rejected/68/42 "2024-11-23T07:41:56Z")
just saw that this query is defined here in this resource file: [operaton/engine/src/main/resources/org/operaton/bpm/engine/impl/mapping/entity/Job.xml at b12e80c7ded6db496c36759c423c2938d86a689b · operaton/operaton · GitHub](https://github.com/operaton/operaton/blob/b12e80c7ded6db496c36759c423c2938d86a689b/engine/src/main/resources/org/operaton/bpm/engine/impl/mapping/entity/Job.xml#L178) Wouldn’t it be possible to just override the resource file? Will test it out. ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [November 23, 2024, 7:53am UTC](https://forum.operaton.org/t/job-execution-rejected/68/43 "2024-11-23T07:53:34Z")
You are right, patching the engine is not possible for most people 😉 Overriding it on the classpath could work, but it might be pretty instable and break with every patch version update. If you want to try it, I added `FOR UPDATE SKIP LOCKED` as a new line after the `${limitAfter}` in line 230. Edit: Can confirm it works: ![grafik](https://forum.operaton.org/uploads/default/original/1X/bb7e6fcc736c67ef755f7b53fa60dc09c4cafcf0.png) ---
**Author:** ![Julian](https://forum.operaton.org/user_avatar/forum.operaton.org/julian/32/30_2.png) [@Julian](https://forum.operaton.org/u/Julian)\ **Post date:** [November 23, 2024, 8:14am UTC](https://forum.operaton.org/t/job-execution-rejected/68/44 "2024-11-23T08:14:42Z")
@jradesenv can you try this and see if and how this affects your test scenario? Would be really interesting to see if you can reproduce the performance improvements. I read the documentation about the SKIP LOCKED SQL command and according to the docs it‘s exactly made for this use case. It‘s of course a hack, you need to copy and patch the file from the exact matching C7 version you‘re using ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [November 23, 2024, 8:51am UTC](https://forum.operaton.org/t/job-execution-rejected/68/45 "2024-11-23T08:51:37Z")
Yeah, tried overriding the file on my resources and it worked! will executed the tests again with the FOR UPDATE SKIP LOCKED and come back here to show results ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [November 23, 2024, 10:27am UTC](https://forum.operaton.org/t/job-execution-rejected/68/46 "2024-11-23T10:27:48Z")
10k process instances 30k jobs with 3 containers 1. job executor default + replacing query with FOR UPDATE SKIP LOCKED it works and theres not a single job acquisition faillure anymore. ![image](https://forum.operaton.org/uploads/default/original/1X/8d3530951fa134c1390bf112ab89dba957fdfe57.png) 1. job executor with pagination + replacing query with FOR UPDATE SKIP LOCKED I can still see some improvement on jobs per minute with these 2 changes together ![image](https://forum.operaton.org/uploads/default/original/1X/2d527368c2b039bd7aa18d457415af78900ca3ce.png) just to compare… 1. job executor default and WITHOUT the addition of FOR UPDATE SKIP LOCKED on the query: the job acquisiition failure happens as expected but i wasnt expecting it to have better performance than with FOR UPDATE… 2. job executor with pagination and WITHOUT the addition of FOR UPDATE SKIP LOCKED on the query: By applying paging/offset I also managed not to have job acquisition failure, and it still appears to be the best performance, but perhaps in a larger test with 100k instances the use of skip locked would avoid overhead in the database by avoiding unnecessary updates that would generate failure. ---
**Author:** ![Julian](https://forum.operaton.org/user_avatar/forum.operaton.org/julian/32/30_2.png) [@Julian](https://forum.operaton.org/u/Julian)\ **Post date:** [November 23, 2024, 10:33am UTC](https://forum.operaton.org/t/job-execution-rejected/68/47 "2024-11-23T10:33:29Z")
Would be interesting to see how it behaves with 5 or 10 containers now ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [November 23, 2024, 10:33am UTC](https://forum.operaton.org/t/job-execution-rejected/68/48 "2024-11-23T10:33:51Z")
AWS RDS Performance Insights of all four tests (it was running on a Postgresql 15.8 t4g.medium): ![image](https://forum.operaton.org/uploads/default/original/1X/59cad9a985027860eacc2cb83e7036cc01089d1a.png) ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [November 23, 2024, 10:52am UTC](https://forum.operaton.org/t/job-execution-rejected/68/49 "2024-11-23T10:52:40Z")
Thanks for doing those comparisons! I’m not sure if I read your summary-images right, where can I see the execution time? I understood that adding FOR UPDATE SKIP LOCKED reduces the acquisition failures, but decreases performance in general? ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [November 25, 2024, 7:02pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/50 "2024-11-25T19:02:34Z")
Saturday my test environment was a mess and today i executed it in a better way. We can really see the improvement of this setting here… 100k process / 300k jobs / 10 pods Default implementation: 40minutes total and 760k job acquisition failed ![image](https://forum.operaton.org/uploads/default/original/1X/7677ae5319d8e724d7566ed34fe7691cf5d096f1.png) Query updated with FOR UPDATE SKIP LOCK: 16minutes total and ZERO job acquisition faiiled ![image](https://forum.operaton.org/uploads/default/original/1X/73cb4f0632bf025194371992498be1edd63492eb.png) Finally achived to use 95-100% of my database cpu in a good way. In the default implementation i was capped at 30% cpu, stuck in job acquiisition fails. Just to show some data RDS waits: ![image](https://forum.operaton.org/uploads/default/original/1X/5f6ca857a893ba7d8a8dd9938cfe899161ec132a.png) here you can see that there was almost no job execution on the first test, and much more job select/acquisition, while on the second test it was almost all about job inserts. ![image](https://forum.operaton.org/uploads/default/original/1X/b5bb172ce9b2f3ac49b3d455c38d85fff40554b3.png) ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [November 25, 2024, 7:18pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/51 "2024-11-25T19:18:41Z")
Thanks for sharing these numbers again! So the “16minutes total and ZERO job acquisition faiiled” is just the FOR UPDATE SKIP LOCK without your custom paging logic, right? That behaves even better than I would have hoped for 👍 I think we could add this as an option to the engine: - Create a new property in the engine configuration (+ Spring Boot + Quarkus), opt in - Allow users to enable this locking mechanism on the job executor - Get many people to try it 😅 ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [November 25, 2024, 7:25pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/52 "2024-11-25T19:25:39Z")
Yep! I’ll will try it plus my custom paging to see if it is still relevant. But the idea of creating a property to enable this behavior seems a great advance as we dont expect users to override those mybatis xml too often 👍 Im still curious to test it against Mysql too, as i have many projects on this too. Will post the results here when it gets done. ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [November 25, 2024, 7:27pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/53 "2024-11-25T19:27:20Z")
For reference: > > > \*\*Is your feature request related to a problem? Please describe.\*\* > \[In our foru…m\](https://forum.operaton.org/t/job-execution-rejected/68/39?u=javahippie) we experimented with adding \`FOR UPDATE SKIP LOCK\` to the select statement for the job executor. This does not work for all supported databases, but sped up execution for many nodes running in parallel. > > \*\*Describe the solution you'd like\*\* > We might consider adding an opt-in configuration flag to enable this behavior, so users running on a database which supports this could benefit from higher concurrency. > > \*\*Describe alternatives you've considered\*\* > An alternative provided in the thread was to provide read offsets for different nodes so they don't all try to read on the most recent jobs. This works too, but might be dangerous if a node crashes and the range that was reserved for it is not read anymore. ---
**Author:** ![Julian](https://forum.operaton.org/user_avatar/forum.operaton.org/julian/32/30_2.png) [@Julian](https://forum.operaton.org/u/Julian)\ **Post date:** [November 25, 2024, 9:05pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/54 "2024-11-25T21:05:14Z")
@jradesenv thanks for all your tests! Looking forward to see the mysql results. ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [November 29, 2024, 12:07am UTC](https://forum.operaton.org/t/job-execution-rejected/68/55 "2024-11-29T00:07:18Z")
Finally tried it against a t4g.medium mysql 8 database and it worked great also (postgresql still performs better as expected): We made like 15k jobs/minute. The same testes took 20minutes to complete ![IMG_7913](https://forum.operaton.org/uploads/default/original/1X/dda051f2687f6c3e799c8b57d045251dde393b78.png) ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [December 1, 2024, 8:37pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/56 "2024-12-01T20:37:47Z")
I decided to use only the updated query instead of my paging solution, but i still used the logic that set a single index for each one of my containers to configure a different priority range for the job executor. This way i have 15 pods total with no concurrency problems (with the FOR UPDATE SKIP LOCKED) and put two of them to work on low priority jobs only (jobPriorityRangeMax(90)). This way all my priority tasks are set from 91 to 100 and the others let as zero. Yesterday i tested for the first time my 500k+ scenario. It took 35min to start all the instances (full logic taking objects from my mongodb to process engine and updating its status). Then restarted the containers with job executor enabled and it took 1:30 to conclude all the 500k+ process instances (which is a complex process with 3 subprocess, parallel gateways for send and receive messages with async before/after) So, im happy with this throughput and it will work 🙂 Just passed by to say thanks for all the sharing we had here and to say that i will keep passing by to discuss new challenges with you all 🤜🤛 ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [December 1, 2024, 9:35pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/57 "2024-12-01T21:35:11Z")
@jradesenv I’m really happy to hear this! The approach with different priority ranges sounds sensible. Also thanks for testing this with MySQL, also! I’m currently preparing a pull request which adds an engine config property which allows users to enable FOR UPDATE SKIP LOCKED in the engine itself as an experimental feature for Operaton. If this works and if we get some positive feedback, I think I’ll offer this as a contribution to Camunda, too, in the spirit of OSS 😉 Also, I think I will write a blog post about this on our companies blog, would it be fine for you if I used the numbers you provided here and mention you? If not, that’s totally fine, I think I can recreate those experiments ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [December 1, 2024, 9:51pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/58 "2024-12-01T21:51:08Z")
Yeah, of course you can share everything we discussed here 🤟 ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [December 7, 2024, 3:40pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/59 "2024-12-07T15:40:48Z")
Just a small update: Implementing this in the engine is not that easy. After digging into the job executor deeper with @kthoms today, I am _a little_ surprised that it worked that well for you. Currently Camunda is not setting any row level locks in the database, but sets a logical lock. The `FOR UPDATE SKIP LOCKED` clause obviously does not know about those locks, but obtains its own physical row lock. As long as this is active, other job executor instances will not retrieve this row. By default, the Job Executor queries all rows, and then sets the logical log in the same transaction. My _assumption_ is, that in the delta between “selecting” and “setting the lock”, other job executors can retrieve this row also, and then try to set the lock, too, which can fail. Due to the additional physical lock, we have prevented this from happening. So the new row-level physical lock does not lock the job in the database for the whole processing time, but locks it so the logical lock can be safely written. Everything written here is a hypothesis yet to be proven, I just wanted to give you an update ---
**Author:** ![jradesenv](https://forum.operaton.org/user_avatar/forum.operaton.org/jradesenv/32/24_2.png) [@jradesenv](https://forum.operaton.org/u/jradesenv)\ **Post date:** [December 7, 2024, 4:02pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/60 "2024-12-07T16:02:17Z")
Hmm, I don’t know if I managed to understand the “problem” with this, because it’s exactly what I expected to happen. By default, camunda has this concurrency problem where several containers are making the same query, getting the same jobs, and the first one manages to set the lock owner and increase the rev\_ while the others will get the job acquisition failed because it no longer finds this row with the same rev\_ (updated rows zero). FOR UPDATE then warns the database that this line must remain locked at the db level until the same transaction makes the update, while SKIP LOCKED is precisely to skip these already locked lines. Perhaps the transaction level READ COMMITED will then help eliminate the rest of the competition that may still occur. Even if there were some job acquisition failing, I understand that it would be minimal, a matter of miliseconds of concurrency, and then it is much better than the standard (where 100k jobs generated 740k failures with 10 containers), and then the logical lock that is done via update still maintains security using rev\_ to avoid the minimum competition that may occur. ---
**Author:** ![javahippie](https://forum.operaton.org/user_avatar/forum.operaton.org/javahippie/32/25_2.png) [@javahippie](https://forum.operaton.org/u/javahippie)\ **Post date:** [December 7, 2024, 4:15pm UTC](https://forum.operaton.org/t/job-execution-rejected/68/61 "2024-12-07T16:15:11Z")
You’re right with that, I rewrote my post several times and in the end it was confusing. The tricky thing is testing. We are now adding row level locking for the first time, until now the whole “locking” concept in the engine is entirely logical locks, so we need to make sure that it plays well with all the other configurations. That definitely got lost in my edits, sorry for the confusion 😉 [Previous page](https://forum.operaton.org/t/job-execution-rejected/68.md?page=2) [Next page](https://forum.operaton.org/t/job-execution-rejected/68.md?page=4)