ISD Maximum Running Jobs
ISD Maximum Running Jobs
Hi,
If you look at operations console -> Workload Management -> Queue Management
There are "Maximum Running Jobs" for various queues. The parameter for the ISD queue is set at 100, but it is not configurable (at least in the console) like the others.
Anyone know how to change the value?
Thanks,
Shaun
If you look at operations console -> Workload Management -> Queue Management
There are "Maximum Running Jobs" for various queues. The parameter for the ISD queue is set at 100, but it is not configurable (at least in the console) like the others.
Anyone know how to change the value?
Thanks,
Shaun
PBT TBIS Consultant
Google is your friend.
https://www.ibm.com/support/knowledgece ... icies.html
I believe that is what you are looking for.
https://www.ibm.com/support/knowledgece ... icies.html
I believe that is what you are looking for.
Not sure why you can't edit it, but it's likely because it isn't that meaningful, provided that these are "always on" ISD Jobs (that have an ISDInput)...
Carefully consider your ISD Jobs...do testing to find out what their high water mark is regarding the performance you need relative to the number of instances that you need to set (within ISD), and then lock that value in as the minimum.
There are very few dynamics here, and you don't want any anyway...for real time Jobs, they need to "be there"...they cannot afford, in most cases, to ever be queued.
Ernie
Carefully consider your ISD Jobs...do testing to find out what their high water mark is regarding the performance you need relative to the number of instances that you need to set (within ISD), and then lock that value in as the minimum.
There are very few dynamics here, and you don't want any anyway...for real time Jobs, they need to "be there"...they cannot afford, in most cases, to ever be queued.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Hi Ernie,
The high water mark we estimated to be at 200-300.
The console is a little confusing that it is under the tab "Queue Management", but the parameter there that I want to change is "Maximum Running Jobs". It is currently set to 100 and when trying to change it get the response "Update or delete is not allowed for non batch queues"
Cheers,
Shaun
The high water mark we estimated to be at 200-300.
The console is a little confusing that it is under the tab "Queue Management", but the parameter there that I want to change is "Maximum Running Jobs". It is currently set to 100 and when trying to change it get the response "Update or delete is not allowed for non batch queues"
Cheers,
Shaun
PBT TBIS Consultant
200 to 300?
are these individual services? do they all have isd input?
...or are you talking about 200 to 300 concurrent requests?
......thats a really huge number for isd always on jobs....
Ernie
are these individual services? do they all have isd input?
...or are you talking about 200 to 300 concurrent requests?
......thats a really huge number for isd always on jobs....
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Well for example now we are testing a workload of ~360 request per minute which we are attempting to handle with three applications created in Information Services Director. The applications are configured with 45, 45 and 10 instances as the min and max.
We may be able to reduce the 100 instances down a bit given more load testing but as the ~360 per minute load is only 40% of the expected workload we will need to create more than a 100 instances.
Are you saying ISD is not suited for the above?
We may be able to reduce the 100 instances down a bit given more load testing but as the ~360 per minute load is only 40% of the expected workload we will need to create more than a 100 instances.
Are you saying ISD is not suited for the above?
PBT TBIS Consultant
Just saying that seems like a lot. A single instance can often handle 200+ requests per second, even with a slow protocol like SOAP over HTTP. Of course it could depend on your tranformations, but if you have 45 instances your throughout could/should be a LOT higher. Something seems very awry here.
Your minimums are at 45? How many stages in the Job? Are these Server Jobs or EE? ....is is a huge machine or grid?
Ernie
Your minimums are at 45? How many stages in the Job? Are these Server Jobs or EE? ....is is a huge machine or grid?
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Well the jobs are very light weight. EE but running on a single node.
The reason for creating so many instances is that even though the job is light weight there is a REST call made by a hierarchical stage in the job that takes anything from 5-12sec to respond. As we need to process a single request in under 30 seconds we decided to create so many instances so that we do not have requests queuing up behind that REST call. In this way we are submitting many parallels REST calls allowing us to come in at under 30 seconds reliably.
The machine is reasonable hefty. 8 cores 32GB of RAM. Saying that while the instances are up and handling requests the CPU idles at 5%. The only signficant resource used is memory. In the case of the 100 instances that is ~20GB.
The client is also looking at upgrading the machine to a size of 48 core and 256GB in the future which may be needed once we start beefing up the jobs with more transformations etc.
So the above is a lot of context and interesting to discuss, but the original post was asking how to change the Maximum Running Jobs for ISD. I have also asked the question to IBM via a PMR.
The reason for creating so many instances is that even though the job is light weight there is a REST call made by a hierarchical stage in the job that takes anything from 5-12sec to respond. As we need to process a single request in under 30 seconds we decided to create so many instances so that we do not have requests queuing up behind that REST call. In this way we are submitting many parallels REST calls allowing us to come in at under 30 seconds reliably.
The machine is reasonable hefty. 8 cores 32GB of RAM. Saying that while the instances are up and handling requests the CPU idles at 5%. The only signficant resource used is memory. In the case of the 100 instances that is ~20GB.
The client is also looking at upgrading the machine to a size of 48 core and 256GB in the future which may be needed once we start beefing up the jobs with more transformations etc.
So the above is a lot of context and interesting to discuss, but the original post was asking how to change the Maximum Running Jobs for ISD. I have also asked the question to IBM via a PMR.
PBT TBIS Consultant
Ok.. sounds very interesting. Just be careful. Start-up in such situations (with that many osh processes needing to come up) can be very problematic with that few cores, along with that many concurrent job instances and EE stages.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Ok... let me know if you need any tips on how to deploy your single Operation across multiple DS Engines with automatic round-robin load balancing.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>