The question here is not about improving performance, instead its on why the jobs is not utilizing additional free resources when more nodes are made available to it .UCDI wrote:you can make routines in other languages that can use multi-threading if you really need the performance
Search found 726 matches
- Sat Feb 11, 2017 8:19 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Job processing speed Vs available nodes
- Replies: 8
- Views: 5948
- Sat Feb 11, 2017 8:16 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Job processing speed Vs available nodes
- Replies: 8
- Views: 5948
As noted I would assume it's simply the fact that you have a BASIC Transformer in the job and that it has no parallel capability, thus creating a choke point. From what little I recall, it is restricted to running on one specific node - head? conductor? Its an SMP server with just one physical node...
- Thu Feb 09, 2017 5:07 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Job processing speed Vs available nodes
- Replies: 8
- Views: 5948
Hi Ray, These jobs were born in the earlier versions so the Server Routine. The routine has nothing but cascaded calls to Change() to replace a list of 200 substrings. We can break it down into stage variables in parallel transformer but this needs to replicated in all places the server routine is c...
- Wed Feb 08, 2017 6:23 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Job processing speed Vs available nodes
- Replies: 8
- Views: 5948
Job processing speed Vs available nodes
Hi, There seems to be a situation which I cant fully comprehend. A job is reading data from sequential file feeding data to Basic Transformer and writing to sequential file. The basis transformer calls server routine which in turn calls Change() over 200 times, so is CPU intensive. The job when run ...
- Tue Feb 07, 2017 5:57 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Identifying Basic transformer stage
- Replies: 4
- Views: 4159
Identifying Basic transformer stage
Having worked for 9 years on DataStage, I was baffled when I wasn't able to say by opening the transformer stage of a job whether it was Basic Transformer or parallel Transformer . Can it be found any where in the stage ? Both say Transformer Stage and I am currently opening Stage variable propertie...
- Wed Nov 30, 2016 6:11 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: User inserted tsort missing in job score
- Replies: 1
- Views: 2198
User inserted tsort missing in job score
Hi, I have inserted link sort over field "ENROLLEE_ID" in a job and I can find it in Generated OSH code but the tsort operator does not exist in the job score why appears confusing to me. Operator combination is disable still cant find the sort operator, should it not exist in the score ? ...
- Mon Nov 28, 2016 12:07 pm
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Clear Preserve Partitioning flag on dataset via command?
- Replies: 3
- Views: 3046
- Mon Nov 28, 2016 9:57 am
- Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- Topic: Clear Preserve Partitioning flag on dataset via command?
- Replies: 3
- Views: 3046
Clear Preserve Partitioning flag on dataset via command?
Hi,
Is there a way to clear the Preserve Partitioning flag of a dataset via command line.
My problem will also be solved if there is any hidden option to clear the PP flag while copying a dataset to another where the source has PP flag set.
Thanks
Is there a way to clear the Preserve Partitioning flag of a dataset via command line.
My problem will also be solved if there is any hidden option to clear the PP flag while copying a dataset to another where the source has PP flag set.
Thanks
- Sat May 14, 2016 1:54 am
- Forum: General
- Topic: Get column names from command line using stage name/link nam
- Replies: 4
- Views: 3382
- Thu May 12, 2016 2:14 pm
- Forum: General
- Topic: Get column names from command line using stage name/link nam
- Replies: 4
- Views: 3382
- Thu May 12, 2016 10:21 am
- Forum: General
- Topic: Get column names from command line using stage name/link nam
- Replies: 4
- Views: 3382
Get column names from command line using stage name/link nam
Hi ,
I am need to get column names using job name, stage name & link name.
The column definitions could be imported into the stage using table definitions or hand coded.
tried the dsjob but doesnt help any ideas will definitely help
Thanks
I am need to get column names using job name, stage name & link name.
The column definitions could be imported into the stage using table definitions or hand coded.
tried the dsjob but doesnt help any ideas will definitely help
Thanks
- Sun Apr 17, 2016 2:06 pm
- Forum: General
- Topic: Searching for file name in XML export
- Replies: 8
- Views: 4587
Searching for file name in XML export
Hi, I need to search for list of impacted jobs which are reading/writing a particular file, the file name is hardcoded in filename property. The DS_JOBOBJECTS search in some cases is not returning complete list hence I have exported 2500+ job in the project into XML format which is around 1.5 GB. Al...
- Thu Apr 14, 2016 5:20 am
- Forum: General
- Topic: Identify if job is in 'COMPILED' status
- Replies: 2
- Views: 2715
Identify if job is in 'COMPILED' status
Hi, I want to check from command line if a job is exactly in 'COMPILED' state as it appears in Director. After having looked at dsapi.h there is no exact code for 'COMPILED' the next close is #define DSJS_NOTRUNNING 99 by which I cant exactly say if its in compiled state. The below IBM link says 99 ...
- Wed Mar 18, 2015 1:26 am
- Forum: General
- Topic: @RECORD Array to Column
- Replies: 14
- Views: 8488
Guess its time to start using metadata repository instead of Universe for more accurate information. Not sure if anybody has figured out a way to get similar report (one fetched by scanning entire DS_OBJECTS record for searching something) with metadata repository. you know what I am looking for, co...
- Tue Mar 17, 2015 1:19 am
- Forum: General
- Topic: @RECORD Array to Column
- Replies: 14
- Views: 8488