Does it exist Linker limits when linking with an object library (I64) ?


Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Does it exist Linker limits when linking with an object library (I64) ?

Post by l.cedric » Thu Sep 24, 2020 10:16 am

Hello,

we are trying to rework our build tools. We wish to use just 1 object library for the whole modules (~ 4000 modules and ~ 83000 symbols) of our application, which will make our link process easier.

But but we are facing this link (or cxxlink) issue :(
link.png
Remarks :
  • If needed, find the tools versions in the above screeenshoot
  • Note the size of our library.
Doubtless, i did not search enough (?), but for our I64 server, I did not find any limits neither for the size, nor number of module or symbol limits :?:

Perhaps a clue / Linker ? ($ sh mem/unit=byte)
linkMem.png
So twice ~ 1 GB ... before the crash ? Is it due to a sort of weird P0 space limit of the i64 Linker ? But I can't understand, as we are on I64 architecture.

Or perhaps a hidden switch to allow the Linker to work with a big library ?

Anyone have any ideas on what to do ?
Thanks for any help.


madsweeney
VSI Expert
Active Contributor
Posts: 40
Joined: Mon Jun 10, 2019 9:23 am
Reputation: 1
Status: Offline

Re: Does it exist Linker limits when linking with an object library (I64) ?

Post by madsweeney » Thu Oct 08, 2020 8:54 am

Cedric,

Can you ask your admin to provide the problem report number given by VSI or HPE for the elevation on the linker? I'll see if I can find out the state of the investigation of the problem.

Dave.
Dave Sweeney
CEO
VMS Software, Inc.
Boston, MA USA


Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Re: Does it exist Linker limits when linking with an object library (I64) ?

Post by l.cedric » Tue Oct 20, 2020 7:45 am

Hi,
just to share the current VSI answers (the problem report number is BZ4494) :

The call has been elevated to our Engineering.

When testing on our VSI cluster :

When all the modules are inserted /selective, the virtual size is :

%ILINK-F-MEMFUL, insufficient virtual address space to complete this link
-SYSTEM-W-REGISFULL, specified region is full
FARIS job terminated at 14-OCT-2020 06:39:34.89

Accounting information:
Buffered I/O count: 78 Peak working set size: 1527552
Direct I/O count: 236 Peak virtual size: 2274064
Page faults: 134090 Mounted volumes: 0
Charged CPU time: 0 00:01:22.48 Elapsed time: 0 00:01:22.69


Which is above the size of P0 but the linker uses also data in P2 space


If the library is recreated with all the modules inserted /noselective :

the link works but they are %ILINK-W-NUDFSYMS, 31 undefined symbols (and you say it is expected because some modules are missing)

Accounting information:
Buffered I/O count: 140 Peak working set size: 26880
Direct I/O count: 1043 Peak virtual size: 219408
Page faults: 2241 Mounted volumes: 0
Charged CPU time: 0 00:00:00.22 Elapsed time: 0 00:00:00.37


So the link is fast and the virtual size is far below 1GB


This leads to think that it could be something else than the size of the library. But if there is a memory
leak in the link operation, it may be long to debug so it is good that you have a workaround.
While Engineering is examining the issue, they gave us answers to some of your questions

• The linker has no hard limits on the number of object modules, number of symbols, etc., that it allows. It allocates memory for these things dynamically,
and it is only when it runs out of P0 memory that it fails.


• The main difference between the librarian's /SELECTIVE_SEARCH option and the linker's /SELECTIVE_SEARCH option is how many passes are involved. When one applies the linker /SELECTIVE_SEARCH option to an object module, the linker makes just one pass through the object and takes from it only symbol definitions that satisfy previous undefined symbol references. But when the linker processes an object library containing object modules marked /SELECTIVE_SEARCH, it needs to make multiple passes through that library. On the first pass, it selects objects that satisfy previous undefined symbol references. But since that first pass may have added new undefined symbol references, it needs to make a second pass through that library. This process continues until the linker cannot find any more modules in the library that satisfy any outstanding undefined symbols. How many passes is that? It depends on how many modules there are in the library, and what order they are in. In the very worst case, the number of passes could be equal to the number of modules in the library.



So using /SELECTIVE on a link operation where a library is not involved will improve the link operation performance by managing less symbols and reducing
the size of the map file but that will need a careful ordering of the modules in the link command.
But using /SELECTIVE when inserting modules in a library can in many cases augment the link processing time because of the additional passes which are needed, depending on the size and complexity of the modules in the library.
Cédric


Topic author
l.cedric
Valued Contributor
Posts: 51
Joined: Thu Jul 18, 2019 8:18 am
Reputation: 0
Status: Offline

Re: Does it exist Linker limits when linking with an object library (I64) ?

Post by l.cedric » Thu Nov 05, 2020 2:36 am

Hi,
just to share the current VSI conclusion :
Our Engineering has reviewed this and found that the root cause is a bad interaction between the librarian’s /SELECTIVE_SEARCH option and object modules generated by the C++ compiler.
So this is by design of the linker algorithm and cannot be changed without breaking upward compatibility.

That’s said they say also that

(a) librarian /SELECTIVE_SEARCH will not speed up links significantly
(b) using /SELECTIVE_SEARCH with libraries containing C++ code is deprecated functionality and should not be used.

As a matter of fact we do not recommend using the librarian /SELECTIVE_SEARCH. In some rare cases even without C++ generated modules this can generate a bad behavior.

Post Reply