problem linking c++ when compiled with 32-bit pointers

Post Reply

Topic author
joukj
Master
Posts: 254
Joined: Thu Aug 27, 2020 5:50 am
Reputation: 0
Status: Offline

problem linking c++ when compiled with 32-bit pointers

Post by joukj » Thu Sep 12, 2024 2:43 am

Hi all,

I use the latest field-test C++ compiler on OpenVMS 9.2-2. I encountered problems when linking a program which was compiled using 32-bit pointer size.

This is what I get

Code: Select all

minuet-jj) cxx/vers
VSI C++ A10.1-2.240805
clang version 10.0.1 (git@bitbucket.org:vms_software/llvm-10.0.1.git 1e3972aa6d4
9533b81ee136553ec6867c04f8b78)
Build date: 08-05-2024
Target: x86_64-OpenVMS
Thread model: posix
InstalledDir: MINUET$DKA100:[SYS0.SYSCOMMON.][SYSEXE]
minuet-jj) sh sys/noproc
OpenVMS V9.2-2  on node MINUET   12-SEP-2024 08:22:19.05   Uptime  49 23:02:53
minuet-jj) ty gen_html.cpp
/*
 * Copyright (c) Meta Platforms, Inc. and affiliates.
 * All rights reserved.
 *
 * This source code is licensed under both the BSD-style license (found in the
 * LICENSE file in the root directory of this source tree) and the GPLv2 (found
 * in the COPYING file in the root directory of this source tree).
 */

#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
using namespace std;


/* trim string at the beginning and at the end */
void trim(string& s, string characters)
{
    size_t p = s.find_first_not_of(characters);
    s.erase(0, p);

    p = s.find_last_not_of(characters);
    if (string::npos != p)
       s.erase(p+1);
}


/* trim C++ style comments */
void trim_comments(string &s)
{
    size_t spos, epos;

    spos = s.find("/*");
    epos = s.find("*/");
    s = s.substr(spos+3, epos-(spos+3));
}


/* get lines until a given terminator */
vector<string> get_lines(vector<string>& input, int& linenum, string terminator)
{
    vector<string> out;
    string line;
    size_t epos;

    while ((size_t)linenum < input.size()) {
        line = input[linenum];

        if (terminator.empty() && line.empty()) { linenum--; break; }

        epos = line.find(terminator);
        if (!terminator.empty() && epos!=string::npos) {
            out.push_back(line);
            break;
        }
        out.push_back(line);
        linenum++;
    }
    return out;
}


/* print line with ZSTDLIB_API removed and C++ comments not bold */
void print_line(stringstream &sout, string line)
{
    size_t spos;

    if (line.substr(0,12) == "ZSTDLIB_API ") line = line.substr(12);
    spos = line.find("/*");
    if (spos!=string::npos) {
        sout << line.substr(0, spos);
        sout << "</b>" << line.substr(spos) << "<b>" << endl;
    } else {
      //  fprintf(stderr, "lines=%s\n", line.c_str());
        sout << line << endl;
    }
}


int main(int argc, char *argv[]) {
    char exclam;
    int linenum, chapter = 1;
    vector<string> input, lines, comments, chapters;
    string line, version;
    size_t spos, l;
    stringstream sout;
    ifstream istream;
    ofstream ostream;

    if (argc < 4) {
        cout << "usage: " << argv[0] << " [zstd_version] [input_file] [output_ht
ml]" << endl;
        return 1;
    }

    version = "zstd " + string(argv[1]) + " Manual";

    istream.open(argv[2], ifstream::in);
    if (!istream.is_open()) {
        cout << "Error opening file " << argv[2] << endl;
        return 1;
    }

    ostream.open(argv[3], ifstream::out);
    if (!ostream.is_open()) {
        cout << "Error opening file " << argv[3] << endl;
        return 1;
   }

    while (getline(istream, line)) {
        input.push_back(line);
    }

    for (linenum=0; (size_t)linenum < input.size(); linenum++) {
        line = input[linenum];

        /* typedefs are detected and included even if uncommented */
        if (line.substr(0,7) == "typedef" && line.find("{")!=string::npos) {
            lines = get_lines(input, linenum, "}");
            sout << "<pre><b>";
            for (l=0; l<lines.size(); l++) {
                print_line(sout, lines[l]);
            }
            sout << "</b></pre><BR>" << endl;
            continue;
        }

        /* comments of type /**< and /*!< are detected and only function declara
tion is highlighted (bold) */
        if ((line.find("/**<")!=string::npos || line.find("/*!<")!=string::npos)
 && line.find("*/")!=string::npos) {
            sout << "<pre><b>";
            print_line(sout, line);
            sout << "</b></pre><BR>" << endl;
            continue;
        }

        spos = line.find("/**=");
        if (spos==string::npos) {
            spos = line.find("/*!");
            if (spos==string::npos)
                spos = line.find("/**");
            if (spos==string::npos)
                spos = line.find("/*-");
            if (spos==string::npos)
                spos = line.find("/*=");
            if (spos==string::npos)
                continue;
            exclam = line[spos+2];
        }
        else exclam = '=';

        comments = get_lines(input, linenum, "*/");
        if (!comments.empty()) comments[0] = line.substr(spos+3);
        if (!comments.empty()) comments[comments.size()-1] = comments[comments.s
ize()-1].substr(0, comments[comments.size()-1].find("*/"));
        for (l=0; l<comments.size(); l++) {
            if (comments[l].find(" *")==0) comments[l] = comments[l].substr(2);
            else if (comments[l].find("  *")==0) comments[l] = comments[l].subst
r(3);
            trim(comments[l], "*-=");
        }
        while (!comments.empty() && comments[comments.size()-1].empty()) comment
s.pop_back(); // remove empty line at the end
        while (!comments.empty() && comments[0].empty()) comments.erase(comments
.begin()); // remove empty line at the start

        /* comments of type /*! mean: this is a function declaration; switch com
ments with declarations */
        if (exclam == '!') {
            if (!comments.empty()) comments.erase(comments.begin()); /* remove f
irst line like "ZSTD_XXX() :" */
            linenum++;
            lines = get_lines(input, linenum, "");

            sout << "<pre><b>";
            for (l=0; l<lines.size(); l++) {
              //  fprintf(stderr, "line[%d]=%s\n", l, lines[l].c_str());
                string fline = lines[l];
                if (fline.substr(0, 12) == "ZSTDLIB_API " ||
                    fline.substr(0, 12) == string(12, ' '))
                  fline = fline.substr(12);
                print_line(sout, fline);
            }
            sout << "</b><p>";
            for (l=0; l<comments.size(); l++) {
                print_line(sout, comments[l]);
            }
            sout << "</p></pre><BR>" << endl << endl;
        } else if (exclam == '=') { /* comments of type /*= and /**= mean: use a
 <H3> header and show also all functions until first empty line */
            trim(comments[0], " ");
            sout << "<h3>" << comments[0] << "</h3><pre>";
            for (l=1; l<comments.size(); l++) {
                print_line(sout, comments[l]);
            }
            sout << "</pre><b><pre>";
            lines = get_lines(input, ++linenum, "");
            for (l=0; l<lines.size(); l++) {
                print_line(sout, lines[l]);
            }
            sout << "</pre></b><BR>" << endl;
        } else { /* comments of type /** and /*- mean: this is a comment; use a 
<H2> header for the first line */
            if (comments.empty()) continue;

            trim(comments[0], " ");
            sout << "<a name=\"Chapter" << chapter << "\"></a><h2>" << comments[
0] << "</h2><pre>";
            chapters.push_back(comments[0]);
            chapter++;

            for (l=1; l<comments.size(); l++) {
                print_line(sout, comments[l]);
            }
            if (comments.size() > 1)
                sout << "<BR></pre>" << endl << endl;
            else
                sout << "</pre>" << endl << endl;
        }
    }

    ostream << "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text
/html; charset=ISO-8859-1\">\n<title>" << version << "</title>\n</head>\n<body>"
 << endl;
    ostream << "<h1>" << version << "</h1>\n";

    ostream << "<hr>\n<a name=\"Contents\"></a><h2>Contents</h2>\n<ol>\n";
    for (size_t i=0; i<chapters.size(); i++)
        ostream << "<li><a href=\"#Chapter" << i+1 << "\">" << chapters[i].c_str
() << "</a></li>\n";
    ostream << "</ol>\n<hr>\n";

    ostream << sout.str();
    ostream << "</html>" << endl << "</body>" << endl;

    return 0;
}
  12-SEP-2024 08:22:29
minuet-jj) cxx/point=32/warn=(noinfo,disable=("comment")) gen_html.cpp
minuet-jj) link  gen_html.obj
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_tr
aitsIcEENS_11allocator32IcEEEC2ERKS5_
	defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11alloca
tor32IcEEEC5ERKS5_ 
	file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1 
	referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11all
ocator32IcEEEC2ERKS5_ 
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_tr
aitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
	defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11alloca
tor32IcEEEC5ERKS5_mmRKS4_ 
	file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1 
	referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11all
ocator32IcEEEC2ERKS5_mmRKS4_ 

What is going wrong?"
What does "group mismatch" mean?

Of course the 64-bit compilation works:

Code: Select all

minuet-jj) cxx/warn=(noinfo,disable=("comment")) gen_html.cpp
minuet-jj) link  gen_html.obj

regards
Jouk


hb
Master
Posts: 152
Joined: Mon May 01, 2023 12:11 pm
Reputation: 0
Status: Offline

Re: problem linking c++ when compiled with 32-bit pointers

Post by hb » Thu Sep 12, 2024 7:44 am

Easy to spot :-)

The compiler defines a weak symbol
_ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
in a section with the group signature
_ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_,
The RTL defines this symbol in a group with signature _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC5ERKS5_mmRKS4_.
You probably immediately see that there is a "2" in what the compiler generates and a "5" in what the RTL provides.

Assuming that such a symbol should be defined by a group with the same signature: the RTL is wrong.

It's a warning. As you should be able to see in a MAP file, the linker uses the symbol defined in the object module. That is, it includes the (code from the) group that is in the object module. If there is no mismatch, the linker would resolve the symbol from the RTL, which is what you usually want.


jreagan
VSI Expert
Master
Posts: 221
Joined: Tue Dec 01, 2020 8:40 am
Reputation: 0
Status: Offline

Re: problem linking c++ when compiled with 32-bit pointers

Post by jreagan » Thu Sep 12, 2024 9:46 am

We have fixed that for the next kit (soon). For these latest compilers, we changed "which" string we use depending on pointer-size. There are now std::string and std::string32 classes with the compiler picking the right one based on usage. There is lots of code that allocates a std::string and then expects to build an OpenVMS descriptor pointing to the data in that object (either the data or c_str methods). Having ALL strings in 64-bit heap breaks that code. The error you saw is that we forgot an "extern" in one of the string class headers and the compiler got confused on instantiating the objects.

User avatar

martinv
Master
Posts: 146
Joined: Fri Jun 14, 2019 11:05 pm
Reputation: 0
Location: Goslar, Germany
Status: Offline
Contact:

Re: problem linking c++ when compiled with 32-bit pointers

Post by martinv » Mon Oct 14, 2024 5:58 am

Problem still exists in VSI C++ V10.1-2 (released 09-Oct-2024), seen in ZMQ 32-bit shareable build.

Code: Select all

LINK    /SHAREABLE=[.X86_64-32]zmqshr32.exe []zmq_share.opt/OPTION,[.X86_64-32]libzmq32.olb/LIBRARY,SYS$DISK:[]tcpip.opt/OPTION
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_
        defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC5ERKS5_
        file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1
        referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
        defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC5ERKS5_mmRKS4_
        file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1
        referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
Never believe that a few caring people can't change the world. For, indeed, that's all who ever have.
(Margaret Mead)


Topic author
joukj
Master
Posts: 254
Joined: Thu Aug 27, 2020 5:50 am
Reputation: 0
Status: Offline

Re: problem linking c++ when compiled with 32-bit pointers

Post by joukj » Mon Oct 14, 2024 8:09 am

My orinial reproducer now seems to work.

Martinv : can you give the source code of what you are doing?

User avatar

martinv
Master
Posts: 146
Joined: Fri Jun 14, 2019 11:05 pm
Reputation: 0
Location: Goslar, Germany
Status: Offline
Contact:

Re: problem linking c++ when compiled with 32-bit pointers

Post by martinv » Mon Oct 14, 2024 9:44 am

Jouk,

this is the ZeroMQ 4.1.8 code directly from GitHub, of course accompanied by hand-crafted DESCRIP.MMS and platform.hpp .
Never believe that a few caring people can't change the world. For, indeed, that's all who ever have.
(Margaret Mead)


jreagan
VSI Expert
Master
Posts: 221
Joined: Tue Dec 01, 2020 8:40 am
Reputation: 0
Status: Offline

Re: problem linking c++ when compiled with 32-bit pointers

Post by jreagan » Tue Oct 15, 2024 12:09 am

If you can submit a case with support, we'll get on that quickly

John


tigran.sargsyan
Visitor
Posts: 2
Joined: Tue Oct 15, 2024 5:17 am
Reputation: 0
Status: Offline

Re: problem linking c++ when compiled with 32-bit pointers

Post by tigran.sargsyan » Tue Oct 15, 2024 5:41 am

Hi Martinv,

Could you please provide a small reproducer?
Is it possible that in linking process there is an object file that was generated by older compiler (released before August 2024)?

Regards
Tigran
Last edited by tigran.sargsyan on Tue Oct 15, 2024 5:41 am, edited 1 time in total.

User avatar

martinv
Master
Posts: 146
Joined: Fri Jun 14, 2019 11:05 pm
Reputation: 0
Location: Goslar, Germany
Status: Offline
Contact:

Re: problem linking c++ when compiled with 32-bit pointers

Post by martinv » Wed Nov 06, 2024 8:56 am

Tigran,

sorry for the late reply. I've been kept busy with other things.

Testing exactly which of the 72 object files in the library causes the link command to emit the warning (while ignoring all the "unresolved symbol" errors) and then isolating what in that file causes the error to obtain a small reproducer would have been quite a lot of effort.

So I thought I'd give instructions on how to reproduce the error using the GitHub sources, and a GNU diff file.

But I noticed that a) there's no GNU patch executable on VMS x86_64 (not a big deal as the changes are few), and b) I hadn't used the latest 4.1.8 sources. So I re-downloaded the latest sources in the libzmq 4-1 branch, re-did my changes, and re-built the 32-bit shareable.

And the error was gone... Hmmm... the only changes made to the system since my post on 14-Oct-2024 were the installations of OPENSSH V8.9-1J over V8.9-1I01, Pascal V6.4-147 over X6.4-145, GIT V2.44-1A, and C V7.6-1 over X7.6-28.

cu,
Martin
Last edited by martinv on Wed Nov 06, 2024 9:39 am, edited 2 times in total.
Never believe that a few caring people can't change the world. For, indeed, that's all who ever have.
(Margaret Mead)


tigran.sargsyan
Visitor
Posts: 2
Joined: Tue Oct 15, 2024 5:17 am
Reputation: 0
Status: Offline

Re: problem linking c++ when compiled with 32-bit pointers

Post by tigran.sargsyan » Tue Nov 19, 2024 8:36 am

Hi Matinv,

Thank you for reply,
Actually, that problem shouldn't be occurred, because we have fixed it before release.
Anyway, please inform us if you will see it again.

Regards,
Tigran

Post Reply