problem linking c++ when compiled with 32-bit pointers

[joukj]

Hi all,

I use the latest field-test C++ compiler on OpenVMS 9.2-2. I encountered problems when linking a program which was compiled using 32-bit pointer size.

This is what I get

Code: Select all

minuet-jj) cxx/vers
VSI C++ A10.1-2.240805
clang version 10.0.1 (git@bitbucket.org:vms_software/llvm-10.0.1.git 1e3972aa6d4
9533b81ee136553ec6867c04f8b78)
Build date: 08-05-2024
Target: x86_64-OpenVMS
Thread model: posix
InstalledDir: MINUET$DKA100:[SYS0.SYSCOMMON.][SYSEXE]
minuet-jj) sh sys/noproc
OpenVMS V9.2-2  on node MINUET   12-SEP-2024 08:22:19.05   Uptime  49 23:02:53
minuet-jj) ty gen_html.cpp
/*
 * Copyright (c) Meta Platforms, Inc. and affiliates.
 * All rights reserved.
 *
 * This source code is licensed under both the BSD-style license (found in the
 * LICENSE file in the root directory of this source tree) and the GPLv2 (found
 * in the COPYING file in the root directory of this source tree).
 */

#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
using namespace std;


/* trim string at the beginning and at the end */
void trim(string& s, string characters)
{
    size_t p = s.find_first_not_of(characters);
    s.erase(0, p);

    p = s.find_last_not_of(characters);
    if (string::npos != p)
       s.erase(p+1);
}


/* trim C++ style comments */
void trim_comments(string &s)
{
    size_t spos, epos;

    spos = s.find("/*");
    epos = s.find("*/");
    s = s.substr(spos+3, epos-(spos+3));
}


/* get lines until a given terminator */
vector<string> get_lines(vector<string>& input, int& linenum, string terminator)
{
    vector<string> out;
    string line;
    size_t epos;

    while ((size_t)linenum < input.size()) {
        line = input[linenum];

        if (terminator.empty() && line.empty()) { linenum--; break; }

        epos = line.find(terminator);
        if (!terminator.empty() && epos!=string::npos) {
            out.push_back(line);
            break;
        }
        out.push_back(line);
        linenum++;
    }
    return out;
}


/* print line with ZSTDLIB_API removed and C++ comments not bold */
void print_line(stringstream &sout, string line)
{
    size_t spos;

    if (line.substr(0,12) == "ZSTDLIB_API ") line = line.substr(12);
    spos = line.find("/*");
    if (spos!=string::npos) {
        sout << line.substr(0, spos);
        sout << "</b>" << line.substr(spos) << "<b>" << endl;
    } else {
      //  fprintf(stderr, "lines=%s\n", line.c_str());
        sout << line << endl;
    }
}


int main(int argc, char *argv[]) {
    char exclam;
    int linenum, chapter = 1;
    vector<string> input, lines, comments, chapters;
    string line, version;
    size_t spos, l;
    stringstream sout;
    ifstream istream;
    ofstream ostream;

    if (argc < 4) {
        cout << "usage: " << argv[0] << " [zstd_version] [input_file] [output_ht
ml]" << endl;
        return 1;
    }

    version = "zstd " + string(argv[1]) + " Manual";

    istream.open(argv[2], ifstream::in);
    if (!istream.is_open()) {
        cout << "Error opening file " << argv[2] << endl;
        return 1;
    }

    ostream.open(argv[3], ifstream::out);
    if (!ostream.is_open()) {
        cout << "Error opening file " << argv[3] << endl;
        return 1;
   }

    while (getline(istream, line)) {
        input.push_back(line);
    }

    for (linenum=0; (size_t)linenum < input.size(); linenum++) {
        line = input[linenum];

        /* typedefs are detected and included even if uncommented */
        if (line.substr(0,7) == "typedef" && line.find("{")!=string::npos) {
            lines = get_lines(input, linenum, "}");
            sout << "<pre><b>";
            for (l=0; l<lines.size(); l++) {
                print_line(sout, lines[l]);
            }
            sout << "</b></pre><BR>" << endl;
            continue;
        }

        /* comments of type /**< and /*!< are detected and only function declara
tion is highlighted (bold) */
        if ((line.find("/**<")!=string::npos || line.find("/*!<")!=string::npos)
 && line.find("*/")!=string::npos) {
            sout << "<pre><b>";
            print_line(sout, line);
            sout << "</b></pre><BR>" << endl;
            continue;
        }

        spos = line.find("/**=");
        if (spos==string::npos) {
            spos = line.find("/*!");
            if (spos==string::npos)
                spos = line.find("/**");
            if (spos==string::npos)
                spos = line.find("/*-");
            if (spos==string::npos)
                spos = line.find("/*=");
            if (spos==string::npos)
                continue;
            exclam = line[spos+2];
        }
        else exclam = '=';

        comments = get_lines(input, linenum, "*/");
        if (!comments.empty()) comments[0] = line.substr(spos+3);
        if (!comments.empty()) comments[comments.size()-1] = comments[comments.s
ize()-1].substr(0, comments[comments.size()-1].find("*/"));
        for (l=0; l<comments.size(); l++) {
            if (comments[l].find(" *")==0) comments[l] = comments[l].substr(2);
            else if (comments[l].find("  *")==0) comments[l] = comments[l].subst
r(3);
            trim(comments[l], "*-=");
        }
        while (!comments.empty() && comments[comments.size()-1].empty()) comment
s.pop_back(); // remove empty line at the end
        while (!comments.empty() && comments[0].empty()) comments.erase(comments
.begin()); // remove empty line at the start

        /* comments of type /*! mean: this is a function declaration; switch com
ments with declarations */
        if (exclam == '!') {
            if (!comments.empty()) comments.erase(comments.begin()); /* remove f
irst line like "ZSTD_XXX() :" */
            linenum++;
            lines = get_lines(input, linenum, "");

            sout << "<pre><b>";
            for (l=0; l<lines.size(); l++) {
              //  fprintf(stderr, "line[%d]=%s\n", l, lines[l].c_str());
                string fline = lines[l];
                if (fline.substr(0, 12) == "ZSTDLIB_API " ||
                    fline.substr(0, 12) == string(12, ' '))
                  fline = fline.substr(12);
                print_line(sout, fline);
            }
            sout << "</b><p>";
            for (l=0; l<comments.size(); l++) {
                print_line(sout, comments[l]);
            }
            sout << "</p></pre><BR>" << endl << endl;
        } else if (exclam == '=') { /* comments of type /*= and /**= mean: use a
 <H3> header and show also all functions until first empty line */
            trim(comments[0], " ");
            sout << "<h3>" << comments[0] << "</h3><pre>";
            for (l=1; l<comments.size(); l++) {
                print_line(sout, comments[l]);
            }
            sout << "</pre><b><pre>";
            lines = get_lines(input, ++linenum, "");
            for (l=0; l<lines.size(); l++) {
                print_line(sout, lines[l]);
            }
            sout << "</pre></b><BR>" << endl;
        } else { /* comments of type /** and /*- mean: this is a comment; use a 
<H2> header for the first line */
            if (comments.empty()) continue;

            trim(comments[0], " ");
            sout << "<a name=\"Chapter" << chapter << "\"></a><h2>" << comments[
0] << "</h2><pre>";
            chapters.push_back(comments[0]);
            chapter++;

            for (l=1; l<comments.size(); l++) {
                print_line(sout, comments[l]);
            }
            if (comments.size() > 1)
                sout << "<BR></pre>" << endl << endl;
            else
                sout << "</pre>" << endl << endl;
        }
    }

    ostream << "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text
/html; charset=ISO-8859-1\">\n<title>" << version << "</title>\n</head>\n<body>"
 << endl;
    ostream << "<h1>" << version << "</h1>\n";

    ostream << "<hr>\n<a name=\"Contents\"></a><h2>Contents</h2>\n<ol>\n";
    for (size_t i=0; i<chapters.size(); i++)
        ostream << "<li><a href=\"#Chapter" << i+1 << "\">" << chapters[i].c_str
() << "</a></li>\n";
    ostream << "</ol>\n<hr>\n";

    ostream << sout.str();
    ostream << "</html>" << endl << "</body>" << endl;

    return 0;
}
  12-SEP-2024 08:22:29
minuet-jj) cxx/point=32/warn=(noinfo,disable=("comment")) gen_html.cpp
minuet-jj) link  gen_html.obj
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_tr
aitsIcEENS_11allocator32IcEEEC2ERKS5_
	defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11alloca
tor32IcEEEC5ERKS5_ 
	file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1 
	referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11all
ocator32IcEEEC2ERKS5_ 
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_tr
aitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
	defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11alloca
tor32IcEEEC5ERKS5_mmRKS4_ 
	file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1 
	referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11all
ocator32IcEEEC2ERKS5_mmRKS4_ 

What is going wrong?"
What does "group mismatch" mean?

Of course the 64-bit compilation works:

Code: Select all

minuet-jj) cxx/warn=(noinfo,disable=("comment")) gen_html.cpp
minuet-jj) link  gen_html.obj

regards
Jouk

[hb]

Easy to spot

The compiler defines a weak symbol
_ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
in a section with the group signature
_ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_,
The RTL defines this symbol in a group with signature _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC5ERKS5_mmRKS4_.
You probably immediately see that there is a "2" in what the compiler generates and a "5" in what the RTL provides.

Assuming that such a symbol should be defined by a group with the same signature: the RTL is wrong.

It's a warning. As you should be able to see in a MAP file, the linker uses the symbol defined in the object module. That is, it includes the (code from the) group that is in the object module. If there is no mismatch, the linker would resolve the symbol from the RTL, which is what you usually want.

[jreagan]

We have fixed that for the next kit (soon). For these latest compilers, we changed "which" string we use depending on pointer-size. There are now std::string and std::string32 classes with the compiler picking the right one based on usage. There is lots of code that allocates a std::string and then expects to build an OpenVMS descriptor pointing to the data in that object (either the data or c_str methods). Having ALL strings in 64-bit heap breaks that code. The error you saw is that we forgot an "extern" in one of the string class headers and the compiler got confused on instantiating the objects.

[martinv]

Problem still exists in VSI C++ V10.1-2 (released 09-Oct-2024), seen in ZMQ 32-bit shareable build.

Code: Select all

LINK    /SHAREABLE=[.X86_64-32]zmqshr32.exe []zmq_share.opt/OPTION,[.X86_64-32]libzmq32.olb/LIBRARY,SYS$DISK:[]tcpip.opt/OPTION
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_
        defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC5ERKS5_
        file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1
        referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_
%ILINK-W-GRPMIS, group mismatch for symbol _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_
        defining group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC5ERKS5_mmRKS4_
        file: SYS$COMMON:[SYSLIB]LIBCXX.EXE;1
        referencing group: _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_11allocator32IcEEEC2ERKS5_mmRKS4_

[joukj]

My orinial reproducer now seems to work.

Martinv : can you give the source code of what you are doing?

[martinv]

Jouk,

this is the ZeroMQ 4.1.8 code directly from GitHub, of course accompanied by hand-crafted DESCRIP.MMS and platform.hpp .

[jreagan]

If you can submit a case with support, we'll get on that quickly

John

[tigran.sargsyan]

Hi Martinv,

Could you please provide a small reproducer?
Is it possible that in linking process there is an object file that was generated by older compiler (released before August 2024)?

Regards
Tigran