Skip to content

Segmentation fault when using static binaries on unix. #364

Closed

Description

@zhayka52

There appears to be a threading issues within the IXDNSLookup.cpp when IXWebSocket is linked as a static library.

To recreate the problem, use the main.cpp example code, and change the server URL to 127.0.0.1:8000 or any other address that does not have a running websocket server. The target platform is Ubuntu 20.04.

Using the standard dynamic linked binary, you will see something like

Connecting to ws://127.0.0.1:8000...

Connection error: Unable to connect to 127.0.0.1 on port 8000, error: Connect error: Connection refused

This message will repeat over and over again as expected.

Now, create a statically linked version of the same main.cpp

Connecting to ws://127.0.0.1:8000...

Connection error: Unable to connect to 127.0.0.1 on port 8000, error: Connect error: Connection refused
Segmentation fault (core dumped)

Upon closer examination, the culprit appears to be DNS related. Given what has been reported in ticket 362, there might be something going with resource allocation that requires more investigation.

This change that seems to work.

struct addrinfo* DNSLookup::resolve(std::string& errMsg,
                                    const CancellationRequest& isCancellationRequested,
                                    bool cancellable)
{
    // THIS does not work
    ///return cancellable ? resolveCancellable(errMsg, isCancellationRequested)
                       //: resolveUnCancellable(errMsg, isCancellationRequested);

   // THIS does work
    return resolveUnCancellable(errMsg, isCancellationRequested);
}

Activity

  1. bsergean commented on Feb 17, 2022

    @bsergean
    Collaborator

    I can't reproduce that on my mac with clang, and on ubuntu linux 21.10 with gcc-11.

    I used the ws example. I just found a bug with the retry logic which I just fixed.

  2. bsergean commented on Feb 17, 2022

    @bsergean
    Collaborator

    Can you dump the coredump, or run this in gdb ?

  3. zhayka52 commented on Feb 17, 2022

    @zhayka52
    Author

    This bug only shows up when the binary is statically linked. I can probably test on other platforms and see if I can the same results.

    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    ixwebsocket/11.3.2 linux ssl/mbedtls 2.16.5 zlib 1.2.11
    Connecting to ws://127.0.0.1:8000...
    [New Thread 0x7ffff7ffa700 (LWP 26793)]

    Connection error: Unable to connect to 127.0.0.1 on port 8000, error: Connect error: Connection refused

    Thread 2 "ws://127.0.0.1:" received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 0x7ffff7ffa700 (LWP 26793)]
    0x0000000000000000 in ?? ()
    (gdb) where
    #0 0x0000000000000000 in ?? ()
    #1 0x000000000040e5b1 in ix::WebSocket::checkConnection(bool) ()
    #2 0x000000000040f608 in ix::WebSocket::run() ()
    #3 0x0000000000537dc4 in execute_native_thread_routine ()
    #4 0x000000000056b139 in start_thread (arg=)
    at pthread_create.c:477
    #5 0x0000000000608ac3 in clone ()

  4. zhayka52 commented on Feb 17, 2022

    @zhayka52
    Author

    This is also confirmed on Centos 7.9.

    gdb) run

    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib64/libthread_db.so.1".
    ixwebsocket/11.4.0 linux nossl
    Connecting to ws://echo.websocket.org...
    [New Thread 0x7ffff7ffb700 (LWP 3080)]

    [New Thread 0x7ffff77f9700 (LWP 3081)]
    [Thread 0x7ffff77f9700 (LWP 3081) exited]
    Connection error: Expecting status 101 (Switching Protocol), got 200 status connecting to ws://echo.websocket.org, HTTP Status line: HTTP/1.1 200 OK

    [New Thread 0x7ffff77f9700 (LWP 3082)]

    Program received signal SIGSEGV, Segmentation fault.
    [Switching to Thread 0x7ffff77f9700 (LWP 3082)]
    0x00007ffff6de15ed in internal_getent () from /lib64/libnss_files.so.2
    Missing separate debuginfos, use: debuginfo-install glibc-2.17-325.el7_9.x86_64
    (gdb)

    I had checked out the latest build from github and made the follow changes

    
    diff --git a/CMakeLists.txt b/CMakeLists.txt
    index e7341bc..7e09a50 100644
    --- a/CMakeLists.txt
    +++ b/CMakeLists.txt
    @@ -12,7 +12,7 @@ set (CMAKE_CXX_STANDARD 11)
     set (CXX_STANDARD_REQUIRED ON)
     set (CMAKE_CXX_EXTENSIONS OFF)
     
    -option (BUILD_DEMO OFF)
    +set (BUILD_DEMO ON)
     
     if (${CMAKE_SYSTEM_NAME} MATCHES "Linux")
       set(CMAKE_POSITION_INDEPENDENT_CODE ON)
    @@ -233,6 +233,7 @@ endif()
     
     option(USE_ZLIB "Enable zlib support" TRUE)
     
    +set (USE_ZLIB false)
     if (USE_ZLIB)
       # This ZLIB_FOUND check is to help find a cmake manually configured zlib
       if (NOT ZLIB_FOUND)
    @@ -315,5 +316,5 @@ endif()
     
     if (BUILD_DEMO) 
       add_executable(demo main.cpp)
    -  target_link_libraries(demo ixwebsocket) 
    +  target_link_libraries(demo ixwebsocket -static) 
     endif()
    diff --git a/main.cpp b/main.cpp
    index 8512537..4278c97 100644
    --- a/main.cpp
    +++ b/main.cpp
    @@ -28,7 +28,8 @@ int main()
     
         // Connect to a server with encryption
         // See https://machinezone.github.io/IXWebSocket/usage/#tls-support-and-con
    -    std::string url("wss://echo.websocket.org");
    +    //std::string url("wss://echo.websocket.org");
    +    std::string url("ws://echo.websocket.org");
         webSocket.setUrl(url);
     
         std::cout << ix::userAgent() << std::endl;
    
    
  5. bsergean commented on Feb 28, 2022

    @bsergean
    Collaborator

    Is the executable a static executable ?

    What does ldd my_exe says ?

  6. zhayka52 commented on Feb 28, 2022

    @zhayka52
    Author

    The binary is statically linked.

    demo: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=5f9873590e43f36045261afb005075298df00dbe, for GNU/Linux 3.2.0, with debug_info, not stripped

    Can you try to replicate the problem using the git diff output provided in this thread? This issue can be consistently reproduced in at least Ubuntu and CentOS.

    While a work around has been found, it might significantly impact DNS performance. The current design appears to allow more efficient lookup by cancelling duplicated calls. Is that what the code is doing?

  7. bsergean commented on Mar 24, 2022

    @bsergean
    Collaborator

    I see. I never tested fully statically linked binaries. I've seen odd threading behavior with threading (pthread) and static binaries so this isn't too surprising.

    The code allows to 'cancel' a hanged dns query, it isn't really for performance reason that the dns/getaddrinfo happens on a background thread. On Mobile platform blocking the main thread is forbidden so this is why we're doing this.

    Maybe a CMake option would be the best way to support this, so that you don't have to use a fork of the library. Or we can try to see if there's a simple modification we can make to make this work.

    Sometimes the ordering of the libraries is important, not sure it's the case here. Are you building glibc yourself ?

  8. changed the title [-]Segmentation fault when using static libraries[/-] [+]Segmentation fault when using static binaries on unix.[/+] on Mar 24, 2022
  9. zhayka52 commented on Mar 25, 2022

    @zhayka52
    Author

    gLibC is the default one from the OS. If the resolveUnCancellable can be used safely used in all system other than Android, then maybe #ifdef (ANDROID) #else block is the easiest way to handle this condition.

  10. bsergean commented on Apr 30, 2022

    @bsergean
    Collaborator

    This code has been running fine on Android, so it's really a problem when making static binaries.

    On Android typically native code will compiled as a shared library, which is why this problem is avoided.

  11. github-actions commented on Jun 30, 2022

    @github-actions

    Stale issue message

  12. bsergean commented on Nov 17, 2026

    @bsergean
    Collaborator

    I've opened #656 with the CMake fallback discussed above:

    cmake -S . -B build -DUSE_ASYNC_DNS=OFF

    This makes hostname resolution synchronous, so a blocking OS resolver can delay stop(). The default remains asynchronous and cancellable. Numeric IPv4/IPv6 addresses are now resolved inline without creating a worker in either mode.

    The PR also removes a raw this pointer from the detached DNS worker: it accesses lookup state only after locking the weak pointer. Cancellation is checked before starting a lookup and after resolving numeric addresses.

    Validation includes the normal unit suite, both resolver modes, and a fully statically linked Linux executable using the CMake-built library with USE_ASYNC_DNS=OFF (30 repeated numeric/localhost lookups). An AddressSanitizer/UndefinedBehaviorSanitizer probe also completed 50 deliberately delayed workers after their lookups were cancelled and destroyed.

    I haven't reproduced the original crash on the reporter's old Ubuntu/CentOS installations. This provides the requested supported workaround; it does not remove glibc's deployment requirements for statically linked getaddrinfo/NSS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions