Skip to content

Conversation

kovdan01
Copy link

@kovdan01 kovdan01 commented May 7, 2021

  • Enhance CMakeLists.txt files.
  • Move to Boost Test from Google Test to support pre-C++11 compilers.
  • Add more configurations on CI matrix builds.
  • Add support for the following types:
    • std::array<std::byte>
    • std::span<char>
    • std::span<unsigned char>
    • std::span<std::byte>

I plan to submit one more PR to add full support for std::span - a nice non-owning data range wrapper from C++20.

@kovdan01 kovdan01 mentioned this pull request May 8, 2021
@redboltz
Copy link
Contributor

redboltz commented May 9, 2021

I merged #952, So #951 should rebase from the new master. In addition, tests for added adaptors are required. Could you do that?

@kovdan01
Copy link
Author

kovdan01 commented May 9, 2021

Yes, I'll do that in a couple of days

@codecov-commenter
Copy link

codecov-commenter commented May 9, 2021

Codecov Report

Merging #951 (d04b1e7) into cpp_master (be4d971) will decrease coverage by 0.12%.
The diff coverage is 80.00%.

@@              Coverage Diff               @@
##           cpp_master     #951      +/-   ##
==============================================
- Coverage       85.74%   85.61%   -0.13%     
==============================================
  Files              72       79       +7     
  Lines            4847     5005     +158     
==============================================
+ Hits             4156     4285     +129     
- Misses            691      720      +29     

@kovdan01 kovdan01 force-pushed the add_span_support branch from 57cbe99 to 4c49c4a Compare May 10, 2021 12:42
@redboltz
Copy link
Contributor

I merged #954 and #955 . They helps detecting UB. Could you rebase again from the new master ?

Add support for the following types:
- `std::array<std::byte>`
- `std::span<char>`
- `std::span<unsigned char>`
- `std::span<std::byte>`
@kovdan01 kovdan01 force-pushed the add_span_support branch from 4c49c4a to 388891e Compare May 10, 2021 13:07
@kovdan01
Copy link
Author

Could you rebase again from the new master ?

Done.

@kovdan01
Copy link
Author

kovdan01 commented May 10, 2021

While trying to add tests for std::span, I have noticed that setting -std=c++20 breaks some tests build:

  • A lot of members of std::allocator are deleted since C++20: https://en.cppreference.com/w/cpp/memory/allocator, which causes errors in msgpack's custom test::allocator.
  • New "spaceship" operator from C++20 (operator<=>) makes some comparisons ambiguous. Example error:
    test/object_with_zone.cpp:1115:46: error: ambiguous overload for ‘operator==’ (operand types are ‘std::enable_if<true, msgpack::v1::type::ext>::type’ {aka ‘msgpack::v1::type::ext’} and ‘msgpack::v1::type::ext_ref’)
    

It looks like that adding C++20 support will not be trivial. I'll do it, but it'll take some time (maybe, a week or more).
Actually, it is not related to adding new adaptors, that's why I think that it'll be better to create a new PR for this. Don't you mind?

@redboltz
Copy link
Contributor

There is no C++20 complete support (language feature and standard libraries) compiler yet.
https://en.cppreference.com/w/cpp/compiler_support

Some of features are already implemented but from my experience, it is usually unstable. I'd like to support C++20 feature after at least one complete support compiler is released.
If some problem would happen, it is difficult to check the problem is caused by msgpack-c implementation, compiler, and/or standard library implementation.
It's my recommendation. But if some feature (mostly adding simple type adaptor) is simple enough, then I accept adding the adaptor. The compiler update would often cause the error in the future, at that time I expect proactive cooperation to fix the issue.

@kovdan01
Copy link
Author

kovdan01 commented May 10, 2021

You are right - none of the compilers has a complete C++20 support (and we might not get it until... 2023?). The problem is that if we don't pass -std=c++20 flag directly (for example, we pass -std=c++17), <span> header is not found, and we can't compile tests for it. But if we enable -std=c++20, some other tests fail to compile. There are some possible solutions:

  1. Set compiler flags for span test separately - let other tests have -std=c++17, and span tests have -std=c++20. I find that it is quite strange and messy and don't like this variant.
  2. Delete all span-related stuff until older codebase is ready to be compiled with -std=c++20 (in terms of this PR it means that we will leave only std::array<std::byte> adaptor). If you do not like the next variant, I can do this and re-submit span- and C++20-related stuff in a separate PR later (when you'll be ready to accept it).
  3. Fix C++20-related errors in old code. Actually, I have already done it in a separate branch, and here you can see the result. The compiler errors could be divided into 2 types:
    • Allocator-related errors. They happen because some functions and types were removed since C++20 (see, for example, this). But they were already deprecated since C++17, and alternatives for them (e.g. std::allocator_traits) exist since C++11 and are completely stable. Note that to fix this we do not need to change the library code: all fixes are applied to tests.

    • operator==-related errors. C++20 introduces a new three-way comparison operator and a bunch of new rules for comparison operators overaload resolution. Citation from N4868 C++ Standard Draft:

      For the relational (7.6.9) and three-way comparison (7.6.8) operators, the rewritten candidatesalso include a synthesized candidate, with the order of the two parameters reversed, for eachnon-rewritten candidate for the expressiony <=> x

      Because of synthesized overloads with reversed order of parameters, we got errors like this:

      test/object_with_zone.cpp:1115:46: error: ambiguous overload for ‘operator==’ (operand types are ‘std::enable_if<true, msgpack::v1::type::ext>::type’ {aka ‘msgpack::v1::type::ext’} and ‘msgpack::v1::type::ext_ref’)
      

      They occur because ext can be implicitly converted to ext_ref, and ext_ref can be implicitly converted to ext. When we try to compare ext and ext_ref, compiler has two options: (1) convert ext_ref to ext and call ext::operator==(const ext&); (2) reverse arguments order, convert ext to ext_ref and call ext_ref::operator==(const ext_ref&). This behavior is correct and predictable according to the standard. To fix the issue, we can either explicitly define comparison operators between ext and ext_ref or mark ext(const ext_ref&) constructor as explicit.

From my perspective, the 3rd variant is applicable - changes are only related to stable and predictable parts of language and standard library. If you think that it is not - I suggest to use the 2nd option.

@redboltz
Copy link
Contributor

Sorry for my late reply, I had busy days. 3rd option seems good to me.
Is this PR 3rd approach ?

Anyway, could you update (rebase) the PR ? The CI reports an error and I have fixed it by #957.

@kovdan01
Copy link
Author

Yes, I have updated the PR (rebased from master & added std::span adaptor tests among with c++20 related fixes), now it is 3rd approach. Now I will focus on enhancing CI scripts, particularly, I will add a build variant with C++20 enabled (of course, I will not delete older variants). Without this enhancement some new code just will not be compiled and tested during CI builds.

@redboltz
Copy link
Contributor

Thanks. I will wait your update.

- Enhance CMakeLists.txt files.
- Move to Boost Test from Google Test to support pre-C++11 compilers.
- Add more configurations on CI matrix builds.
- Other minor fixes
@kovdan01
Copy link
Author

kovdan01 commented Jul 1, 2021

@redboltz Hello, sorry for a long delay. I have added appropriate build configurations on CI, and… accidently did a lot of other changes that I believe are important and helpful for the project. The initial motivation was that I simply did not catch the pattern – why that 10 particular sets of options were considered in CI matrix build, and others were not (see this).

If you think that some changes need to be rejected or should be separated into other PR’s – please let me know. Here is a brief overview of what was done:

  • Enhance CMakeLists.txt files. This includes:
    • Using target_ versions of commands instead of the global ones (e.g. link_libraries). These global commands are considered legacy and should not be used in CMake 3.0 and above.
    • Delete include_directories-like commands where not needed – when you write target_link_libraries(your_target library::name), essential library include directories will already be used.
    • Use nice cmake options to avoid compiler-specific stuff, for instance, set C++ standard version just like this: SET (CMAKE_CXX_STANDARD 11)
    • Add an install target for msgpackc-cxx library. Previous library release supports a “canonical” CMake-way of using it – with find_package(msgpack REQUIRED) and target_link_libraries(target msgpackc-cxx). Currently library in cpp_master does not support it. Although msgpackc-cxx is a header-only library, it is a good idea to create a CMake package for it.
    • Other minor fixes.
  • Move to Boost Test from Google Test. The problem with GTest is that latest version that supports pre-C++11 compilers is 1.8.x (2018), while actual version is 1.11.0. Meanwhile, msgpack supports pre-C++11 compilers, and, in my opinion, it is not cool when you are not able to compile C++03 tests without compiling an outdated version of GTest from sources itself instead of installing a pre-compiled library with your package manager (it’s true for macOS and Windows too, not just for Linux). Boost Test supports old compilers, and msgpack depends on Boost already – that’s why I think that Boost Test is a nice choice.
  • Add more configurations on CI matrix builds. Needless to explain.
  • Enhance C++ version detection. Using __cplusplus macro on MSVC is not a good idea – it is set to 199711L unless /Zc:__cplusplus is specified (see https://docs.microsoft.com/en-us/cpp/build/reference/zc-cplusplus). Instead, it is recommended to use _MSVC_LANG. See MSGPACK_CPP_VERSION macro definition in msgpack/cpp_version.hpp header.
  • Other minor fixes.

@kovdan01 kovdan01 changed the title Add support for more binary buffer types Modernize codebase Jul 1, 2021
@kovdan01 kovdan01 force-pushed the add_span_support branch from a1a59fc to 3365d76 Compare July 1, 2021 18:06
@kovdan01 kovdan01 force-pushed the add_span_support branch 2 times, most recently from 4dca6a5 to 754aba2 Compare July 7, 2021 09:39
VERSION_GREATER_EQUAL is available since cmake 3.7 only
@kovdan01 kovdan01 force-pushed the add_span_support branch from 754aba2 to 25d6f84 Compare July 7, 2021 15:58
@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

@kovdan01 Thank you for the great work !
I just started a review.

Before I post the review comments, I answer your question.

I have added appropriate build configurations on CI, and… accidently did a lot of other changes that I believe are important and helpful for the project. The initial motivation was that I simply did not catch the pattern – why that 10 particular sets of options were considered in CI matrix build, and others were not (see this).

It is based on https://en.wikipedia.org/wiki/Orthogonal_array_testing but manually configured. The motivation is how to shorten the test time. The resource (especially macos) is limited. To be honest, the criteria of picking case in ambiguous.
It seems that your PR do the all combination of tests using matrix. Testing all combination is ideal if the test time is practical.
I created the new branch based on this PR just only for measure the time.
https://github.com/redboltz/msgpack-c/actions/runs/1106831751
If CI finished in practical time, I will apply your approach.

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

I'm watching https://github.com/redboltz/msgpack-c/actions/runs/1106831751 progress.
macos and windows builds run parallelly. But linux builds not run. Maybe enqueued.
When I checked Github Actions behavior, macos, linux, and windows have separated resorces if I remember correctly.

Why linux build is not started ? It seems that macos and linux share (vm or container) resources. Is this right?
Also I noticed that 5 builds run parallelly on macos.

Maybe we should check up to date Github Actions document.

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

I found this post (3 months ago).
https://stackoverflow.com/questions/67295639/how-many-simultaneous-builds-can-be-run-on-github-actions

Maybe total limit is 20.
In our case,

macos linux windows total
running 5 0 3 8

My expectation as follows:

macos linux windows total
running 5 12 3 20

You can test on your account using the following modification:
redboltz@3546c3f

IMPORTANT
I guess that the total build limit might be applied by account. I'm not sure by account or by project (repository).

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

It seems that Windows build "Build dependencies" takes a long time. It is about 30 minutes.

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

The current CI on the PR should be improved. The current total time in unacceptable. So far, only 36 builds are finished (they all macos) and takes 34 minutes. Maybe total time will be several hours. I continue measuring.

Problem and possible solution candidate

  • Linux build doesn't run.
    • Need to fix some how.
  • Even if Linux builds works parallelly, macos could be a bottle neck.
    • Give up the all matrix build. It is covered by the Linux build. For macos, pick up typical case some how.
  • Windows build dependencies takes a long time.
    • I'm not sure what is happening. Use pre-built library as long as you can do.

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

Expected total duration is less that 30 minutes.
https://github.com/msgpack/msgpack-c/actions/runs/825255984

It is a kind of turn around time that someone post a PR and got the first response (from the CI).

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

2 hours later, macos buils were finished. linux build started.
It seems that 17 parallel builds are running.

NOTE: windows 3 builds were all finished.

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

time macos linux windows
0:00:00 start (5) queued start(3)
0:35:00 queued finished
2:00:00 finished start(20)
5:00:00 finished

macos build spent 2 hours.
linux build spent 3 hours.
windows build spent 35 minutes.

@redboltz
Copy link
Contributor

redboltz commented Aug 7, 2021

The PR linux build has the following matrix:

matrix
        arch: [32, 64]
        cxx: [98, 11, 14, 17, 20]
        sanitize: ["no", "undefined"]
        api: [1, 2, 3]
        char_sign: ["unsigned", "signed"]
        x3_parse: ["OFF", "ON"]

Is this from

if [ ${{ matrix.pattern }} == 0 ]; then
?

I think that you need to pick up some practical patterns as follows:

index api cxx arch char_sign sanitize x3_parse
0 3 17 64 signed undefined ON
1 2 20 32 unsigned no OFF
2 ...
...

if [ ${{ matrix.pattern }} == 0 ]; then
was the result of the pick up.

I think that at most 5 build for macos, at most 15 builds for linux, and windows 3 builds don't need to update.
Windows build requires shrink dependency build time.

Then, the total CI time could be practical.

@kovdan01
Copy link
Author

kovdan01 commented Aug 9, 2021

@redboltz Thanks for your reply! I see your point – builds took really long time. Updated the PR and left 12 Linux, 5 macOS and 3 Windows builds.

Regarding that

Windows build requires shrink dependency build time.

I did not change this part – dependencies are installed via vcpkg as before. Github actions cache is used to cache those dependencies, so builds will be fast after first run (while cache is valid). Anyway, Windows builds are not slower than before at the moment.

@redboltz
Copy link
Contributor

Thank you for updating the PR. I'm very busy now so I couldn't check the PR soon but I will check it when things have calmed down .

Just a comment for CI running time.

The PR spent 51m 49s. It still long.

See the following results:
https://github.com/msgpack/msgpack-c/actions/runs/845095110
https://github.com/msgpack/msgpack-c/actions/runs/825255984

They are recent merged PR.
You can see it as follows:

  1. Open some PR
  2. Click Checks TAB
  3. Click CI in the Left sidebar.
  4. You can see Total duration
  5. Click linux, macos, or windows folder like icon, then you can see individual time.

You can find the bottle neck.

@redboltz
Copy link
Contributor

It seems that you updated CMakeLists.txt.
README.md said that cmake >= 3.0.0. Is this still correct?

@redboltz
Copy link
Contributor

Thank you for updating the PR. I'm very busy now so I couldn't check the PR soon but I will check it when things have calmed down .

Just a comment for CI running time.

The PR spent 51m 49s. It still long.

See the following results:
https://github.com/msgpack/msgpack-c/actions/runs/845095110
https://github.com/msgpack/msgpack-c/actions/runs/825255984

They are recent merged PR.
You can see it as follows:

  1. Open some PR
  2. Click Checks TAB
  3. Click CI in the Left sidebar.
  4. You can see Total duration
  5. Click linux, macos, or windows folder like icon, then you can see individual time.

You can find the bottle neck.

Could you load balance the parallel CI builds ?
Some of builds require short time, and some of tasks require long time.

I realized that

# matrix config
if [ ${{ matrix.pattern }} == 0 ]; then
export CXX=clang++
ACTION="ci/build_cmake.sh"
export ARCH="64"
export CHAR_SIGN="unsigned"
export API_VERSION="2"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 1 ]; then
export CXX=clang++
ACTION="ci/build_cmake.sh"
export ARCH="32"
export CHAR_SIGN="signed"
export API_VERSION="2"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 2 ]; then
export CXX=clang++
ACTION="ci/build_cmake.sh"
export CXX17="ON"
export ARCH="64"
export CHAR_SIGN="signed"
export API_VERSION="3"
export X3_PARSE="ON"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 3 ]; then
export CXX=clang++
ACTION="ci/build_cmake.sh"
export CXX17="ON"
export ARCH="32"
export CHAR_SIGN="unsigned"
export API_VERSION="2"
fi
if [ ${{ matrix.pattern }} == 4 ]; then
export CXX=g++
ACTION="ci/build_cmake.sh"
export CXX17="ON"
export ARCH="64"
export CHAR_SIGN="signed"
export API_VERSION="2"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 5 ]; then
export CXX=g++
ACTION="ci/build_cmake.sh"
export CXX17="ON"
export ARCH="32"
export CHAR_SIGN="unsigned"
export API_VERSION="3"
export X3_PARSE="ON"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 6 ]; then
export CXX=g++
ACTION="ci/build_cmake.sh"
export ARCH="64"
export CHAR_SIGN="unsigned"
export API_VERSION="2"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 7 ]; then
export CXX=g++
ACTION="ci/build_cmake.sh"
export ARCH="32"
export CHAR_SIGN="signed"
export API_VERSION="1"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 8 ]; then
export CXX=g++
ACTION="ci/build_cmake.sh"
export ARCH="32"
export CHAR_SIGN="signed"
export API_VERSION="2"
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 9 ]; then
export CXX=clang++
ACTION="ci/build_regression.sh"
export ARCH="64"
export SAN="UBSAN"
export MSGPACK_FUZZ_REGRESSION="ON"
export CTEST_OUTPUT_ON_FAILURE=1
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi
if [ ${{ matrix.pattern }} == 10 ]; then
export CXX=clang++
ACTION="ci/build_regression.sh"
export ARCH="64"
export SAN="ASAN"
export MSGPACK_FUZZ_REGRESSION="ON"
export CTEST_OUTPUT_ON_FAILURE=1
export SANITIZE="-fsanitize=undefined -fno-sanitize-recover=all"
fi

is the result of load balancing.

@redboltz
Copy link
Contributor

I tried to build the branch on my local environment.

[kondo@archboltz] $ rm -rf build                                                                                                                  [kovdan01-add_span_support][~/work/msgpack-c]
[kondo@archboltz] $ mkdir build                                                                                                                   [kovdan01-add_span_support][~/work/msgpack-c]
[kondo@archboltz] $ cd build                                                                                                                      [kovdan01-add_span_support][~/work/msgpack-c]
[kondo@archboltz] $ cmake ..                                                                                                                [kovdan01-add_span_support][~/work/msgpack-c/build]
-- The CXX compiler identification is GNU 11.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /usr/lib64/cmake/Boost-1.76.0/BoostConfig.cmake (found version "1.76.0")
-- Found Doxygen: /usr/bin/doxygen (found version "1.9.1") found components: doxygen dot
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kondo/work/msgpack-c/build
[kondo@archboltz] $ make                                                                                                                    [kovdan01-add_span_support][~/work/msgpack-c/build]
[kondo@archboltz] $                                                                                                                         [kovdan01-add_span_support][~/work/msgpack-c/build]

But nothing happens. On cpp_master, started building after make.

@redboltz
Copy link
Contributor

redboltz commented Aug 17, 2021

It seems that you replaced gtest with boost test. It is nice. Great work!

I found the following description in the README.md.
I think that it is still correct but for testing, I guess that boost library build and link are required.
Could you add something about that to the README.md ?

Dependency

msgpack-c requires boost library.
msgpack-c depends on only boost headers. You don't need to link boost libraries.

@kovdan01
Copy link
Author

kovdan01 commented Aug 17, 2021

@redboltz

I tried to build the branch on my local environment. But nothing happens.

MSGPACK_BUILD_TESTS is OFF by default, so to build tests you need to pass -DMSGPACK_BUILD_TESTS=ON to cmake configuration command. I can change the default value to ON if you want - just thought that when a person wants to install a header-only library, they do not want to build its tests.

To install the library, run make install (or cmake --build . --target install - that is considered even more preferable because works not only with make but with other generators like ninja or visual studio)

README.md said that cmake >= 3.0.0. Is this still correct?

Currently cmake 3.1.0 or newer is required. Fixed it in readme.

I think that it is still correct but for testing, I guess that boost library build and link are required.
Could you add something about that to the README.md ?

Updated. Thanks!

Could you load balance the parallel CI builds ?

Yes, I'll do that and will write about results.

By the way, I briefly described the changes in the comment on Jul 1 (maybe you did not notice that). For example, it is described what particularly was changed in cmake files (and not only that).

@redboltz
Copy link
Contributor

Could you load balance the parallel CI builds ?

Yes, I'll do that and will write about results.

Thanks. This is not so creative work but it is needed to get result in practical time.

By the way, I briefly described the changes in the comment on Jul 1 (maybe you did not notice that). For example, it is described what particularly was changed in cmake files (and not only that).

I confirmed #951 (comment) , thanks.

@kovdan01
Copy link
Author

@redboltz Previously I used both gcc and clang for every Linux build. It seems that there was no way to enhance build time while using both compilers, that's why now I use gcc for 6 Linux builds and clang for the remaining 6 Linux builds. Currently the same approach is used in cpp_master (only one compiler per build).

It seems that after this fix build time is OK (at least as for me).

@redboltz
Copy link
Contributor

Thank you for updating. It looks nice build time result. I will check details this weekend.

@redboltz
Copy link
Contributor

@kovdan01 Thank you for updating the PR.
Finally, all issues have been solved. I appreciate your great effort!

@redboltz redboltz merged commit 76f5af0 into msgpack:cpp_master Aug 29, 2021
@kovdan01
Copy link
Author

@redboltz Thank you for reviewing such a huge PR!

@kovdan01 kovdan01 mentioned this pull request Aug 30, 2021
@redboltz redboltz added the C++ label Sep 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants