Skip to content

Add job for testing simsimd dynamic dispatch to nightly build-and-deploy workflow #5007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Mar 19, 2025
Merged

Add job for testing simsimd dynamic dispatch to nightly build-and-deploy workflow #5007

merged 8 commits into from
Mar 19, 2025

Conversation

royi-luo
Copy link
Contributor

@royi-luo royi-luo commented Mar 6, 2025 •
edited
Loading

Description

Add a job to the nightly build-and-deploy workflow that tests to see if simsimd's dynamic dispatch is working as expected. The job will download the nightly build of the kuzu shell and run the test on a newly-selected CI runner. Ideally this would over time test out all the combinations of build + execution machines.

Contributor agreement

@royi-luo royi-luo self-assigned this Mar 6, 2025
@royi-luo royi-luo changed the title Add nightly CI workflow for testing simsimd dynamic dispatch Add job for testing simsimd dynamic dispatch to nightly build-and-deploy workflow Mar 6, 2025
@royi-luo royi-luo force-pushed the royi/simsimd-dispatch-tests branch from 967b01b to ed0d140 Compare March 6, 2025 18:53
royi-luo
royi-luo commented Mar 6, 2025
View reviewed changes
@royi-luo royi-luo force-pushed the royi/simsimd-dispatch-tests branch from 156cda3 to 37146db Compare March 7, 2025 15:01
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 7, 2025

Benchmark Result

Master commit hash: 8cfb6553038cd144cfb0211783441aac326e4919
Branch commit hash: f46df2e10aeebe87555cec397e5b21bb40172ce0

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 725.38 748.66 -23.28 (-3.11%)
aggregation q28 6524.48 7316.74 -792.26 (-10.83%)
filter q14 128.13 155.57 -27.44 (-17.64%)
filter q15 132.97 151.37 -18.40 (-12.16%)
filter q16 341.66 351.07 -9.41 (-2.68%)
filter q17 447.95 453.38 -5.44 (-1.20%)
filter q18 1969.86 1921.66 48.21 (2.51%)
filter zonemap-node 91.45 97.36 -5.90 (-6.06%)
filter zonemap-node-lhs-cast 89.62 97.47 -7.85 (-8.06%)
filter zonemap-node-null 88.78 97.22 -8.44 (-8.69%)
filter zonemap-rel 5672.72 5742.78 -70.06 (-1.22%)
fixed_size_expr_evaluator q07 681.28 695.10 -13.82 (-1.99%)
fixed_size_expr_evaluator q08 970.67 978.62 -7.95 (-0.81%)
fixed_size_expr_evaluator q09 964.87 974.26 -9.39 (-0.96%)
fixed_size_expr_evaluator q10 255.04 270.85 -15.82 (-5.84%)
fixed_size_expr_evaluator q11 256.03 270.22 -14.19 (-5.25%)
fixed_size_expr_evaluator q12 233.37 248.16 -14.78 (-5.96%)
fixed_size_expr_evaluator q13 1593.92 1572.27 21.65 (1.38%)
fixed_size_seq_scan q23 115.75 126.42 -10.67 (-8.44%)
join q29 731.31 962.57 -231.26 (-24.03%)
join q30 1686.65 2178.69 -492.04 (-22.58%)
join q31 5.12 4.82 0.30 (6.14%)
join SelectiveTwoHopJoin 44.30 45.07 -0.77 (-1.71%)
ldbc_snb_ic q35 9.74 10.16 -0.42 (-4.10%)
ldbc_snb_ic q36 97.02 101.01 -3.99 (-3.95%)
ldbc_snb_is q32 4.45 3.94 0.51 (13.04%)
ldbc_snb_is q33 12.25 13.69 -1.44 (-10.51%)
ldbc_snb_is q34 1.17 1.15 0.01 (1.27%)
multi-rel multi-rel-large-scan 1421.68 1415.65 6.03 (0.43%)
multi-rel multi-rel-lookup 12.24 11.19 1.05 (9.37%)
multi-rel multi-rel-small-scan 203.06 209.17 -6.11 (-2.92%)
order_by q25 132.51 138.81 -6.30 (-4.54%)
order_by q26 448.64 464.55 -15.90 (-3.42%)
order_by q27 1383.28 1409.30 -26.02 (-1.85%)
recursive_join recursive-join-bidirection 297.89 311.90 -14.02 (-4.49%)
recursive_join recursive-join-dense 7163.30 7142.15 21.15 (0.30%)
recursive_join recursive-join-path 23242.56 23743.69 -501.14 (-2.11%)
recursive_join recursive-join-sparse 630.38 898.59 -268.21 (-29.85%)
recursive_join recursive-join-trail 7052.39 7074.84 -22.45 (-0.32%)
scan_after_filter q01 169.13 185.28 -16.15 (-8.72%)
scan_after_filter q02 154.38 166.52 -12.14 (-7.29%)
shortest_path_ldbc100 q37 87.07 102.05 -14.99 (-14.69%)
shortest_path_ldbc100 q38 386.35 336.07 50.28 (14.96%)
shortest_path_ldbc100 q39 57.51 65.48 -7.97 (-12.17%)
shortest_path_ldbc100 q40 320.08 406.31 -86.22 (-21.22%)
var_size_expr_evaluator q03 2146.21 2113.38 32.83 (1.55%)
var_size_expr_evaluator q04 2264.23 2218.90 45.34 (2.04%)
var_size_expr_evaluator q05 2678.23 2721.70 -43.46 (-1.60%)
var_size_expr_evaluator q06 1346.37 1359.69 -13.32 (-0.98%)
var_size_seq_scan q19 1440.85 1435.45 5.40 (0.38%)
var_size_seq_scan q20 2650.31 2684.94 -34.63 (-1.29%)
var_size_seq_scan q21 2242.85 2246.30 -3.44 (-0.15%)
var_size_seq_scan q22 130.01 128.73 1.28 (0.99%)

@codecov Codecov
Copy link

codecov bot commented Mar 7, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.99%. Comparing base (f8f1517) to head (61778d2).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #5007   +/-   ##
=======================================
  Coverage   87.98%   87.99%           
=======================================
  Files        1418     1418           
  Lines       64597    64597           
  Branches     7627     7627           
=======================================
+ Hits        56838    56843    +5     
+ Misses       7730     7725    -5     
  Partials       29       29

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@royi-luo royi-luo force-pushed the royi/simsimd-dispatch-tests branch from 37146db to e5fef28 Compare March 14, 2025 20:02
royi-luo
royi-luo commented Mar 14, 2025
View reviewed changes
scripts/test-simsimd-dispatch.py
@@ -0,0 +1,45 @@
import gdb
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is the best place to put the testing script, let me know if there is a better place

@royi-luo royi-luo requested a review from mewim March 14, 2025 20:26
@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: 5cfad24b83046f5d6320e8eeb4658edc45945603
Branch commit hash: 0312cffc7aa8a031576aec7b53da347c3fae9f85

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 732.49 726.84 5.65 (0.78%)
aggregation q28 6638.49 6592.86 45.64 (0.69%)
filter q14 126.32 126.03 0.29 (0.23%)
filter q15 124.07 130.19 -6.13 (-4.71%)
filter q16 346.76 345.42 1.34 (0.39%)
filter q17 446.93 445.50 1.43 (0.32%)
filter q18 2005.31 1992.13 13.17 (0.66%)
filter zonemap-node 90.86 89.96 0.90 (1.00%)
filter zonemap-node-lhs-cast 90.34 88.72 1.61 (1.82%)
filter zonemap-node-null 90.84 88.68 2.16 (2.43%)
filter zonemap-rel 5630.13 5682.22 -52.08 (-0.92%)
fixed_size_expr_evaluator q07 679.32 681.79 -2.47 (-0.36%)
fixed_size_expr_evaluator q08 966.09 968.32 -2.23 (-0.23%)
fixed_size_expr_evaluator q09 964.54 982.49 -17.95 (-1.83%)
fixed_size_expr_evaluator q10 254.41 259.36 -4.95 (-1.91%)
fixed_size_expr_evaluator q11 253.75 259.31 -5.56 (-2.14%)
fixed_size_expr_evaluator q12 232.89 232.95 -0.07 (-0.03%)
fixed_size_expr_evaluator q13 1568.33 1581.16 -12.84 (-0.81%)
fixed_size_seq_scan q23 106.84 110.60 -3.76 (-3.40%)
join q29 796.53 692.18 104.36 (15.08%)
join q30 1715.17 1620.06 95.11 (5.87%)
join q31 5.47 6.12 -0.65 (-10.62%)
join SelectiveTwoHopJoin 56.20 52.50 3.70 (7.05%)
ldbc_snb_ic q35 9.93 10.17 -0.24 (-2.32%)
ldbc_snb_ic q36 99.19 98.50 0.69 (0.70%)
ldbc_snb_is q32 3.44 3.19 0.25 (7.89%)
ldbc_snb_is q33 13.65 15.28 -1.62 (-10.63%)
ldbc_snb_is q34 1.22 1.33 -0.10 (-7.87%)
multi-rel multi-rel-large-scan 1663.87 1649.40 14.47 (0.88%)
multi-rel multi-rel-lookup 4.95 6.87 -1.92 (-27.92%)
multi-rel multi-rel-small-scan 200.41 191.79 8.62 (4.50%)
order_by q25 132.82 133.29 -0.47 (-0.35%)
order_by q26 463.51 451.33 12.18 (2.70%)
order_by q27 1395.00 1398.01 -3.01 (-0.22%)
recursive_join recursive-join-bidirection 291.31 294.89 -3.57 (-1.21%)
recursive_join recursive-join-dense 7232.74 7222.83 9.91 (0.14%)
recursive_join recursive-join-path 23533.52 23705.25 -171.73 (-0.72%)
recursive_join recursive-join-sparse 638.00 634.19 3.81 (0.60%)
recursive_join recursive-join-trail 7121.42 7123.99 -2.57 (-0.04%)
scan_after_filter q01 171.89 169.99 1.90 (1.12%)
scan_after_filter q02 157.41 155.38 2.02 (1.30%)
shortest_path_ldbc100 q37 94.10 85.04 9.06 (10.65%)
shortest_path_ldbc100 q38 326.60 318.40 8.20 (2.58%)
shortest_path_ldbc100 q39 61.05 50.82 10.22 (20.12%)
shortest_path_ldbc100 q40 350.70 360.86 -10.16 (-2.82%)
var_size_expr_evaluator q03 2096.43 2128.84 -32.41 (-1.52%)
var_size_expr_evaluator q04 2298.48 2284.35 14.14 (0.62%)
var_size_expr_evaluator q05 2733.02 2734.66 -1.65 (-0.06%)
var_size_expr_evaluator q06 1355.27 1335.68 19.59 (1.47%)
var_size_seq_scan q19 1459.33 1444.95 14.37 (0.99%)
var_size_seq_scan q20 2642.78 2633.13 9.65 (0.37%)
var_size_seq_scan q21 2262.00 2308.03 -46.04 (-1.99%)
var_size_seq_scan q22 124.45 127.85 -3.40 (-2.66%)

@royi-luo royi-luo marked this pull request as ready for review March 17, 2025 14:45
mewim
mewim approved these changes Mar 19, 2025
View reviewed changes
Copy link
Member

@mewim mewim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

andyfengHKU and others added 2 commits March 19, 2025 10:05
@andyfengHKU @royi-luo
Add dispatch test script
bb82d9f 
Update testing script

Update script

Remove testing script

Update rust build

Update tests so that it passes in wasm

Add testing script

Fix script

Fix indentation

Fix script

Fix indentation

Fix script

Add CI workflow for simsimd dispatch test

Move job to build and deploy workflow

Fix yaml

Add test trigger

Add dependency on build job

Move CI job to separate workflow
@royi-luo
Avoid building non-x86 binaries before running dispatch test
d8851c5
@royi-luo royi-luo force-pushed the royi/simsimd-dispatch-tests branch from e5fef28 to d8851c5 Compare March 19, 2025 14:11
@github-actions GitHub Actions
Copy link

Benchmark Result

Master commit hash: f8f1517ff8f82d5c6f2a4ed54dad9d6e74d25f8b
Branch commit hash: 39972b3a6b555ff57914585a5eac3e5f62a358a3

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 726.18 725.67 0.51 (0.07%)
aggregation q28 6601.62 6581.62 20.00 (0.30%)
filter q14 124.14 125.63 -1.49 (-1.18%)
filter q15 127.44 129.30 -1.86 (-1.44%)
filter q16 341.97 341.21 0.77 (0.22%)
filter q17 444.57 445.49 -0.93 (-0.21%)
filter q18 2000.22 1994.05 6.17 (0.31%)
filter zonemap-node 88.79 89.33 -0.53 (-0.60%)
filter zonemap-node-lhs-cast 88.95 89.82 -0.87 (-0.97%)
filter zonemap-node-null 88.67 90.71 -2.04 (-2.25%)
filter zonemap-rel 5712.71 5774.07 -61.36 (-1.06%)
fixed_size_expr_evaluator q07 680.19 682.75 -2.57 (-0.38%)
fixed_size_expr_evaluator q08 964.09 966.94 -2.85 (-0.29%)
fixed_size_expr_evaluator q09 967.83 963.20 4.63 (0.48%)
fixed_size_expr_evaluator q10 253.68 256.12 -2.44 (-0.95%)
fixed_size_expr_evaluator q11 253.25 255.77 -2.52 (-0.98%)
fixed_size_expr_evaluator q12 232.56 233.03 -0.47 (-0.20%)
fixed_size_expr_evaluator q13 1562.05 1563.00 -0.95 (-0.06%)
fixed_size_seq_scan q23 111.68 114.00 -2.32 (-2.03%)
join q29 749.31 694.26 55.06 (7.93%)
join q30 1615.72 1584.47 31.25 (1.97%)
join q31 5.77 5.51 0.25 (4.58%)
join SelectiveTwoHopJoin 41.04 48.82 -7.77 (-15.92%)
ldbc_snb_ic q35 9.29 10.32 -1.03 (-9.98%)
ldbc_snb_ic q36 94.95 93.25 1.70 (1.82%)
ldbc_snb_is q32 2.43 5.24 -2.81 (-53.63%)
ldbc_snb_is q33 13.51 11.99 1.52 (12.65%)
ldbc_snb_is q34 1.09 1.18 -0.09 (-7.79%)
multi-rel multi-rel-large-scan 1732.02 1698.53 33.49 (1.97%)
multi-rel multi-rel-lookup 10.88 11.04 -0.16 (-1.48%)
multi-rel multi-rel-small-scan 203.52 200.83 2.69 (1.34%)
order_by q25 129.10 130.01 -0.91 (-0.70%)
order_by q26 445.66 448.21 -2.55 (-0.57%)
order_by q27 1410.91 1406.05 4.86 (0.35%)
recursive_join recursive-join-bidirection 296.98 302.97 -5.99 (-1.98%)
recursive_join recursive-join-dense 7175.21 7154.89 20.32 (0.28%)
recursive_join recursive-join-path 23207.80 23072.60 135.20 (0.59%)
recursive_join recursive-join-sparse 628.41 634.47 -6.06 (-0.96%)
recursive_join recursive-join-trail 7131.71 7055.52 76.19 (1.08%)
scan_after_filter q01 172.40 168.48 3.92 (2.32%)
scan_after_filter q02 157.26 153.98 3.29 (2.13%)
shortest_path_ldbc100 q37 93.55 87.20 6.35 (7.29%)
shortest_path_ldbc100 q38 310.85 242.00 68.85 (28.45%)
shortest_path_ldbc100 q39 62.23 61.01 1.22 (2.00%)
shortest_path_ldbc100 q40 381.99 351.73 30.26 (8.60%)
var_size_expr_evaluator q03 2094.52 2137.31 -42.79 (-2.00%)
var_size_expr_evaluator q04 2294.63 2283.26 11.37 (0.50%)
var_size_expr_evaluator q05 2754.99 2738.81 16.18 (0.59%)
var_size_expr_evaluator q06 1360.64 1368.27 -7.63 (-0.56%)
var_size_seq_scan q19 1437.67 1441.43 -3.76 (-0.26%)
var_size_seq_scan q20 2677.81 2678.92 -1.10 (-0.04%)
var_size_seq_scan q21 2255.29 2257.67 -2.38 (-0.11%)
var_size_seq_scan q22 128.31 126.07 2.24 (1.78%)

@royi-luo royi-luo merged commit fb62bab into master Mar 19, 2025
28 checks passed
@royi-luo royi-luo deleted the royi/simsimd-dispatch-tests branch March 19, 2025 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mewim mewim mewim approved these changes

Assignees

@royi-luo royi-luo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@royi-luo @mewim @andyfengHKU