Add script to diff zjit stats from output csv #471
+459
−21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Still only print the stats when not already enabled.
Example output:
================================================================================
ZJIT Stats Comparison
================================================================================
ruby-mdev (baseline):
ruby 4.1.0dev (2026-01-30T17:29:15Z master 1298f9ac1a) +ZJIT dev +PRISM [arm64-darwin25]
ruby-bdev:
ruby 4.1.0dev (2026-01-30T17:29:15Z profile-block-arg 1298f9ac1a) +ZJIT dev +PRISM [arm64-darwin25]
BENCHMARK TIMINGS (lower is better)
--------------------------------------------------------------------------------
lobsters:
ruby-mdev avg: 1.076s min: 0.944s ★ (baseline)
ruby-bdev avg: 1.103s min: 0.967s +2.5% (slower)
railsbench:
ruby-mdev avg: 1.580s min: 1.532s ★ (baseline)
ruby-bdev avg: 1.587s min: 1.539s +0.4% (slower)
MEMORY USAGE
--------------------------------------------------------------------------------
lobsters:
ruby-mdev maxrss: 510.3MB zjit_mem: 65.9MB
ruby-bdev maxrss: 510.6MB zjit_mem: 65.5MB
railsbench:
ruby-mdev maxrss: 200.2MB zjit_mem: 30.1MB
ruby-bdev maxrss: 203.9MB zjit_mem: 29.8MB
NOT INLINED C METHODS (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
#<Module:0x000000012a06a460>#object_id 241 → 0 ▼ -100.0%
Kernel#instance_variable_defined? 416 → 8 ▼ -98.1%
Numeric#nonzero? 9,789 → 1,556 ▼ -84.1%
... and 11 more
railsbench:
Array#hash 133 → 285 ▲ +114.3%
#<Module:0x000000012804a500>#enum_for 180 → 0 ▼ -100.0%
String#<=> 9,896 → 1,674 ▼ -83.1%
... and 10 more
CALLS TO C FUNCTIONS FROM JIT CODE (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
#<Module:0x000000012a06a460>#object_id 241 → 0 ▼ -100.0%
Kernel#instance_variable_defined? 416 → 8 ▼ -98.1%
Class#new 561 → 22 ▼ -96.1%
... and 18 more
railsbench:
Array#hash 133 → 285 ▲ +114.3%
#<Module:0x000000012804a500>#enum_for 180 → 0 ▼ -100.0%
Numeric#nonzero? 9,896 → 1,674 ▼ -83.1%
... and 15 more
NOT OPTIMIZED METHOD TYPES FOR SEND (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
optimized 0 → 4,591 ▲ new
railsbench:
optimized 19,471 → 49,608 ▲ +154.8%
NOT OPTIMIZED METHOD TYPES FOR SUPER (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
attrset 2,642 → 1,866 ▼ -29.4%
SEND FALLBACK REASONS (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
send_not_optimized_method_type 1,682 → 6,273 ▲ +272.9%
one_or_more_complex_arg_pass 1,145,141 → 2,821,156 ▲ +146.4%
send_no_profiles 2,068,333 → 196,661 ▼ -90.5%
... and 3 more
railsbench:
send_not_optimized_method_type 19,471 → 49,608 ▲ +154.8%
one_or_more_complex_arg_pass 2,641,602 → 4,786,718 ▲ +81.2%
send_no_profiles 4,046,018 → 1,069,740 ▼ -73.6%
... and 1 more
SETIVAR FALLBACK REASONS (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
new_shape_needs_extension 946 → 170 ▼ -82.0%
railsbench:
new_shape_needs_extension 935 → 161 ▼ -82.8%
GETBLOCKPARAMPROXY HANDLER (showing differences > 10.0%)
--------------------------------------------------------------------------------
railsbench:
no_profiles 1,146 → 350 ▼ -69.5%
COMPLEX ARGUMENT-PARAMETER FEATURES (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
caller_blockarg 21,564 → 1,697,716 ▲+7772.9%
caller_kwarg 23,322 → 135,011 ▲ +478.9%
caller_splat 33,663 → 37,288 ▲ +10.8%
railsbench:
caller_kwarg 944 → 110,104 ▲+11563.6%
caller_blockarg 184,024 → 2,329,272 ▲+1165.7%
caller_splat 170,800 → 260,718 ▲ +52.6%
SIDE EXIT REASONS (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
obj_to_string_fallback 1,948 → 443 ▼ -77.3%
railsbench:
obj_to_string_fallback 2,021 → 370 ▼ -81.7%
NOT ANNOTATED C METHODS (showing differences > 10.0%)
--------------------------------------------------------------------------------
lobsters:
#<Module:0x000000012a06a460>#object_id 241 → 0 ▼ -100.0%
Class#new 561 → 22 ▼ -96.1%
String#<=> 9,789 → 1,556 ▼ -84.1%
... and 10 more
railsbench:
Array#hash 133 → 285 ▲ +114.3%
#<Module:0x000000012804a500>#enum_for 180 → 0 ▼ -100.0%
Numeric#nonzero? 9,896 → 1,674 ▼ -83.1%
... and 11 more
rwstauner
changed the title
k0kubun
reviewed
Jan 31, 2026
| end.parse! | ||
|
|
||
| if ARGV.empty? | ||
| puts "Usage: #{$0} [options] <output.json> [benchmark ...]" |
Member
There was a problem hiding this comment.
Can we document the usage in README and/or the top of this file as well? It's not immediately clear where output.json is located or how it's supposed to be generated.
k0kubun
approved these changes
Jan 31, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Example output:
Suggestions welcome for more useful output format/features/etc.