Hello everyone,
Here, at Access Softek Toolchains we collect buildbot numbers we think the LLVM community would be interested to check.
In this report we’d like to share the following stats:
- The longest time each builder was red during the week;
- "Status change ratio" by active builder (percent of builds that changed the builder status from green to red or from red to green);
- Number of commits by project;
- Number of completed builds, failed builds and average build time for successful builds per active builder;
- Average waiting time for a revision to get build result per active builder (response time).
...
Time Periods
To make it easy to compare the numbers, we pulled stats from equal monthly periods. In this report, we compare data in May 2020 to June 2020. To be more precise, the dates are as follows:
May 10, 2020 – June 6, 2020 (further in this report will be marked as May 2020) and June 7, 2020 – July 4, 2020 (further in this report will be marked as June 2020)
...
Raw Data
Feel free to download raw data in the .CSV format by clicking on the three dots in the top-right corner next to each chart you see below.
...
About Us
A few words about who we are and why we share these stats:
Access Softek Toolchains is an active contributor and a partner of the open source compiler technology ecosystem – the LLVM Project. We host the LLVM Lab in our office in Berkeley, California and manage the continued integration (LLVM buildbot) for all projects under the LLVM umbrella.
We also help companies to drive adoption of the LLVM infrastructure and get the best possible software development tools. We have worked on a number of projects such as a front-end compiler, back-end code generation and optimization, linkers, code instrumentation, etc. for clients like Sony, Google and Nintendo.
If you need help with developing, porting, representing and maintaining compiler toolchains based on Clang and LLVM, please drop us a line. Also, please see this page for more details about our expertise.
...
Feedback Wanted
This is just a first report in a series we’ve planned ahead. We hope you’ll like the data and the way we’ve put it together.
If you have any suggestions or ideas on how to improve the reports, or if there is other data you’d like to see in the coming reports, please let us know.
...
1. The Longest Time Each Builder Was Red During the Month
Period: May 10, 2020 – June 6, 2020 (May 2020) VS June 7, 2020 – July 4, 2020 (June 2020)
Format: HH:MM:SS
...
As seen from the charts above, the longest time being red in May – June 2020 by far were the following builders:
| Builder Name |
Was Red, May 2020 (HH:MM:SS) |
| llvm-sphinx-docs |
135:14:21 |
| libcxx-libcxxabi-libunwind-armv7-linux-noexceptions |
130:46:04 |
| libcxx-libcxxabi-libunwind-armv7-linux |
128:34:27 |
| libcxx-libcxxabi-libunwind-armv8-linux |
119:51:43 |
| libcxx-libcxxabi-libunwind-armv8-linux-noexceptions |
119:51:27 |
| clang-native-arm-lnt-perf |
115:00:42 |
| sanitizer-x86_64-linux |
109:51:18 |
| Builder Name |
Was Red, June 2020 (HH:MM:SS) |
| llvm-clang-win-x-armv7 |
139:09:35 |
| lld-x86_64-freebsd |
109:33:39 |
| sanitizer-x86_64-linux-android |
97:09:18 |
| mlir-nvidia |
96:23:08 |
| clang-cmake-thumbv7-full-sh |
86:10:53 |
...
The least time being red during the same time period goes to the following builders:
| Builder Name |
Was Red, June 2020 (HH:MM:SS) |
| libcxx-libcxxabi-x86_64-linux-debian-noexceptions |
00:20:29 |
| libc-x86_64-debian |
00:49:26 |
| openmp-clang-x86_64-linux-debian |
00:58:35 |
| openmp-gcc-x86_64-linux-debian |
00:59:47 |
| clang-aarch64-linux-build-cache |
01:02:41 |
| lld-perf-testsuite |
01:03:20 |
| Builder Name |
Was Red, June 2020 (HH:MM:SS) |
| lld-perf-testsuite |
01:32:57 |
| ml-opt-devrel-x86-64 |
01:33:11 |
| openmp-clang-x86_64-linux-debian |
01:43:54 |
| clang-cmake-armv7-selfhost-neon |
02:01:02 |
| ml-opt-rel-x86-64 |
02:35:30 |
...
The largest shifts for the time a builder was red in June 2020 over May 2020 are:
| Builder Name |
Was Red, May 2020 (HH:MM:SS) |
Was Red, June 2020 (HH:MM:SS) |
Change (HH:MM:SS) |
| libcxx-libcxxabi-libunwind-armv7-linux |
128:34:27 |
16:43:02 |
-111:51:25 ↓ |
| clang-native-arm-lnt-perf |
115:00:42 |
03:18:34 |
-111:42:08 ↓ |
| libcxx-libcxxabi-libunwind-armv8-linux |
119:51:43 |
14:10:40 |
-105:41:03 ↓ |
| libcxx-libcxxabi-libunwind-armv7-linux-noexceptions |
130:46:04 |
48:08:45 |
-82:37:19 ↓ |
| llvm-sphinx-docs |
135:14:21 |
56:14:14 |
-79:00:07 ↓ |
| Builder Name |
Was Red, May 2020 (HH:MM:SS) |
Was Red, June 2020 (HH:MM:SS) |
Change (HH:MM:SS) |
| lld-x86_64-freebsd |
03:27:43 |
109:33:39 |
+106:05:56 ↑ |
| llvm-clang-win-x-armv7 |
39:58:13 |
139:09:35 |
+99:11:22 ↑ |
| mlir-nvidia |
20:09:29 |
55:48:49 |
+76:13:39 ↑ |
| sanitizer-x86_64-linux-android |
22:48:06 |
97:09:18 |
+74:21:12 ↑ |
| clang-s390x-linux-multistage |
11:20:51 |
81:56:32 |
+70:35:41 ↑ |
...
2. Status Change Ratio
"Status change ratio" by active builder (percent of builds that changed the builder status from green to red or from red to green). Period: May 10, 2020 – June 6, 2020 (May 2020) VS June 7, 2020 – July 4, 2020 (June 2020)
...
...
The highest status change ratio in May – June 2020 had the following builders:
| Builder Name |
Builds |
Changes |
Status Change Ratio (%) |
| libcxx-libcxxabi-libunwind-armv7-linux |
36 |
10 |
27.8 |
| libcxx-libcxxabi-libunwind-armv7-linux-noexceptions |
36 |
10 |
27.8 |
| sanitizer-x86_64-linux |
576 |
137 |
23.8 |
| clang-cmake-aarch64-lld |
302 |
69 |
22.8 |
| clang-x64-windows-msvc |
394 |
90 |
22.8 |
| Builder Name |
Builds |
Changes |
Status Change Ratio (%) |
| clang-cmake-aarch64-full |
463 |
139 |
30 |
| libcxx-libcxxabi-libunwind-armv7-linux |
21 |
5 |
23.8 |
| libcxx-libcxxabi-libunwind-armv8-linux-noexceptions |
31 |
7 |
22.6 |
| libcxx-libcxxabi-libunwind-armv7-linux-noexceptions |
23 |
5 |
21.7 |
| clang-cmake-aarch64-lld |
363 |
72 |
19.8 |
| clang-x64-windows-msvc |
420 |
83 |
19.8 |
...
The lowest status change ratio in June 2020 had exact same builders as in May 2020:
| Builder Name |
Builds |
Changes |
Status Change Ratio (%) |
| lldb-sphinx-docs |
1642 |
0 |
0 |
| clang-sphinx-docs |
1499 |
0 |
0 |
| libcxx-sphinx-docs |
1284 |
0 |
0 |
| lld-sphinx-docs |
1268 |
0 |
0 |
| libunwind-sphinx-docs |
1192 |
0 |
0 |
...
The largest shifts for the status change ratio in June over May 2020:
| Builder Name |
Status Change Ratio, May 2020 (%) |
Status Change Ratio, June 2020 (%) |
| libcxx-libcxxabi-libunwind-armv8-linux ↓ |
20.3 |
7.5 |
| clang-cmake-armv7-selfhost-neon ↓ |
12.8 |
1.2 |
| clang-cmake-armv7-selfhost ↓ |
12 |
1.5 |
| clang-cmake-armv7-full ↓ |
13.8 |
4 |
| sanitizer-x86_64-linux ↓ |
23.8 |
14.2 |
| Builder Name |
Status Change Ratio, May 2020 (%) |
Status Change Ratio, June 2020 (%) |
| clang-cmake-aarch64-full ↑ |
16.6 |
30 |
| libcxx-libcxxabi-x86_64-linux-debian-noexceptions ↑ |
2.6 |
9.7 |
| clang-ppc64be-linux-lnt ↑ |
10 |
16.3 |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx14 ↑ |
0 |
5.5 |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx17 ↑ |
0 |
5.5 |
...
3. Number of Commits by Project
Period: May 10, 2020 – June 6, 2020 (May 2020) VS June 7, 2020 – July 4, 2020 (June 2020)
...
Changes of project commits in June compared to May 2020:
| project |
Commits, May 2020 |
Commits, June 2020 |
Change (%) |
| mlir |
0 |
176 |
+ ∞ |
| flang |
0 |
59 |
+ ∞ |
| libunwind |
2 |
8 |
+300 |
| libcxxabi |
7 |
16 |
+129 |
| openmp |
10 |
18 |
+80 |
| libc |
20 |
31 |
+55 |
| cfe |
437 |
563 |
+29 |
| clang-tools-extra |
81 |
104 |
+28 |
| llvm |
1523 |
1595 |
+5 |
| compiler-rt |
61 |
61 |
0 |
| libcxx |
74 |
66 |
-11 |
| lld |
77 |
68 |
-12 |
| lldb |
197 |
172 |
-13 |
| polly |
6 |
3 |
-50 |
...
Share of each project in the overall number of commits during the observed period:
| Project |
% of Total Commits, May 2020 |
% of Total Commits, June 2020 |
| llvm |
61 |
54.3 |
| cfe |
17.5 |
19.1 |
| mlir |
0 |
6 |
| lldb |
7.9 |
5.9 |
| clang-tools-extra |
3.2 |
3.5 |
| lld |
3.1 |
2.3 |
| libcxx |
3 |
2.2 |
| compiler-rt |
2.4 |
2.1 |
| flang |
0 |
2 |
| libc |
0.8 |
1.1 |
| openmp |
0.4 |
0.6 |
| libcxxabi |
0.3 |
0.5 |
| libunwind |
0.1 |
0.3 |
| polly |
0.2 |
0.1 |
...
4. Number of Completed Builds, Failed Builds and Average Build Time for Successful Builds Per Active Builder
Period: May 10, 2020 – June 6, 2020 (May 2020) VS June 7, 2020 – July 4, 2020 (June 2020)
...
The following builders had the largest ratio of the failed builds during May – June 2020:
| Builder Name |
All Builds, May 2020 |
Red Builds, May 2020 |
% of Red Builds, May 2020 |
| clang-cmake-x86_64-avx2-linux-perf |
621 |
621 |
100 |
| clang-sphinx-docs |
1499 |
1499 |
100 |
| lld-sphinx-docs |
1268 |
1268 |
100 |
| clang-hexagon-elf |
317 |
316 |
99.7 |
| reverse-iteration |
351 |
349 |
99.4 |
| clang-x64-ninja-win7 |
247 |
233 |
94.3 |
| llvm-sphinx-docs |
1217 |
1073 |
88.2 |
| Builder Name |
All Builds, June 2020 |
Red Builds, June 2020 |
% of Red Builds, June 2020 |
| clang-hexagon-elf |
308 |
308 |
100 |
| clang-sphinx-docs |
1542 |
1542 |
100 |
| lld-sphinx-docs |
1291 |
1291 |
100 |
| lld-x86_64-freebsd |
502 |
501 |
99.8 |
| reverse-iteration |
347 |
346 |
99.7 |
| clang-cmake-x86_64-avx2-linux-perf |
617 |
615 |
99.7 |
| clang-x64-ninja-win7 |
267 |
259 |
97 |
| llvm-sphinx-docs |
1247 |
1178 |
94.5 |
...
The following builders had no failed builds during the observed period:
| Builder Name |
All Builds, May 2020 |
Red Builds, May 2020 |
| lldb-sphinx-docs |
1642 |
0 |
| libcxx-sphinx-docs |
1284 |
0 |
| libunwind-sphinx-docs |
1192 |
0 |
| libcxx-libcxxabi-x86_64-linux-ubuntu-gcc-tot-latest-std |
75 |
0 |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx14 |
65 |
0 |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx17 |
63 |
0 |
| Builder Name |
All Builds, June 2020 |
Red Builds, June 2020 |
| lldb-sphinx-docs |
1664 |
0 |
| libcxx-sphinx-docs |
1297 |
0 |
| libunwind-sphinx-docs |
1244 |
0 |
| clang-aarch64-linux-build-cache |
1212 |
0 |
...
...
The following builders had the highest increase in average build time in June compared to May 2020:
| Builder Name |
Avg Build Time, May 2020 |
Avg Build Time, June 2020 |
Change (%) |
| sanitizer-windows |
00:06:04 |
00:09:24 |
+54.9 ↑ |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03 |
00:06:04 |
00:07:47 |
+28.3 ↑ |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx14 |
00:07:30 |
00:08:37 |
+14.9 ↑ |
| clang-ppc64be-linux-multistage |
00:54:27 |
01:02:07 |
+14.1 ↑ |
| clang-ppc64be-linux-lnt |
00:27:50 |
00:31:40 |
+13.8 ↑ |
| libc-x86_64-debian-dbg |
00:02:19 |
00:02:38 |
+13.7 ↑ |
| libcxx-libcxxabi-x86_64-linux-ubuntu-cxx11 |
00:07:15 |
00:08:13 |
+13.3 ↑ |
...
Average build time reduced significantly for the following builders:
| Builder Name |
Avg Build Time, May 2020 |
Avg Build Time, June 2020 |
Change (%) |
| clang-tools-sphinx-docs |
00:00:44 |
00:00:14 |
-68.2↓ |
| clang-cmake-armv7-global-isel |
02:06:22 |
00:55:39 |
-56↓ |
| lld-x86_64-freebsd |
00:32:02 |
00:14:12 |
-55.7↓ |
| clang-cmake-aarch64-global-isel |
01:52:19 |
00:51:30 |
-54.1↓ |
| clang-cmake-armv7-lnt |
01:24:50 |
00:39:27 |
-53.5↓ |
| clang-cmake-armv7-selfhost |
01:17:53 |
00:39:56 |
-48.7↓ |
| clang-cmake-armv8-lld |
02:49:56 |
01:28:55 |
-47.7↓ |
| clang-cmake-armv7-selfhost-neon |
01:17:34 |
00:41:15 |
-46.8↓ |
| clang-cmake-aarch64-lld |
03:22:45 |
01:51:17 |
-45.1↓ |
| clang-cmake-armv7-quick |
00:25:16 |
00:15:01 |
-40.6↓ |
...
5. Average Waiting Time for a Revision to Get Build Result Per Active Builder (Response Time)
Period: May 10, 2020 – June 6, 2020 (May 2020) VS June 7, 2020 – July 4, 2020 (June 2020)
...
Response time fell significantly for the following builders in June compared to May 2020:
| Builder Name |
Avg Wait Time, May 2020 |
Avg Wait Time, June 2020 |
Change (%) |
| clang-cmake-armv7-quick |
00:35:25 |
02:15:52 |
+283.6 ↑ |
| llvm-avr-linux |
00:39:37 |
01:21:30 |
+105.7 ↑ |
| mlir-nvidia |
03:14:13 |
06:02:01 |
+86.4 ↑ |
| clang-ppc64le-linux-lnt |
02:31:24 |
04:34:33 |
+81.3 ↑ |
| clang-cmake-aarch64-quick |
01:41:42 |
03:04:02 |
+81 ↑ |
| libcxx-libcxxabi-libunwind-armv8-linux-noexceptions |
03:55:51 |
06:44:45 |
+71.6 ↑ |
| libcxx-libcxxabi-libunwind-aarch64-linux-noexceptions |
04:13:10 |
07:00:59 |
+66.3 ↑ |
| clang-aarch64-linux-build-cache |
01:37:06 |
02:37:29 |
+62.2 ↑ |
| libcxx-libcxxabi-libunwind-armv8-linux |
04:06:38 |
06:17:31 |
+53.1 ↑ |
...
The following builders had the highest response time improvement during the same period:
| Builder Name |
Avg Wait Time, May 2020 |
Avg Wait Time, June 2020 |
Change (%) |
| clang-ppc64le-rhel |
04:43:11 |
00:27:38 |
-90 ↓ |
| ppc64le-lld-multistage-test |
06:09:21 |
01:35:36 |
-74 ↓ |
| lldb-aarch64-ubuntu |
01:14:54 |
00:26:31 |
-65 ↓ |
| clang-cmake-x86_64-sde-avx512-linux |
00:17:28 |
00:08:16 |
-53 ↓ |
| openmp-clang-x86_64-linux-debian |
00:08:34 |
00:04:51 |
-43 ↓ |
| flang-aarch64-ubuntu |
01:29:35 |
00:52:21 |
-42 ↓ |
| lldb-arm-ubuntu |
01:41:39 |
01:09:08 |
-32 ↓ |
| sanitizer-ppc64le-linux |
07:04:47 |
05:15:33 |
-26 ↓ |
| llvm-sphinx-docs |
00:03:10 |
00:02:22 |
-25 ↓ |
...
Have a suggestion?
We hope you like the stats we’ve collected in this report.
If you have any suggestions or ideas on how to improve it, or if there is other data you’d like to see in the coming reports, please let us know.
Thank you,
Access Softek Toolchains Team
