Automated performance monitoring in WordPress core


Gathering performance metrics automatically is a way to track performance over time and ensure that WordPress continues to improve. Automated performance tooling will also help the core team identify issues and resolve them with less effort.
Why add automated performance testing?
Adding automated performance testing will help us monitor performance changes in WordPress core continuously. It gives us a track record to capture how core performance is being enhanced over time, and it allows us to catch regressions early and accurately identify underlying causes. Similar to our unit test suite, automated performance testing would help protect core from introducing large performance regressions by catching problems immediately and tracking performance over time. Automating testing also means saving contributor effort by replacing a time consuming manual process.
The core performance team is focused on improving core performance. Examples of this work include introducing changes that reduce the number of database queries or improve caching. Each new performance team feature must show measurable gains, and each new WordPress release is performance tested by the team. In the 6.1 release cycle, this led to the discovery of some significant performance regressions, e.g. in this Gutenberg issue or this Trac ticket. Automated testing would catch this type of regression as soon as it was introduced, making it much easier to resolve.
It is worth noting that Gutenberg has already done extensive work on performance, tracking metrics with each commit and publicizing details with each release.
What automated performance testing would do in core
Similar to Gutenberg, WordPress core would gather a set of automated performance metrics along with the existing test runs (e.g. unit tests, coding standards) we already have for each new commit. These metrics can be used to identify the exact point a performance regression is introduced into core. At milestones like a major release, the metrics can be compared against the previous release to gauge progress.
Goals for the initial version
The scope of the initial version of automated performance testing is intentionally kept limited so we can deliver more quickly, then we can iterate.
The initial version will include the following features:
- For each core commit a GitHub action will run a set of automated performance tests, collecting key data points about how WordPress is performing (such as total load time and total query time) on block and classic themes.
- The system will collect server timing metrics using the standard WordPress environment and current recommended version of PHP.
Future Enhancements
Several areas of work are considered out of scope for the initial implementation, primarily to keep the focus limited for the initial release — not because they aren’t good ideas! Once we build a solid foundation for tests and are confident in the metrics we are collecting, we can consider additional improvements.
- Collecting additional metrics: initially we will focus on key server timing metrics
- Collecting metrics for other contexts: initial metrics will only measure the home page of the latest core block and classic themes with their default demo content.
- Slack or ticket reporting: initial work will focus only on collecting and recording metrics at each commit.
FAQ
Will the data be stable enough to be useful?
Performance results can vary in a CI environment, making regressions harder to detect. To mitigate this, the system will run several iterations and use the median value.
What metrics will be collected exactly?
Initially we will only collect a few key metrics to keep the dashboard simple, focused on the total load time. Once we have established the new tool, we can consider including additional metrics, or adding hooks to make the test runs extensible.
What about testing older PHP versions?
To reduce the time/cost associated with running these tests, they will be limited to the current recommended version of PHP. Unlike unit tests, performance tests are unlikely to produce significantly different deltas for different PHP versions: regressions are likely to be across PHP versions.
Why not test wp-admin or more routes?
In order to keep the time and cost of running these tests low, and the dashboard simplified, initial testing is intentionally being kept somewhat minimal. In the future it would be good to consider adding other common routes such as the wp-admin dashboard and single post page.
Thanks to @tweetythierry @flixos90 @joemcgill @mukesh27 for reviewing this post and to @youknowriad for the inspiration (and foundation) of the Gutenberg performance metrics.
nnnn
The core performance team is focused on improving core performance. Examples of this work include introducing changes that reduce the number of database queries or improve caching. Each new performance team feature must show measurable gains, and each new WordPress release is performance tested by the team. In the 6.1 release cycle, this led to the discovery of some significant performance regressions, e.g. in this Gutenberg issue or this Trac ticket. Automated testing would catch this type of regression as soon as it was introduced, making it much easier to resolve.
nnnn
It is worth noting that Gutenberg has already done extensive work on performance, tracking metrics with each commit and publicizing details with each release.
nnnn
What automated performance testing would do in core
nnnn
Similar to Gutenberg, WordPress core would gather a set of automated performance metrics along with the existing test runs (e.g. unit tests, coding standards) we already have for each new commit. These metrics can be used to identify the exact point a performance regression is introduced into core. At milestones like a major release, the metrics can be compared against the previous release to gauge progress.
nnnn
Goals for the initial version
nnnn
The scope of the initial version of automated performance testing is intentionally kept limited so we can deliver more quickly, then we can iterate.
nnnn
The initial version will include the following features:
nnnn
- n
- For each core commit a GitHub action will run a set of automated performance tests, collecting key data points about how WordPress is performing (such as total load time and total query time) on block and classic themes. nnnn
- The system will collect server timing metrics using the standard WordPress environment and current recommended version of PHP. n
nnnn
Future Enhancements
nnnn
Several areas of work are considered out of scope for the initial implementation, primarily to keep the focus limited for the initial release u2014 not because they arenu2019t good ideas! Once we build a solid foundation for tests and are confident in the metrics we are collecting, we can consider additional improvements.
nnnn
- n
- Collecting additional metrics: initially we will focus on key server timing metrics nnnn
- Collecting metrics for other contexts: initial metrics will only measure the home page of the latest core block and classic themes with their default demo content. nnnn
- Slack or ticket reporting: initial work will focus only on collecting and recording metrics at each commit. n
nnnn
FAQ
nnnn
Will the data be stable enough to be useful?
nnnn
Performance results can vary in a CI environment, making regressions harder to detect. To mitigate this, the system will run several iterations and use the median value.
nnnn
What metrics will be collected exactly?
nnnn
Initially we will only collect a few key metrics to keep the dashboard simple, focused on the total load time. Once we have established the new tool, we can consider including additional metrics, or adding hooks to make the test runs extensible.
nnnn
What about testing older PHP versions?
nnnn
To reduce the time/cost associated with running these tests, they will be limited to the current recommended version of PHP. Unlike unit tests, performance tests are unlikely to produce significantly different deltas for different PHP versions: regressions are likely to be across PHP versions.
nnnn
Why not test wp-admin or more routes?
nnnn
In order to keep the time and cost of running these tests low, and the dashboard simplified, initial testing is intentionally being kept somewhat minimal. In the future it would be good to consider adding other common routes such as the wp-admin dashboard and single post page.
nnnn
nnnn
Thanks to @tweetythierry @flixos90 @joemcgill @mukesh27 for reviewing this post and to @youknowriad for the inspiration (and foundation) of the Gutenberg performance metrics.
nnn#core-performance, #performance, #testing","contentFiltered":"
Gathering performance metrics automatically is a way to track performance over time and ensure that WordPress continues to improve.u00a0 Automated performance tooling will also help the core team identify issues and resolve them with less effort.
nnnn
Why add automated performance testing?
nnnn
Adding automated performance testing will help us monitor performance changes in WordPress core continuously. It gives us a track record to capture how core performance is being enhanced over time, and it allows us to catch regressions early and accurately identify underlying causes. Similar to our unit test suite, automated performance testing would help protect core from introducing large performance regressions by catching problems immediately and tracking performance over time. Automating testing also means saving contributor effort by replacing a time consuming manual process.
nnnn
The core performance team is focused on improving core performance. Examples of this work include introducing changes that reduce the number of database queries or improve caching. Each new performance team feature must show measurable gains, and each new WordPress release is performance tested by the team. In the 6.1 release cycle, this led to the discovery of some significant performance regressions, e.g. in this Gutenberg issue or this Trac ticket. Automated testing would catch this type of regression as soon as it was introduced, making it much easier to resolve.
nnnn
It is worth noting that Gutenberg has already done extensive work on performance, tracking metrics with each commit and publicizing details with each release.u00a0u00a0
nnnn
What automated performance testing would do in core
nnnn
Similar to Gutenberg, WordPress core would gather a set of automated performance metrics along with the existing test runs (e.g. unit tests, coding standards) we already have for each new commit. These metrics can be used to identify the exact point a performance regression is introduced into core. At milestones like a major release, the metrics can be compared against the previous release to gauge progress.
nnnn
Goals for the initial version
nnnn
The scope of theu00a0 initial version of automated performance testing is intentionally kept limited so we can deliver more quickly, then we can iterate.u00a0
nnnn
The initial version will include the following features:
nnnn
- For each core commit a GitHub action will run a set of automated performance tests, collecting key data points about how WordPress is performing (such as total load time and total query time) on block and classic themes.u00a0 nnnn
- The system will collect server timing metrics using the standard WordPress environment and current recommended version of PHP.u00a0u00a0 n
Future Enhancements
nnnn
Several areas of work are considered out of scope for the initial implementation, primarily to keep the focus limited for the initial release u2014 not because they arenu2019t good ideas! Once we build a solid foundation for tests and are confident in the metrics we are collecting, we can consider additional improvements.
nnnn
- Collecting additional metrics: initially we will focus on key server timing metrics nnnn
- Collecting metrics for other contexts: initial metrics will only measure the home page of the latest core block and classic themes with their default demo content. nnnn
- Slack or ticket reporting: initial work will focus only on collecting and recording metrics at each commit.u00a0 n
FAQ
nnnn
Will the data be stable enough to be useful?
nnnn
Performance results can vary in a CI environment, making regressions harder to detect. To mitigate this, the system will run several iterations and use the median value.
nnnn
What metrics will be collected exactly?
nnnn
Initially we will only collect a few key metrics to keep the dashboard simple, focused on the total load time. Once we have established the new tool, we can consider including additional metrics, or adding hooks to make the test runs extensible.
nnnn
What about testing older PHP versions?
nnnn
To reduce the time/cost associated with running these tests, they will be limited to the current recommended version of PHP. Unlike unit tests, performance tests are unlikely to produce significantly different deltas for different PHP versions: regressions are likely to be across PHP versions.
nnnn
Why not test wp-admin or more routes?
nnnn
In order to keep the time and cost of running these tests low, and the dashboard simplified, initial testing is intentionally being kept somewhat minimal. In the future it would be good to consider adding other common routes such as the wp-admin dashboard and single post page.
nnnn
Thanks to @tweetythierry @flixos90 @joemcgill @mukesh27 for reviewing this post and to @youknowriad for the inspiration (and foundation) of the Gutenberg performance metrics.
n
","permalink":"https://make.wordpress.org/core/2023/02/02/automated-performance-monitoring-in-wordpress-core/","unixtime":1675348524,"unixtimeModified":1675348524,"entryHeaderMeta":"","linkPages":"","footerEntryMeta":"","tagsRaw":"core-performance, performance, testing","tagsArray":[{"label":"core-performance","count":10,"link":"https://make.wordpress.org/core/tag/core-performance/"},{"label":"performance","count":163,"link":"https://make.wordpress.org/core/tag/performance/"},{"label":"testing","count":25,"link":"https://make.wordpress.org/core/tag/testing/"}],"loginRedirectURL":"https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fcore%2F2023%2F02%2F02%2Fautomated-performance-monitoring-in-wordpress-core%2F&locale=en_US","hasPrevPost":false,"prevPostTitle":"","prevPostURL":"","hasNextPost":false,"nextPostTitle":"","nextPostURL":"","commentsOpen":true,"is_xpost":false,"editURL":null,"postActions":"","comments":[{"type":"comment","id":"44335","postID":"102272","postTitleRaw":"Automated performance monitoring in WordPress core","cssClasses":"comment byuser comment-author-javiercasares even thread-even depth-1","parentID":"0","contentRaw":"
To reduce the time/cost associated with running these tests, they will be limited to the current recommended version of PHP.
nnI think it should check PHP 7.4 and PHP 8+. This allows to see data of about 75% of the installations.","contentFiltered":"
To reduce the time/cost associated with running these tests, they will be limited to the current recommended version of PHP.
n
I think it should check PHP 7.4 and PHP 8+. This allows to see data of about 75% of the installations.
n","permalink":"https://make.wordpress.org/core/2023/02/02/automated-performance-monitoring-in-wordpress-core/#comment-44335","unixtime":1675348865,"loginRedirectURL":"https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fcore%2F2023%2F02%2F02%2Fautomated-performance-monitoring-in-wordpress-core%2F%23comment-44335&locale=en_US","approved":true,"isTrashed":false,"prevDeleted":"","editURL":null,"depth":1,"commentDropdownActions":"","commentFooterActions":"
","commentTrashedActions":"","mentions":[],"mentionContext":"","commentCreated":"1675348865","hasChildren":false,"userLogin":"JavierCasares","userNicename":"javiercasares"},{"type":"comment","id":"44336","postID":"102272","postTitleRaw":"Automated performance monitoring in WordPress core","cssClasses":"comment byuser comment-author-dansoschin odd alt thread-odd thread-alt depth-1","parentID":"0","contentRaw":"This sounds like a great program to ensure regressions are minimized. Given how much emphasis peer CMS platforms place on "advertising" their performance and benchmarking it against the industry leaders, investing in tools to ensure WordPress continues to perform optimally makes a lot of sense. And, given how many sites are powered by WordPress, even minor gains in performance (including those unnoticeable to most people) add a lot of value to web hosts and lowering overall internet traffic burdens/bandwidth. Great work here!","contentFiltered":"
This sounds like a great program to ensure regressions are minimized. Given how much emphasis peer CMS platforms place on u201cadvertisingu201d their performance and benchmarking it against the industry leaders, investing in tools to ensure WordPress continues to perform optimally makes a lot of sense. And, given how many sites are powered by WordPress, even minor gains in performance (including those unnoticeable to most people) add a lot of value to web hosts and lowering overall internet traffic burdens/bandwidth. Great work here!
n","permalink":"https://make.wordpress.org/core/2023/02/02/automated-performance-monitoring-in-wordpress-core/#comment-44336","unixtime":1675351800,"loginRedirectURL":"https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fcore%2F2023%2F02%2F02%2Fautomated-performance-monitoring-in-wordpress-core%2F%23comment-44336&locale=en_US","approved":true,"isTrashed":false,"prevDeleted":"","editURL":null,"depth":1,"commentDropdownActions":"","commentFooterActions":"
","commentTrashedActions":"","mentions":[],"mentionContext":"","commentCreated":"1675351800","hasChildren":false,"userLogin":"dansoschin","userNicename":"dansoschin"}],"postFormat":"standard","postMeta":{"isSticky":false},"postTerms":{"category":[{"label":"Core","count":64,"link":"https://make.wordpress.org/core/category/core/"}],"post_tag":[{"label":"core-performance","count":10,"link":"https://make.wordpress.org/core/tag/core-performance/"},{"label":"performance","count":163,"link":"https://make.wordpress.org/core/tag/performance/"},{"label":"testing","count":25,"link":"https://make.wordpress.org/core/tag/testing/"}],"post_format":[]},"pluginData":[],"isPage":false,"mentions":["tweetythierry","flixos90","joemcgill","mukesh27","youknowriad"],"mentionContext":"","isTrashed":false,"userLogin":"adamsilverstein","userNicename":"adamsilverstein"}]