Intel GFX CI
What Is It About?
We are continously testing i915 (a Linux kernel device driver for Intel Graphics) in an automated fashion, with a goal of having production ready upstream.
Our CI system is composed of multiple Intel machines spanning many generations and display types. You can find our full hardware list here.
This is the crux of this CI system. We test each patch that goes in our driver (i915) or in the DRM drivers test suite we use (IGT GPU Tools) before it lands in the repository and compare the results with the post-merge baseline (we filter out known bugs). This shifts the cost of integration on the people making the change and helps us avoid time-consuming bisection and reverts.
Since we accept patches through mailing lists, this is where you can find the results - they are sent out as a replies to the original mail. We run BAT and FULL IGT (if BAT is successful). On a usual day you can expect result emails for a BAT within 1 hour and for FULL IGT within 6 hours. You will receive the emails even if the run is successful.
Here are the mailing lists we currently support:
|Mailing List||List Info||Archive||Patchwork|
Trybot mailing list is for trying out changes on our infrastracture before they are ready for a review. It has the lowest priority of all the mailing lists. It supports both IGT and kernel patches.
IGT patches sould have
i-g-t tag in the subject, e.g.
[PATCH i-g-t] My
patch. This can be automated with:
git config --local format.subjectPrefix "PATCH i-g-t"
If you are interested in knowning the testing latency’s distribution, you should check out the dedicated latency page.
Forcing Tests In BAT And Changing Configuration
To force some IGT tests in CI you just need to add an extra patch to your
series that makes the desired alteration to
tests/intel-ci/fast-feedback.testlist file - CI uses it directly.
Make sure to mark such patch by adding
HAX in the subject, and if not
obvious - explain the reason for the forced testing in the commit message.
To change kernel’s configuration or command line parameters just submit a
patch that changed the relevant part (e.g.
submit it as an extra patch with
HAX in the subject.
To test force IGT test list changes for a kernel patch series you have to apply the above to our guide on Testing Combined IGT And Kernel Changes.
Testing Combined IGT And Kernel Changes
See Testing Combined IGT And Kernel Changes for details.
Git Trees Tested Post-Merge
We test many trees post-merge. Some of them are our baseline for pre-merge testing (drm-tip and IGT GPU Tools), while the others helps us to make sure that what we submit upstream works and catch any potential issues cased by changes in other drivers/trees before we actually integrate with them.
The testing is sparse, i.e. we poll the branch for changes periodically, and if something has changed we run it through our CI, even if that means multiple commits/merges.
|IGT GPU Tools*||docs||repo|
*: IGT results are part of the drm-tip visualisation
Repositories With CI Tags
git ls-remote and don’t forget to fetch tags with
git fetch -t!
Results Filtering And Bug Tracking
Bugs caught by our system are filled to the following bug trackers:
We also maintain a set of filters that tie those bugs to failures we see in our CI using machine names/types and patterns in dmesg/test output. This way we are able to filter known issues out of pre-merge results to decrease noise for developers, keep track of bug’s life cycle, and reproduction rate. The tool makes us able to confirm that a supposed fix is indeed fixing things.
You will be able to browse the CI issues with the related data using our tool called cibuglog once we will have it deployed publicly. In the meantime you can browe its source code.
The Kinds Of Runs
Basic Acceptance Tests (aka BAT)
BAT is the most basic run in our portfolio - it consists of tests that help us ensure that the testing configuration is in a working condition. It gates all the sharded runs - if it breaks, then the build is deemed to broken to use more of the CI time on it. It utilizes fast-feedback.testlist, which you can find in the IGT repository.
Full IGT (aka sharded runs)
If the BAT run is succesful, then we continue with the sharded run. Much broader set of tests (everything you can find in IGT filtered through our blacklist.txt) is executed.
Those runs are complementary to the ones above, used mainly to gather extra data and increase coverage. As the name suggests, they are run when CI would be idle otherwise (i.e. there nothing to test for the regular pre-merge/post-merge).
- drmtip - Full IGT but on the same hosts that BAT uses, thus increasing coverege.
- KASAN - same as above but runs with The Kernel Address Sanitizer enabled.
- 100 re-runs - re-run BAT 100 times with fixed IGT and kernel for extra data on flip-floppers.
Resume runs are using the same test list as Full IGT but are done on a single machine. In case of hangs/incompletes the run is resumed. They are not a part of regular CI and are tiggered on demand.
- IRC: #intel-gfx-ci @ freenode
- Mailing list: i915-ci-infra @ lists.freedesktop.org
- hardware/CI maintainer: Tomi Sarvela - tomi.p.sarvela @ intel.com
- cibuglog/metrics maintainer: Karol Krol - karol.krol @ intel.com