Intel GFX CI

What Is It About?

We are continously testing i915 and Intel Xe (a Linux kernel device drivers for Intel Graphics) in an automated fashion, with a goal of having production ready upstream.

Hardware List

Our CI system is composed of multiple Intel machines spanning many generations and display types. You can find our full hardware list here.

Pre-Merge Testing

This is the crux of this CI system. We test each patch that goes in our driver (i915), Intel Xe or in the DRM drivers test suite we use (IGT GPU Tools) before it lands in the repository and compare the results with the post-merge baseline (we filter out known bugs). This shifts the cost of integration on the people making the change and helps us avoid time-consuming bisection and reverts.

Since we accept patches through mailing lists, this is where you can find the results - they are sent out as a replies to the original mail. We run BAT and FULL IGT (if BAT is successful). On a usual day you can expect result emails for a BAT within 1 hour and for FULL IGT within 6 hours. You will receive the emails even if the run is successful.

Here are the mailing lists we currently support:

Mailing List List Info Archive Patchwork
intel-gfx@lists.freedesktop.org info archive patchwork
igt-dev@lists.freedesktop.org info archive patchwork
intel-gfx-trybot@lists.freedesktop.org info archive kernel patchwork
IGT patchwork
intel-xe@lists.freedesktop.org info archive patchwork

And here is a guide for reading pre-merge results emails.

Trybot - available only for i915

Trybot mailing list is for trying out changes on our infrastracture before they are ready for a review. It has the lowest priority of all the mailing lists. It supports both IGT and kernel patches.

IGT patches sould have i-g-t tag in the subject, e.g. [PATCH i-g-t] My patch. This can be automated with:
git config --local format.subjectPrefix "PATCH i-g-t"

Queues

You can check our pre-merge queues for BAT and Full/Sharded. They are helpful for assessing how busy the CI is. See Queues for details.

If you are interested in knowning the testing latency’s distribution, you should check out the dedicated latency page.

Forcing Tests In BAT And Changing Configuration

To force some IGT tests in CI you just need to add an extra patch to your series that makes the desired alteration to tests/intel-ci/fast-feedback.testlist file - CI uses it directly.

Make sure to mark such patch by adding HAX in the subject, and if not obvious - explain the reason for the forced testing in the commit message. Example.

To change kernel’s configuration or command line parameters just submit a patch that changed the relevant part (e.g. Kconfig or i915_params.h) and submit it as an extra patch with HAX in the subject. Example.

Testing Combined IGT And Kernel Changes - available only for i915

To test force IGT test list changes for a kernel patch series you have to apply the above to our guide on Testing Combined IGT And Kernel Changes for details.

Git Trees Tested Post-Merge

We test many trees post-merge. Some of them are our baseline for pre-merge testing (drm-tip, IGT GPU Tools and Intel Xe), while the others helps us to make sure that what we submit upstream works and catch any potential issues cased by changes in other drivers/trees before we actually integrate with them.

The testing is sparse, i.e. we poll the branch for changes periodically, and if something has changed we run it through our CI, even if that means multiple commits/merges.

CI Results Documentation Repository
drm-tip docs repo
IGT GPU Tools* docs repo
Linus’ tree ??? repo
linux-next docs repo
drm-intel-fixes docs repo
drm-intel-next-fixes docs repo
drm-intel-next docs repo
drm-intel-gt-next docs repo
drm-misc-fixes docs repo
drm-misc-next-fixes docs repo
drm-intel-media ??? repo
drm-next ??? repo
Intel Xe ??? repo
xe-kmd-backports ??? repo
i915-kmd-backports ??? repo

*: IGT results are part of the drm-tip visualisation

Repositories With CI Tags

Try git ls-remote and don’t forget to fetch tags with git fetch -t!

Firmware Repositories

Modern drivers require firmware blobs for full functionality. These blobs are available from the drm-firmware repository. There are 2 branches you need to pay attention to:

When new pushes are detected to either one of these branches, we take i915 and xe dirs from intel-staging, intel-ci from intel-for-ci and deploy these 3 dirs onto test machines next to the usual firmware tree as supplied by the base OS (usually Ubuntu).

Results Filtering And Bug Tracking

Bugs caught by our system are filled to the following bug trackers:

We also maintain a set of filters that tie those bugs to failures we see in our CI using machine names/types and patterns in dmesg/test output. This way we are able to filter known issues out of pre-merge results to decrease noise for developers, keep track of bug’s life cycle, and reproduction rate. The tool makes us able to confirm that a supposed fix is indeed fixing things.

You will be able to browse the CI issues with the related data using our tool called cibuglog once we will have it deployed publicly. In the meantime you can browe its source code.

The Kinds Of Runs

Basic Acceptance Tests (aka BAT)

BAT is the most basic run in our portfolio - it consists of tests that help us ensure that the testing configuration is in a working condition. It gates all the sharded runs - if it breaks, then the build is deemed to broken to use more of the CI time on it. It utilizes fast-feedback.testlist for i915 and xe-fast-feedback.testlist for Xe tests, which you can find in the IGT repository.

Full IGT (aka sharded runs)

If the BAT run is succesful, then we continue with the sharded run. Much broader set of tests (everything you can find in IGT filtered through our blacklist/blocklist) is executed.

i915 - all IGT tests with blacklist.txt. Xe - all IGT tests with xe.blocklist.txt.

Idle Runs

Those runs are complementary to the ones above, used mainly to gather extra data and increase coverage. As the name suggests, they are run when CI would be idle otherwise (i.e. there nothing to test for the regular pre-merge/post-merge).

Other Runs

Resume runs are using the same test list as Full IGT but are done on a single machine. In case of hangs/incompletes the run is resumed. They are not a part of regular CI and are tiggered on demand.

Contacts