Intel

A Conversation with Kernel Developers from Intel, Red Hat and SUSE

Bryan Lunduke — Tue, 30 Apr 2019 11:30:00 +0000

Three kernel developers describe what it's really like to work on the kernel, how they interact with developers from other companies, some pet peeves and how to get started.

Like most Linux users, I rarely touch the actual code for the Linux kernel. Sure, I've looked at it. I've even compiled the kernel myself on a handful of occasions—sometimes to try out something new or simply to say I could do it ("Linux From Scratch" is a bit of a right of passage).

But, unless you're one of the Linux kernel developers, odds are you just don't get many opportunities to truly look "under the hood".

Likewise, I think for many Linux users (even the pro users, sysadmins and developers), the wild world of kernel development is a bit of a mystery. Sure, we have the publicly available Linux Kernel Mailing List (LKML.org) that anyone is free to peruse for the latest features, discussions and (sometimes) shenanigans, but that gives only a glimpse at one aspect of being a kernel developer.

And, let's be honest, most of us simply don't have time to sift through the countless pull requests (and resulting discussions of said pull requests) that flood the LKML on a daily basis.

With that in mind, I reached out to three kernel developers—each working at some of the most prominent Linux contributing companies today—to ask them some basic questions that might provide a better idea of what being a Linux kernel developer is truly like: what their days look like and how they work with kernel developers at other companies.

Those three developers (in no particular order):

Dave Hansen, Principal Engineer, System Software Products at Intel.
Josh Poimboeuf, Principal Software Engineer on Red Hat Enterprise Linux.
Jeff Mahoney, Team Lead of Kernel Engineering at SUSE Labs.

Intel, Red Hat and SUSE—three of the top contributors of code to the Linux kernel. If anyone knows what it's like being a kernel developer, it's them.

I asked all three the exact same questions. Their answers are here, completely unmodified.

Bryan Lunduke: How long have you been working with the Linux kernel? What got you into it?

Dave Hansen (Intel): My first experience for the Linux kernel was a tiny little device driver to drive the eight-character display on an IBM PS/2, probably around 20 years ago. I mentioned the project on my college resume, which eventually led to a job with IBM's Linux Technology Center in 2001. IBM is where I started doing the Linux kernel professionally.

Go to Full Article

Fun Little Tidbits in a Howling Storm (Re: Intel Security Holes)

Zack Brown — Thu, 21 Feb 2019 13:00:00 +0000

by Zack Brown

Some kernel developers recently have been trying to work around the massive, horrifying, long-term security holes that have recently been discovered in Intel hardware. In the course of doing so, there were some interesting comments about coding practices.

Christoph Hellwig and Jesper Dangaard Brouer were working on mitigating some of the giant speed sacrifices needed to avoid Intel's gaping security holes. And, Christoph said that one such patch would increase the networking throughput from 7.5 million packets per second to 9.5 million—a 25% speedup.

To do this, the patch would check the kernel's "fast path" for any instances of dma_direct_ops and replace them with a simple direct call.

Linus Torvalds liked the code, but he noticed that Jesper and Christoph's code sometimes would perform certain tests before testing the fast path. But if the kernel actually were taking the fast path, those tests would not be needed. Linus said, "you made the fast case unnecessarily slow."

He suggested that switching the order of the tests would fix it right up. He added:

In fact, as a further micro-optimization, it might be a good idea to just specify that the dma_is_direct() ops is a special pointer (perhaps even just say that "NULL means it's direct"), because that then makes the fast-case test much simpler (avoids a whole nasty constant load, and testing for NULL in particular is often much better).

But that further micro-optimization absolutely *requires* that the ops pointer test comes first. So making that ordering change is not only "better code generation for the fast case to avoid extra cache accesses", it also allows future optimizations.

Regarding Linus' micro-optimization, Christoph explained:

I wanted to do the NULL case, and it would be much nicer. But the arm folks went to great lengths to make sure they don't have a default set of dma ops and require it to be explicitly set on every device to catch cases where people don't set things up properly, and I didn't want to piss them off....But maybe I should just go for it and see who screams, as the benefit is pretty obvious.

Linus also suggested that for Christoph's and Jesper's tests, the dma_is_direct() function should be sure to use the likely() call. And this was interesting because likely() is used to alert the compiler that a block of code is more "likely" to be run than another in order to optimize it. And, Christoph wasn't sure this was true. He said, "Yes, for the common case, it is likely. But if you run a setup where you say always have an iommu, it is not, in fact, it is never called in that case, but we only know that at runtime."

Go to Full Article

New Intel Caching Feature Considered for Mainline

Zack Brown — Wed, 22 Aug 2018 12:07:07 +0000

by Zack Brown

These days, Intel's name is Mud in various circles because of the Spectre/Meltdown CPU flaws and other similar hardware issues that seem to be emerging as well. But, there was a recent discussion between some Intel folks and the kernel folks that was not related to those things. Some thrust-and-parry still was going on between kernel person and company person, but it seemed more to do with trying to get past marketing speak, than at wrestling over what Intel is doing to fix its longstanding hardware flaws.

Reinette Chatre of Intel posted a patch for a new chip feature called Cache Allocation Technology (CAT), which "enables a user to specify the amount of cache space into which an application can fill". Among other things, Reinette offered the disclaimer, "The cache pseudo-locking approach relies on generation-specific behavior of processors. It may provide benefits on certain processor generations, but is not guaranteed to be supported in the future."

Thomas Gleixner thought Intel's work looked very interesting and in general very useful, but he asked, "are you saying that the CAT mechanism might change radically in the future [that is, in future CPU chip designs] so that access to cached data in an allocated area which does not belong to the current executing context wont work anymore?"

Reinette replied, "Cache Pseudo-Locking is a model-specific feature so there may be some variation in if, or to what extent, current and future devices can support Cache Pseudo-Locking. CAT remains architectural."

Thomas replied, "that does NOT answer my question at all."

At this point, Gavin Hindman of Intel joined the discussion, saying:

Support in a current generation of a product line doesn't imply support in a future generation. Certainly we'll make every effort to carry support forward, and would adjust to any changes in CAT support, but we can't account for unforeseen future architectural changes that might block pseudo-locking use-cases on top of CAT.

And Thomas replied, "that's the real problem. We add something that gives us some form of isolation, but we don't know whether next generation CPUs will work. From a maintainability and usefulness POV that's not a really great prospect."

Elsewhere in a parallel part of the discussion, Thomas asked, "Are there real world use cases that actually can benefit from this [CAT feature] and what are those applications supposed to do once the feature breaks with future generations of processors?"

Reinette replied, "This feature is model-specific with a few platforms supporting it at this time. Only platforms known to support Cache Pseudo-Locking will expose its resctrl interface."

To which Thomas said, "you deliberately avoided to answer my question again."

Go to Full Article

Some of Intel's Effort to Repair Spectre in Future CPUs

Zack Brown — Wed, 25 Jul 2018 11:30:00 +0000

by Zack Brown

Dave Hansen from Intel posted a patch and said, "Intel is considering adding a new bit to the IA32_ARCH_CAPABILITIES MSR (Model-Specific Register) to tell when RSB (Return Stack Buffer) underflow might be happening. Feedback on this would be greatly appreciated before the specification is finalized." He explained that RSB:

...is a microarchitectural structure that attempts to help predict the branch target of RET instructions. It is implemented as a stack that is pushed on CALL and popped on RET. Being a stack, it can become empty. On some processors, an empty condition leads to use of the other indirect branch predictors which have been targeted by Spectre variant 2 (branch target injection) exploits.

The new MSR bit, Dave explained, would tell the CPU not to rely on data from the RSB if the RSB was already empty.

Linus Torvalds replied:

Yes, please. It would be lovely to not have any "this model" kind of checks.

Of course, your patch still doesn't allow for "we claim to be skylake for various other independent reasons, but the RSB issue is fixed".

So it might actually be even better with _two_ bits: "explicitly needs RSB stuffing" and "explicitly fixed and does _not_ need RSB stuffing".

And then if neither bit it set, we fall back to the implicit "we know Skylake needs it".

If both bits are set, we just go with a "CPU is batshit schitzo" message, and assume it needs RSB stuffing just because it's obviously broken.

On second thought, however, Linus withdrew his initial criticism of Dave's patch, regarding claiming to be skylake for nonRSB reasons. In a subsequent email Linus said, "maybe nobody ever has a reason to do that, though?" He went on to say:

Virtualization people may simply want the user to specify the model, but then make the Spectre decisions be based on actual hardware capabilities (whether those are "current" or "some minimum base"). Two bits allow that. One bit means "if you claim you're running skylake, we'll always have to stuff, whether you _really_ are or not".

Arjan van de Ven agreed it was extremely unlikely that anyone would claim to be skylake unless it was to take advantage of the RSB issue.

That was it for the discussion, but it's very cool that Intel is consulting with the kernel people about these sorts of hardware decisions. It's an indication of good transparency and an attempt to avoid the fallout of making a bad technical decision that would incur further ire from the kernel developers.

Note: if you're mentioned above and want to post a response above the comment section, send a message with your response text to ljeditor@linuxjournal.com.

Go to Full Article

Working around Intel Hardware Flaws

Zack Brown — Mon, 30 Apr 2018 12:07:07 +0000

by Zack Brown

Efforts to work around serious hardware flaws in Intel chips are ongoing. Nadav Amit posted a patch to improve compatibility mode with respect to Intel's Meltdown flaw. Compatibility mode is when the system emulates an older CPU in order to provide a runtime environment that supports an older piece of software that relies on the features of that CPU. The thing to be avoided is to emulate massive security holes created by hardware flaws in that older chip as well.

In this case, Linux is already protected from Meltdown by use of PTI (page table isolation), a patch that went into Linux 4.15 and that was subsequently backported all over the place. However, like the BKL (big kernel lock) in the old days, PTI is a heavy-weight solution, with a big impact on system speed. Any chance to disable it without reintroducing security holes is a chance worth exploring.

Nadav's patch was an attempt to do this. The goal was "to disable PTI selectively as long as x86-32 processes are running and to enable global pages throughout this time."

One problem that Nadav acknowledged was that since so many developers were actively working on anti-Meltdown and anti-Spectre patches, there was plenty of opportunity for one patch to step all over what another was trying to do. As a result, he said, "the patches are marked as an RFC since they (specifically the last one) do not coexist with Dave Hansen's enabling of global pages, and might have conflicts with Joerg's work on 32-bit."

Andrew Cooper remarked, chillingly:

Being 32bit is itself sufficient protection against Meltdown (as long as there is nothing interesting of the kernel's mapped below the 4G boundary). However, a 32bit compatibility process may try to attack with Spectre/SP2 to redirect speculation back into userspace, at which point (if successful) the pipeline will be speculating in 64bit mode, and Meltdown is back on the table. SMEP will block this attack vector, irrespective of other SP2 defenses the kernel may employ, but a fully SP2-defended kernel doesn't require SMEP to be safe in this case.

And Dave, nearby, remarked, "regardless of Meltdown/Spectre, SMEP is valuable. It's valuable to everything, compatibility-mode or not."

SMEP (Supervisor Mode Execution Protection) is a hardware mode, whereby the OS can set a register on compatible CPUs to prevent userspace code from running. Only code that already has root permissions can run when SMEP is activated.

Go to Full Article

diff -u: Intel Design Flaw Fallout

Zack Brown — Mon, 19 Mar 2018 14:41:06 +0000

by Zack Brown

For weeks, the world's been talking about severe Intel design flaws affecting many CPUs and forcing operating systems to look for sometimes costly workarounds.

Linux patches for these issues are in a state of ongoing development. Security is always the first priority, at the expense of any other feature. Next would probably be the general speed of a running system for the average user. After that, the developers might begin piecing together any features that had been pulled as part of the initial security fix.

But while this effort goes on, the kernel developers seem fairly angry at Intel, especially when they feel that Intel is not doing enough to fix the problems in future processors.

In response to one set of patches, for example, Linus Torvalds burst out with, "All of this is pure garbage. Is Intel really planning on making this shit architectural? Has anybody talked to them and told them they are f*cking insane?" He went on, "the IBRS garbage implies that Intel is _not_ planning on doing the right thing for the indirect branch speculation. Honestly, that's completely unacceptable." And then he said:

The whole IBRS_ALL feature to me very clearly says "Intel is not serious about this, we'll have an ugly hack that will be so expensive that we don't want to enable it by default, because that would look bad in benchmarks". So instead they try to push the garbage down to us. And they are doing it entirely wrong.

He went on, even more disturbingly, to say:

The patches do things like add the garbage MSR writes to the kernel entry/exit points. That's insane. That says "we're trying to protect the kernel". We already have retpoline there, with less overhead.

So somebody isn't telling the truth here. Somebody is pushing complete garbage for unclear reasons. Sorry for having to point that out....As it is, the patches are COMPLETE AND UTTER GARBAGE....WHAT THE F*CK IS GOING ON?

At one point, David Woodhouse offered a helpful technical summary of the whole situation for those of us on the edge of our seats:

This is all about Spectre variant 2, where the CPU can be tricked into mispredicting the target of an indirect branch. And I'm specifically looking at what we can do on *current* hardware, where we're limited to the hacks they can manage to add in the microcode.

The new microcode from Intel and AMD adds three new features.

One new feature (IBPB) is a complete barrier for branch prediction. After frobbing this, no branch targets learned earlier are going to be used. It's kind of expensive (order of magnitude ~4000 cycles).

Go to Full Article

Meltdown/Spectre Status for Red Hat and Oracle

Charles Fisher — Mon, 05 Feb 2018 20:34:57 +0000

by Charles Fisher

The Red Hat family of operating systems addressed Meltdown and Spectre in its v3.10 kernel quickly, but relied too much upon Intel's flawed microcode and was forced to revert from a complete solution. Oracle implemented alternate approaches more suited to its v4.1 UEK, but both kernels continue to lack full Spectre coverage while they wait for Intel. Conspicuously absent from either Linux branch is Google's retpoline, which offers far greater and more efficient coverage for all CPUs. Auditing this status is a challenge. This article presents the latest tools for vulnerability assessments.

A frenzy of patch activity has surrounded this year's Meltdown and Spectre CPU vulnerability disclosures. Normally quiet microcode packages for Intel chips have seen four updates in the month of January, one of which was finally to roll back flawed code that triggers random reboots. For enterprise-grade hardware, Intel's quality control has left much to be desired.

It is likely premature to deploy new monitoring and compliance tools, and a final solution for this set of vulnerabilities will wait until correct microcode is obtained. Still, it may be important for many organizations to evaluate the patch status of servers running Linux kernels packaged by Oracle and/or Red Hat.

Meltdown patches exist now and should be deployed immediately on vulnerable servers. Remediating all Spectre vulnerabilities requires not only the latest kernels, but also a patched GCC to compile the kernel that is capable of implementing "retpolines", or compatible microcode from your CPU vendor.

RHCK

Red Hat was one of the first Linux distributions to publish guidance on Meltdown and Spectre. It established three files as "kernel tunables" in the /sys/kernel/debug/x86 directory to monitor and control these patches: pti_enabled for Meltdown, ibpb_enabled for Spectre v1 and ibrs_enabled for Spectre v2. Only the root user can access these files.

When these files contain a numerical zero, the patches are not active. If allowed for the CPU, a numerical one may be written to the file to enable the relevant remediation, and a zero may be written later to disable it. This is not always allowed—AMD processors are not vulnerable to Meltdown, and the value in the pti_enabled file is locked to zero and cannot be changed. If the fixes are active and show 1, the performance of the CPU may be reduced. Compatible microcode is required to enable all patches on vulnerable CPUs, which adds new assembler/machine language op codes that erase vulnerable kernel data from CPU pipelines and caches to close the exploit.

Go to Full Article