Thursday, January 12, 2017

Whose bug is it?

My favorite quote from chapter 1, page 1 of http://dl.acm.org/citation.cfm?id=2742705 includes "'If you can fix a hardware bug in firmware, it’s not a bug but a documentation issue.'  —An anonymous hardware manager." I still recall being aghast upon hearing that utterance early in my career, but over time I grew to understand the wisdom. In fact, this inaugural chapter from 2015 book further elaborates on work-around's of hardware concerns that can be implemented in firmware.

This same sentiment was echoed in the position paper https://github.com/vincentjzimmer/Documents/blob/master/Neither-Seen-Nor-Heard.pdf and presentation https://github.com/vincentjzimmer/Documents/blob/master/Neither_Seen_Nor_Heard_Presentation-HLDVT-2010.pdf for the 2010 IEEE International High-Level Design Validation and Test Workshop. Specifically, the firmware can be modified at the 23rd hour to fix a work-around in hardware, leading to a potentially long list of 'firmware changes' during a product's postmortem. This sentiment is expressed in 'Incentives to fix issues in firmware cause root cause to be commonly incorrectly assigned to firmware' of the presentation and the following section of the paper:

      "There could be confusion between the root cause and the fix
      for issues. In particular, there is great pressure to resolve
      hardware issues in firmware due to the order-of-magnitudes
      difference in the cost of resolution. A “spin” of a chip may
      take many weeks and cost millions of dollars whereas a
      firmware fix may cost a few thousand dollars and a day or two
      of total work. In fact, many modern chips are designed so the
      firmware can configure the chips to work around issues in the
      field rather than having the hardware recalled. The public is
      simply told that there is a firmware issue when, in fact, the
      firmware resolved what was a hardware issue. "

I also recall being given a harried call on a Friday night about a the number of 'firmware bugs' on a product. I replied to the caller with some of the sentiments above, including the caveat 'I think that we can get the firmware update out by Monday, but I don't believe we can get a new stepping of the SOC by then.' Regrettably programs don't roll up the reason for the firmware changes and people are just shocked by the cardinality.

And often these bugs are not easy to find Bohrbugs, but Heisenbugs or Hindenbugs where there is a subtle interaction between host firmware and opaque hardware state machines. A week of investigation may yield a solution wherein one line of code changing one bit in a register access yields a solution. A few months ago a firmware manager at a conference related to me that the promotion process in his company entailed upper management reviewing the Git commits of the engineers. The firmware manager had to defend the smaller number of code changes versus the OS kernel guys as each delivering similar business value. But I still recall the final quote from the manager when he smiled and said 'The coolest part is that they are reviewing code changes, right?'



Sometimes these work-around's mitigate errata in the hardware (board, silicon, etc), and sometimes the changes are for cost savings. An example of the latter I recall includes a board design where the integrated Super I/O could be decoded as I/O addresses 0x2E/0x2F or 0x4E/0x4F by application of a pull-down or pull-up resistor, respectively. The hardware design engineer omitted the resistor in order to save costs on the Bill of Materials (BOM), so the boot firmware had to probe for what port value being decoded on each machine restart. And this was a cost savings of a single surface mount resistor.

The above discourse isn't meant to be an apologist view about firmware bugs, though. In general there are many bugs based upon programming flaws, not just hardware work-around's. If one is interested in the latter, take a look at
https://github.com/tianocore/tianocore.github.io/wiki/Reporting-Issues. But hopefully this posting will provide an alternate view into firmware and firmware changes.


Sunday, January 1, 2017

Saying good bye to 2016

I meant to do a final blog of '16 but I instead opted to catch the fireworks at the Seattle Space Needle



I like the end of the year as it hosts the Chaos Communications Conference (33c3). I recall seeing Trammell Hudson's Thunderstrike 2 https://media.ccc.de/v/32c3-7236-thunderstrike_2 over the '15 holiday, and for this '16 holiday I watched his talk on Heads, variously described in
http://hackaday.com/2016/12/29/33c3-if-you-cant-trust-your-computer-who-can-you-trust/
https://trmm.net/heads_33c3
https://www.youtube.com/watch?v=UqxRPLfrpfA

I like Trammell's threat model write up at https://trmm.net/Heads_threat_mode, too. Today we only have the higher level http://www.uefi.org/sites/default/files/resources/Intel-UEFI-ThreatModel.pdf.

It was interesting to see his mention of Intel (R) FSP and also reference our work on pre-OS DMA protection https://github.com/vincentjzimmer/Documents/blob/master/A_Tour_Beyond_BIOS_Using_Intel_VT-d_for_DMA_Protection.pdf.

Another talk of interest for the pre-OS was the review of porting UEFI Secure Boot for virtual machines https://media.ccc.de/v/33c3-8142-virtual_secure_boot. This entails the Open Virtual Machine Format (OVMF) http://www.tianocore.org/ovmf/ variant of EDKII that executes upon QEMU and is used as the guest firmware in projects like KVM, Virtualbox, etc.

The latter talk included a reference to the EDKII lock box https://www.kraxel.org/slides/virtual-secure-boot/#sb-virtual
work https://github.com/vincentjzimmer/Documents/blob/master/A_Tour_Beyond_BIOS_Implementing_S3_resume_with_EDKII_V2.pdf and emulating the full System Management Mode (SMM) infrastructure. The addition of more of the SMM infrastructure in https://github.com/tianocore/edk2/tree/master/UefiCpuPkg was positively mentioned, too.

Speaking of security and 33c3, an interesting read about researchers and industry was posted to
http://laforge.gnumonks.org/blog/20161206-it_security_culture_telecoms/. As long as the flaws are responsibly disclosed such that the conference presentations aren't zero-day events, I cannot argue with their sentiment.

One common element discussed in Heads and the Virtual Secure Boot topics entailed availability of full platforms. In that area there is great progress in having a set of full EDKII platform code in source that works with an Intel(R) FSP for the embedded Apollo Lake (APL) https://ark.intel.com/products/codename/80644/Apollo-Lake#@Embedded SOC (formerly known as "Broxton") in the repository https://github.com/tianocore/edk2-platforms/tree/devel-MinnowBoard3/.

Regarding security and treatment of EDKII https://github.com/tianocore/edk2 issues, we have moved our advisory update to gitbook from the former two PDF postings
https://www.gitbook.com/book/edk2-docs/security-advisory/details. These recent postings represent fixes that honored the industry request for six month embargo of the project updates. Going forward we'd like to auto-generate the advisory from Bugzilla https://github.com/tianocore/tianocore.github.io/wiki/Reporting-Security-Issues, but for now the document is manually curated. There have also been discussions of moving from the advisory document issue enumeration to things like CVE's https://cve.mitre.org/ which is an investigation in progress, too.

Moving into 2017, maybe I'll catch up to George Westinghouse's https://en.wikipedia.org/wiki/George_Westinghouse number of issued US Patents. I left 2016 with 354 issued, whereas George has 361 https://en.wikipedia.org/wiki/List_of_prolific_inventors.

2017 should also feature an update to a couple of UEFI books, including Beyond BIOS
https://www.degruyter.com/view/product/484468 and Harnessing the UEFI Shell
https://www.degruyter.com/view/product/484477. Beyond BIOS was originally published in 2006, so this update will mark over a decade since its first appearance.

It has been an interesting run on this project, with over 17 years on the EFI team and nearly 20 years at Intel. I look forward to what the next wave of technology will bring in '17 and beyond.