Bug 75 - [EFI-32] booting "mixed" (x86_64) Linux
Summary: [EFI-32] booting "mixed" (x86_64) Linux
Status: CONFIRMED
Alias: None
Product: Syslinux
Classification: Unclassified
Component: syslinux (show other bugs)
Version: None
Hardware: PC Linux
: --- normal
Assignee: H. Peter Anvin
URL:
Depends on:
Blocks:
 
Reported: 2017-03-09 21:06 PST by Gerardo Exequiel Pozzi
Modified: 2020-01-28 12:32 PST (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gerardo Exequiel Pozzi 2017-03-09 21:06:05 PST
Hello

While adding support to Arch Linux ISO to boot x86_64 on EFI-32 systems, I encountered this issue.
Booting syslinux on EFI-32 (QEMU/OVMF, but also on real hardware) Linux 4.9.11 and 4.10.1 resets (ovmf) / hang (real-hw), this happens just after "jump" loading vmlinuz or if specified after initrd.

Linux 4.4.52 boots OK, also if booting 4.9/4.10 on EFI-64.

CONFIG_EFI_MIXED is enabled in all cases.

If more info is needed please let me know.

Thanks.
Comment 1 Gerardo Exequiel Pozzi 2017-03-09 21:14:00 PST
This happens 6.03 + some patches from our distro and 6.04-pre1 build from git.
Comment 2 Ady 2017-03-10 00:46:00 PST
(@Gerardo, you probably already know some of the following details. I am adding them here in the tracker for convenience.)

This _sounds_ similar to the problem booting some specific hardware such as some Chromebooks, which depends on kernel version / configuration.

Example: Acer C720 boots OK with kernel 4.8.x but failed with kernel 4.9.x+. This was supposed to be resolved by:

* 2015Feb:
"load_linux: correct a type"
 http://repo.or.cz/w/syslinux.git/commit/83aad4f69065509ba5b1c080edccfed316a4cff0 

 http://repo.or.cz/syslinux.git/patch/83aad4f69065509ba5b1c080edccfed316a4cff0 


* 2015Apr:
"com32/lib/syslinux/load_linux.c: update prot_mode_base"
 http://repo.or.cz/w/syslinux.git/commit/0a2dbb3392ee710838bea6bda80d4daad6b54780 

 http://repo.or.cz/syslinux.git/patch/0a2dbb3392ee710838bea6bda80d4daad6b54780 


These 2 commits are already part of Syslinux 6.04-pre1, so probably there is some additional issue to resolve for Syslinux EFI_IA32 to work correctly with kernels 4.9.x+ (considering that other UEFI bootloaders boot OK).

Additionally, the following items should be considered:

* OVMF / KVM / TIANOCORE  have been known to show problems with syslinux.efi before, and the problems were resolved by extra patches in those (rather than having to patch syslinux.efi).

* The building architecture has influenced the behavior OK/KO of the resulting binaries (corresponding to architectures different than the building arch).

* Testing with official pre-built binaries instead of building your own and testing with binaries built on other Linux distributions should be considered. I have had different experiences with Syslinux 6.04-pre1, depending on where I took the binaries from (e.g. upstream Syslinux binaries vs. Fedora25's). Unfortunately, there are very few distributions providing Syslinux binaries for EFI_IA32 (their syslinux.spec / rpm's are incorrect, or they mix / confuse the architecture BIOS' "x86" with EFI's "IA32").

* Considering the previous point, there are already-built binaries available from current Debian Sid (which include the specific patches I mentioned above) you could test (these binary packages can be downloaded from any OS):
** ldlinux.e32 can be found in "syslinux-common_6.03+dfsg-14.1_all.deb".
** syslinux.efi for EFI_IA32 can be found in "syslinux-efi_6.03+dfsg-14.1_all.deb".

* At the time I am writing this, Syslinux 6.04-pre1 is already more than a year old. Testing binaries built from current git master head http://repo.or.cz/syslinux.git/shortlog/refs/heads/master might be worth a try. Unfortunately, ATM the Syslinux code is still based on _old_ deps., including the gnu-efi submodule (based on its commit ab54e2b40e914d0ca01dc3d44c8d4eb8517bf999 from 2014Feb).

* Considering that ArchLinux tends to use recent / modern versions of packages that influence the resulting Syslinux binaries, please take a look at http://www.syslinux.org/wiki/index.php/Building for some of the known building conflicts.

* A recent attempt to build Syslinux binaries in Debian FTBFS, "ldlinux.elf: Not enough room for program headers". The problem was resolved by:
 "adding --no-dynamic-linker to the linker command lines" 
for the package version "6.03+dfsg-14.1" (which I mentioned above). For more info see http://www.syslinux.org/archives/2017-February/025531.html .

* Please remember that ATM syslinux.efi is capable of booting from FAT filesystems only. Booting optical media or USB devices being dd'd from ISO images will fail to boot in UEFI mode.
Comment 3 Gerardo Exequiel Pozzi 2017-03-10 18:51:24 PST
(@Ady: Yes I am aware, we have a open ticket about CB in our bugtracker)

Some other tests:
* [OK] Boot i686 Linux-4.10.1 on EFI-32 (OVMF) with syslinux-6.04-pre1 [build from source]
* [FAIL] Boot x86_64 Linux-4.10.1 on EFI-32 (OVMF) with syslinux-6.04-pre1 and 6.03 [pre-build from tarball]

I did not built against master because I did not see any relevant changes, most of them are for PXE, but I can do if needed.

For my test I am just using Qemu on FAT just to minimize potential issues with "hybrid-ISO" or "El Torito".

Thanks for all the info Ady!
Comment 4 Ady 2017-03-11 08:51:27 PST
(In reply to Gerardo Exequiel Pozzi from comment #3)

> 
> Some other tests:
> * [OK] Boot i686 Linux-4.10.1 on EFI-32 (OVMF) with syslinux-6.04-pre1
> [build from source]
> * [FAIL] Boot x86_64 Linux-4.10.1 on EFI-32 (OVMF) with syslinux-6.04-pre1
> and 6.03 [pre-build from tarball]


IMHO, testing this case with 6.03 at this point is a waste of time. Testing with git master head is much more relevant for this case. I would also suggest testing with the 2 binaries from Debian Sid (see my comment #1 above).

As I mentioned in my prior post, the behavior of the resulting binaries in each build environment are not consistent (e.g. I have seen a different behavior when using 6.04-pre1 binaries from Fedora 25 than when using the upstream pre-built official binaries from kernel.org).


> 
> I did not built against master because I did not see any relevant changes,
> most of them are for PXE, but I can do if needed.


At the time I am writing this post, the git master head has commits (after 6.04-pre1) related to the building environment, not only related to PXE. Bug #71 is just one of several issues presented after 6.04-pre1, and it was partially resolved (emphasis on _partially_).


> 
> For my test I am just using Qemu on FAT just to minimize potential issues
> with "hybrid-ISO" or "El Torito".
> 
> Thanks for all the info Ady!


As I mentioned before, OVMF has presented its own issues with Syslinux in the past (and the problem was not always in Syslinux). So, I'd like to shamelessly request a few more details / tests if possible.


_ You originally posted that older kernels boot OK and the failure starts with kernel 4.9.x. Are the older kernels (that boot OK) for i686, or are they for x86_64 instead?

_ You mentioned tests in real HW. Which one? (please be as specific as possible)

_ Are all these kernels successfully booting with grub2-efi32 (or any other EFI_IA32 bootloader) in these same environment(s)?

_ Would you please post the exact size (Bytes) of these kernels? Are these kernels publicly available somewhere?

_ If it is not too much to ask, could you please test also in VirtualBox in _EFI mode_?

_ Would you please post the exact complete QEMU command line you used? This should help in order to replicate the behavior.

_ Would you please post the complete content of syslinux.cfg? In theory this should not be a factor in this case, but it might help in replicating the issue.

Thank you for the testing and feedback.


> (@Ady: Yes I am aware, we have a open ticket about CB in our bugtracker)

off-topic:
In ArchLinux, the Syslinux package version 6.03-7 was supposed to close a couple of those tickets. Moreover, Syslinux 6.04-pre1 should had been provided in ArchLinux a long time ago in [testing] (without moving it to [core]) and the Syslinux git master head should be in AUR, as it used to be. These two alternatives would had helped ArchLinux's users (and they would also help in closing additional open tickets in ArchLinux).
Comment 5 Mortan 2017-03-12 19:32:40 PDT
(In reply to Ady from comment #4)
Hi, I opened the report on Arch's bug tracker (https://bugs.archlinux.org/task/53182) and have been helping to test this on real i386-efi only hardware.

Here are the questions that I can answer:
>_ You mentioned tests in real HW. Which one? (please be as specific as possible)
Lenovo ideapad MIIX 300-10IBY
This review website has details on it: https://www.notebookcheck.net/Lenovo-IdeaPad-Miix-300-10IBY-Convertible-Review.159389.0.html

>_ Are all these kernels successfully booting with grub2-efi32 (or any other EFI_IA32 bootloader) in these same environment(s)?
With Linux 4.9.11-1 on an existing installation I was able to boot from one of these x86_64 i386-efi only computers with:
pacman -S --needed dosfstools efibootmgr grub intel-ucode
grub-install --target=i386-efi --efi-directory=/boot --removable
grub-mkconfig -o /boot/grub/grub.cfg

I have not tested Linux 4.10.1-1 yet.

>_ Would you please post the exact size (Bytes) of these kernels? Are these kernels publicly available somewhere?
The binaries and sources can be located here: https://www.archlinux.org/packages/core/x86_64/linux/

>_ If it is not too much to ask, could you please test also in VirtualBox in _EFI mode_?
I don't believe that VirtualBox supports setting the EFI and the CPU to two different architectures.

>_ Would you please post the complete content of syslinux.cfg? In theory this should not be a factor in this case, but it might help in replicating the issue.
Here is the current patch file: https://bugs.archlinux.org/task/53182?getfile=15071

If you have access to an Arch installation you can build the image that we are testing with:
sudo pacman -S --needed archiso git
cp -r /usr/share/archiso/configs .
curl -JLO https://bugs.archlinux.org/task/53182?getfile=15071
git apply 0001-configs-releng-Add-32-bit-EFI-support.patch
sudo ./configs/releng/build.sh -v
Comment 6 Ady 2017-03-13 00:37:37 PDT
Thank you for the information.

I am aware that you are using ArchLinux for your tests. Unfortunately, to be able to replicate the situation, I'd like to request, if it is possible, for the information to be provided in a "plain" form, less dependent on having to set up an ArchLinux installation for these tests.

For example, instead of providing a patch, please post the whole content of your:
EFI_SYSTEM_PARTITION/EFI/BOOT/SYSLINUX.CFG 
or of:
EFI_SYSTEM_PARTITION/EFI/BOOT/SYSLIA32.CFG 
in a plain form.

One reason for such request, besides being able to test under others OS, is that there are specific issues that can be particular to ArchLinux. For example, at the time I am writing this post, the Syslinux package in ArchLinux (6.03-7) is not capable of executing a LABEL entry other than the DEFAULT from CLI (Note: such behavior does not happen in every OS, and the reasons for it are off-topic here). Another example would be the specific versions of Syslinux's (building) dependencies, which could be part of ArchLinux but could be different under other OS (and this is particularly relevant for Syslinux, as I already expressed in my prior posts).

BTW, since we are troubleshooting, the config files for Syslinux for these tests should be as KISS as possible (but not less than needed :).


@Gerardo,

To replicate the problem, either we have to have access to specific hardware, or we need to trust qemu or similar emulation tools (up to a certain point, since OVMF/KVM and others have shown their own problems, as I expressed already). Thus, posting the command line you used in qemu in a plain form should be helpful. Would you please post it?

If possible, I'd like to request a test using Syslinux current git master head http://repo.or.cz/syslinux.git , please?

In a nutshell, let's make this report (about Syslinux EFI_IA32 + (x86_64 kernels >= 4.9.x)) different than https://bugs.archlinux.org/task/53182 (very much related, but different). If we solve this one, then the one for ArchLinux would be solved too.

Thank you again for the tests and feedback.
Comment 7 Gerardo Exequiel Pozzi 2017-03-13 20:56:20 PDT
@Ady: OK please give some time for all these tests during next days.
@Mortan: Yes you can set EFI32 from vboxmanage.

syslinux.cfg looks very basic:
--
default arch64

label arch64
    linux vmlinuz.efi
--

qemu command also is the most basic
--
qemu-system-x86_64 -m 1G -bios ovmf_ia32.bin -drive file=efiboot.img
--

Filesystem of efiboot.img is 32M FAT32:
--
/mnt/dsk/
└── EFI
    └── boot
        ├── bootia32.efi
        ├── ldlinux.e32
        ├── syslinux.cfg
        └── vmlinuz.efi
--
Comment 8 Gerardo Exequiel Pozzi 2017-03-15 23:56:38 PDT
Ugh! *big* sorry I made a mistake, when I used linux-4.4.52 was an 32-bit kernel in a EFI32, no "mixed" mode, this is why was able to boot.
In other words, 4.4 fails like any other 64-bit kernel version when booting in EFI32.

For now I tested:

* syslinux upstream (6.04-pre1)
* archlinux package (6.03-7)
* debian package (6.03+dfsg-14.1)
* building syslinux from source at master (b4cc12b9)

all with the same results:

* efi32 -> linux32 [OK]
* efi32 -> linux64 [FAIL]
* efi64 -> linux32 [OK]
* efi64 -> linux64 [OK]

This is on OVMF (r21286.7babb4372e-1)

Pending:
* EFI from vbox
* using grub
Comment 9 mirh 2017-12-02 09:20:23 PST
Any update for this?
Comment 10 Ady 2018-01-27 09:17:00 PST
The possibility of booting 64-bit kernels from 32-bit EFI firmware was added in version 3.15 of Linux, around the first quarter of 2014.

It is important to note that this feature in Linux kernels is achieved by means of the EFI handover protocol, not by the EFI boot stub.

Around the same time (2013Q4 - 2014Q1) Matt Fleming (author of the EFIstub) stopped contributing code to syslinux.efi.

My guess is that something in (efi32) syslinux.efi + ldlinux.e32 regarding the EFI handover protocol is incomplete / incorrect / inaccurate / buggy / outdated.

A confirmation about this guess could come from testing the same 64-bit kernel(s) booting in (virtual) UEFI IA32 firmware by means of:

_ efi32/syslinux.efi + ldlinux.e32

_ GRUB2

and comparing their results.

Testing EFI bootloaders that depend exclusively on the EFIstub to be able to boot Linux kernels would _not_ be needed.
Comment 11 mirh 2020-01-19 17:10:08 PST
That something could be this maybe
https://www.spinics.net/lists/linux-efi/msg17576.html

It is also mentioned everyone should be able to try mixed mode on his own system with OVMF/qemu.
Comment 12 mirh 2020-01-28 12:32:17 PST
Duh, the whole thing was basically rewritten for 5.6. And funnily enough, it seems like 64 bit UEFI booting 32 bit kernels is also a thing. 

https://lore.kernel.org/lkml/20200128074850.GA27168@gmail.com/