[14:23] <Martyn> Good morning everyone!
[16:43] <Martyn> I am -><- this close to screaming at the UEFI code
[16:43] <Martyn> I hate doing first-article bring-up of new chips sometimes
[16:44] <Martyn> Booting can be a real pain in the ass
[16:46] <Martyn> Anyone here have extensive experience with UEFI and arm64?
[16:50] <mrutland> Martyn: I have some
[16:50] <Martyn> HEY!  You're online!
[16:51] <mrutland> Martyn: what trouble are you having?
[16:52] <Martyn> okay, so I am trying to bring up on a Juno board
[16:52] <mrutland> Those have upstream support in EDK2, so what exactly are you trying to do?
[16:53] <Martyn> I'm trying to boot over PCIe
[16:53] <Martyn> so I need to initialize the PCIe bus early, in UEFI
[16:54] <Martyn> ( this is on the A57 )
[16:55] <Martyn> However, once I do that initialization and copy/unpack the kernel into a bootable memory address -- The kernel fails to boot as -it- tries to re-initialize it
[16:56] <mrutland> Martyn: which Juno revision?
[16:57] <Martyn> r2
[16:57] <Martyn> r2p0
[16:58] <Martyn> oh wait -- that's wrong
[16:58] <Martyn> I have Cortex A57 - r1p1
[16:58] <Martyn> and the a53 is r0p3
[16:58] <Martyn> so it's juno r1
[16:59] <mrutland> Hmm, that _should_ work. Which FW, which kernel, and are you using ACPI or DT?
[17:00] <Martyn> I'm using devicetree, kernel 4.6-rc5 ( current mainline ), and unsure which firmware
[17:01] <Martyn> Also, I've tested booting from SATA just to make sure that I'm not having some weird XAUI issue, and I can boot from SATA just fine
[17:01] <Martyn> also I can boot from 10GigE ( also XAUI )
[17:02] <mrutland> Martyn: So which PCIe device(s) do you have plugged in in the failing case?
[17:02] <Martyn> but when I try to cluster boot over PCIe .. somehow initializing the PCIe bus early results in a deadlock condition once the mainline kernel starts to boot.  I don't have a 1wire debug available, or KEIL debug over JTAG, so I'm kind of stuck
[17:03] <mrutland> Martyn: I was under the impression that the SATA was over PCIe, so what do you mean by "boot over PCIe"? Which device?
[17:03] <Martyn> just a PCIe debug board I built.  I'm clocking the data in, and all the data is well ordered
[17:04] <Martyn> I'm trying to do something San Mehat did back at VA Linux Systems back in the early 2000's ... I am putting together a TCP/IP stack over PCIe, so that I can use one a64 card to boot another directly.
[17:05] <mrutland> Martyn: given that SATA works (over PCIe), the issue sounds specific to your debug board; I'm not sdure I can be of much help. :/
[17:06] <Martyn> so the board that's plugged into the PCIe bus is fairly simple -- a SPEAr 1340 w/ single channel PCIe 2.0 RC/EP port
[17:06] <Martyn> and all it does it clock in the data -- the kernel does load correctly into memory, and I have uncompressed it into RAM and can even start booting it
[17:07] <Martyn> the -problem- is that the entire PCIe bus locks up right afterwards.  I don't think the PCIe bus likes being re-initialized by when the kernel reboots
[17:07] <Martyn> reboots/boots
[17:07] <Martyn> Is there a way to prevent the kernel from attempting to re-initialize the PCIe bus, given that I've already done it during UEFI?
[17:12] <Martyn> oh .. interesting.  I just tried to checkout a fresh copy of EDK2 -- issues with git
[17:12] <Martyn> warning: remote HEAD refers to nonexistent ref, unable to checkout.
[17:12] <mrutland> Martyn: I don't follow why that would only be a problem when your board is plugged in? Surely it re-initialises it anyway?
[17:13] <Martyn> That's what I was thinking... PCIe has to be initialized to get SATA working, so I figured this would be safe as houses
[17:13] <Martyn> however, I'm not doing anything significantly more complex than what happens during SATA boot
[17:14] <Martyn> I am copying data from the card I built, the data copies in, and as soon as the kernel does a PCIe probe -- BANG
[17:14] <Martyn> deadlock
[17:15] <Martyn> Bah .. I need to get JTAG on this
[17:19] <mrutland> It may be worth asking on the linux-arm-kernel mailing list; this sounds like a PCIe issue, and people there are more familiar with the ARM/ARM64 details for that than I am
[17:19] <Martyn> Yeah.  With the A57/v8 chips starting to get a lot more popular out there ... I figured that y'all might be doing some unusual cluster booting techniques like I am here
[17:19] <Martyn> Will do
[17:20] <Martyn> LKML can be ... contentious :)
[17:20] <Martyn> LAKML is also .. contentious :)
[17:22] <mrutland> I boot my Juno with GRUB and TFTP, which works well enough for me
[17:23] <mrutland> There are some nice people on LAKML (and LKML), if you can ignore the noise
[17:25] <Martyn> Heh .. this work doesn't end with the Juno, of course.  just like with Smooth-Stone/Calxeda -- early steps
[17:25] <Martyn> Well, for now I'll stick to booting on 10gigE
[17:25] <Martyn> But I need to crack this PCIe issue
[19:03] <Martyn> mrutland : GAH!  solved it.
[19:04] <Martyn> mrutland : It was an out-of-order transaction happening on the PCIe bus... for some weird reason, the PCIe IP on the Juno isn't allowing it.
[19:05] <Martyn> IS there a limitation on the a57's architecture that breaks when there are out-of-order transactions on PCIe? ( like there was on the A9's? )
[19:06] <Martyn> Is relaxed ordering allowed?
[19:06] <Martyn> ( If I turn off the relaxed ordering bit in the TLP, things work )
[19:07] <Martyn> that would be bad news for things like 8 and 16 lane graphics cards...