[06:29] * alkisg got access to one of the schools with black screens, let's see... i5-6400, HD Graphics 530 [8086:1912] (rev 06) [06:30] Updating from ppa: libegl-mesa0 libegl1-mesa libgbm1 libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libglx-mesa0 libosmesa6 libwayland-egl1-mesa libxatracker2 mesa-va-drivers mesa-vdpau-driver [07:13] tjaalton: sorry for the direct ping; I updated to the mesa from the PPA, and lightdm still fails. Anything else I can try? Any logs to keep, while the server is booted without nomodeset? [07:14] boot an earlier kernel [07:15] .44 or .29? [07:15] * alkisg tries with this first: vmlinuz-4.15.0-44-generic [07:15] -44 is broken [07:15] OK, 29 then, from the repositories [07:15] 4.15.0.20.23 500 [07:16] Sorry I meant that one, that is available from the stock repositories [07:16] that's the metapackage [07:16] apt install linux-image-4.15.0-20-generic linux-modules-4.15.0-20-generic linux-modules-extra-4.15.0-20-generic [07:16] that should do it [07:23] GRUB_DEFAULT='gnulinux-advanced-00d0b350-2f42-4574-8abe-b53ede7da033>gnulinux-4.15.0-20-generic-advanced-00d0b350-2f42-4574-8abe-b53ede7da033'; update-grub; reboot [07:25] tjaalton: the same. `uname -r` = 4.15.0-20-generic, gput-manager, lightdm ==> failed [07:26] Xorg.log reports success even though it fails: https://termbin.com/6l2b [07:26] Plain `xinit` reports: i965: Failed to submit batchbuffer: Invalid argument [07:26] downgrade mesa then [07:27] I'll need a bit of help on that, which packages does it involve, any handy commands? Trying... [07:27] you just mentioned the list of packages that got updated.. [07:27] apt install foo=ver [07:27] That was the updates from the ppa, not the whole list [07:28] echo $(dpkg -l '*mesa*' | awk '/^ii/ { print $2 }') [07:28] libegl-mesa0:i386 libegl1-mesa:i386 libgl1-mesa-dri:i386 libgl1-mesa-glx:i386 libglapi-mesa:i386 libglu1-mesa:i386 libglx-mesa0:i386 libosmesa6:i386 libwayland-egl1-mesa:i386 mesa-utils mesa-va-drivers:i386 mesa-vdpau-drivers:i386 [07:28] apt-cache policy libgl1-mesa-dri for instance will show that the original version is still there [07:28] Are those enough, to downgrade? [07:28] maybe, and it'll complain if there are more dependencies [07:28] Ty, trying [07:28] I'll try 18.0.0~rc5-1ubuntu1 [07:29] that's what main has [07:29] and bionic shipped with [07:29] I have those available: 18.2.8-0ubuntu0~18.04.1~ppa1, 18.2.2-0ubuntu1~18.04.1, 18.0.0~rc5-1ubuntu1 [07:29] Which one should I try? I think the first 2 failed [07:30] the last [07:30] So I guess, try with the stock bionic? Yup. [07:36] apt install libegl-mesa0:i386=18.0.0~rc5-1ubuntu1 libegl1-mesa:i386=18.0.0~rc5-1ubuntu1 libgl1-mesa-dri:i386=18.0.0~rc5-1ubuntu1 libgl1-mesa-glx:i386=18.0.0~rc5-1ubuntu1 libglapi-mesa:i386=18.0.0~rc5-1ubuntu1 libglx-mesa0:i386=18.0.0~rc5-1ubuntu1 libosmesa6:i386=18.0.0~rc5-1ubuntu1 libwayland-egl1-mesa:i386=18.0.0~rc5-1ubuntu1 mesa-va-drivers:i386=18.0.0~rc5-1ubuntu1 mesa-vdpau-drivers:i386=18.0.0~rc5-1ubuntu1 libgbm1=18.0.0~rc5-1ubunt [07:36] u1 libegl1-mesa=18.0.0~rc5-1ubuntu1 libegl1=1.0.0-2ubuntu2 libglvnd0=1.0.0-2ubuntu2 libgl1-mesa-glx=18.0.0~rc5-1ubuntu1 libgl1=1.0.0-2ubuntu2 libglx0=1.0.0-2ubuntu2 [07:36] ...a long list there, probably cut by irc :D [07:36] why would it want to downgrade libegl1 et al [07:37] Version mismatches [07:37] huh? [07:37] ah ok [07:37] nevermind [07:37] doesn't matter [07:37] I build all this list slowly, because each one complained about some other package version mismatch, [07:37] now it wants to remove vlc and kio etc, but I don't know how else to do it [07:39] The following packages will be REMOVED: kinit kio kolourpaint libgles2 libkf5kdelibs4support5 libkf5kdelibs4support5-bin libkf5notifications5 libkf5parts-plugins libkf5parts5 libkf5sane5 libkf5wallet-bin libkf5wallet5 libkwalletbackend5-5 libwayland-egl1 phonon4qt5 phonon4qt5-backend-vlc vlc vlc-plugin-video-output [07:39] I can put them back on after we test, shall I continue? [07:40] downgrade wayland [07:40] too [07:41] They're using MATE, so I hope it won't even load wayland [07:41] and libgles2 [07:42] apt install libgles2=1.0.0-2ubuntu2 ; reboot [07:44] tjaalton: that worked [07:44] downgrade fixed it? [07:44] Yes [07:44] ok, file a bug upstream and on lp [07:45] Thank you, doing so [07:45] wondering if it's lightdm which is triggering it [07:46] Maybe MATE as well [07:46] But plain xinit doesn't work either [07:46] I.e. it's possible that ubuntu gnome users don't even see that, I don't know [07:46] how did you run xinit if it didn't have any display? [07:46] From something like ssh [07:47] (epoptes, remote control) [07:47] k [07:48] does xubuntu use the same bits? [07:49] probably not [07:49] Unfortunately I don't have a hardware that fails locally, and it's very hard to test other distros than mate remotely [07:49] I'll try to find some affected hardware tomorrow [07:50] ubuntu mate [07:50] I'll try that on snb [07:50] I file it in https://bugs.freedesktop.org/enter_bug.cgi under Drivers/DRI/i915? It warns "that's the old driver" [07:50] i965 [07:50] Ty [07:51] unless it's migrated to gitlab, not sure [07:51] They also have a 18.3 there, can I try that in Ubuntu? [07:51] xorg-edgers or something? [07:53] I can push that to x-swat updates [07:58] lspci => 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 530 [8086:1912] (rev 06) Subsystem: ASUSTeK Computer Inc. HD Graphics 530 [1043:8694] Kernel modules: i915 [07:58] does dmesg have anything btw? [07:58] it's sandybridge [07:58] tjaalton: i915? [07:58] yes [07:59] And I should report it under i965? [07:59] the dri driver is i965 [07:59] ty [07:59] dmesg is clean [07:59] https://termbin.com/ip4e [08:00] hmm [08:01] it's skylake actually [08:01] A mail was just sent to ltsp-discuss, "black screen on cinnamon" [08:01] I think they're using lightdm too [08:03] k [08:04] https://bugs.freedesktop.org/show_bug.cgi?id=109583 [08:04] Freedesktop bug 109583 in Drivers/DRI/i965 "Black screen on skylake after 18.0 => 18.2 update" [Normal,New] [08:04] Filing in launchpad now... [08:05] check lightdm logs btw [08:05] if they have the same error [08:06] I don't think systemd even starts lightdm as it depends on gpu-manager [08:06] Ah, I can try restarting gpu-manager, I think that allows systemd to start lightdm and let it fail there.. [08:06] so how does it fail then? [08:07] and what's failing.. [08:08] so you're running 32bit on skylake? [08:08] gpu-manager, with some error message about not able to initialize the cards etc [08:08] Yes. Hmm, I'll try to see if 64bit installations are affected [08:09] Let me update mesa so that I see the errors again [08:09] just run full dist-upgrade [08:09] Yeah that's the easy one :) [08:10] Launchpad bug: https://bugs.launchpad.net/mesa/+bug/1815172 [08:10] Launchpad bug 1815172 in mesa (Ubuntu) "Black screen on skylake after 18.0 => 18.2 update" [Undecided,New] [08:10] great [08:15] i wonder if that xinit is doing the right thing, or if it's trying to use remote x somehow [08:15] tjaalton: nah I use it a lot of times, it works great for troubleshooting [08:15] ok then [08:15] I can even get remote access after that, I know that step [08:15] *remote xorg, with vnc [08:16] systemctl status gpu-manager: https://termbin.com/kkix [08:16] 18.3.3 test build on cosmic fine, now bionic [08:16] Thanks [08:17] gidarakos: could you tell the school to leave the server open for the whole weekend? [08:17] *running [08:17] tseliot: ^ [08:17] tseliot: the gpu-manager error [08:17] systemctl status lightdm: https://termbin.com/ybev [08:18] check it's logs then [08:18] tseliot: probably nothing [08:20] Clean lightdm.log, after doing: systemctl start gpu-manager; systemctl start lightdm: https://termbin.com/ssrb [08:20] gpu-manager runs for a bit, and allows the lightdm service to run without dependency issues, then enter the failed state [08:20] then look at the x log [08:20] [+0.18s] DEBUG: XServer 0: X server stopped [08:21] it was respawning every 0.2s [08:21] Last xorg.log: https://termbin.com/92rx [08:21] huh [08:22] I never see any issues in xorg log in these tests [08:22] so it's a client crashing that takes the server down without leaving a trace in the log [08:22] enable apport, check /var/crash [08:23] Isn't xinit a simpler test there? [08:23] maybe [08:23] As it doesn't involve lightdm and anything? [08:24] xinit again gives a clean xorg.log, with just this in stderr: i965: Failed to submit batchbuffer: Invalid argument | xinit: giving up | xinit: unable to connect to X server: Connection refused | xinit: server error [08:24] ok [08:25] Apport is enabled afaik, but nothing in /var/crash [08:25] https://wiki.ubuntu.com/Apport#How_to_enable_apport [08:26] alright [08:29] uploaded 18.3.3 to ppa:ubuntu-x-swat/updates [08:29] Ty, adding... [08:30] not built yet [08:31] alkisg: Ok, I'll ask for it and I'll let you know.. [08:31] 18.2.8-0ubuntu0~18.04.1~ppa1 500 for now, waiting [08:31] Thank gidarakos [08:33] alkisg: you are welcome! :) [08:47] Another school that has the issue, said: 64bit installation, Intel Corporation HD Graphics 530 [8086:1912] (rev 06)   Subsystem: Dell HD Graphics 530 [1028:06b7] [08:47] So fortunately it doesn't just happen on 32bit systems :) [08:47] k [09:06] Upgrading to 18.3.. [09:08] Same issue [09:10] cool [09:10] no need to bisect [09:11] Eh, unless we end up needing to bisect between 18.0 and 18.2, to pinpoint the issue :D [09:11] nah [09:12] what does xinit try to run, btw? [09:12] tjaalton: would it make sense to try with a -hwe kernel, something newer than 4.15? [09:12] xinit runs xterm [09:12] k [09:12] When not provided any params [09:13] right [09:33] tjaalton: is that a hybrid system? [09:33] no [09:34] Debug log: https://termbin.com/6n12 [09:34] then I doubt the error is fatal [09:34] After: echo 0xf > /sys/module/drm/parameters/debug; xinit; dmesg [09:34] right, it didn't prevent lightdm from starting [09:44] alkisg: chris (ickle) is also on #intel-3d [09:57] hum, should ubuntu mate support efi boot? [09:58] probably not [09:58] doesn't have an EFI dir [10:00] trying it on a kabylake [10:02] seems to hang [10:05] hmm no, it was probably respawning the desktop and didn't accept any input other than power button [10:07] yep, works with nomodesest [10:07] -set [10:11] oh right, 32bit doesn't support efi.. [10:17] alkisg: try with the hwe kernel [10:19] linux-image-4.18.0-14-generic [10:19] and the modules [10:27] alkisg: try the kernel from proposed [10:28] alkisg: nevermind [10:28] but try hwe kernel [10:41] 64bit daily image seems to work fine here [11:03] alkisg: I'm running 64bit mate with stock bionic kernel & X and things work fine :/ [11:23] Back [11:24] tjaalton: AFAIK all ubuntu CDs have the EFI dir, but in the fat special file system, not in the iso9xx noe [11:24] *one [11:24] I do have MATE installed under EFI [11:24] Ah 32 bit, ok, yeah [11:24] * alkisg tries with -hwe [11:26] apt install linux-generic-hwe-18.04 [11:27] I'll install 32bit version next [11:28] tjaalton: Does it run with 4.15 for you? [11:28] 64bit did [11:28] Which card, the same as the schools? [11:28] kabylake, same gen9 as skylake [11:29] Maybe it depends on the specific model? [11:29] Rebooting with 4.18... [11:31] 4.18.0-14-generic => failed [11:31] k [11:32] skl gt2 is pretty much the same as kbl gt2, which this is [11:33] A school did report failure with their 64bit installation, so I don't know what to say [11:33] could be something else [11:44] meh, installing 32bit messed up grub/efi [12:08] Do you get the rescue prompt? [12:11] yes [12:11] set root=(hd0,gpt1) [12:11] configfile /boot/grub/grub.cfg [12:11] I think this should work... [12:11] Point it to your 64bit instalaltion [12:11] After it boots, run grub-install /dev/sda [12:12] so what after configfile? [12:12] it just clears the screen [12:13] Ouch, it should have displayed the grub menu [12:13] set root=(hd0,gpt ==> tab there for autocompletion [12:13] If you installed in 2 partitions, select the other one [12:14] nah [12:15] Eh, then manually load linux/initrd [12:15] linux /boot/vmlinuz root=/dev/sda1 [12:15] initrd /boot/initrd.img [12:15] boot [12:15] And if grub.cfg was empty,you'll need `update-grub` as well, to generate a proper one [12:16] i'll just fix it from a live session [12:26] ha, was able to boot the 32bit install after all, and it works fine [12:39] Another affected school has a slightly different card: [12:39] root@pc02:~# lspci -nn -k | grep -A 2 VGA 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:5912] (rev 04) Subsystem: ASRock Incorporation HD Graphics 630 [1849:5912] Kernel driver in use: i915 [12:44] that's kbl gt2, which I have [12:45] Hrm. Stranger and stranger. [13:04] Should I mention in the bug report that we're using modesetting instead of the intel driver? (the Ubuntu defaults) [13:04] no [13:04] it's seen on the X log [13:04] OK, ty [13:22] tjaalton: you can boot an efi installation from grub-rescue-cd, by typing that "configfile" command above, if you feel like spending the time to test that... [13:54] yeah I fixed it already [14:16] * alkisg tests with linux-image-4.20.7-042007-generic_4.20.7-042007.201902061234_i386.deb ... [14:18] oh right, I was about to push mesa 19.0rc2 somewhere.. [14:20] They're asking me for the output of `/usr/bin/glxinfo -B`, how should I provide that, with nomodeset or with AccelMethod=none? [14:20] I don't have a display without those, and I guess no glx in either case? [14:21] you should probably get the hw on your desk [14:21] otherwise it'll be painful [14:21] as it is [14:22] I only have one so far in my city, and it's the school server... I'll see if I can replace the server with something else, and keep the disk maybe... [14:22] The others are in other cities [14:24] you don't have skylake? [14:25] I guess legacy bios is the key [14:25] but I can't wipe my machines [14:25] I only have this in my office: 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) [14:25] these machines [14:25] Wait, I think it works fine with 4.20 [14:25] Let me reboot without xorg.conf... [14:26] that's a haswell [14:26] gen7 gpu [14:26] I don't have anything more recent here :( [14:26] But I tried 4.20 in the remote school, and I think it works, I'll be sure after the reboot [14:26] cool [14:26] Not sure how that will help us for a "real" solution, but it's surely progress [14:28] Yup, it seems to work ifne [14:28] fine [14:29] it's a kernel bug then [14:30] fixed between 4.18..4.20 [14:30] ickle might have ideas.. post to the upstream bug [14:30] looking at changes to i915_gem_execbuffer.c gives some hints [14:32] So, mesa=18.0 works, mesa=18.2 breaks, yet it's a kernel issue? :D [14:32] Meh these programmers! :P [14:32] * alkisg has been a programmer / sysadmin for 25 years now... [14:35] needs a newer kernel :) [14:50] alkisg: you could bisect the kernel by using mainline builds between 4.18 and .20 [15:03] alkisg: if you're still around, boot v4.19.0, then .3 [15:03] .3 has plenty of changes for i915 [15:06] tjaalton: thanks, will do [15:06] tjaalton: so the end result will probably be a kernel cherrypick for 4.15 in bionic? [15:07] es [15:07] yes [15:07] ty [16:28] tjaalton: if you do build a kernel, please build both 32bit and 64bit, so that schools test both of them... [16:28] alkisg: need to push to a ppa [16:28] but can do [16:28] ppa with tons of space, I have that [16:29] Great, even better than wget [16:33] could probably cross-build locally, but meh [16:33] I'll verify the build first as 64bit [16:33] locally, then upload [16:41] alkisg: maybe I'll just revert softpin support from mesa, as ickle suggested.. more likely to get in the release [16:41] Sure, whatever is easier [16:41] I can test mesa then,if you want me to [16:45] yes, getting late, maybe not today, we'll see [16:46] I'll check the code first how to disable it [16:46] np, take your time, thanks for all the help [17:01] must be this https://pastebin.com/tAeRSD1v [17:01] so I'll revert that [17:15] alkisg: I've pushed it to https://launchpad.net/~canonical-x/+archive/ubuntu/x-staging [17:18] and closed the upstream bug [17:18] tjaalton: thanks, will test when it builds. It's 18.2 with that line reverted? [17:18] yeah [17:18] Great, ty [17:18] the kernel patches will be backported later [18:00] alkisg: i386 seems to be built [18:06] ty,testing... [18:10] * alkisg hopes he didn't mess up grub this time :D [18:10] mesa doesn't touch it [18:10] kernel update does [18:11] The hard part is configuring grub to boot an older kernel [18:11] From 4.19 back to 4.15 [18:11] uninstall 4.19 [18:11] Not good when it's running, apt refuses [18:11] complains [18:11] Hmm maybe that changed from the last time I tried [18:12] tjaalton: success! [18:12] Can I copy this to the schools ppa, for everyone to get? [18:12] sure [18:12] tjaalton: Mesa i965 was hitting consistent deadlocks with userptr, so we switched entirely to softpin recently [18:12] I assume it'll be overriden by the next stable update, right? [18:12] it'll be in proposed soon [18:13] alkisg: you reported that this happened with 64bit systems too? [18:13] janesma: well, changing the kernel takes more time than rolling a revert via mesa [18:13] janesma: yes, a school said it saw it on a 64bit system [18:13] yep [18:13] and kernel won't get changed for 18.04.2 image which is next week [18:15] why wouldn't see the same thing with upstream mesa 18.3.3 and linux 4.18.2, on a 64bit system? [18:15] The same card? [18:16] tjaalton didn't see it either, I don't know why [18:16] beats me, but this was with legacy bios [18:16] it was a skylake machine... I think all of this should be the same for gen9 [18:16] which might or might not play a part [18:16] I tested on a kbl laptop [18:17] yeah, gen9 has the same feature set (skylake kabylake coffeelake) [18:17] 100% symmetry in the mesa regressions [18:17] we don't test different boot configurations (legacy/uefi), and I can't see how that would affect graphics.... but if it does then we have a new validation problem. [18:18] I'll test next week on hw that I can wipe clean [18:18] I'm hoping that this is down to running a 32bit kernel. [18:19] I managed to boot that too, still worked [18:21] I'd like to figure out how to reproduce this and make sure we aren't missing something in Mesa CI. [18:22] I think disabling softpin will cause intermittent hangs under load for ubuntu users. Not nearly as bad as a black screen. [18:22] that's a good tradeoff for now [18:22] It's not configurable by kernel cmdline or xorg.conf, right? [18:24] mesa 19.0 has the switch to softpin, in 731c4adcf9b11 [18:25] not sure what that will do to oibaf/padoka users. I guess they will be getting newer kernel and mesa, so they won't suffer. [18:38] alkisg: bionic will get 18.2.2 + the revert soon, 18.2.8 will follow after 18.04.2 is released [18:42] alkisg: are you still able to verify this build? [18:42] tjaalton: I only have access to one server now, and it works with this build, yes [18:43] remove 18.2.8 with ppa-purge, then enable proposed and install the new update from there, then verify bionic for the bug [18:43] Schools will need it on monday anyway, so if it'll get the build until then, I can do the verification done step [18:43] Will do [18:43] it just got accepted, so the build takes a while again [18:43] Tomorrow is fine [18:43] https://launchpad.net/ubuntu/+source/mesa/18.2.2-0ubuntu1~18.04.2 [18:44] already building [18:44] yeah tomorrow is fine I guess [18:45] thanks for everything so far [18:45] almost 2100, time to grab a beer [18:45] Thank you too! [18:45] too late to go out for a run [18:45] Oh, same timezone, where are you? [18:45] espoo, finland [18:45] * alkisg ran 10 km earlier [18:45] Ah, greece here, so yeah same timezone [18:46] yup, I remember [18:46] * alkisg sends a virtual beer to tjaalton :) [18:46] hope we'll get to crete again next year [18:46] though it's a tourist trap but we like it [18:47] Here's a nice trick I found to revert to the stock versions, even when ppa-purge can't do it; google translate needed: http://alkisg.mysch.gr/steki/index.php?topic=7370.0 [18:47] Yeah I spent 5 of my best months in crete [18:47] what happened ?-) [18:48] Nah, just talking in general, nothing happened here [18:48] I had to research it for schools with veeery broken installations [18:48] that were installing .debs from wherever they could find them [18:48] I mean in crete, why only 5mo :) [18:48] Haha [18:48] I was in the army then [18:49] ah right [18:49] For 23 months, 5 in crete [18:49] And even though the army generally isn't a good period.... crete was soooo nice.... [18:49] I remember f-16 flyovers [18:49] and many tourist women too :D [18:49] yes yes, very receptive [18:49] I assume [19:04] tjaalton: if you find a way to reproduce this locally, can you update me? [19:07] So to only update mesa from proposed, I guess this will do it (after apt sources + pinning): [19:07] apt update libegl-mesa0:i386 libglapi-mesa:i386 libxatracker2:i386 libegl1-mesa:i386 libx11-6:i386 libgbm1:i386 libwayland-egl1-mesa:i386 libx11-data:i386 libgl1-mesa-dri:i386 libosmesa6:i386 libgl1-mesa-glx:i386 mesa-vdpau-drivers:i386 libx11-xcb1:i386 mesa-va-drivers:i386 libglx-mesa0:i386 [19:08] I don't know why these libx11 things were in the upgraded-from-the-ppa list previously though... [19:11] tjaalton, janesma: apt policy libegl-mesa0:i386 => 18.2.8-0ubuntu0~18.04.1 from bionic-proposed/main [19:11] How can you upload 18.2.2 when it has 18.2.8? [19:12] Ah, maybe standard repository rules don't apply to the proposed queue :) [19:33] alkisg: 18.2.8 got removed [19:33] janesma: hope I can, next week [21:04] yup, proposed works, i'll comment in launchpad [21:09] Hmm it'll be hard to do the cosmic verification [21:09] don't worry about that [21:10] thanks for testing [21:10] :) [21:10] Thanks for the package :) [21:57] alkisg: and released to updates :) [21:57] now 18.2.8 back in [22:09] Yey!!!! :) [23:36] is there a change to get vega20 fimwares from agd5f into disco before release? [23:36] *chanCe