=== Aztec03 is now known as SmokinGrunts === SmokinGrunts is now known as Aztec03 [05:04] ddstreet: #1466926 is dead for now after the insights on verification, please go for yours WITHOUT the changes my upload had [06:20] Good morning === fnordahl_ is now known as fnordahl === tinwood_ is now known as tinwood [08:27] rbasak: you are on SRU today IIRC - would you mind considering accepting the improved open-vm-tools that is in x-unapproved for 1741390? [08:49] cpaelzer: ack [09:05] Anyone got experience installion 16.04 on a Dell T130 with a Perc H330 using software raid? Refuses to find boot drive and not sure if its a install problem or H330 problem. [09:21] cpaelzer: was this an SRU regression from the previous backport? We need a Test Case and the issue resolved for Bionic first I think, unless there's some reason this is urgent? [09:27] rbasak: it is resolved in bionic [09:27] the bug has an update [09:28] well wait there are two things in flight [09:28] I need to ensure not to mix them [09:29] 1741390 is a regression when backporting the new version [09:29] it wasn't seen in Bionic yet [09:29] (I already asked to check there, as it needs a special setup) [09:29] but so far the assumption is that with newer systemd it is fine [09:29] if they report it is not, then I can still fix Bionic [09:30] so for now we have [09:30] Bionic likely unaffected (not known better), Xenial fixup in x-unapproved [09:30] the other one that is resolved in bionic was a libvirt issue, that I work on atm [09:30] sorry [09:31] there also is a refresh to the Xenial SRU incoming - so they are similar cases [09:32] otoh the vmware guy "expects" it to affect Bionic [09:32] as I said that part just isn't clear [09:32] I can push a Bionic upload with the same fix (one line removal) to Bionic first if you'd prefer that [09:34] rbasak: the other one is 1758428 - do you want me to fix this in Bionic and then get a ping again for the refreshed SRU? [09:35] as I said the biggest inhibitor here is the uncertainty that I can't test it on my own (uses some special vmware api call to trigger) [09:35] it might even be easy but I don't know how (yet) [09:35] * cpaelzer stops flooding the chan without rbasak rpelying :-) [09:36] Sorry I just realised there's more to this than I thought and I've been reading up [09:37] hehe [09:37] What is the Xenial backport based upon? [09:37] the version in Bionic [09:37] in recent history we hit (and fixed) several things only affecting it in systemd-level in xenial (but good in bionic) [09:37] Would the version then not be 2:10.2.0-3ubuntu2~16.04.1 or something like that? [09:38] And the changelog doesn't have the Bionic entry [09:38] rbasak: it is based on the version in bionic a few weeks ago, and ther ubuntu1 and ubuntu2 in Bionic are reverts of each other [09:39] I see, OK. [09:39] so 10.2.0-3ubuntu2 essentially = 10.2.0-3 [09:39] therefore there was no need to rebase the backport [09:40] rbasak: you are not wrong to ask for a Bionic fix of 1758428, it is just that I know for sure it is in Xenial (therefore the upload) and not yet sure it is in Bionic (so no upload yet) [09:40] It's all starting to make sense not :) [09:40] now :) [09:40] lol+ [09:43] cpaelzer: 2:10.2.0-3ubuntu0.16.04.1 has not yet been published, right? [09:44] rbasak: correct [09:45] rbasak: it was held by finding the new issue on the verifications step [09:45] so I ask to accept the new one into x-proposed "over" the former [09:45] we might still even hold back the release of it until the Bionic case is calrified [09:45] but having in x-proposed allows all the parties to retest [09:46] But 2:10.2.0-3ubuntu0.16.04.1 hasn't even been in xenial-proposed, AFAICT? [09:48] it was put to x-unapproved on 2018-03-22 [09:48] and you are right, it wasn't in proposed yet (people are testing the ppa while waiting for it) [09:49] So I'd have squashed 2:10.2.0-3ubuntu0.16.04.2 into 2:10.2.0-3ubuntu0.16.04.1 [09:50] But I don't think it's necessarily worth changing that now [09:50] That did confuse me a bit though. [09:50] hmm === Aztec03 is now known as WhiteboyDigs [09:51] Technically 2:10.2.0-3ubuntu0.16.04.1 is higher than 2:10.2.0-3, whereas for a backport you generally want it to be lower. [09:51] on the second upload 2:10.2.0-3ubuntu0.16.04.2 I ensured to have a better -v on buildpackage [09:51] so it includes all the history since Xenial anyway === WhiteboyDigs is now known as Aztec03 [09:51] But in this case, Artful has already moved on [09:51] one more or less doesn't change the long list of entries [09:51] So in practice that also may not matter [09:51] Your -v is fine [09:52] But that cannot show me everything, so I was inferring the rest based on an assumption that you'd only burn the minimum version strings for an Ubuntu upload. [09:53] s/Artful/Bionic/ above. === pesari_ is now known as pesari [10:02] rbasak: so next steps? [10:03] rbasak: I thought we could accept the Xenial sru into proposed and wait on the confirm that Bionic is affected before making useless changes there [10:03] after you read into the case, do you agree or suggest differently? [10:04] cpaelzer: is there a reason this can't wait until the package is settled in Bionic? Also, what about Artful? [10:04] rbasak: several parties depend on the backport, but we can postpone it for now and reconsider if the bionic case needs too long [10:05] artful is not considered at all, the decision was to only provide that for last LTS [10:05] after bionic is out we will only do the same for Bionic until 20.04 is there [10:11] thanks rbasak, despite causing more work you are right :-) [10:11] cpaelzer: I'll write up a review in the bug. [10:12] I updated the respective bugs, let the peers on them have some time to confirm the state in Bionic and then kick off a new round [10:13] rbasak: when writing your review, please consider my latest post and feel free to cancel at least the first open-vm-tools in x-unapproved [10:13] keeping the other one there until resolved === ivoks_ is now known as ivoks === trekkie1701c_ is now known as trekkie1701c === aluria` is now known as aluria === Beret- is now known as Beret [10:16] rbasak: I also updated the planning (to no more ignore interim releases) [10:19] dpb1: ^^ discussion affecting planned open-vm-tools SRUs [10:20] dpb1: I updated the plan you drafted already [10:20] TL;DR 4 instead of 2 such SRUs per year needed [10:20] I have some hope that post Bionic those will be less painful backports than now [10:29] cpaelzer: leaving it in the queue will mean every other SRU team member will spend time looking before discovering it's blocked and moving on. [10:29] I'd prefer to reject for this reason. You can keep your upload around and re-upload without changes easily enough I think. [10:30] well then, cancelling from unapproved does not imply version bumps, so ok === niedbalski_ is now known as niedbalski === alai` is now known as alai === miguel is now known as Guest32093 [13:36] hey all, on Launchpad trying to log into the bug tracker, is there any reason my Ubuntu One account isn't being accepted and i'm being told to 'Go away' for being a 'Bad bot'? [13:55] cpaelzer: so, we need it in artful too? [13:59] dpb1: yes [13:59] cpaelzer: ok [13:59] I can do that, but as this was a little box of backport-surprises so far I don't expect it to be different there [13:59] dpb1: if you want to talk about the plan I can HO if you want [14:00] I hope the updates to the sheet are good and understandable [14:00] cpaelzer: it's probably enough, but I haven't checked [14:01] i believe Gargravarr's issue was resolved in #ubuntu [14:06] okay, so, root reason for me wanting to log into the bug tracker - i'm hitting the same sssd problem again, a lot harder this time [14:09] are sssd and libpam-ldapd the only two options for doing LDAP user auth with Ubuntu? [14:09] the latter works as intended but doesn't have native caching so won't work for laptops taken off-site, and my attempts to set up external caching failed === dosaboy_ is now known as dosaboy [16:26] rbasak: so i'm running into a real mess with this SSSD problem, but my suspicion is that the bug you linked me to is not the whole story [16:27] i'm starting to think this is related to the intel-microcode [16:27] this is the last package installed on my laptop before it broke (which occurred on the next boot) [16:28] and i'm running sssd on the current 4.13.0-38 kernel on a desktop without it crashing [16:28] should i post my findings onto the same bug ticket or should i look at opening a new one? === Daviey_ is now known as Daviey [16:39] Gargravarr, could you post a link to the ticket (sorry, it's not in my scrollback) [16:40] it's fine [16:40] https://bugs.launchpad.net/cloud-images/+bug/1746806 [16:40] Launchpad bug 1746806 in linux (Ubuntu) "sssd appears to crash AWS c5 and m5 instances, cause 100% CPU" [Critical,In progress] [16:40] thanks [16:40] i'm just trying uninstalling the intel-microcode package [16:40] fingers.cross() [16:41] just a bit of background: I use LDAP and sssd here too, and recently had a machine (my main desktop) hard-lock on boot after a kernel upgrade. Booting with prior kernel worked fine; after a bit of investigation, removing intel-microcode resolved things for me [16:42] waveform: i'm finding similar. a colleague just had the issue and i rolled him back to 4.13.0-26 from 4.13.0-38 and it booted [16:42] nope. tried uninstalling -> reboot -> log in as root from a TTY -> run sssd -id 9 and netcat it to a different machine -> switch to lightdm -> login as LDAP user -> login succeeds, but machine freezes [16:42] I *think* in my case (I'd have to check my notes somewhere), -36 worked for me and -37 was the one that locked up [16:43] it seems to be a random alignment of intel-microcode, CPU and kernel [16:43] we have Skylake desktops and Kaby Lake laptops. so far the desktops are unaffected [16:44] but someone in the bug ticket mentions having a Skylake desktop [16:44] mine's pretty old - it's an i7-2600K [16:44] (sandy-bridge) [16:44] i still have a 2520m in use in my ThinkPad [16:45] now, if i recall correctly, the whole point of microcode is that it gets loaded into the CPU [16:46] so there's a good chance that uninstalling it won't fix the problem [16:47] so the Kaby Lake laptop has frozen, and the CPU fan is pumping out a fair amount of heat, so i assume it's spinning the cores again [16:48] hmmm, it certainly did in my case - I removed both intel-microcode and the "no longer required" package that went with it - I'll just check what that was... [16:48] ah, iucode-tool [16:48] ta, i'll check they're both gone [16:49] never installed on this laptop, it seems [16:49] ah wait, typo'd [16:49] waveform: did you reboot after uninstalling? [16:49] yes, I did [16:50] I was in the older kernel to do the uninstall, then rebooted into the newer one [16:50] i have to use the 4.4-series kernel on this laptop due to intel wifi driver hell [16:50] found my notes: that was the one change I made between booting and finding -37 failed, and -37 worked. Incidentally, would be interesting to know if -36=works, -37=fails for you as well - that might at least confirm we're looking at the same thing (and narrow down the changesets that need investigating) [16:51] incidentally, I did also encounter this with 4.4-series and found that -116 to -117 were the culprits there [16:52] -116 was fine until last week when i updated the microcode [16:52] -117 is no improvement [16:53] okay, sssd starting up (on -116) [16:53] if this breaks, i'll roll back to -112 or something [16:54] wow, premature celebration at its finest. that got far further than normal, Firefox loaded [16:54] THEN the machine froze [16:54] the 4.4 series I'm not so clear on as my jump to 4.13 was literally a re-install of ubuntu desktop on that machine (after being unable to fix it over the weekend, and needing it working in a hurry I just did "oh sod it, I'll reinstall" thing only to find that it still froze after re-installing all the stuff I had installed before :) [16:54] yeah. we generally use 4.13 here for HWE [16:55] but i ran into error after error with the Intel drivers (my laptop has an Intel card, everyone else has Qualcomm) [16:55] eventually i got it stable by downgrading the card and using the basic 4.4 [16:55] it was only on 4.13 that I attempted to remove intel-microcode and found that fixed things - hence my 4.4 assumption is purely based on -117 being the first version that failed for me (I don't know whether intel-microcode removal would fix things there, and can't really rollback to test it!) [16:56] fair enough [16:56] -117 isn't even officially released afaik [16:57] Gargravarr: -119 was just released [16:57] my mistake [16:58] ah, could well be 119! Handwriting ... bah [16:58] well, the official announcement hasn't arrive just yet, the kernel is still "hot" :) [16:58] sdeziel: it shows in apt-cache so that counts as released [16:59] right, trying -112 (and will add -119 as well (which might well be lucky, it's my grandmother's house number...)) [16:59] Gargravarr: btw, the microcode is a per-boot operation. So, removing the update actually does revert things. [16:59] dpb1: thanks for clarifying [17:00] is a plain 'remove' operation enough or does it need to be purged? [17:00] and the BIOS could also load a fresh microcode if you updated it recently [17:00] hmm, I did purge (out of habit) but if a package is only half-installed I'd slightly surprised if it was still "doing things" [17:00] yes, it's actually a very weird process. [17:01] oh joy. i upgraded the BIOS out of desperation too [17:01] ah, I definitely didn't do that (it's too old for there to be any updates available!) [17:01] Gargravarr: so, you are hitting this issue in a cloud, or just on your local workstation? [17:01] dpb1: on multiple Dell laptops [17:01] ok [17:02] not hit it in a cloud context (yet) [17:02] Gargravarr: I just asked since the bug title was about aws [17:02] thx [17:02] indeed [17:03] Guys, "dijuremo" in #ubuntu-kernel has been reporting and testing this for several weeks, I suggest you join in that channel and pool observations and resources [17:03] if it makes a difference, the machines i've hit it on so far have all been Kaby Lake [17:03] TJ-: thanks for the hint [17:03] thanks, will do [17:03] Gargravarr: did you test the fix in comment #43 [17:03] bug 1759920 [17:03] bug 1759920 in linux (Ubuntu Artful) "intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-image-4.13.0-37-generic)" [High,Confirmed] https://launchpad.net/bugs/1759920 [17:03] dpb1: i did. made no difference [17:03] Seems like we have several problems with the Spectre_v2 microcode [17:04] Gargravarr: and did you post on the LP issue? [17:04] LP? [17:04] Gargravarr: sorry, I don't actually know who you are. :) [17:04] oh, Launchpad? [17:04] launchpad [17:04] yes [17:04] no, i was drafting a reply when i started noticing the microcode bit [17:04] OK [17:04] at least writing there would be good so that Kamal knows [17:05] yeah, i guess that would help [17:05] thanks [17:05] I posted something on *an* LP issue, but it wasn't that one - I'll see if I can dig it up and link it in [17:05] waveform: is it sssd in your case too? [17:06] well ... I am running sssd but I never tied the issue to that specifically. I just noted my machine freezing at the login screen, and eventually removing intel-microcode sorted things [17:06] ok [17:06] my machines have a tendancy to freeze *before* the login screen [17:06] basically as soon as the sssd service loads at boot, so the splash screen freezes [17:08] dpb1: i posted comment #48 on the AWS ticket [17:08] thanks. [17:08] hmmm ... I did note I could boot in recovery mode (without network) on the new kernel and the freeze wouldn't occur. The lack of networking would obviously preclude sssd, so there's perhaps a link there. I could install intel-microcode, make a local user and disable sssd - see if I can re-create it that way if that's any use [17:08] waveform: did you drop to a root shell there and then, or continue the boot? [17:09] i could get a root shell from recovery, but as soon as i tried to resume boot, it re-occurred [17:09] I dropped to the root shell a few times to poke around, and did notice that certain actions like continuing the boot would always freeze the system but until I never narrowed down *precisely* what started before the crash (even tried videoing the screen - but it was too blurry to read :) [17:10] but until I *reinstalled ... [17:11] i figured out that disabling the sssd service brought the system back up [17:11] this started happening on an XPS that i was re-imaging [17:11] the script errored out and the machine froze immediately after restarting sssd [17:11] interesting. [17:11] the logs clued me in that sssd was responsible [17:11] Gargravarr: this is xenial + the HWE? [17:12] that one was, yes [17:12] we're 90% Xenial/HWE with a few exceptions [17:13] the one that broke today was HWE, 4.13.0-38 [17:14] okay, so got the laptop up with 4.4.0-112, let's see what happens invoking sssd [17:15] nope, frozen again, immediately after successful auth [17:16] I wonder if there's a clue on the sssd compilation optins [17:17] interesting - ah, finally found the bug I commented on and yup, it's now marked as a dup of 1759920 (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1760377) [17:17] Launchpad bug 1759920 in linux (Ubuntu Artful) "duplicate for #1760377 intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-image-4.13.0-37-generic)" [High,Confirmed] [17:17] TJ-: i've been running sssd in the foreground and collecting the logs on another machine [17:18] with the debug level set at 9, so got some pretty verbose logs if it'll help [17:19] Gargravarr: would not hurt to put on the bug. [17:19] ah, I was wrong about -116 working for me, it was -112 that worked - which is curious given it's failing for you [17:23] Gargravarr: I mean that the most likely explanation, if it is caused by the microcode update, is the spectre_v2 changes. Some of that could have an interaction with compile time options that aren't spectre-aware ... have you tried disabling some of the kernel mitigations, e.g. adding to kernel command line "nospectre_v2" [17:23] i have not [17:23] kernel flags are a dark art to me :) i'll try that [17:24] looks like tyhicks has good advice, I'll stay with #ubuntu-kernel [17:24] just reading the rest of that long bug thread - intriguing stuff [17:26] it'd be more intriguing if Meltdown/Spectre weren't such a !"£$%^ mess :) [17:27] this is my first job as a sysadmin after being a developer for years, i only started last year and it feels like i'm fighting not just The Bad Guys trying to get into my systems, but the vendors themselves making it impossible to secure them! [17:28] oh indeed - I'm rather interested in why sssd is such a good trigger for it (and ultimately what the root cause is ... other than chipzilla's incompetence ;) [17:29] yeah, on that level it is interesting [17:29] i found it quite hard to get my head around why an auth daemon could cause such a low-level problem [17:30] my rough guess would be it's doing something "fancy" with memory (trying to ensure certain things are secure / never reach swap / etc. but that's total speculation on my part at present) [17:31] indeed, but the interesting thing is that it does this after successful auth, too [17:31] by which point the secure memory should have been deleted [17:32] TJ-: the nospectre_v2 hasn't made a difference, frozen again [17:33] ahh, #61 has some meat on it [17:34] i'll read through in a minute [17:35] when you have the latest microcode from Intel, a new code path is taken in the kernel when it is switching between two tasks [17:36] and apparmor is implicated by the looks of it? [17:36] (sorry, still reading :) [17:36] if the old task is confined by apparmor, the kernel will attempt to generate an audit message during the task switch [17:37] unfortunately, a lock is already held from early in the task switch code that the audit code then attempts to take again [17:37] that results in a hard lockup [17:38] so it's a fight between the kernel and apparmor? [17:38] ah, thanks very much for the clarification - I got some of that from the ticket, but your explanation is very clear! [17:41] Gargravarr: it is just a bad interaction between several kernel subsystems due to the complexity of switching between tasks [17:45] (and presumably that that complexity has just been increased with the advent of all the spectre/meltdown mitigations) [17:45] right, this bit of problematic code is trying to decide if new microcode features need to be triggered when switching between two tasks [17:46] tyhicks: man that's some complex stuff right there. :) === sarnold_ is now known as sarnold [17:55] bah, got a three year old to go and deal with - will be back later - good luck with the testing! [18:00] i'll trade you [18:00] spectre debugging for childcare [18:02] woah [19:00] cpaelzer: fixed pg-repack uploaded [19:00] submitted to debian as well === tomaw_ is now known as tomaw [19:55] Gargravarr, heh - I'd happily swap! spectre may be complex and certainly beyond my current knowledge in several areas but ultimately it's deterministic and logical [19:55] ... this is more than I can say for the thought processes of a 3-year old :) [19:56] huh I'd have thought three year olds would make many demands :) [19:56] very determined [19:57] lots of speculative execution too [20:00] it's the context switching that gets me - far too many topic changes within a minute! [20:01] almost leading to a meltdown? [20:02] :D [20:41] can package descriptions in debian/control, and I mean the short description here, be longer than 80 chars? [20:41] I'm not finding anything about this in https://www.debian.org/doc/debian-policy/#s-f-description [20:42] does the lint tool give any hints? [20:44] nope, it seems to ignore it [20:44] ahasenack, see section 3.4.1 [20:45] "The single line synopsis should be kept brief—certainly under 80 characters." [20:45] does that include the "Description: " bit? [20:45] I'd think no [20:46] it refers to just the first line of the Description: field by my understanding (read section 3.4 before it, which refers to 5.6.13 which you originally linked to) [20:47] the dpkg -s output includes "Description: ", so if the intent is to be able to display dpkg -s in a 80-char terminal, then the whole thing should be less than 80 chars [20:49] the example in 5.6.13 includes "single-line-synopsis" as part of their Description, which would seem to refer to 3.4.1 ? [20:49] in other words, the first line of Description: should be strictly <80 chars, the following lines are effectively separate and form the extended description [20:50] * ahasenack gets creative with English and brings up a thesaurus [20:50] (or at least, that's my reading :) [20:52] Description: Persistent Memory low level remote access support library - debug build [20:52] meh [20:53] * ahasenack drops "low level" [20:56] good call - that sort of detail can always go in the extended description. I'd think "debug build" could too (e.g. python2 / 3 packages don't specify whether they're 2 or 3 in the synopsis, typically only in the extended description) [20:57] I have libfooN, libfoo-dev and libfoo-debug is a bit controversial (it was part of libfoo-dev). It's not just debugging symbols, it has extra assertions, checks, slower code, logging [21:01] hmmm, just having a quick comparison of various -dbg package synopsis' vs their "parent" - doesn't seem to be any particular standard [21:02] this is not a -dbg package. The -dbg packages are being automatically generated [21:02] I think most -dbg packages ought to be dropped these days for the ddebs instead [21:02] yes [21:02] but a package with extra assertions might be useful thing to have, ala electric fence or valgrind etc [21:03] ah, not something I'm familiar with - I'll have to go read up on ddebs [21:04] there's precious little docs :/ [21:04] http://ddebs.ubuntu.com/ [21:05] https://wiki.ubuntu.com/Debug%20Symbol%20Packages ? [21:06] yup :) === berglh_ is now known as berglh === Guest20669 is now known as devil_ === mwhudson_ is now known as mwhudson === bradm_ is now known as bradm [22:52] do you know how to stream webrtc from /dev/video0 ? [22:58] cat /dev/video0 | webrtc-program # sorry, I got nothing. [22:59] firefox or chromium is my best guess [23:01] sarnold, talking to me? [23:01] dpb1, lol [23:02] stanford_AI: yeah [23:02] sarnold, I would like to publish webRTC from the terminal, not from a browser [23:03] yeah, that sounds like a good idea. I just haven't seen anything except browsers implement it.. [23:06] sarnold, what else could I use for video streaming from linux? [23:07] there was something I did [23:07] * dpb1 looks it up [23:10] oh, I did rtsp [23:11] webrtc is the chat thing, right? [23:11] like hangouts [23:11] I wouldn't be anymore help than google [23:12] crtmpserver looked promising, but the webpage is dead :/