/srv/irclogs.ubuntu.com/2017/01/30/#ubuntu-devel.txt

naccdoko: thanks, will work on it on monday02:28
cpaelzergood morning!06:46
rbasako/07:23
cpaelzero/ rbasak07:35
=== Darcy is now known as Spydar007
dupondjelets hope we quickly have a firefox update :P08:00
dupondjeseems its broken on all ubuntu's now :x08:00
rbasakdupondje: is there a bug on this please?08:16
dupondjehttps://bugs.launchpad.net/bugs/165992208:18
ubottuLaunchpad bug 1659922 in firefox (Ubuntu) "Firefox 51.0.1 does not display pages/shows blank pages." [Critical,Triaged]08:18
dupondjethe fix is easy :)08:19
rbasakdupondje: looks like this bug is a complete regression but only for users with an enforcing Firefox AppArmor profile. But this is not the default. Is this correct?08:26
rbasakdupondje: what's the fix? I don't see one.08:27
rbasakAh, https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1627239/comments/3 may be a reasonable fix.08:28
ubottuLaunchpad bug 1627239 in firefox (Ubuntu) "Web pages not rendering with e10s enabled and AppArmor profile in enforce mode" [High,Confirmed]08:28
dupondjerbasak: well don't know whats the default apparmor status is ... :)08:39
rbasakdupondje: AFAICT, Firefox ships with a non-enabled-by-default AppArmor profile. Users can enable it, but it restricts Firefox. The new Firefox does stuff that the AppArmor profile doesn't permit to the point of complete breakage.08:47
=== jamesh_ is now known as jamesh
caribouwho's the resident expert on systemd nowadays ?10:36
caribouI'd appreciate a second look at LP: #165460010:36
ubottuLaunchpad bug 1654600 in unattended-upgrades (Ubuntu Xenial) "unattended-upgrade-shutdown hangs when /var is a separate filesystem" [High,Confirmed] https://launchpad.net/bugs/165460010:36
seb128caribou, not sure we have a "resident" one but you can try xnox or slangasek10:40
caribouseb128: yeah, thought so10:40
infinitycaribou: Your analysis sounds reasonable.  systemctl list-dependencies will give you a tree that you might sort of be able to parse to understand what your changes do.10:44
infinitycaribou: Which is better than testing and being subject to "oh, I beat the race, I must have fixed the bug" logic.10:44
caribouinfinity: that was my worry, especially since the window where the lock might be taken can be quite short10:45
caribouinfinity: that and I'm not too found of a unit *starting* when we're shutting down but that's debatable10:45
infinitycaribou: It's a oneshot, starting or stopping amount to the same thing.10:46
infinitycaribou: (But it may be true that claiming Start or Stop might subtly change ordering)10:46
caribouinfinity: apparently not in that case as ExecStart will only be triggered after the Before= are shutdown10:47
caribouinfinity: anyway thanks for looking10:47
infinitycaribou: Though, ExecStop isn't even documented in systemd.unit... Did you make it up? :)10:48
caribouinfinity: documented in services10:48
infinityAhh, indeed.  Silly docs.10:48
infinitycaribou: Docs claim that ExecStop won't run at all unless a service is actually running, though.10:51
infinitycaribou: Which for a oneshot, then gets trickier.10:52
infinitycaribou: See "Example 3. Stoppable oneshot service"10:52
caribouinfinity: hence the RemainAfterExit10:53
caribouinfinity: well, this seems to work, I'm just worried about "just being lucky"10:53
infinitycaribou: Yeah, but that won't do anything without an ExecStart, surely.10:53
infinity(or something to start)10:54
caribouinfinity: k,I'll double-check that10:54
infinityMaybe it will, though.  I dunno.10:54
infinityEasy to check.  "start" it, and then check status.10:54
infinityIf it's running, you won, if not, you didn't, and your stop will never run.10:54
ginggsinfinity: is it ok to sync libv8-3.14 now? it builds on ppc64el but not powerpc, and powerpc is no longer a release architecure in ubuntu, right?11:27
infinityginggs: We haven't dropped it yet.11:27
infinityginggs: We don't have a concept of "release" and "non-release" architectures.  It's in until it's out.11:27
infinityginggs: But no reason it can't be synced after we definitely do drop it.11:28
infinity"Add support for powerpc, ppc and ppc64el platform merging patch from Ubuntu"11:29
infinityThat implies it added ppc32 support too, though?11:29
ginggsinfinity: ok, well i'm not sure it will not build ubuntu, but it didn't in debian11:29
ginggsinfinity: and i can't test in a PPA :(11:30
infinityHrm.  It looks like it wouldn't indeed.11:30
infinityCurious what they broke in taking our patch. :P11:30
infinityginggs: Ask me in a few days, and I can maybe help clean it up.11:31
ginggsinfinity: ok, thanks11:31
=== hikiko is now known as hikiko|ln
=== _salem is now known as salem_
=== hikiko|ln is now known as hikiko
dupondjechrisccoulson: how are things going with firefox? :)13:43
elopioLaney: are you around? We are having issues with the armhf autopktest again.14:50
seb128elopio, he's on vac today (doesn't mean he's not going to look to IRC at some point but you probably shouldn't count on it)14:51
elopioseb128: thanks for the info. I've sent an RT, but I'm not sure if they have control over those machines.14:59
seb128elopio, you can maybe try people in the foundations team as well15:00
seb128they are the ones supposed to maintain that service15:00
seb128or at least p_itti was in their team so I guess they had some handover15:00
elopioslangasek: anybody else from your team can poke the armhf autopkgtest machines?15:03
elopioinfinity: and for the record, now the problem is rustup, which doesn't know what to do when it receives armv8l :)15:04
cjwatsonYou probably always want to look into fixing the package that's failing as well15:07
cjwatsonNot to say that the infra shouldn't take the conservative route, but it's still a bug in the package.15:07
elopiocjwatson: I will report a bug in rust. It's no package, it's a curl | bash: https://github.com/rust-lang/rustup.sh/blob/master/rustup.sh#L155115:18
slangasekelopio: so, what the issue regarding armhf autopkgtest machines?  I don't see excessive armhf-specific queues, has this already been resolved?15:33
cjwatsonslangasek: apparently they're running without the compat_uts_machine=armv7l (or whatever it is exactly) command line hack to stop uname reporting armv8l15:35
elopioslangasek: related to bug https://bugs.launchpad.net/ubuntu/+source/linux/+bug/152062715:38
ubottuLaunchpad bug 1520627 in linux (Ubuntu Xenial) "New personality for more accurate armv7l emulation on arm64" [High,Fix released]15:38
elopiouname regressed to return armv8l15:38
slangasekelopio: ah, you're saying there's been a change in behavior on the test runners?15:40
chilukslangasek mind sponsoring https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/1660372 since you are the debian uploader?15:52
ubottuLaunchpad bug 1660372 in cifs-utils (Ubuntu) "merge cifs-utils with debian 2:6.6-5" [Undecided,New]15:52
chilukalso cyphermox can I get sponsorship for https://bugs.launchpad.net/ubuntu/+source/clock-setup/+bug/166036215:53
ubottuLaunchpad bug 1660362 in clock-setup (Ubuntu) "Merge clock-setup with debian 0.131 for zesty" [Undecided,New]15:53
chilukalso slangasek cyphermox if this request annoyed you https://wiki.ubuntu.com/chiluk/CoreDevApplication  ... my DMB coredev meeting is this afternoon.15:55
* cyphermox looks15:56
slangasekchiluk: ah; I saw a mail from you that suggested I had already missed the deadline before I'd even read the mail, so I hadn't followed up :)15:57
chilukslangasek ... DMB didn't reach quorum last meeting.15:57
slangasek"the debian uploader" - well, I'm /a/ Debian uploader, who doesn't upload it15:57
chilukslangasek... stop beign so pedantic.15:58
chilukUploaders: Steve Langasek <vorlon@debian.org>, Christian Perrier <bubulle@debian.org>, Noèl Köthe <noel@debian.org>, Jelmer Vernooij <jelmer@debian.org>, Mathieu Parent <sathieu@debian.org>15:58
chilukyou were listed first...  must make you important..15:58
slangasekchiluk: it's ok you have slavic background I give you pass for definite and indefinite article confusion15:59
chilukis true.15:59
slangasekchiluk: half of your merge changelog is false because it is not a remaining change16:02
slangasekelopio: so when did this regress?  I'm not aware of any changes we've made to the armhf autopkgtest runners recently16:04
slangasekelopio: also, that kernel bug says it's fixed for xenial in 4.4.0-3.17; porter-arm64 is running a later linux-lts-xenial generic kernel, and uname -m returns armv8l.  Perhaps this regressed in the kernel?16:09
chilukslangasek you are absolutely correct.16:09
elopioslangasek: we noticed on Friday. Last time this happened, Iain said something about rebooting the machines.16:10
slangasekelopio: I can't imagine why rebooting the machines would fix the uname output...16:10
slangasekchiluk: ok, changelog edited locally, and uploading16:11
elopio¯\_(ツ)_/¯16:11
chilukthanks slangasek...shame on me..16:11
slangasekelopio: can you point me to specific autopkgtest logs showing the failure?16:11
chilukI checked for the keyutils.. but didn't occur to me that you might have fixed debian.16:11
cjwatsonslangasek,elopio: it's certainly worth checking /proc/cmdline if you can ...16:12
cjwatsonmaybe they somehow got rebooted without the override param16:12
elopioslangasek: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-snappy-dev-snapcraft-daily/xenial/armhf/s/snapcraft/20170129_193742_803b1@/log.gz16:12
elopiosearch for rustup: unknown CPU type: armv8l16:12
cjwatsonthat seems more likely than a kernel regression, just in an Occam's razor kind of way :)16:13
slangasekoh, this requires a kernel parameter? it doesn't just dtrt under linux32?16:18
cjwatsonslangasek: No, for reasons, basically that it was too late by the time we noticed this was a problem and the 32-on-64 ABI was already defined16:20
cjwatsonslangasek: my memory was indeed correct and it's spelled compat_uts_machine=armv7l16:21
cjwatsonslangasek: LP builders run with that16:21
elopiocjwatson: btw, do you know why is it that our armhf executions are slower?16:27
cjwatsonelopio: executions where?16:28
elopiocjwatson: on those autopkgtest machines.16:36
elopiothis is armhf: Ran 28 tests in 3581.469s16:36
elopiothis is amd64: Ran 28 tests in 2746.484s16:36
cjwatsonelopio: I know nothing about the autopkgtest machines.16:37
cjwatsonCan't help you.16:37
cjwatsonelopio: My comments above are just from sharing experience with how the LP builders are configured.16:38
elopioI know. I was just wondering :)16:39
elopionot a big deal anyway because now we are moving some platforms to the nightly run.16:40
dobeyelopio: different load on the machines when the tests ran?16:46
elopiodobey: this was on sunday morning, I think no other tests were running. But maybe.16:47
hallynIs there a way for non Canonical employees to verify whether someone has signed the CLA?16:56
jgrimmhallyn, "Canonical Contributor Agreement" team participation16:59
hallynjgrimm: not sure that's reliable - i'm not listed there17:11
jgrimmhallyn, wonder if you are part of a team that has membership17:13
jgrimmhallyn, or something missing because you are a former canonical employee?  no idea17:14
hallynok - thanks.  it's not terribly important, guy had a merge request against vmbuilder.  but that's being forked on github anyway so i asked him to resubmit as  PR there where no CLA requirement exists17:15
jgrimmhallyn, if i look at your lp profile I'm able to see that you've signed.. is that field not visible if you look at someone elses id?17:16
jgrimmhallyn, cool17:17
hallynjgrimm: where do you see that?17:21
jgrimmhallyn, i was wrong. sorry17:21
hallynok17:21
hallynhm.     i don't seem able to push to lp:vmbuilder18:02
hallynwonder whether it has to do with the switch to git18:02
hallynhm, no, i seem to have been dropped from vmbuilder-dev18:04
nacchallyn: wouldn't it be lp:vm-builder?18:05
nacchallyn: oh i see there's a distinct 'upstream'18:05
hallynno18:05
hallynyeah that naming was always funky18:06
hallynanyway, now someone needs to approve my membership, but there's noone to do that18:06
hallynmaybe mvo18:06
hallynoh, smoser18:06
hallynsmoser: can you hit approve on my request to join vmbuilder?18:06
smoserhere ? https://launchpad.net/~vmbuilder/+members#proposed18:07
smoseri dont see you18:07
hallynsorry, no, vmbuilder-dev18:08
smoserapparently, i cannot18:09
smosersoren ^ ?18:09
hallynlol18:09
rbasak!dmb-ping19:00
ubottubdmurray, BenC, cyphermox, infinity, micahg, rbasak, sil2100: DMB ping.19:00
elopioslangasek: did you find anything about armv8l ?19:35
slangasekelopio: I'm just getting sshed in over there now, it's been a stack of meetings this morning19:45
elopioslangasek: thank you.19:50
elopioslangasek: Sergio told me to mention the alternative of skipping the rust tests for now in armhf, as this is blocking our SRU.20:01
elopioI know you don't like skips so I'm scared while I say this :)20:02
elopioI can make sure that they pass in rpi3.20:03
slangasekbarry, tdaitx: what do you know about administering the armhf lxd autopkgtest runners?  I don't seem to have creds to ssh into the guests, and can't reach the host20:13
slangasekelopio: which SRU is blocking on this?20:14
elopioslangasek: https://bugs.launchpad.net/ubuntu/+source/snapcraft/+bug/165994620:14
ubottuLaunchpad bug 1659946 in snapcraft (Ubuntu Zesty) "[SRU] New stable micro release 2.26" [Undecided,New]20:14
sergiusensslangasek: I didn't push yet to xenial or yakkety as I wanted zesty to pass20:32
sergiusensslangasek: we can either disable the tests that use rust which has the machine detection problem or wait for the fix20:32
barryslangasek: i don't believe i have access either unfortunately20:41
slangasekbarry: hrm, so this wasn't part of the training from pitti?20:49
barryslangasek: no.  i suppose we need to talk to is to give more of us access20:49
slangasekmmk20:50
chilukDmitrii-Sh: can i get an upload for https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/165522520:59
ubottuLaunchpad bug 1655225 in qemu (Ubuntu) "Under heavy load qemu hits bdrv_error_action: Assertion `error >= 0' failed" [Medium,New]20:59
chilukDmitrii-Sh: I have a feeling you are asleep by now... but maybe you can catch it in the morning.21:00
Dmitrii-Shchiluk: 1 sec21:00
chilukoh awesome.21:00
chilukyou just seem to be doing lots of qemu uploads recently21:00
chilukso I figured I'd ping you.21:00
Dmitrii-Shchiluk: oh, well I cannot do uploads myself yet21:01
mwhudsoninfinity: hey, does this look like a glibc bug do you? https://bugs.launchpad.net/snapd/+bug/1657504/comments/1821:02
ubottuLaunchpad bug 1657504 in Snapcraft "asciinema 1.3.0 snap is segfaulting on 14.04" [Critical,In progress]21:02
chilukoh ok.. ah I understand you have just packaged up public patches.21:02
Dmitrii-Shchiluk: those were patches I was seeking sponsorship for basically21:02
chilukthanks Dmitrii-Sh I'll find someone else to sponsor the uploa.. I should have checked your perms.21:02
Dmitrii-Shchiluk: ok, np at all21:03
chilukgood night Dmitrii-Sh21:03
Dmitrii-Shchiluk: thx21:03
infinitymwhudson: Maybe?  But I'd have to see more debug output.  I'm shocked to discover we're using runpath for this.  I guess I naively assumed snaps ran chrooted *in* the core snap somehow.21:17
infinitymwhudson: So, yes, this may well be a bug, as this is very infrequently exercised code, because Who Does That?21:18
mwhudsoninfinity: this is for classic snaps21:19
mwhudsonwhich are a different breed of hack21:19
infinitymwhudson: On the other hand, some googling suggests the behaviour is correct, and the docs are muddy.21:23
mwhudsoninfinity: hm ok21:27
mwhudsonthe docs are not muddy, they are wrong :)21:28
infinityhttps://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/125363821:29
ubottuLaunchpad bug 1253638 in eglibc (Ubuntu) "dynamic linker does not use DT_RUNPATH for transitive dependencies" [Undecided,Confirmed]21:29
infinitymwhudson: That Qt blog seems to be frequently referenced from anyone else discussing the issue.  I'd like to think it being widely-known for 6+ years would mean someone might have mentioned it upstream. :P21:32
mwhudsoninfinity: what DT_RUNPATH is only 13 years old, do you think people are actually using it yet? :)21:33
mwhudsoninfinity: open since 2012 https://sourceware.org/bugzilla/show_bug.cgi?id=1394521:34
ubottusourceware.org bug 13945 in dynamic-link "RUNPATH behaviour is not transitive" [Normal,New]21:34
infinity'The current gABI says "One object's DT_RUNPATH entry does not affect the21:35
infinitysearch for any other object's dependencies."'21:35
mwhudsonoh heh https://sourceware.org/ml/libc-hacker/2002-11/msg00011.html21:35
ubottusourceware.org bug 2002 in general "frysk's custom /usr/lib/frysk/pkgconfig/glib-java.pc is corrupt" [Normal,Resolved: fixed]21:35
mwhudsonubuntulog_: lol21:36
mwhudsonubottu: lol21:36
mwhudson"The application or library can use whatever21:36
mwhudsonprefix it put in its DT_RUNPATH on the strings it passes to dlopen directly."21:36
mwhudsonexcept the library is sodding nss21:36
slangasekelopio, sergiusens: I would prefer having the failing test selectively disabled in the package on armhf (yes, my views on skip-test are well known! ;). but yes, I'm willing to ignore the test failure manually on a one-time basis if that helps.21:36
infinitymwhudson: So, yeah.  I think ld and dlopen are spec-compliant here.21:36
slangasekelopio, sergiusens: of course, disabling it in the package would mean working around an infrastructure issue, so ignore that I even said that. Yes, I'll skip-test this for you.21:37
mwhudsoninfinity: fair enough21:37
infinitymwhudson: One could argue that libc itself might want a RUNPATH to be able to find its own modules.21:37
infinitymwhudson: And I'd entertain that line of reasoning.21:37
sergiusensslangasek: sounds good21:37
* sergiusens sees runpath/rpath conversations everywhere now21:37
infinitymwhudson: But I tihnk what snappy is trying to do here is RPATH (or LD_LIBRARY_PATH), not RUNPATH.21:37
mwhudsoninfinity: the thing that makes this extra fun is that the core snap is sometimes at / (for strict confinemt) and sometimes at /snap/core/current (for classic)21:38
mwhudsonsergiusens: you can't escape!!!21:38
infinitymwhudson: Fun.  glibc isn't exactly the only library with plugins.  It's just the most obvious one.21:39
infinitymwhudson: And I bet some *do* set RUNPATH, and that'll break if you're relocating them.21:39
mwhudsonyep21:39
infinityKinda sounds like this all needs LD_LIBRARY_PATH wrappers of doom.21:39
mwhudsoninfinity: zyga is talking about patching the dynamic linker in the core snap21:40
mwhudsonwhich is kinda the extreme version of that i guess21:40
mwhudsoninfinity: tbf we don't have to care about all libraries here21:40
infinityPatching it to do what?21:40
mwhudsonjust ones in the core snap21:40
mwhudsoninfinity: change search paths depending on whether the process is a classic confinement snap or not21:41
infinityI'm fundamentally opposed to ld.so (and, by extension, glibc) differing between a classic Ubuntu system and the ubuntu core snap.21:41
infinityFor so many reasons.21:42
infinitySo very many.21:42
zygainfinity: oh, thouse would not differe21:42
zygainfinity: on core we don't support confinement: classic snaps21:42
infinityzyga: Then expand on "patching the linker in the core snap".21:42
zygainfinity: we could ship the same glibc everywhere21:42
mwhudsonups kinda outstayed my welcome in this cafe biab21:42
zygainfinity: the core snap is used as the "runtime part" of supporting confinement: classic snaps on "classic" ubuntu21:43
zyga(the word classic has three meanings, sorry about that)21:43
zyga(classic snap on core, classic ubuntu and classic confinement)21:43
infinityzyga: Yes.  I get that.21:43
infinityzyga: I'm curious how you intend to patch the linker without patching the linker.21:43
zygainfinity: I intend to patch the linker and ship it everywhere in ubuntu (and upstream if possible, but not required)21:43
sergiusensslangasek: I'll be pushing something shortly then21:44
zygainfinity: simply the effect would only be visible on classic ubuntu running classic confinement snaps21:44
zygainfinity: (or any classic distro for that matter)21:44
zygabut that case is perhaps more complex so let's solve the easier case first21:45
infinityzyga: FWIW, as both glibc maintainer in Ubuntu and an upstream committer, I'm likely to push back on this unless it's got some very sane rationale, obvious thought about security considerations for heuristic lookup relocations, and some really solid likely/unlikely branching that makes it a near no-op for most cases.21:46
infinityzyga: But happy to see initial rough cuts of what you *mean* at least, so I can maybe steer in another direction. :P21:46
zygainfinity: ok, you have much better understanding of the internals than I do, perhaps just the simple goal statement would work; if you think that goal is hard to achieve (or unlikely for other reason) we can abort the process early21:47
zygainfinity: the goal is to have a way where people can unpack debs into their confinement:classic snaps (or just build something locally without voodoo) and have that work*21:48
zygainfinity: without the use of mount namespaces21:48
zygainfinity: so that it uses predictable (good) dynamic linker and shared libraries21:48
infinityAnd how do you envision that looking on the filesystem?21:48
zygainfinity: on the filesystem it looks as follows21:48
infinityCan I have both ubuntu-core 16 and ubuntu-core 18 snaps installed?  How do I know which one is which when I do a magic lookup?21:49
zygainfinity: the core snap is mounted at /snap/core/current (current is a symlink) but we are allowed to rely on it21:49
zygainfinity: yes, they would be different base snaps (gimme a sec)21:49
infinityKay.  I'll go for a smoke while you type and come back to the full explanation. :)21:49
zygainfinity: the snap in question (say hello-classic) is monted at a similar location /snap/hello-classic/current/21:50
zygainfinity: when you execute hello-classic wrapper generated by snapd, via snap-run/snap-confine/snap-exec the predictable dynamic linker from the core snap is invoked (side-stepping any linke in the distro)21:50
zygainfinity: dynamic libraries should be resolved from the core snap, from the snap in question (hello-world) and not from the classic distro21:51
zygainfinity: when you exec something then we need to be careful (hence glibc patches) to intercept that and use our linker again21:52
zygainfinity: unless the executable is outside of /snap, where the normal behavior is expected (no magic)21:52
infinitySo, all binaries have a wrapper?21:52
zygainfinity: similar redirection should happen when dlopen is used, it should not rely on libraries from classic distribution21:52
zygainfinity: no, just declared apps21:52
zygainfinity: but those are the only entry points21:52
infinityIf you have a wrapper, I'm not sure why you need a special linker.  You can set LD_LIBRARY_PATH21:53
zygainfinity: but once you run (say declared app foo) you can freely exec any internal executable on the core snap, in your snap or on the host distro21:53
zygainfinity: that would leak to the child processes and it is not sufficient to ensure we use the right dynamic linker21:53
zygainfinity: that's my idea (so far), please tell me what you think21:54
zygainfinity: as a closing remark: classic confinement works "almost" allright as long as you build software from source21:54
zygainfinity: but it breaks if you want to use prebuilt software that people are more familiar with21:55
zygainfinity: (the only thing that breaks is the glibc bug that mwhudson found)21:55
* zyga turns all ears21:55
mwhudsonzyga: which apparently is not a bug and maybe snapcraft should use rpath not runpath after all21:55
infinityzyga: How do you propose finding the "correct" linker without specifying it manually?21:59
infinityhttp://paste.ubuntu.com/23896033/21:59
infinityNote that if I don't specify it there, I get the host system's linker/libc/libnss, which also works fine.22:00
infinityOr, you're proposing the host ld.so first checks the binary's path, then forks the ld.so in the ubuntu-core root?22:02
zygainfinity: snap-exec would know that it runs a confinement-classic app nffsdfsdfsfdsdfsdfadasd22:03
zygare22:03
zygasorry, lost connection22:03
zygainfinity: snap-exec would run the desired application through the right linker if it was instructed to do so by snapd22:03
chilukhey cyphermox.. here's the qemu bug I'm needing sponsorship on https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/165522522:03
ubottuLaunchpad bug 1655225 in qemu (Ubuntu) "Under heavy load qemu hits bdrv_error_action: Assertion `error >= 0' failed" [Medium,New]22:03
zygainfinity: so to be clear, running executables directly from (e.g. from shell) would not work22:03
zygainfinity: it would only work if it got initiated by snap-exec22:04
zygainfinity: and if we implement glibc changes to exec, from any process that uses classic confinement22:04
zygainfinity: looking at the pastebin now22:04
infinityzyga: Okay, so no linker changes needed there if snap-exec is divining which one to call.  ld.so is itself an executable for this very use-case.22:04
sergiusensmwhudson: let me do a test run with rpath22:04
zygainfinity: not directly, we'd run the linker instead of the executable in snapd22:05
zygainfinity: the host linker would never see the executable22:05
mwhudsonsergiusens: s/--enable-new-dtags//g or whatever?22:05
mwhudsonsergiusens: would be interesting to see22:05
zygainfinity: I don't know if this would work for the case that mwhudson found but perhaps we were just using the feature incorrectly due to bugs in the documentation22:06
zyga(though reading the elf spec I find this aspect confusing)22:06
zygainfinity: (the dlopen case)22:06
infinityzyga: Oh, s/RUNPATH/RPATH/ will definitely fix mwhudson's issue, but it does nothing for your pre-build binary case.22:06
zygainfinity: correct22:07
infinityAnd fixing the prebuilt binary case means you don't need RPATH in the from-source case either.22:07
zygainfinity: but there (I hope) we can patch the linker to chage its behavior22:07
infinitySo that seems like a distraction.22:07
zygainfinity: oh? sorry, can you explain please?22:07
zygainfinity: how would it work for pre-build binaries?22:07
infinityzyga: You read my statement backwards. :)22:07
zygainfinity: ah22:08
zygainfinity: correct22:08
infinityzyga: I'm saying that if there's a solution for pre-built binaries, from-source doesn't need RPATH hacks, as they can be built with the same assumptions as a deb was.22:08
zyga(kids-going-to-bed-moment, sometimes distracting)22:08
zygainfinity: yes, I understand and totally agree22:08
zygathanks22:08
infinityI'm still knee-jerking at the claim that the linker needs any special knowlege of this.22:09
zygainfinity: I think someone has to change the default search path, can it be anything but the linker?22:09
zygainfinity: does the linker take any arguments we could use?22:09
* zyga runs --help to see 22:10
infinity--help won't help much. ;)22:10
zygainfinity: do you think setting --library-path is sufficient?22:10
infinity--library-path is effectively the same as LD_LIBRARY_PATH.22:11
zygammm, that's very promising then22:11
zygainfinity: but that still leaves the exec case open22:11
infinityThough, may fix your concern about leak-on-fork.22:11
zygainfinity: yes, it does fix the leak22:11
zygainfinity: and seems like a much cleaner solution22:11
infinity(FWIW, tangentially related, but I think relying on 'current' symlinks is incredibly short-sighted here)22:12
infinityRight now, you have one core ABI.  When 18 happens, you'll have two.  Without a flag day where all apps upgrade at once, you *need* any library-path wrapping magic to use versioned paths.22:13
zygainfinity: I think the plan for core will change before 18, AFAIK the idea is that we will have many base snaps and still exactly one core22:13
zygainfinity: and what you think of core will largely become base22:14
zygainfinity: (core will be a small snap with mostly just snapd itself)22:14
infinityAnd linker/libc?22:14
zygainfinity: if we had many bases then each classic confinement snap will link against a specific core22:14
zygainfinity: unclear, this is hand-wavy territory22:15
zygainfinity: I think it is safe to say that we'll see base-16 and base-18 (perhaps base-16 will just be called base for legacy reasons)22:15
infinityI mean, I know we've been kind enough to give you a stable ABI for almost 2 decades, but one should always prepare for their world being rocked by libc7. ;)22:15
zygainfinity: if we were to say that we cannot rely on the current symlink we can still resolve things correctly by running ld.so directly, I suspect22:15
zygainfinity: was it two decades already? wow22:16
infinityAnyhow, I'd think an app meant to run against core-16 should know that, and the wrappers should be set up to know that.22:16
zygainfinity: in case abi does change we should be fine as we can always run the right dynamic linker for each particular snap22:16
zygainfinity: right, the snap/app knows that and so does snapd22:16
sergiusensmwhudson: https://asciinema.org/a/bjzuw6ujlg3l39cx1ai1bn6ii22:17
zygainfinity: if we don't use rpath/dt-runpath and instead use ld.so directly I think we don't have a problem with (eventual) abi transitions, right?22:17
infinityzyga: It certainly makes life easier.22:17
zygainfinity: we only have the exec patch for glibc that lives in core/base22:17
zygainfinity: (still assuming that is needed unless we can find a way to make do away with it)22:18
infinityzyga: Which exec patch?22:18
infinityDid we just run full circle? :)22:18
zygainfinity: the (theorethical) patch that would allow a process to exec anything and still DTRT22:18
zygainfinity: maybe, bare with this for a moment22:19
zygainfinity: a process running under classic confinement can exec anything it wants from the core snap22:19
zygainfinity: if it runs /snap/core/current/bin/true we need to intercept that22:19
infinityzyga: IMO, any snap that's on the path should be taking care of its own runtime setup.  So, if it needs a wrapper, the thing on the path is a wrapper.22:19
zygainfinity: or it would again run with the system dynamic linker, right?22:19
zygainfinity: that's at odds with reusing prebuilt software where you don't know anything about that and at odds with not leaking LD_LIBRARY_PATH22:20
infinityzyga: Ahh.  I see.22:20
infinityzyga: So, if a snap runs /bin/true, you want it to run the core version, not the host version.22:20
zygainfinity: I think it is fine to limit this special behavior to just processes started by a snap22:20
infinity(true probably being a stupid example, but I get the point)22:20
zygainfinity: well, not really, /bin/true should be the host true22:21
zygainfinity: but /snap/core/*/bin/true should be the core version22:21
zygainfinity: e.g. imagine a bash shell snap22:21
infinityWhy would a snap ever call something in that namespace that way?22:21
zygainfinity: it should be able to run stuff normally22:21
zygainfinity: when you said namespace did you mean to refer to linux namespaces or was it just a word clash?22:21
infinityMy brain predates linux namespaces. :)22:22
zygainfinity: classic confinement snaps are allowed to use core resources _and_ host resources if they want to22:22
zygahehe :)22:22
zygainfinity: so if you build a quick snap with stuff from firefox.deb22:22
infinityThat feels like a strange design choice.22:22
zygainfinity: it should work fine even if it wants to run something from your distro22:22
zygainfinity: as long as it works fine by itself (I mean, it should not link to distro libs but it can run any executable and expect it to work normally)22:23
zygainfinity: that's the design we've got, perhaps I misunderstand mark's intent but I think it feels like a "this is what you can count on but you can use anything you like"22:23
naccdoko: in ldb 2:1.1.26-1ubuntu1, you changed -Xldb.so to -Xldb. in override_dh_makeshlibs. Was that intentional?22:23
zygainfinity: it should allow people to use snaps quickly without learning new things22:24
dokonacc: I assume that's an oversight22:24
naccdoko: and, iiuc, the reason to switch from -c4 to -c3 in that release was since a new library was being introduced for python3?22:24
zygainfinity: or even having to use devmode confinement (and have various other issues)22:25
naccdoko: ok, np, just wanted to confirm, I'll probably ahve other questions : )22:25
zygainfinity: if we can deliver that22:25
infinityzyga: But doesn't it seem odd to have people calling things in that bizarre path?22:25
zygainfinity: I think that's a big win22:25
zygainfinity: no, why? if I tell firefox snap to use "evince" from my system to open PDFs then it should just do that22:25
zygainfinity: it depends on each case but I think it is an important "no-barriers" design decision22:25
infinityI don't want to muddly my code with calls to /snap/core/current/usr/bin/thing when what I really want is to call 'thing' and get a working one if one exists.22:26
zygainfinity: move off debs/rpms/random stuff to stuff with no barriers in place22:26
infinityOr are you prepending all the /snap/blah paths to PATH and hoping for the best?22:27
zygainfinity: I'm doing neither, I don't think I understand your concern22:27
infinityCause that way does indeed lie madness.  You're calling into a chroot without chrooting.22:27
zygainfinity: can you make an exaple, I think this is not mad and it should work OK but perhaps we're confusing something22:28
infinityzyga: So, you say that people should be free to use /snap/core/*/bin/command despite NOT being chrooted.  How do you expect them to call it?  On PATH, or directly, or?22:28
zygainfinity: I think I see your point, I didn't mention this but it should not be allowed (supported) to run /snap/* executables from the classic distro without going through /snap/bin and the snap-exec in the way22:29
zygaon the way22:29
zygainfinity: once that has happened (you ran via snap-exec) you should be able to execute any executable on the system correctly though22:30
zygainfinity: as long as you give your app and entry point (which we control) this seems like a reasonable limitation22:30
zygainfinity: e.g. a bash classic snap would work normally22:30
zygainfinity: so would any less demanding program22:30
infinityzyga: No, I'm not talking about how to run a snap from classic.  I'm talking about a snap itself forking.22:30
zygainfinity: ok, so a snap itself forks22:31
zygainfinity: if it chooses to exec something from the host that's fine (no patches needed), right?22:31
infinityzyga: And you're saying a forking snap should be able to call binaries inside core.22:31
infinityzyga: How is it calling them?22:31
zygainfinity: it would go through the systmem linker22:31
zygainfinity: yes22:31
zygainfinity: and that's the place where I believe we need to patch glibc22:31
infinityHow.  Is.  It.  Calling.  Them?22:32
zygainfinity: so that exec is changed to exec via core's ld.so22:32
infinityYou bring up a "bash snap" (I assume you mean a shell script).  Is that shell script calling things as /snap/core/current/bin/thing?22:32
zygainfinity: (I meant bash itself deployed as a snap)22:32
infinityWell, bash itself is just a binary like any other.22:33
zygainfinity: yes but let's say you want to snap install bash22:33
infinityThere's nothing special about it.22:33
zygainfinity: (some new version)22:33
zygainfinity: and use that on your system for everything22:33
zygainfinity: (e.g. your login shell)22:33
zygainfinity: (* not everything but daily stuff)22:33
zygainfinity: you'd expect that bash to have no unreasonable limitations22:33
jdstrandI don't want to get embroiled in this cause infinity will be able to recomend the best course of action, but, if I ship 'foo' as a snap, aiui, /snap/bin/foo should use core's libraries/etc but 'foo' exec'ing 'ture' should get the host's 'true'. it going to /snap/core/current/bin/true seems like a huge corner case22:33
infinityThen /snap/bin/bash is a wrapper that configures it correctly.  This isn't relevant to the forking thing.22:34
zygainfinity: correct22:34
infinitySo, again, no glibc patching.  Check.22:34
zygainfinity: what if in bash I run /snap/core/current/bin/true22:34
jdstrandif it wants to call its own stuff, it needs to use /snap/bin22:34
infinityWhy are you running that?22:34
jdstrandyes, why?22:34
zygainfinity: because any app can run any command from core transparently as a normal course of action22:35
infinityHow is that a use-case we want to waste any time supporting?22:35
zygainfinity: and that is supported today in other snaps22:35
infinityAny confined app can, yes.22:35
zygainfinity: and removing this would surely break something22:35
infinityclassic apps are meant to run as if they're on the hose system.22:35
zygainfinity: (e.g. stuff running stuff via system() and what not)22:35
jdstrandI think the use cases are incredibly few22:35
infinitys/hose/host/22:35
zygajdstrand: I disagree, we specifically say that you can rely on anything in the core snap being there22:35
zygajdstrand: this translates to people using that22:36
infinityzyga: Here's the fundamental difference.  A confined app runs /bin/true and gets the core version.  To get the core version in a classic app, you have to obtusely call /snap/core/version/bin/true22:36
zygajdstrand: and I don't think we want binary wrappers for any executable in the core snap22:36
zygainfinity: correct22:36
jdstrandzyga: classic, aiui, is about running your stuff as a snap and able to call the host's stuff without issue. calling into /snap/.../ without going through /snap/bin is totally unreasonable imho22:36
infinityzyga: The latter use-case makes zero sense to me.  This isn't about transparently supporting something.22:36
zygainfinity: hmm, I disagree, if I know that core snap has well-defined "awk" I may just decide to use it22:37
zygainfinity: why would it be unreasonable?22:37
zygainfinity: core snaps run in an unknown world where the only save heaven is the core snap22:37
zygainfinity: if we say that they cannot run any executable in the core snap then that feels like a broken story22:37
infinityzyga: Because calling into a weird chroot path is exactly the opposite of transparently running on the host system? :P22:37
jdstrandzyga: the one case I could see as interesting is foo wants to calls something in its own $SNAP, but that is easily remedied by exposing that thing being in /snap/bin22:38
infinityIf your goal is for classic to have all the same features as a confined snap, you will fail.22:38
infinityPeriod.22:38
zygajdstrand: except that's not something that they do today22:38
jdstrandzyga: but classic is fundamentally different than devmode or strict22:38
zygainfinity: that's not my goal but the goal is to offer classic as a simple way to have more things working as a snap, I think that not allowing to run executables from the core snap is at odds with that idea22:38
zygajdstrand: technically yes but we should do our best to make that seem less so22:39
zygajdstrand: it's a step towards devmode towards strict22:39
infinityzyga: And, to be clear, to support your use-case, you need to patch the HOST libc, not the core one.  Which means either this only works on Ubuntu, or you need to patch *all* host libcs.  If it only works on Ubuntu, we have a set of things people can rely on for classic snaps, it's called "Essential: yes" in dpkg/apt.22:39
zygainfinity: can you tell me why I'd have to patch the host libc?22:39
jdstrandzyga: my understanding of what is desired is to be able to run your thing and anything on the host, not your thing, anything on the host and any arbitrary thing in /snap that happens to have the x bit set22:39
infinityzyga: Because you're calling /snap/foo/bin/true, and the kernel is invoking the host ld.so.22:40
infinityzyga: Welcome to ELF.22:40
zygainfinity: I know, but my idea was that since this would *only* be allowed from classic confinement snaps22:40
zygainfinity: those would run with our libc22:40
zygainfinity: and those could have patched exec that injects the linker at that time22:40
zygainfinity: (as I said this doesn't apply to someone just wanting to run /snap/core/current/bin/true from outside of a snap)22:41
jdstrandwhy would they only use our libc?22:41
jdstrandclassic isn't supported on other distros?22:41
zygajdstrand: because on all distros they would link to our libc22:41
zygajdstrand: via the ld.so trick we discussed earlier22:41
infinityEh?22:41
infinityOne of your stated goals for classic is to not rebuild binaries in special ways.22:42
zygaI think there's something fundamental that I'm either wrong about or I didn't state earlier22:42
jdstrandthe ld.so trick was a snap run thing (ie /snap/bin wrapper), no?22:42
zygainfinity: correct,22:42
zygajdstrand: correct22:42
jdstrandso, this thing is running now22:42
zygajdstrand: and at that time you run against our libc22:42
jdstrandit execs /snap/random/binary22:42
zygajdstrand: becuase our linker ran /snap/random/binary22:43
jdstrandthat is going to use the host's libc22:43
zygajdstrand: with altered library paths22:43
zygajdstrand: no22:43
zygajdstrand: that's the part I disagree aobut22:43
zyga*about22:43
infinityThat's not how linkers work.22:43
zygajdstrand: wait, sorry, (lag on irc)22:43
zygainfinity: I know what you mean22:43
zygamy assertion is as follows:22:43
zygaany app that *starts* executing via snap-exec + ld.so trick uses our libc?22:44
zygacan we all agree on that?22:44
jdstrandthat sounds reasonable to me22:44
infinityI run /snap/bin/foo, which is a wrapper that calls into the core linker and loads foo with the core ld.so and libc.  So far, we're on the same page.  Then foo invokes /snap/core/current/bin/true, THAT will load with the system ld.so22:44
zygayes22:44
zygaI agree and that's why I keep saying I think we need to patch our libc to do something different then22:44
infinityNo.22:44
zygahow can a process run another application? only via exec right?22:44
zygainfinity: I agree that as things stand now /snap/core/current/bin/true will load with the distro/system ld.so22:45
infinityThere are many entry points to the fork syscall.22:45
infinityWhat you want is a kernel patch, not a libc patch.  And you don't want a kernel patch either. :P22:45
zygainfinity: fork != exec, fork doesn't hurt us22:45
zygainfinity: I bet we can do this with just libc, what else is there? what am I missing?22:46
* jdstrand reasserts that running /snap/core/current/bin/true would not be a case we should support. classic gives you your snap, /snap/bin and running host binaries. period. anything random executable in /snap is not supported by snapd proper, but a snap could invoke it on its own22:47
jdstrand(by setting whatever enviornment it wanted)22:47
infinityzyga: fork is exactly the case we were discussing.22:47
zygajdstrand: that's not what mark described; I'm not arguing with you22:47
zygajdstrand: I don't disagree with your POV but if we're asked to support /snap/core in its entirety that's what we need to do22:47
jdstrandzyga: I wasn't in those meetings but what I've read on the subject, I'm surprised /snap/core/current/bin/true is meant to be a supported use case22:48
zygainfinity: can you tell me how forking is a problem?22:48
zygajdstrand: I can ask mark / gustavo to clarify22:48
infinityzyga: You want to be able to fork /snap/core/current/bin/true ... You can't.22:48
zygajdstrand: but regardless I'd like to understand if we can technically do it22:48
infinityAnd I don't understand why you'd want to.22:48
zygainfinity: I think you exec /snap/core/current/bin/true, not fork it (then we agree)22:49
zygainfinity: fork just forks the process, right?22:49
* jdstrand said his piece and wanders off22:49
infinityexec replaces a process with another process, fork creates a child process.22:49
infinityYou may well want both, but both are fundamentally silly in this context.22:49
zygainfinity: and exec is a library call via glibc (I agree that it would not block someone going at the raw syscall)22:49
zygainfinity: ok we are in agreement on this; fork is not a problem because it doesn't interact with any of the parts at play, exec is the only point where we have issues22:50
infinitysystem(), for instance, is a fork.22:50
infinityCalling any process from a shell script is a fork.22:50
zygainfinity: fork + exec22:50
infinityCalling anything from python is a fork.22:50
zygainfinity: yes22:50
zygainfinity: sure but I don't see any issues in fork alone22:50
infinitySome languages use the raw syscall to do so.22:50
zygainfinity: you just have a new process running the same thing at that time22:51
infinity...22:51
zygainfinity: that's interesting and that's indeed a problem22:51
jdstrandI will say one other thing. executing something in /snap/random/... is not a path towards devmode since that isn't at all supported by devmode22:51
infinityBUT YOU WANT TO ALLOW PEOPLE TO FORK /snap/core/current/bin/true22:51
infinityJust no.22:51
zygainfinity: (please don't shout at me, I'm trying to understand technical side, I want people to fork+exec /snap/core/bin/true and my assertion is still that with patched exec that's well possible to achieve22:51
infinityClassic snaps on Ubuntu have a defined set of things available to them.  It's called Essential.22:52
zygajdstrand: except that /snap/core/current/bin/random may be allowed by the default policy or by an interface, then we do allow that22:52
infinityClassic snaps on other distros might not have such a set, but we can't "fix" that without patching the HOST.22:52
sarnoldinfinity: fork(2) absolutely must remain available. THat's how most unixy processes do threading.22:52
jdstrandzyga: but not at /snap/core/current/bin/random. /snap/core/current/bin/random isn't in your PATH22:52
infinitysarnold: Erm, yes.  But not relevant to this discussion.22:52
zygajdstrand: that's true22:52
zygainfinity: I don't want to argue if it is a good idea to support that thing22:53
jdstrandso a classic snap has to go *completely* out of its way to call something in /snap/random/...22:53
zygainfinity: just to understand if it is technicaly doable22:53
infinityzyga: It's not doable without patching the host.  Host libc for reasonable coverage, host kernel for total coverage.22:53
zygajdstrand: ok, I agree but I think we need to spin this around a little22:53
zygajdstrand: let's forget about core22:53
infinityzyga: Neither of those seems like a thing we should do.22:53
jdstrandif it is going to do all that, it can call with the right linker or we can provide a helper for it to use that calls the linker22:54
zygajdstrand: a snap can run its own apps22:54
zygajdstrand: and if that's just a random app they have inside they should be ale to run it22:54
zygajdstrand: and they cannot run it on !classic confinement if they expose it as an app22:54
infinitySure, running their own stuff is simple, though, as they should be exposing it with the same wrapper style as the primary binary was.22:54
zygainfinity: that's not as simple as you think, we have some strong semantics of what that means22:54
zygainfinity: and they don't have to use any wrappers, today, to run executables from the same snap22:55
jdstrandzyga: core doesn't run anything. as for the random app of its own, that is the only interesting case. I think exposing that in /snap/bin is not unreasonable. both would be in its path22:55
jdstrandbut, there is perhaps some thinking that needs to be done there22:55
zygajdstrand: but that also lets users see it and it may not be something you wanted22:55
zygajdstrand: and it would have different name22:55
zygajdstrand: e.g. /snap/bin/$SNAP_NAME.foo22:55
zygajdstrand: so breaks unpacking debs22:55
infinitySo snap needs a libexec concept.22:55
jdstrandI understand. that is why I said it is the interesting case22:56
* zyga is still convinced that we need to path exec to achieve transparency (except syscalls, I totally agree with infinity on that)22:56
* jdstrand has to run22:56
zygathe libexec idea is intereting22:56
zygaI think it is worth exploring22:56
zygae.g. for snapctl22:56
zygajdstrand: o/22:56
zygainfinity: sorry for being so tought in this conversation, I really want to understand what we are standing on now22:57
zygainfinity: and I'm not desigining anything, just implementing a design (that I may misunderstand as it stands)22:57
infinityWell, designs can sometimes be wrong, too.22:57
zygainfinity: I'll talk to gustavo/mark about what are the guarantees we expect people to have in classic confinement22:57
infinityShockingly.22:57
zygainfinity: totally22:57
zygainfinity: but wrong != impossible22:57
zygainfinity: and I was trying to probe what's doable22:57
infinityAnd Jamie and I seem to agree that a design that says classic snaps can execute things in the core snap is probably not quite right.22:58
zygainfinity: even if that's unorthodox behavior to say the least22:58
zygainfinity: I think that's fine and as I said a moment ago, the case of running stuff from their own snap without going through wrappers is far more interesting22:58
infinityThe corollary there is: Not impossible != right22:58
zygainfinity: and perhaps the answer is indeed libexec22:58
zygainfinity: as that has many advantages (simplicity and flexibility)22:59
infinityAnd yes, being able to run your own subordinate binaries is a much more interesting and valid use-case.22:59
zygainfinity: but it also has some downsides that I can think of quickly22:59
infinityAs it's a very UNIX thing to do, and dozens of applications do it.22:59
zygainfinity: e.g. hardcoded paths that don't go through path search22:59
zygainfinity: e.g. gtk/glib running some internal helpers that have baked-in paths based on prefix22:59
infinityThe general way is either to ship subordinates in /usr/bin (in which case, they're expected to be on path, and should be wrapped in /snap/bin) or to ship in /usr/lib/$app/ and called directly, which needs a snap mapping think, probably.23:00
sarnoldbtw why not just recompile with ./;configure --prefix=/snap/ or whatever?23:00
zygainfinity: and people using --prefix=/snap/$SNAP_NAME/current/usr23:00
zygainfinity: I don't think that libexec idea can support that23:00
zygaeh, laggy again, probably lost connectiong23:00
zyga*connection23:00
infinityzyga: I didn't literally mean we need a BSD-style libexec (and, indeed, if snap mirrors the FHS, it's library subdirs per app), but that we're dealing with the concept of libexec, that is executables in a library directory.23:02
infinityzyga: That is an interesting problem to discuss (and should be entirely doable with the same launching/wrapping techniques used for /snap/bin)23:03
infinityOr, if you prefer "executables off the path", we just happen to stuff them in library dirs. :P23:03
zygasarnold: debs23:04
zygasarnold: people love to have snaps that just unpack 12 debs23:04
zygasarnold: and expect them to run23:04
zygasarnold: and if we can support that, we win hearts23:04
infinityBut, unlike "all the things in /bin and /usr/bin in the core snap", the off-path executables in a package are a known quantity, identifiable, taggable, and wrappable.23:04
zygainfinity: I only agree if "good practice" == "actual practice"23:04
zygainfinity: if that breaks gedit I'll want exec interception23:04
zygainfinity: if gedit/gimp/whatever run fine then I won't argue :)23:04
* zyga needs to EOD23:04
zygaas irssi tells me it's past midnight and my wife tells me I should stop working at some stage23:04
zygainfinity: can we use that idea even if an app ships hardcoded path to run something?23:04
sarnoldzyga: gnight :)23:05
infinityHardcoded paths are problematic in new and exciting ways. :P23:05
infinityBut I also need to escape this discussion at least for today.  So you should EOD so I stop typing.23:06
zygainfinity: take care, thank you very much for the conversation :)23:14
infinityzyga: Have a good night unwinding and not arguing on the internet. ;)23:17
infinity(Though, there's plenty to argue about these days that isn't work related too :/)23:17

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!