=== PRaXiS is now known as MaGiSTeR [01:58] How do I access packages merged from debian? [02:04] define "access" [04:42] Good morning [04:49] infinity: denneed02 is unreachable (at least via apache) for about 3 days now; it's active and building in https://launchpad.net/builders though [04:49] infinity: does apache need some prodding there? otherwise we'll lose ppc ddebs [04:49] pitti: Erk, really? [04:49] * infinity looks. [04:51] pitti: So, it seems angry building wine, and has been for 17h. But that doesn't account for the previous two days you say you couldn't reach it... [04:51] infinity: well, I got a metric ton of cron warning mails; let me look when they started [04:52] pitti: Should be reachable now? [04:52] infinity: ok, only since yesterday, sorry [04:53] infinity: confirmed, works again; thanks [04:53] * pitti mops them up [04:54] pitti: Do you have any prototype for how you plan to populate ddebs.u.c from the librarian once we make the switch? [04:54] pitti: The time is drawing nearer, finally. [04:55] infinity: no, I don't; if we have the ddebs in LP but need to do the publishing on ddebs.u.c with ddeb-retriever, this will once again mean some rewrite [04:56] but at least it'll get massively simpler [04:56] pitti: Well, we still want the apt-gettable archive, I think, and putting in a second publisher instance to create ddebs.u.c, while perhaps the ultimate goal, won't happen right away. [04:56] pitti: But I think it should be as simple as walking all the Packages files, backtracking to the source:version tuple that produced the binaries, then gathering the ddebs from the API. [04:57] right [04:57] as opposed to now, where I have to do this crazy queueing, and try to match a ddeb to a release/pocket/component/source [04:58] pitti: Really not sure how we'll manage the publishing with a real publisher thing, when all our current content isn't in LP. Though, I guess the publisher doesn't actually ever remove files from disk that it doesn't know about. [04:58] pitti: So, maybe we could just start publishing in the middle of your populated archive, and it would Just Work. [04:58] (And we'd have to manually clean out pool when we retire old series') [04:59] wgrant: Would that work? [04:59] wgrant: I mean, assuming other things, like having a parallel publisher for ddebs will work at all. :P [05:00] infinity: I think this will become incrementally easier over time; after a full release or so we can rebuild everything which hasn't been rebuilt since then, and then LP has a full set of ddebs to publish [05:00] pitti: Sure, at some point, we can just decide the past doesn't matter, but that point isn't for a long time. [05:00] pitti: Because trusty. [05:00] unless we can import the current ddebs somehow? [05:01] pitti: We've talked about that, and every "solution" seemed crazier than the last. [05:01] infinity: They also wouldn't be in indices... [05:01] pitti: But it's not beyond the realm of possibility to staple the ddebs onto existing build records. [05:01] Yes it is. [05:02] wgrant: No, not beyond the realm of "possibility", just also crazy, see above. :P [05:02] well, otherwise we'll just have to wait until trusty goes away, and keep the ddeb-retriever thing until then [05:02] pitti: We can ditch ddeb-retriever as soon as we can publish LP ddebs in the middle of your archive. [05:02] Well, no [05:02] or that [05:02] It'll still have to sort of exist [05:03] Something has to download them from LP [05:03] It'll just be about 1000x less awful. [05:03] wgrant: So, yeah, backscroll. [05:03] I'm currently trying to get my desktop to boot. Backscroll comes later :P [05:03] wgrant: We started with "let's fix ddeb-retriever to pull from API backrefs to the librarian". [05:04] wgrant: And then I moved on to "wait, a parallel publisher wouldn't in any way damage the files already on disk, cause it literally cares nothing for files it knows nothing about, so we COULD just publish ddebs in the middle of pitti's archive". [05:04] wgrant: And skip ddeb-retriever entirely. [05:04] wgrant: Assuming a parallel publisher is doable at all. [05:04] LP will not be able to publish ddebs.ubuntu.com without significant work. [05:04] wgrant: Right, so that's a "next phase" thing, and a simpler ddeb-retriever is phase 1. [05:04] And anything LP publishes might have external ddebs in the pool, but they wouldn't be able to be in indices. [05:05] the publishing is actually the most painful part, as it takes about 2 hours, or more than 24 if I remove the cache (which I need to do from time to time once publishing time hits the 6 h mark or so) [05:05] pitti: Some of us have some experience that might be able to help speed that up. [05:06] pitti: ddeb-retriever scares the poop out of me, but if we reduced it to the 20 lines or so probably required to parse Packages, backtrack, find API ref for ddebs, pull to disk, then it'll suddenly be something I'd be happy to contribute to. :P [05:07] pitti: Perhaps we could prototype on a parallel miniddebs archive that's using a PPA w/ ddebs as its source. [05:07] infinity: scares> heh, same; it's half of a soyuz implementation, and a really bad one [05:09] Doing the backref hunting via lplib instead of direct SQL will probably be slow as heck, but I imagine that won't matter much if we're not averaging more than a few thousand files per run. [05:09] yeah, that too [05:09] and we have to do it once per every other hour or so [05:10] Once per ftpmaster pulse, ideally. [05:10] we can probably iterate over the builds since the last time, and keep track of when the last run happened [05:10] When we get it working and proven-not-crap, we'll trigger it from the publisher. [05:10] that should only iterate over the recent builds [05:10] infinity: too slow [05:10] Trigger, not blokc. [05:10] apt-ftparchive is excruciatingly slow [05:10] ^ [05:11] pitti: It shouldn't be. You're using it wrong. :) [05:11] pitti: We'll fix that. [05:11] Particularly since you're not publishing sources. [05:11] Binary publishing is really quick when you have it set up right. [05:11] pitti: Or, it's decided that ddebs aren't debs and it's not caching them properly, which would be a 1-char fix somewhere. [05:11] It only took us 7 years to get there, too. [05:12] infinity: it does cache them in the big db; as I said, if I remove that it takes > 24 hours to run dpkg/tar on all of them [05:12] Of course, ddebs are bigger than debs, and thus slower to hash, but whatever. The fact that ddeb-retriever has otherwise slightly less work to do than frpmaster should balance that out. [05:12] yeah, any hint how to make it faster would be greatly appreciated [05:12] pitti: I'm confident we can find he bottlenecks and stomp them out. [05:13] pitti: But let's start with prototyping the new/slim variant againt a ddeb-publishing PPA, and go from there. [05:13] pitti: So I have a codebase to work with that doesn't make me want to stab my eyes out and mail them to Germany. [05:13] heh [05:13] (brb) [05:14] wgrant: Do the trigger points in the publisher have knowledge of which pockets were dirty? [05:14] wgrant: I'm guessing no, but that could be passed in the env, perhaps? [05:14] infinity: I think so. [05:14] But if not, we can tell them, [05:14] Maybe not needed for this, I guess, I could just cmp old and new Packages before deciding who needs a-f love. [05:14] But in general, might be handy to pass along if it's not. [05:15] Oh [05:15] X randomly started after 150 seconds. [05:15] I guess that... works. [05:17] Is this your way of telling me not to reboot today? [05:17] I just upgraded my nvidia-plagued desktop to vivid [05:18] It's running nvidia-343 from xorg-edgers. [05:18] So it's a bit of a special case. [05:18] Ahh. [05:21] Right, I'm going to get an early night tonight and see if this strange feeling of enthusiasm carries forward to Monday morning. [05:22] pitti: Let's revisit this in a week or two and see what we can come up with for a prototype that won't suck when the ddeb world improves. [05:23] infinity: yeah; for that it would be good to actually have some ddebs somewhere in LP to try this on [05:23] pitti: Easily done in a PPA, like I said. [05:23] but I figure this will use something like getPackageUploads(created_since_date=last_run) [05:24] and custom_type='raw-ddeb' or similar [05:24] so that we only process the recent uploads, not the entire archive [05:24] pitti: I'd rather have it match the Packages files we mirror from ftpmaster identically, so no need for date windowing. [05:24] pitti: Rather, we'll just be processing deltas from last run to current. [05:24] infinity: that's quite a lot more bookkeeping then, though? [05:25] yes, we still need to match them up against the archive's Packages.gz of course [05:25] pitti: Shouldn't be too bad. [05:26] pitti: ddebs aren't actually custom uploads. They have BPPHs, but they just don't hit disk. [05:26] for now I think we can only rip out the wgetting of http://*.buildd/ and replace that with getPackageUploads(created_since_date=last_run) [05:26] So you'd basically want to look up the SPPHs that you have, call getBuiltBinaries(), and then filter to ddebs. [05:26] I think that ridiculously involved matching against the archive's Packages.gz needs to stay [05:26] wgrant: but still from getPackageUploads(created_since_date=)? [05:26] Or watch new entries in getPublishedBinaries, but I'm not sure how GC for that would work on your end. [05:27] Right, that could work too. [05:27] There are a few approaches, but it's not like raw-static-translations. [05:27] I just can't query all packages in all releases all the time [05:27] No, but we can keep local state. [05:27] that's like three magnitudes too slow [05:27] So if we walk Packages, then only query things we've never queried before, we're getting the delta. [05:28] Then GC stuff that's no longer referenced. [05:28] Personally, I would watch the Packages files, and keep a local map of (source, version) -> [ddebs] [05:28] Then run a-f with a file list generated from the above steps. [05:28] When you see a source version drop out of all the Packages files, you drop those ddebs. [05:28] When you see a new one appear, you query for which ddebs it should have and download them. [05:28] wgrant: I think we just said more or less the same thing. :) [05:28] wgrant: that's what ddeb-retriever more or less does [05:28] Probably. I wasn't listening to you :P [05:28] Typical. [05:28] Kids these days. [05:29] it downloads all Packages.gz and builds a packagename -> pocket -> version -> arch -> component map (this takes some 20 mins already) [05:29] pitti: Right, except now you can ask LP for the map, rather than having to do the hilarious madness with guessing which ddeb filenames map to what after you download them from the backdoor :) [05:29] Hmm. [05:29] That really shouldn't take more than 5 seconds. [05:29] Unless germanium is secretly a 6502. [05:29] That would explain a lot, actually. [05:29] germanium's decently speedy. [05:29] and then match ddebs against that mapping [05:30] I hope that can get massively simpler, if I can get the proper releases/versions/components etc. from LP [05:30] Anyway, we should indeed find an appropriately configured PPA and fix the temporary, short-term, 8 year solution to work for another 8 years :) [05:30] right now, the downloaded tarballs tell me just about nothing [05:30] heh [05:30] Right. [05:30] LP can get you the map from source -> ddebs, and we might even want to add a new API call or two to make that easier if it proves too slow. [05:31] No more guessing from tarball contents. [05:31] The main problem for you is working out which sets of ddebs you want. [05:31] And that should be easy to calculate from the Packages files. [05:31] yes [05:32] and then of course building the lists for a-f and calling that, which takes effing log (but is conceptually easy) [05:33] Building the lists should be near instant if we do it right with some local intelligence and caching. [05:33] a-f runs might be slower than peop, cause germanium's not actually as fast as I remember. :P [05:33] But we'll work that out. A big lock around the whole thing so it can take longer than ftpmaster and skip a run here and there should make it good enough. [05:33] a-f should take less than 15 seconds per DAS when configured correctly, I suspect. [05:33] Even on germanium. [05:33] 42.5 MB/s read speed [05:33] (on germanium's disk) [05:34] It's seek speed and a-f cache maintenance that's the bigger concern. [05:34] yeah [05:34] Unless you keep your a-f cache clean, you are basically dead. [05:34] As we discovered 18 months ago. [05:34] All tractable. [05:34] wgrant: yeah, I don't know the secret recipe to do that [05:34] As we discovered from the FEATURE CELSO IMPLEMENTED SIX YEARS AGO AND FORGOT TO TELL ANYONE ABOUT. [05:34] We have the power. [05:34] wgrant: I just drop the db from time to time, take the "runs for 30 hours" hit, and go on [05:34] Not that I'm bitter. [05:34] Heh [05:34] Yeah, that's what we used to do. [05:35] 15.6 MB/s write speed on germanium [05:35] that's really not great [05:35] and that was a linear cp of an 843 MB file [05:35] Now we run a daily cronjob which removes files that no longer exist in the pool. [05:35] so I don't know how much it sucks on random access [05:35] And keeps the cache snappy. [05:35] wgrant: Actually just triggered on a timer from the publisher itself, but whatever. [05:35] wgrant: you mean remove from the db? [05:35] pitti: Right. [05:35] So the DB doesn't grow without bound. [05:35] infinity: Well yeah, but effectively. [05:36] Important difference, cause racing those caches with an external job could have dire consequences. [05:36] Or just fail entirely if a-f actually uses proper BDB locking. [05:36] Pick one. [05:36] Dire consequences are the spice of life. [05:37] If germanium really is too slow for this, one might ask IS nicely to slap germanium's array on a newer machine. [05:37] But I don't think it'll limit us when we get it set up right. [05:38] Agreed. [05:38] Bed for realz. [05:38] Night. [05:38] sleep well [06:10] ... my Firefox just decided I'm Arabic. [06:11] Oh, no. Not Arabic. But some language I don't read that goes right to left. [06:11] Oh, yes. Arabic indeed. [06:12] So confused. [07:48] mvo: guten Morgen [07:48] mvo: do you know if there's a fix for apt's test suite regression with the new dpkg? [07:49] pitti: I don't think so, did I miss a mail from autopkgtest? [07:50] mvo: notifications aren't back on (will do with jibel today), so falling back to "pitti annoys people on IRC" [07:50] mvo: https://jenkins.qa.ubuntu.com/job/vivid-adt-apt/? [07:50] pitti: heh :) ok [07:51] mvo: might be fixed in Debian git already, as dpkg has been there for a while [07:52] pitti: right [07:53] pitti: I will take care of it [07:53] mvo: danke! [08:14] utlemming: I tested http://cloud-images.ubuntu.com/vivid/20141031.4/ amd64, working well; thanks! [08:14] utlemming: I suppose /current will appear once this gets cron'ed? [08:47] Noskcaj, pitti: since that gsettings-desktop-schemas changed some default value, do we override for e.g "button-layout" in ubuntu-settings to avoid regressions/behaviour change on upgrade? [08:47] same for "action-middle-click-titlebar" [08:48] do we need override* [08:52] mvo: "Jenkins Fixed - vivid-adt-apt 7" -- cheers! [08:52] bonjour seb128 [08:52] pitti, salut, ça va bien ? [08:52] seb128: sorry, I didn't notice a change in the general desktop or control center [08:52] what changed? [08:53] pitti, did you restart unity? is middle click on the title bar still doing minimize? [08:53] seb128: oui, alors j'ai me levé à 3 h :/ [08:53] :-( [08:53] I tried in a guest session and an existing user, but I didn't try title bar clicking; hang on [08:54] seb128: that maximizes/unmaximizes a window; is that different than before? [08:54] (not minimize) [08:54] pitti, well, I just read the diff [08:54] http://launchpadlibrarian.net/188976878/gsettings-desktop-schemas_3.12.2-0ubuntu1_3.14.1-1ubuntu1.diff.gz [08:55] [08:55] - ':minimize,maximize,close' [08:55] + 'appmenu:close' [08:55] enum="org.gnome.desktop.GDesktopTitlebarAction"> [08:55] - 'lower' [08:55] + 'none' [08:55] pitti, ^ [08:55] pitti: your welcome [08:55] so it seems like it's going to create behaviour changes, we might need to add those back in our ubuntu-settings override [08:56] seb128: yeah, but I figure the Ubuntu theme or so overrides this; I didn't check where that happens, just that it happens [09:00] pitti, ok,I didn't get the update, it looks in theory that middle click on the wm decoration should stop minimizing after that update [09:00] but maybe compiz has some override in some way [09:00] I'm going to check later when I get the update [09:53] hi developers, have a nice week! [10:13] pitti: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/archivepublisher/model/ftparchive.py cleanCaches has the magic you need. It should be pretty easy to stitch that in right now, independent of any other work. [10:38] cjwatson: thanks! === _salem is now known as salem_ [11:50] /wi/win 9 === MacSlow is now known as MacSlow|lunch === timrc-afk is now known as timrc [13:22] some help in understanding schroot and sbuild needed plz [13:23] i always get errors using sbuild 'Permission denied', 'could not clean source directory' with sbuild [13:23] have setup btrfs-snapshot schroot, to which i can login and do things === MacSlow|lunch is now known as MacSlow === salem_ is now known as _salem [14:29] mdeslaur: may i package gpgme to provide one library for gpg2 and one for gpg1, conflicting with each other? also what needs to be done to get gpg2 by default on the desktop? [14:29] gtk3 pinentry? [14:31] xnox: I know nothing about gpgme, so I'm not the one to ask, sorry [14:31] xnox: as for gpg2, I thought the problem was that it required an agent to work, and that was an issue [14:32] mdeslaur: so was it not you who uploaded https://launchpad.net/ubuntu/+source/gpgme1.0/1.5.1-6ubuntu1 ? [14:32] xnox: yes, because i TIL for a security update [14:33] xnox: but I'm knowledgeable about it [14:33] s/I'm/I'm not/ [14:33] mdeslaur: we ship an agent by default - gnome-keyring one, which doesn't support everything that gpg2 needs however. we'd need a change of the default agent. [14:33] mdeslaur: ok. [14:33] xnox: and we'd start an agent by default on the server? [14:34] mdeslaur: on the server, gnupg fallsback to forking tty agent by itself. [14:47] Laney, did you upload -v3 for gdcm? [14:48] seems so, wonderful [14:48] thanks [14:49] thanks, I can ping you if it gets accepted (and published, to avoid other rebuilds) [14:58] np === Sweetsha1k is now known as Sweetshark === _salem is now known as salem_ === Pici is now known as to === to is now known as Pici [15:29] hallyn, stgraber: hmm @ https://jenkins.qa.ubuntu.com/job/vivid-adt-lxc/7/ARCH=i386,label=adt/console -- did anything in cgmanager recently change wrt. the cgroup controllers? [15:30] pitti: yeah, cgmanager 0.33 has an updated api, which lxc uses - and uses wrongly. there is a fix in upstream lxc, but i guess it needs to go into vivid [15:30] there surely is bug 1346734, but we don't run systemd in vivid yet by default [15:30] bug 1346734 in systemd (Ubuntu) "Unprivileged LXC containers don't work under systemd" [Medium,Triaged] https://launchpad.net/bugs/1346734 [15:30] no it's not that [15:30] hallyn: ah, splendid; so I can chalk this off as "understood/handled" [15:30] it's that lxc tries to pass "all" to getpidcgroup [15:30] hallyn: thanks [15:30] (which isn't suported) [15:30] np. [15:31] stgraber: ^ were you going to push a new lxc to vivid soonish? [15:31] pitti: I'll upload a fix to vivid once upstream Ci is back online and I confirm the fix works [15:31] stgraber: merci === salem_ is now known as _salem === _salem is now known as salem_ === salem_ is now known as _salem === _salem is now known as salem_ [16:08] bregma, omg are you going to implement https://bugs.launchpad.net/bugs/793482 ? [16:08] Ubuntu bug 793482 in Compiz "[WISHLIST] Windows in global expose (Super+W) should have title underneath" [Low,Triaged] [16:09] brendand, hopefully at some point, but realistically "low" priority means we may suffer from the inevitable heat-death of the universe before we get to it [16:15] bregma, well it's only taken 3 years to be triaged [16:16] bregma, i can wait another 3 for it to be implemented :) [16:16] (or do it myself, yes i know) [16:16] patches welcome etc etc [16:17] brendand, I'm sure you can just whip something up in your spare time? [16:19] bregma, actually if i could get a hint or two about where to look... [16:24] stgraber: pitti: I went ahead and pushed lxc with the attach regression fixed to vivid, as that's pretty high prio [16:24] brendand, I'd start in the Compiz source at plugins/expo/src/expo.cpp [16:24] hallyn: ah, great, thanks [16:25] bregma, and does that already have some visibility to the window objects (note i know nothing about compiz) [16:25] ? [16:26] bregma, so that i would just have to fish out the title of the window [16:29] anyone know how to get the whoopsie identifier for a user from the CL? [16:29] you can get it from the gui by going Dash -> Security & Privacy | Diagnostics -> Show Previous Error Reports [16:29] (you can get it using C like this- https://bugs.launchpad.net/ubuntu/+source/sosreport/+bug/1319160/comments/2) [16:29] Ubuntu bug 1319160 in sosreport (Ubuntu) "Collect /var/crash info or the users personal crash key" [Medium,Triaged] [16:30] brendand, I can't be certain the name is available without digging down through the code -- at which point the fix would become trivial [16:32] bregma, well you should find out soon :) [16:46] gQuigs: there is dbus api to query it. [16:49] gQuigs: gdbus call -y -d com.ubuntu.WhoopsiePreferences -o /com/ubuntu/WhoopsiePreferences -m com.ubuntu.WhoopsiePreferences.GetIdentifier [16:49] or similar with dbus-send [16:51] xnox: thanks! [17:23] xnox: you'd talked about a casper patch for /var/crash the other day. Is that something you could do or give me an idea about? [17:30] bdmurray: in a meeting atm. i'll respond later. [17:31] xnox: okay, thanks [18:11] bdmurray: i concluded that it works correctly already, by virtue that we had to disable whoopsie to get rid of the pop-ups. [18:11] bdmurray: if it was working, simply because we are crashing all over the place and the upload daemon starts after the first crash, then indeed this needs to be fixed in casper. [18:12] bdmurray: something simple like "touch /var/crash/.anyfile; rm /var/crash/.anyfile" after the overlay has been setup for the target. [18:12] this will create an inode & /var/crash directory in the overlay and then inotify will work for the whoopsie daemon. [18:12] bdmurray: i have not created a patch for this, but will be happy to review it if you write one. [18:14] xnox: so the overlay work is already done we just need to "initialize" it so inotify will work? [19:01] @pilot in === udevbot changed the topic of #ubuntu-devel to: Archive: open | Devel of Ubuntu (not support or app devel) | build failures -> http://qa.ubuntuwire.com/ftbfs/ | #ubuntu for support and discussion of lucid -> utopic | #ubuntu-app-devel for app development on Ubuntu http://wiki.ubuntu.com/UbuntuDevelopment | See #ubuntu-bugs for http://bit.ly/lv8soi | Patch Pilots: barry === roadmr is now known as roadmr_afk === roadmr_afk is now known as roadmr === timrc is now known as timrc-afk === timrc-afk is now known as timrc === salem_ is now known as _salem [22:40] @pilot out === udevbot changed the topic of #ubuntu-devel to: Archive: open | Devel of Ubuntu (not support or app devel) | build failures -> http://qa.ubuntuwire.com/ftbfs/ | #ubuntu for support and discussion of lucid -> utopic | #ubuntu-app-devel for app development on Ubuntu http://wiki.ubuntu.com/UbuntuDevelopment | See #ubuntu-bugs for http://bit.ly/lv8soi | Patch Pilots: