[10:57] sigh, ssl [11:01] sigh, Turnip unit tests :) [11:06] at that stage of 'how did it ever work' [11:11] :( same here, except I know how.... I commented out unit tests and now: "Ran 246 tests in 27.055s OK", that's how it worked :) [11:11] *my* unit tests [11:11] I'm trying to work out how the dev certificate ever worked [11:13] hmmm tough one to debug .. [11:14] tomwardill: I thought there was some commentary about how that was put together [11:14] cjwatson: there's a script for it [11:14] Right [11:14] I've successfully altered and used the script [11:15] but this morning, another cert generated using the script doesn't work [11:15] Peculiar [11:16] yeah [11:16] I'm sure I've done something silly somwhere [11:25] I guarantee I've done something silly somewhere [11:25] (What were we talking about?) [11:27] Incidentally, I've got the test suite to start up on Python 3. 15000 lines of diff or thereabouts, so I'll be trickling in various MPs over time [11:27] Whole hundreds of tests pass! [11:27] Let's quietly ignore the thousands that don't for now [11:28] :) [11:28] +1 [11:28] cjwatson: niiiiice [11:28] (And I'm also going to be firmly putting this aside for a bit even though it's a fun toy, because there are a few other things I really need to get done) [11:29] can you dig^Wpush it in any useful form? [11:29] Not really yet, but certainly an aim [11:30] Well I mean I suppose I can throw the branch up on the understanding that it's all a pile of poo [11:30] https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+ref/py3 [11:30] git push origin here-be-dragons ;) [11:30] (fab, thanks :)) [11:31] I will not be supporting this in any way shape or form for the moment :) [11:31] And it'll rebase freely [11:32] Oh also it relies on lazr.restful work that isn't pushed anywhere yet [11:32] That's more annoying to casually push because bzr [11:32] is there any expectation that branches are in any way supported? [11:33] er [11:33] that non-"mainline" branches etc. etc. [11:33] Well maybe not supported, but I mean "don't ask me questions about how to get it running" [11:33] right sure [11:34] My lazr.restful branch is https://code.launchpad.net/~cjwatson/lazr.restful/py3-declarations/+merge/378548 plus https://paste.ubuntu.com/p/nX9zfPPg4q/ [11:34] The biggest remaining problem with lazr.restful is that it runs into cgi.FieldStorage bugs on Python 3 and I haven't worked out how to sort those out, especially since cgi.FieldStorage is going away in 3.9 or so [11:35] Makes it hard to get the test suite going properly [11:35] It's no doubt workaroundable somehow, just haven't spent the relevant time on it [11:37] Oh and also https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+ref/py3 for dependencies [11:58] tomwardill: iirc you asked something about Buildbot versions a while back? [11:59] tomwardill: I know ~0 about Buildbot but I did just clock that we're running 0.8.5 and latest stable is 2.6.0 [11:59] well, that's fun :) [12:01] (I know long-term we're looking at a variety of things we can/should do there, but part of me wonders about the ROI on even a small version bump?) [12:10] SpecialK|Canon: This is more a function of praseodymium still running precise than anything else [12:11] (We run distro buildbot packages, or backports of them, I forget which) [12:11] We should at least get that upgraded to xenial; praseodymium is a weird and critical system that I have roughly 0 visibility into, which is more or less why we haven't [12:13] Makes sense, thanks [12:15] SpecialK|Canon: https://portal.admin.canonical.com/C110859 for history of the last major buildbot care-and-feeding work we did [12:16] And I suppose the superseded https://portal.admin.canonical.com/C106558 too [12:17] I think precise → xenial is likely rather less rough than lucid → precise was but I've never tested it ... [12:17] I can't find sluagh and radande on https://wiki.canonical.com/Launchpad/Instances - are they pining for the fjords? [12:19] That page doesn't really list build/test infrastructure [12:19] right ok [12:20] And indeed, there's probably more upgrade complication on the worker side [12:21] Due to the pre-release implementation of ephemeral containers that they're currently using [12:21] But hopefully 0.8.12 master and 0.8.5 workers can interoperate so we could decouple that upgrade? Not sure [12:23] I'm unlikely to be able to look at it for ... a while, but would be happy to braindump what I know on somebody else if need be [12:24] Fortunately buildbot is such that we don't need continuous availability. If the answer ends up being hold breath, upgrade, fix up on the other side, that may be workable as long as we're reasonably confident we'll be able to put the pieces back together [12:25] Is there an enumeration of "what buildbot does for us" anywhere? [12:27] I'm wondering to what extent we can avoid "a careful upgrade from a 2012 version" and just go for "drop it and go with something different, whether that's latest-stable buildbot or a Jenkins or similar" [12:27] I'm not suggesting we do the latter lightly, or that it's without risk/cost etc. [12:30] We're not wedded to buildbot if something else works. The main thing is making sure that test runs don't get substantially slower, since they already take a long time; our current setup does careful IIRC 24-way parallelisation spreading tests across multiple containers on beefy-for-the-time machines [12:30] I have no idea how long it would take if we just did a naive implementation on current Jenkins/scalingstack, and those would be useful numbers to have [12:30] Ah, yes, I'd forgotten that nuance, thanks [12:31] We do have some automation around buildbot runs which deal with merging {master,db-devel} into {stable,db-stable} when their corresponding test runs pass, and merging master into db-devel when tests on master pass [12:32] That would be easy enough to reinvent on top of something else if we needed to [12:32] I don't think there's really anything else worth mentioning [12:33] I'd also say that I don't know of any problems particularly attributable to the old version of buildbot at the moment (we have buildbot unreliability in various ways, but AFAIK those are all due to bugs in the LP test suite, often related to parallel testing). Do you? [12:34] Of course keeping tools current is a worthy goal in and of itself [12:35] And there are possibly interesting things we could do with less basic tooling. But just trying to get a sense of why you're asking [12:36] Oh sorry yes [12:36] Standard "can we make things easier/better by upgrading/removing this and what's the ROI like?@ [12:36] *" [12:37] And I was reminded of buildbot now I'm on canonical-launchpad and get emails about failures [12:40] I think I fixed one of the main sources of buildbot unreliability in https://github.com/testing-cabal/txfixtures/pull/13 + https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379456 [12:41] Ah yes I saw that - nice [12:41] Just to be clear I'm not saying I think buildbot is unreliable, just that the emails prompted me to go looking at it [12:44] The remaining ones seem to be lp.services.job.tests.test_celeryjob.TestRunMissingJobs.test_run_missing_ready_does_not_return_results (e.g. http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/865/steps/shell_9/logs/summary), lp.buildmaster.tests.test_interactor.TestSlave failing in various ways (e.g. ... [12:44] ... http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/864/steps/shell_9/logs/summary, http://lpbuildbot.canonical.com/builders/lp-db-devel-xenial/builds/860/steps/shell_9/logs/summary), lp.testing.layers.RabbitMQLayer:setUp timing out (e.g. http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1052/steps/shell_9/logs/summary) [12:45] And to be fair there is one thing that is an issue with buildbot, although it was triggered by upgrading zope.testrunner; buildbot has an old version of subunit that doesn't understand the much more robust v2 protocol, so it sometimes raises exceptions when trying to parse the subunit output from builds, which is v annoying [12:46] All the remaining issues are hard to reproduce locally or I'd have fixed them already ;-) [12:47] But if any of us ever happen to run across one of them locally, and we aren't dealing with some kind of seriously urgent fire at the time, then I'd urge dropping everything else to take the opportunity to figure it out - it will pay off [12:50] Neat, cheers [12:53] I think upgrading praseodymium to xenial would let us fix the subunit thing. Probably [15:07] gah, sigh [15:07] still had a py3 virtalenv in my turnip instance [15:07] this did not end well [15:08] or infact, start well [15:37] Would anyone be up for reviewing this small series? https://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+merge/379682 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379683 https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379684 [15:37] I believe it should significantly improve our deployment speeds [15:39] This is sort of as discussed in the Tuesday weekly meeting a week or two ago: instead of worrying about making sure every deployment target can build wheels, arrange for a full set of wheels to be part of the deployment artifact [15:40] And entirely sidesteps the fact that pip's automatic wheel cache isn't relocatable [15:42] nice [15:55] Anyone object to me self-approving https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379509 (six-xmlrpclib)? It's big but basically mechanical [15:57] cjwatson: no objections - I skim-reviewed just for lightweight value [15:58] Thanks, I'll take that as sufficient and the test suite can find anything I might have missed [15:59] cjwatson: is it possible to login as someone when using 'make iharness'? [16:00] I'm getting a bunch of ForbiddenAttribute errors with pretty much everyting (including DistributionSet.getByName) [16:01] tomwardill: iharness uses execute_zcml_for_scripts, so logging in isn't usually necessary [16:01] hmm [16:02] Can I see your current code? [16:03] cjwatson: https://pastebin.canonical.com/p/7R5HG32NKK/ [16:03] Oh, that's a known annoyance of iharness [16:04] If you just let the default REPL try to print things for itself then it wants to get at __dict__ of objects and the proxy tends to forbid that [16:04] But you can assign to variables, or you can do print(foo) [16:04] aha! [16:05] yep, print(ds_set.getByName('ubuntu')) works [16:05] thanks [16:05] Would be nice to work out a way to make that less irritating, but it's easy to work around once you know about it [16:14] Also https://code.launchpad.net/~cjwatson/lpjsmin/tox/+merge/379695 to start bringing another dependency into shape [16:15] And https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379654 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/379657 are simple (and much shorter) import rearrangements [16:19] ooo, I've made a recipe [16:19] now to work out how to dispatch that [16:20] cjwatson: +1 to all [16:20] I haven't done the views for that yet IIRC, so that might be mildly tricky [16:20] the death of buildout makes me sad though [16:20] I suppose you can .requestBuild in harness [16:20] yeah, that's why I'm in iharness poking things [16:20] Yeah, buildout was nice in its day [16:21] Even in 2014 I was still going "OK, how the hell do I get this thing to build", though [16:21] Not sure how much that's about buildout and how much about the way our projects use(d) it [16:22] https://pypi.org/project/isotoma.recipe.django/1.2.2/ [16:22] I think that was everyone's response to buildout tbh [16:23] Then https://code.launchpad.net/~cjwatson/lpjsmin/py3/+merge/379696 makes it actually work on py3 [16:23] Was short enough I didn't bother trying to split it up further [16:25] Yeah, recipes were quite nice when you found/wrote one tht worked [16:45] cjwatson: if I've done requestBuild() do I need to do anythign to process that and dispatch it? [16:45] (I've got LP running via 'make run' and have buildd-manager running) [16:49] tomwardill: transaction.commit() [16:49] tomwardill: But not otherwise; check buildd-manager logs to see if it's complaining about anything, e.g. a missing base image [16:50] ah, that is feasible [16:50] * tomwardill locates logs [16:52] Was playing with tomwardill's lp-btw (sorry I forgot the name already) [16:52] hmm, nothing in the buildd logs, the build appears to be in the state 'NEEDSBUILD' [16:55] tomwardill: the other trick here is to make sure you have at least one virtualised buildd [16:55] or it won't ever dispatch [16:56] aha! [16:56] that might be it [16:56] you can make it be fake-virtualised via a trick I forget [16:56] what's with all the relinking stuff we do with the rocketfuel* scripts? [16:56] (i.e. so it doesn't actually try to call ppa-reset) [16:56] 2020-02-24 16:56:09+0000 [QueryProtocol,client] exceptions.TypeError: ('Could not adapt', , ) [16:56] looks like I have a bug! [16:57] yay! [16:57] Ah yes, configure builddmaster.vm_resume_command to /bin/true or something [16:57] SpecialK|Canon: relinking? [16:58] oh that [16:58] SpecialK|Canon: it was particularly good for bzr to avoid having a bazillion copies of dependency checkouts for every branch you ever had, but it's still handy for worktrees [16:58] naively it looks like it's reimplementing virtualenvs [16:58] no [16:59] it's for sourcecode/, which is not virtualenv [16:59] it's sort of more like codetree [16:59] (in fact I have an unreviewed MP somewhere to make it use codetree ...) [16:59] I recognise that the README says not to ask about sourcecode/ [16:59] but uh [16:59] but the symlink stuff is to avoid having to have lots of checkouts [17:00] https://paste.ubuntu.com/p/kKftSPNTXM/ [17:00] (let's leave aside that my primary worktree links into an old bzr directory ... I should fix that) [17:01] Don't we have lots of checkouts anyway? Isn't that what rocketfuel-get does just beforehand? [17:01] right, ok [17:01] To be fair there may well be cruft in rocketfuel-get; I don't use it much [17:02] But I think the idea is that after it pulls it goes around and updates all your worktrees (née branches) [17:02] Was more necessary with bzr [17:04] Actually sorry when you say "more like codetree" [17:04] https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/373737 [17:04] I thought the primary value of codetree was for cross-language / cross-visibility stuff? [17:04] the relinking stuff isn't like codetree, but sourcecode/ in general is [17:04] vs. e.g. `pip install --editable` [17:04] ah ok [17:05] Not particularly; codetree is for cheap-and-cheerful vendoring-by-checkout [17:05] I very much enjoy the number of deleted lines in that MP! [17:05] With pinned versions [17:07] See https://git.launchpad.net/launchpad/tree/utilities/sourcedeps.conf, which you'll probably notice looks awfully like a codetree input file [17:07] (I think it was originally config-manager, back in the day) [17:08] https://bazaar.launchpad.net/~codetree-maintainers/codetree/trunk/view/head:/README.md#L14 I think you're right! [17:08] Oh wait you mean sourcedeps was [17:08] Well, both [17:09] It's possible the pile of stuff in lib/devscripts/ predates config-manager being a separate thing. I haven't bothered to go back and dig through the history [17:09] But in any case, no point maintaining all of that in-tree IMO [17:11] It is separately true that everything in sourcecode/ (now that mailman is gone) could in principle be moved into the virtualenv, and that this would be an improvement [17:12] There are some details of bzr/brz plugins that I'm not sure about [17:12] So in that sense, yes, this is reimplementing virtualenv and should die [17:13] It's just a matter of working out all the details at this point [17:14] Nodnod - I hope it's clear I'm not suggesting it's easy/straightforward! Just trying to get a clearer understanding in terms of things I'm more familiar with [17:15] Yeah [17:15] loggerhead is https://bugs.launchpad.net/loggerhead/+bug/383360 / https://bugs.launchpad.net/loggerhead-breezy/+bug/1831661 [17:15] Bug #383360: Loggerhead doesn't declare its dependencies in setup.py

[17:15] Bug #1831661: loggerhead can't reasonably be installed using pip due to self-import in setup.py [17:15] I'm hoping old_xmlplus and pygettextpo can go away; I have a semi-tested patch to convert the latter to Babel [17:16] For the former, I think I was complaining the other day about how painful it was to parse XML DTDs in Python; that's why [17:16] Some of the others relate to work jelmer has been doing to build more previously-in-plugins functionality directly into breezy [17:17] (I'm explaining in detail here since I hope somebody will be enthusiastic about picking up some individual part of the puzzle ...) [17:18] Could you explain in detail in a LP bug / Trello card please? I'm enthusiastic but also timeslices... [17:19] https://bugs.launchpad.net/launchpad/+bugs?field.searchtext=sourcecode :) [17:19] touche ;) [17:19] * SpecialK|Canon needs to sort out his compose key [17:32] ooh, it tried to dispatch a build [17:32] but failed because the builder is missing some configuration [17:50] cjwatson: my buildd manager / buildd appears to have got itself stuck in 'cleaning', despite me restarting everything (I think) [17:50] oh wait [17:50] literally as I type that, it does a thing [17:50] "Scanning 10.104.9.248 failed with: builder_proxy_auth_api_admin_secret is not configured." [17:54] Right, probably in a slightly inappropriate timeout loop or similar [17:54] whereabouts should I put that configuration? [17:55] configs/$LPCONFIG/launchpad-lazr.conf [17:57] oh, it's LP side, not buildd side? [18:01] tomwardill: Yep [18:01] buildd-manager contacts the snap proxy to get a token to dispatch to the builder [18:01] aaah, right [18:02] * tomwardill configures, restarts everything, waits for it to wake up [18:02] It's complex to set up at the moment; if you don't have it there already then you could just unset builder_proxy_host? [18:02] But maybe you have it [18:03] aah, I don't have it, so yeah, that makes most snese [18:03] * tomwardill waits more