=== igitoor_ is now known as igitoor
tomwardillGoing to start landing the yarn/sass upgrades and testing on dogfood08:01
tomwardillobject now if this is a bad plan :)08:01
tomwardillokay, last buildbot run failed with a bunch of JS timeouts, going to retry09:40
tomwardillbah, failed again10:07
tomwardillcjwatson: is there anything special I need to do to ensure the nodeJS version is upgraded in the buildbot base images?10:07
cjwatsontomwardill: https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/Buildbot#Updating_the_LXC_environment10:08
cjwatsontomwardill: You'll need to ask IS to do that10:08
tomwardillaha, that would probably be the casue10:08
cjwatson(On both workers)10:08
tomwardilllooking more promising now10:40
tomwardill\o/ success10:57
* tomwardill lands more things10:57
tomwardillokay, still getting timeouts12:42
tomwardillgot an actual problem then12:42
tomwardillcjwatson: how do I run these tests via firefox?13:28
cjwatsonfind the absolute path to the associated .html, paste that in13:28
cjwatson[4/4] Building fresh packages...13:29
cjwatson[1/1] ⠁ node-sass: g++ '-DNODE_GYP_MODULE_NAME=libsass' '-DUSING_UV_SHARED=1' '-DUSING_V8_SHARED=1' '-DV8_DEPRECATION_WARNINGS=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DLIBSASS_VERSION="3.5.5"' -I/home/cjwatson/.node-g13:29
cjwatsontomwardill: ^- is node-sass meant to be doing that?  I thought you'd squashed that13:29
tomwardillyes, I had/have13:33
tomwardillcjwatson: have you got the latest source-dependencies?13:34
cjwatsonAfter an initial false start anyway13:34
cjwatsonAnd I upgraded my base container13:34
cjwatsonMaybe a cache that defeats your attempts to tell whether your fix worked?13:35
tomwardillcjwatson: can you add --verbose to the end of the yarn line in the css_combine Makefile13:36
tomwardillit should output whether it's found the .node binding before it starts compilation13:36
cjwatsonOh.  That overflowed my terminal's scrollback13:55
* cjwatson tries again with |less13:55
tomwardilloh, yes13:56
tomwardillit is very verbose13:56
tomwardillbut somehwere in the middle will be the output about the binding13:56
tomwardillmy local lxd has decided it's going to drop all of it's internet connection13:58
tomwardillwhich is not ideal13:58
cjwatsonverbose 3.23 Copying "/home/cjwatson/.cache/yarn/v6/npm-node-sass-4.14.1-99c87ec2efb7047ed638fb4c9db7f3a42e2217b5-integrity/node_modules/node-sass/binding.gyp" to "/home/cjwatson/src/canonical/launchpad/git/launchpad/yarn/node_modules/node-sass/binding.gyp".14:00
cjwatsonlooks possibly relevant14:00
cjwatsonHm, maybe, not sure14:03
cjwatson[4/4] Building fresh packages...14:04
cjwatsonverbose 6.952 node-sass build Binary found at /home/cjwatson/src/canonical/launchpad/git/launchpad/download-cache/yarn/node-sass-4.14.4-linux-x64-57_binding.node14:04
cjwatsonverbose 7.674 Binary found at /home/cjwatson/src/canonical/launchpad/git/launchpad/download-cache/yarn/node-sass-4.14.4-linux-x64-57_binding.node14:04
cjwatsonThat file shows as modified in git14:04
tomwardillah, so if it ever compiles it, it will replace your existing one14:05
* cjwatson copies it aside and restores it14:05
tomwardillbecasue input and output are synonymous, apparently14:05
cjwatsonJust going to try nuking ~/.cache/yarn/14:05
cjwatsonAh of course14:11
cjwatsonMy container is 32-bit14:11
tomwardillyes, that would do it14:11
cjwatsonBut it helpfully writes it back to the path you gave anyway, so replaces *-x64-* with a 32-bit build14:12
cjwatsontomwardill: So um where did the 4.14.4 bit come from?14:13
cjwatsonhttps://www.npmjs.com/package/node-sass <- newest version is 4.14.114:14
tomwardillyes, I just looked at that and realised my mistake there14:15
* tomwardill gives up on today, it's clearly faulty and needs to be returned14:15
tomwardillcjwatson: okay, now I've finally managed to test it (via X11 forwarding firefox), the JS tests still pass locally14:16
cjwatsontomwardill: I'd suggest building two worktrees, one from before it started failing, one from current master14:18
cjwatsontomwardill: And then diff the two after running 'make' in each14:18
cjwatsonI'll fix up the versions and arches and stuff14:18
cjwatsonhttps://code.launchpad.net/~cjwatson/lp-source-dependencies/+git/lp-source-dependencies/+merge/384913 and https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/38491414:37
tomwardillcjwatson: diff doesn't show anything immediately obvious and I can't make the JS tests fail locally15:00
tomwardillcjwatson: I'm out of ideas for how to reproduce this15:09
cjwatsonTrying to see if I can find anything15:09
* tomwardill fetches more biscuits15:11
cjwatsontomwardill: Ah look15:23
cjwatsontomwardill: http://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1295/steps/shell_8/logs/stdio15:23
cjwatsonFrom the build step before the test step15:23
cjwatsontomwardill: So same problem as in my setup before the MPs I posted above15:23
cjwatsonExcept it doesn't have the bits necessary to build it15:23
tomwardillserves me right for just ctrl-f for 'Failure'15:23
cjwatsonOr rather, it tries to connect to the network and network says no15:24
tomwardillright, so it's a 32 bit container15:24
cjwatsonIt's not often you need to look at the output of the build step, but when you've just frobbed the build system and things are being weird ...15:25
tomwardillI've just +1'd your MPs15:25
tomwardillare we actually deployed in a 32 bit world then?15:26
cjwatsonI suspect buildbot does this because it's doing something like 20 tests in parallel and running them in 32-bit mode uses less memory15:26
tomwardillyeah, that would make sense15:27
cjwatsonI admit I hadn't actually realised buildbot was 32-bit (or maybe I knew once and forgot)15:27
* cjwatson adds a note about that to https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/Buildbot15:29
cjwatsonAlso I kind of hate that the test proceeds even though the build fails.  If somebody wants to figure out how to fix that, it might be a good intro to lpbuildbot15:30
tomwardillcjwatson: I've just mentally added it to my 'work out buildbot and move it to bionic/focal' task15:30
tomwardillwhich I might do next, I think it's sufficiently annoyed me for me to have a look at it now :)15:30
cjwatsonIt is *meant* to halt the build on failure15:30
cjwatsonLook at bzrbuildbot.shell.ShellCommand (yes, bzrbuildbot is now horribly misnamed but ...)15:31
tomwardillcan you point me at the code/setup/docs for buildbot?15:31
cjwatsonPublic branch in lp:lpbuildbot, some private customisations in lp:~canonical-launchpad-branches/lpbuildbot/production (make changes in the former if you can - it gets merged into the latter)15:31
cjwatsonI think all other documentation that exists is probably in https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/Buildbot15:32
cjwatsonoh.  Actually the bug is probably in lp-setup-lxd-build15:33
cjwatsonlp-setup-lxc-build, rather15:33
cjwatsonWhich is in the most horribly confusing bit of the whole setup15:33
tomwardillrighto :)15:33
cjwatsonWhen parallel buildbot tests were being developed, there was an effort to get all the code needed for that sort of thing out of puppet and into an "lpsetup" package15:34
cjwatsonAnd this was sort of 80% done15:34
cjwatsonIt's in lp:lpsetup and deployed via a recipe build into ppa:launchpad/ubuntu/ppa15:35
SpecialK|CanonCan this end up on a wiki page if it isn't already please?15:36
cjwatsonOr maybe those scripts aren't15:36
cjwatsonIf I understood it well enough to not be misleading I would have documented it a long time ago :-/15:36
cjwatsonI think possibly lp-setup-lxc-* never got fully extracted from puppet15:37
cjwatsonWhich is bad because we kind of need to fix those, partly for this but also to use lxd15:37
SpecialK|Canon"this is a thing probably worth looking at" / "here is possibly what happened" are still valuable documentation15:37
SpecialK|CanonEr, sentence fail, but hopefully you get the idea15:37
SpecialK|CanonI'd rather have your vague half-understandings on a page (with appropriate notes as to confidence in the quality) than not at all!15:38
cjwatsonOK, I'll see what I can do15:38
SpecialK|CanonThank you!15:39
cjwatsontomwardill: I'd suggest getting an SRE to give you the current contents of /usr/local/bin/lp-setup-* in the appropriate container on e.g. sluagh, and then comparing that with lp:lpsetup and lp:canonical-is-puppet to work out which one of those it comes from15:41
cjwatsonShould be able to tell from the comments at the top of lp-setup-lxc-build, say15:41
tomwardillthat makes sense15:41
* tomwardill does that now15:41
cjwatsonAnd if they aren't the lpsetup versions, we should check the effective diffs between them, and probably make them be the lpsetup version15:42
cjwatsonBecause while lpsetup is kinda weird it's better than a defunct-for-us config management system15:42
tomwardillpoked buildbot15:58
cjwatsonBuild step worked now, so that should hopefully work better16:02
tomwardillexcellent :)16:05
tomwardillwonder why it worked that once16:05
cjwatsonIt's interesting, since it failed before the sass move16:06
cjwatsonhttp://lpbuildbot.canonical.com/builders/lp-devel-xenial/builds/1291/steps/shell_8/logs/stdio is a somewhat different failure16:06
cjwatsonOh right16:06
cjwatsonSo two different failures16:06
cjwatsonIt failed because you landed the yarn upgrade before buildbot workers had a newer nodejs16:06
cjwatsonYou got that fixed, and then nodejs on the worker was able to parse yarn and the build passed16:07
cjwatsonThen you landed the node-sass upgrade, which failed due to the 32-bit thing16:07
tomwardillaah, right16:07
tomwardillof course16:07
cjwatsonAll appropriately deterministic :)16:07
tomwardillbut because I never looked at the build step, they had the same apparent failure mode to me16:07
tomwardillwonder what new and exciting failure I can come up with the next landing!16:11
tomwardillstay tuned to find out!16:11
cjwatsonI'd have let you at least fix it had I realised that it was the buildbot failure rather than just weird thing on my machine :)16:13
tomwardillheh, no worries :)16:16
* tomwardill has also broken his glasses today, so it's not like it's been the best day all roudn16:17
tomwardillbuildbot success!16:41
tomwardilllanding the next in the series16:41
tomwardillanyone got time for this horrible diff? https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/38483416:55
pappacenaI can check that17:19
* pappacena immediatelly regret this decision17:20

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!