[01:16] hey tych0, there? [01:16] would be easier to chat here for a moment instead of your bug [01:16] I see others reporting the problem earlier on here [01:43] bigjools: here now [01:43] tych0: howdy [01:44] tych0: was just wondering what you were doing when not seeing the proxy used? [01:44] just standard stuff [01:44] fastpath with trusty commissioning and precise images [01:45] i honestly don't have any idea whether it happens all the time or not [01:45] sometimes i am paying attention and sometimes i just let it run :-) [01:45] I think it's a bug in the preseed for fastpath [01:45] is there a file it drops somewhere or something? [01:45] i can check for that [01:45] next time i see it [01:45] I vaguely remember talking to smoser about this but his suggestion to fix it didn't work and then it fell by the wayside [01:46] the proxy config in the preseed is curtin-specific [01:46] ah, ok [01:47] well the d-i is d-i specific, so ... :) [01:47] yep [01:47] i assume it edits some apt preferences file or drops something in preferences.d? [01:48] oh [01:48] this node has 90curtin-aptproxy [01:48] look in contrib/preseeds_v2/curtin_userdata [01:48] was that the fix? [01:48] well apparently it doesn't work :) [01:48] well [01:48] it might [01:48] i didn't file it about this node [01:48] i just know i've seen it before [01:49] ok [01:49] if there is something i should look for [01:49] i can look for that next time i know i see it [01:49] i guess maybe the absence of this file is a good place to start [01:49] I am not sure tbh. If I start to look at this I will first examine the squid log as the node boots [01:49] yeah [01:49] then delve into the node itself [01:50] ok [01:50] there's different things that get pulled off the internets [01:52] bigjools: do you want my copy of /var/log/maas for the other issue? [01:52] i'm not sure if it'll be useful or not [01:52] it does sound like a bug in celery [01:52] whatever happens, celery shouldn't crash [01:53] tych0: it's a celery bug imo [01:53] and I agree [01:53] yep [01:53] we can work around it with a wrapper [01:53] eyah, that might be good [01:53] well there *is* a wrapper already, we just need to make it restart [01:53] i just sat down this morning and nothing would work [01:53] made me a sad panda [01:53] same happened to me yesterday [01:54] restarted the worker and boom, loads of jobs went through that had been queued [01:54] which is kinda dangerous [01:54] yeah [01:54] same here [01:54] a couple of extra power on power off cycles to make things interesting :-) [01:54] I think we need to put expirations on all the jobs [01:54] some have that but not all [01:55] but we'll revisit this as part of work to harden maas [02:09] rharper: why is the tgt stuff ebing dropped? [02:09] err [02:09] jtv: ^^ [02:10] The new import script doesn't use our pre-existing tgt setup. [02:11] jtv: right, that's why I'm asking... what does the new script do? [02:11] That link we had in /etc/tgt/conf.d was pointing to a file that no longer existed. [02:11] It uses tgt-admin to create the targets. [02:11] roaksoax: the new script constructs a tgt conf on the fly and inserts it using tgtadmin [02:11] IIRC it does write a "metadata" file, and a master config, but all in /var/lib/maas. [02:12] bigjools: ok... where do these get stored, any ideas? [02:12] /var/lib/maas/boot-resources [02:12] roaksoax: they are ephemeral [02:13] No there is a tgt config file. [02:13] it is not used after insertion though [02:13] you can query tgt using "tgtadmin -s" [02:13] with no config [02:14] the config is for the admin script, not tgt. [02:14] bigjools: tgt-admin [02:14] bigjools: ok cool then [02:14] roaksoax: yeah I forgot which is which, there's a helper wrapper and the main thing [02:15] I think tgtadmin is the wrapper [02:15] bigjools: tgtadmin doesn't exist [02:15] roaksoax: it does on my box :) [02:16] tgt-admin, with the dash [02:16] there are two scripts [02:16] Not on my machine. [02:16] bigjools: it does not exist on my system [02:17] after upgrade [02:17] Not on my Trusty machine. [02:17] * bigjools boggles [02:17] let me boot my server, one sec [02:17] I am 100% sure I have seen both scripts [02:19] From an installed package that isn't a dependency? [02:20] ah ok it's tgtadm [02:20] my bad [02:22] ok cool [02:25] Uh-oh. [02:25] roaksoax: what did you think about my bug on the packaging? [02:25] https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1300507 [02:25] Ubuntu bug 1300507 in maas (Ubuntu) "Rabbit password is reset on every upgrade which forces lockstep cluster restarts" [Undecided,New] [02:25] Dear friends, who takes care of re-installing the tgt targets after reboot? [02:25] bigjools: i'm looking into it [02:25] roaksoax: ok cheers [02:26] bigjools: ^^ [02:26] I just rebooted my maas server and got no output from "sudo tgt-admin -s" [02:26] jtv: the script [02:26] uhmmm [02:26] jtv: oh,,,, oh dear! [02:26] that's the bug we found then [02:26] then that's why we need config files [02:26] Could well be. [02:26] jtv: which means we need that confiug back [02:26] fuuuuuuuuuuu [02:26] yup [02:26] New one. Not the old one. [02:27] jtv: the new one is still ephemeral IIRC [02:27] it gets written once for each target [02:27] No, there's a single config with all targets. [02:27] In the "current" snapshot. [02:27] jtv: are you *sure*? [02:27] anyway I will file a bug [02:27] arse [02:27] I have 4 definitions in my maas.tgt. [02:28] jtv: ok I think this code changed since I last looked, it seems ok now [02:28] What code? [02:29] the script [02:29] https://bugs.launchpad.net/maas/+bug/1300548 [02:29] Ubuntu bug 1300548 in MAAS "tgt targets do not persist after a reboot" [Critical,Triaged] [02:32] And now, having smashed everyone's hopes and dreams, I can reboot in peace. [02:32] heh [02:33] bigjools: so there's something weird [02:33] bigjools: i have no idea why this happens [02:33] bigjools: i have not make any changes that would cause this in comparison to trusty I think [02:35] Oh dear. Shouldn't have rebooted. Suddenly I'm back in low-res. [02:36] roaksoax: the password reset thing? [02:36] bigjools: yup [02:36] This is horrible. Big, fuzzy pixels. Luckily I kept my pre-Saucy hack somewhere to get back to a decent resolution. [02:36] roaksoax: it has always happened [02:36] bigjools: i've never seen it [02:37] we just ignored the problem until now; for some reason the cluster is getting restarted first which has made it apparent [02:37] We're nog going to be limited to 1920×1200 for the release version, are we? [02:37] jtv: hi-dpi is a feature of unity now IIRC [02:37] It worked in Saucy, and until today, Trusty. [02:37] roaksoax: but basically every upgrade I ever do resets the rabbit p/w [02:38] bigjools : does that file exist in the inital precise package? [02:38] bigjools: and has it changed since then? [02:38] which file? [02:38] maas_local_celeryconfig.py [02:38] * bigjools checks [02:40] roaksoax: it's new after precise [02:40] I think [02:40] bigjools: ok so if we iupgrade from precise to trusty, then we still need to run that file [02:40] bigjools: err to run that psasword change [02:41] the actual file is not in my tree it gets generated [02:41] roaksoax: why do we need to regenerate passwords at all? [02:41] we cannot do this, it screws over remote clusters [02:42] bigjools: why? becuase the upgrade should generate passwords for systems wher the file didn't exist before [02:42] roaksoax: one sec [02:42] bigjools: there's a reason why it is there. if we upgrade say precise to trusty directly, and we don't have that, then we see failure because the psasword would never get generated [02:43] roaksoax: this is going to break when updating on remote clusters, it can only work when cluster is local [02:44] otherwise they will be out of sync [02:44] bigjools: well then we should recommend upgrading regions first and then upgrade clusters [02:45] so the cluster gets regenerated [02:45] get new pass [02:45] bigjools: what should actually happen, ois region should tell the clusters to update their password automatically [02:45] roaksoax: it still forces a lockstep upgrade. I don't know how we can get around that [02:45] roaksoax: agreed [02:46] roaksoax: I'll file a bug a bout that [02:48] roaksoax: https://bugs.launchpad.net/maas/+bug/1300554 [02:48] Ubuntu bug 1300554 in MAAS "If the rabbit password changes, clusters are not informed" [High,Triaged] [02:48] bigjools: cool, otherwise we can just note that in a release upgrade [02:48] bigjools: err release note [02:48] roaksoax: fine for now [02:48] thanks [02:48] bigjools: that for upgrades that bug will happen and the "fix" is to restart the clusters [02:48] anywayi'm off [02:48] night [02:48] roaksoax: something must have changed in apt for it to screw the local cluster [02:49] nn roaksoax [02:49] different ordering of installs [02:49] ok cheers roaksoax, sleep well [02:49] * bigjools thinks about writing release notes [02:52] bigjools: btw.. i was thinking thay maybe tgtadmin might have a way to export the condig into a file for persistancy [02:52] wouls be worth looking into that [02:52] roaksoax: yeah, good point, thanks [02:53] --dup! [02:53] err [02:53] --dump [02:53] any better jtv? [02:53] Well this gets me back to the previous bad setting. :( [02:53] heh [02:53] bigjools: cool. we should dump each time a new entry gets added [02:53] And my previous xrandr incantations didn't work. [02:53] anyway... night! [02:53] jtv: just noticed tgt-admin has a --dump which might be useful [02:53] Good night. [02:53] cheers roaksoax [02:53] bigjools: I doubt it. [02:54] We already write the full config anyway. [02:54] just an option [02:54] ok [02:58] Wow, and the letter "b" is broken in my dash. That's a weird one. [03:09] * bigjools eats lunch === CyberJacob|Away is now known as CyberJacob [09:01] rvba, ping [09:03] rvba, bigjools, allenap, I managed to pin down the probable cause of bug 1299114 [09:03] bug 1299114 in MAAS "'ValidationError' object has no attribute 'error_dict' when creating a network" [Critical,Triaged] https://launchpad.net/bugs/1299114 [09:04] dimitern: Excellent! However, I have to pop out for a few minutes. I’ll ping when I’m back. [09:04] http://paste.ubuntu.com/7188828/ - it seems the error happens only when you try to add a network with a ip (network) address that matches an existing network [09:05] Which is probably the cause of the error in the first place. [09:05] There's a separate check for that, and it must be broken. [09:05] yep, it's just does not have a good error message [09:06] Not what I meant. :) The ValidationError probably needs to be constructed in some particular way. [09:07] dimitern: could you update the bug with your new information? [09:08] Hmm... I wonder why we have a NetworksListingForm and a NetworkListForm. [09:10] Looks like a leftover from parallel work producing the same form from two people, probably one as just a placeholder. [09:10] jtv, just did [09:12] Thanks. [09:13] I think this means that the exception at the very end of src/maasserver/models/network.py isn't quite right for what form validation wants. [09:35] dimitern: any chance we could get the full traceback of the exception? [09:35] Is that in the logs? [09:35] Ah, found it. [09:38] Wow, this could have been clearer. Yes, it's documented all over the place that validate_unique must raise ValidationError if it finds a clash, but not that that ValidationError must be one that was constructed from a dict, not an error message like in the documentation examples! [09:51] jtv: Django’s ValidationError is a horror-show. Reading its code is not only like seeing inside the sausage factory, but also seeing the animal’s skulls being caved in. [09:53] jtv: src/provisioningserver/tests/test_maas_import_pxe_files.py [10:00] Yes, that'd be the one. [10:13] dimitern, just for you. :) https://code.launchpad.net/~jtv/maas/bug-1299114/+merge/213623 [10:15] I get Internal Server Error when trying to connect the MAAS GUI in a fresh installation on 12.04 | http://askubuntu.com/q/441870 [10:25] jtv, great, thanks! [11:25] allenap, rvba, bigjools, I'd appreciate if someone can spare some time to review this gomaasapi CL https://codereview.appspot.com/82460044/ [11:26] dimitern: I’ll take a look. [11:26] allenap, ta! === fader is now known as fader_ === fader_ is now known as fader === cmagina-away is now known as cmagina === sputnik1_ is now known as sputnik13net === kevin is now known as Guest2886 === roadmr is now known as roadmr_afk === roadmr_afk is now known as roadmr === CyberJacob is now known as CyberJacob|Away [23:13] mwhudson: did you try arm with the latest maas in trusty at all? [23:14] or the faily ppa [23:14] (sic) [23:14] bigjools: not recently [23:14] i will be soon [23:14] mwhudson: ok. I need a 3rd party to verify it works on arm [23:14] once IS do something with networking [23:14] since there's a ton of changes in image imports [23:14] bigjools: timeframe? [23:15] mwhudson: now? :) [23:15] heh [23:15] there is an rt you can make noise on if you like... [23:15] bigjools: the hyperscale guys might have been doing stuff [23:16] ok [23:16] i'll be testing on midway [23:18] the more the better [23:18] thanks