[00:24] roaksoax: in case you are not really asleep yet, there's one more fix that needs to get in to maas, the WSGIImportScript change to fix the bson problem. It's possibly only a small packaging change for the Apache config. [00:24] might have to SRU it though [00:24] either way, it's urgent of course [01:44] bigjools: Any idea how I can figure out why MAAS is no longer serving TFTP? [01:44] pserv.log shows: TFTP Listener started at 172.30.171.5:69 [01:45] But it's not. [01:46] Oh, wait, might have fied it. [01:46] can you co.. [01:53] So, I chwoned pserv.log back to maas:mas. [01:53] ... [01:53] But it still can't do its thing: chinstrap.canonical.com/~jpds/maas.png [01:54] Interesting how it doesn't say that it's starting up... [01:55] restart it [01:55] it didnt shut down [01:56] If i stop it, I get another Received SIGTERM, shutting down. line. [01:56] start doesn't being anything up. [01:56] when you stop it, do you see the process actually go away? [01:56] Yep, it's no longer listening on port 69. [01:57] that means nothing, is the process still there? [01:57] No, no pserv. [01:57] ok [01:57] $ sudo service maas-pserv start [01:57] what user does it run as? [01:57] I have a process, port 69, user maas. [01:57] Nothing in logs. [01:58] is the log file +w? [01:58] Yep. [01:59] apparmor ? [01:59] having said that your last error was hours ago [01:59] so once it's up, can you use a cmd line tftp client? [02:00] looks like that part worked actually, from the bios screen [02:00] unless the dots mean it's waiting and not downloading [02:05] Interestinly, if I download from the cmd line, I actually get logs in pserv.log. [02:05] Sorry, suffering the _worst_ Internet connection I've seen here. [02:06] jpds: do you have negative bandwidth ? [02:06] davecheney: I wouldn't be surprised, took me 5 minutes to write that last line it kept dropping out. [02:07] tftp> get pxelinux.0 [02:07] Received 26837 bytes in 0.0 seconds [02:07] Works fine, VM on the same phyiscally host as the MAAS VM, doesn't work... :-/ [02:08] sounds like your VM network setup is wrong then [02:08] bigjools: They're both sitting on br0. [02:09] well something is wrong [02:17] bigjools: Can you see anything breaking if I "sudo -u postgres pg_dump maasdb" and import that into a new MAAS install? [02:18] yes [02:18] but if all the nodes are Ready then it's ok [02:18] I think [02:22] They're ready, I don't want to re-enlist everything though. [02:27] Does anybody know why bug 1232174 was reset from Fix Committed to Triaged? [02:27] bug 1232174 in MAAS "maas-import-pxe-files fails because Saucy's images information is not available." [Critical,Triaged] https://launchpad.net/bugs/1232174 [02:29] nfi [02:29] does it work? [02:29] btw our sampledata seems screwed [02:29] I thought it did, but... Wanted to try just now, but couldn't get a large canonistack instance. [02:30] you get "#### INVALID STRING ####" in the node listing for each node's name [02:30] I think that's hard-coded IDs in the sample data fixture. [02:30] clicking on them gets a crash [02:30] Yes. [02:31] Say the sample data creates a nodegroup, and the database assigns it ID 3. [02:31] Unfortunately the fixture assumes it gets ID 1. [02:31] I don't know how exactly that would work, given referential integrity, but ISTR that was how it happened. [02:53] bigjools: yep that's on my radar [02:53] roaksoax: cool, SRU? [02:53] bigjools: i think we might be able to get in critical fixes [02:54] even more better [03:00] bigjools: what other critical fixes we have? [03:01] roaksoax: that's the only one I know of [03:01] I'm doing some doc changes now but that won't affect the release [03:01] bigjools: this aaffects maas-import-ephemerals, and I';m filking a new one [03:01] https://bugs.launchpad.net/simplestreams/+bug/1237990 [03:01] Ubuntu bug 1237990 in simplestreams "FileStore checksumming will always fail on resumed download" [Undecided,Fix committed] [03:02] bigjools: i think we should unify maas-import-pxe-files and maas-import-ephemerals config to be just one [03:03] roaksoax: +1 [03:03] it will happen when we convert mipf [03:07] bigjools: https://bugs.launchpad.net/maas/+bug/1238376 [03:07] Ubuntu bug 1238376 in MAAS "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Undecided,New] [03:08] bigjools: is the bug clear or am I too tired? [03:08] reading [03:10] roaksoax: hmmm, jtv any comments ^ [03:11] * jtv reads [03:13] If you still have the shell-style config for maas-import-ephemerals, it includes the maas-import-pxe-files by default. [03:14] So AFAICS the bug only applies to a fresh install. [03:14] And there, the behaviour seems reasonable per se — as long as we can make the configuration clear to the user. [03:15] jtv: might also affect in upgrades I think I had issues with upgrades as well [03:15] that's how I think I discovered it first [03:15] Let me just check the old config file... [03:15] i can't really remember though [03:15] jtv: but either way, on a fresh install, if before we used to modify 1 file, now we need to modify 2 [03:15] which can become painful [03:15] The old /etc/maas/import_ephemerals config started with: [03:16] [ ! -f /etc/maas/import_pxe_files ] || . /etc/maas/import_pxe_files [03:16] jtv: right but that upgrade should remove import_pxe_files [03:16] err [03:16] import_ephemerals [03:16] It moves it, but after reading the variables it sets. [03:16] technically speaking, and due to packaging standards, it shold be removfed on upgrade IIRC [03:16] So the resulting config _should_ include any settings from /etc/maas/import_pxe_files [03:17] right [03:17] jtv: so I've seen upgrades where it generated a new pserv.yaml [03:17] while I've seen others that didn't [03:17] i think it requires more corner case escenarios [03:17] but the thing is that should not happen in fresh installs either [03:17] because we are supposed to only run maas-import-pxe-files which also runs maas-import-ephemerals [03:18] What exactly should not happen in fresh installs? [03:18] If you've seen a fresh install rewrite pserv.yaml, that's a bug. [03:18] jtv: not fresh, an upgrade [03:19] jtv: in fresh install.. because we run maas-import-pxe-files, which runs maas-import-pepehemrals, then it should inhering [03:19] whatever config for release is in /etc/maas/import_pxe_files [03:19] It doesn't. I think we should update the documentation for this. [03:20] jtv: right, it doesn't iunherit, so that *is* a bug [03:20] jtv: that's a "regression" [03:20] from how it used to work [03:20] Or "change in behaviour" as we software engineers like to justify ourselves. :) [03:20] a regression because maas-import-pxe-files *does* run maas-import-ephemerals [03:20] lol [03:21] jtv: yeah unfortantely users say otherwise and I deal with that :) [03:21] but anytways [03:21] simply because maas-import-pxe-files runs maas-import-ephemerals, then it is a regression and should be fixed [03:21] jtv: that's why i suggested bigjools to somehow unify the config between maas-import-pxe-files and maas-import-ephemerals [03:21] but either way that's an standing bug [03:22] jtv: there's users been hitting that issue already (internal and in important projects that caused blockers) [03:22] That's fast. [03:22] We have been unifying the config, but this is a transitional issue. [03:22] so I'm happy it wasn't just me [03:23] jtv: yeah, transitional should mean backwards compatible [03:23] and fresh installs are not [03:23] and might affect upgrades [03:23] i;m not 100% sure it doesn't cause I found the issues on an upgrade [03:23] It shouldn't be too hard to preserve the old behaviour — it's just a bit ugly to keep running one shell config but moving the other into the unified config. [03:23] yeah [03:23] anyway [03:23] i'm off to bed [03:24] have a good night [03:24] It's important to know when you hit those issues. [03:24] Good night... dream of MAAS. :) [03:24] ¡Buenas! [03:24] nn roaksoax [04:09] https://code.launchpad.net/~julian-edwards/maas/more-docs/+merge/190525 but no rush I am going to eat [04:34] Any way I can force a rewrite of dhcpd.conf ? [04:37] I tried editing eth0 on the web UI, but it's not writing dhcpd.conf for some reasons. [04:37] reason* [04:58] Did it by hand, and got MAAS to work. [05:59] jpds: edit the cluster IIRC [06:01] bigjools: Yeah, I tried that, it doesn't write the file. [06:02] Tried removing the interface too. [06:02] And readding it. [06:04] why did you need to rewrite it? [06:04] bigjools: I didn't have one, this is a fresh MAAS install. [06:04] sounds like your cluster's celeryd is not workig [06:05] maas-cluster-celery start/running, process 599 - hmm. [06:05] or perhaps dhcp not set to managed? [06:05] Ah, /var/log/maas/celery.log as HTTPErrors. [06:06] or the cluster is not accepted [06:07] INFO/PoolWorker-4] The DHCP leases file does not exist. This is only a problem if this cluster controller is managing its DHCP server. If that's the case then you need to install the 'maas-dhcp' package on this cluster controller. [06:07] if editing the cluster doesn't write it one of those conditions is not met [06:07] It is installed, so I guess the file was just missing to begin with. [06:09] that error is orthogonal to this [06:12] Aha, i seem to have a pending cluster. [06:18] jtv: we don't have any loud announcement of pending clusters do we? [06:18] No, they're just listed on the settings page. === CyberJacob|Away is now known as CyberJacob [06:37] jtv: " If multiple tags attached to a node have the kernel_opts defined, the first one ordered by name is used." [06:37] ok? [06:39] bigjools: yes, thanks [06:39] cheers [07:21] hi all [07:21] i have installed maas [07:21] but facing issues with dhcp [07:22] when i am trying to get ip for machine through dhcp [07:22] i am geting the ip from the range i have configured in /etc/dhcp/dhcpd.conf [07:22] but when i am trying the same conf through maas [07:23] it is not able to fetch the ip [07:23] please help me in configuring dhcp with maas correctly [07:24] http://maas.ubuntu.com/docs/install.html#configure-dhcp followed the steps in this link === CyberJacob is now known as CyberJacob|Away [10:11] bigjools: I think DOWNLOAD in masa-import-pxe-files should have a -c by default. [11:45] Hi smoser, I'm getting this http://paste.ubuntu.com/6221791/ when running the import script… I assume this is a transient issue… ? [12:49] rvba, smoser's not around today [12:49] rvba, I'd hoped that commits to published simplestream data might be atomic - does it resolve if you re-run straight away? [12:50] jamespage: I ran the script twice… but maybe this is due to the caching proxy we use in the lab… I'm investigating… [12:51] rvba, that could certainly created some issues depending on config [14:17] Hello everyone! [14:18] I tryied to deploy the node, but after deployment it returns: 2013-10-11 07:15:54 DEBUG juju.state open.go:88 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused [14:18] I guess there a problem in different version on the mongodb [14:20] and I guess I should add the stable-juju repository to this file: /etc/apt/sources.list before installation, but I don't know how [15:00] nesusvet: sounds like something the Juju folks would be better able to diagnose. It doesn't seem to be anything in MAAS per se. [15:00] thanks [15:11] rvba: just hit that checksum error too [15:11] On a "precise" download this time. [15:12] But also 20131010, strangely enough. [15:15] roaksoax: are you here? If so, could you review a packaging change for me? It's https://code.launchpad.net/~jtv/maas/pkg-bug-1238376/+merge/190658 [15:17] jtv: branch needs fixing [15:17] huge diff :) [15:17] Uh-oh [15:17] I bet I proposed for the wrong branch! [15:18] roaksoax: hoping this one'll work out better... https://code.launchpad.net/~jtv/maas/pkg-bug-1238376/+merge/190694 [15:18] Definitely a smaller diff... [15:19] roaksoax: I also landed another critical fix in r1702, so maybe I should just bump the revision number to that... [15:19] Thanks for the review. [15:28] yep [15:29] jtv: if you guys can make a list of critical bug fixes would be great [15:31] roaksoax: there's no "us guys" at the moment I'm afraid... I think the rest of the squad's gone home, and I'm about to do the same. [15:32] I know there's bug 1238376 and 1238681. [15:32] bug 1238681 in MAAS "Legacy import config treats arches/releases as lists of characters" [Critical,Fix committed] https://launchpad.net/bugs/1238681 [15:32] bug 1238376 in MAAS "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Critical,Fix committed] https://launchpad.net/bugs/1238376 [15:33] jtv: next week is fine [15:33] over email better ;) [15:34] roaksoax: this listing should give you a pretty good idea, on an ongoing basis: https://bugs.launchpad.net/maas/+bugs?field.searchtext=&orderby=-date_last_updated&search=Search&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.importance%3Alist=CRITICAL&field.importance%3Alist=HIGH&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&fie [15:34] ld.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on [15:35] Hum. Pretty long URL. [15:35] Call it http://bit.ly/1bLZzBX for short. [15:36] roaksoax: that's critical bugs, fix committed & fix released, sorted from most recently changed to less recently changed. [15:40] roaksoax, have you seen this before: http://pastebin.ubuntu.com/6222957/? [15:40] roaksoax, triggered during a test run for the saucy daily package [15:42] jtv: awesome! thanks yoiu [15:42] matsubara: looking [15:42] matsubara: yeah, it is pretty self explanatory [15:42] matsubara: do you have the images for the release? [15:42] roaksoax, well, it did run maas-import-pxe-files [15:42] matsubara: make sure the directory and images exist [15:43] so I guess maas-import-pxe-files is failing quietly [15:43] matsubara: probably is, or maas-import-ephemerals [15:45] roaksoax, there's a /var/lib/maas/ephemeral/precise/ephemeral/i386 dir but there's not a amd64 dir [15:45] matsubara: so you only imported i386 images [15:45] you need both, amd64 and i386 [15:47] roaksoax, the test ran maas-import-pxe-files, that should be enough to get both right? [15:47] I'm investigating why it didn't, just wondered if you had seen it before [15:47] matsubara: by default it should get both yes [15:48] matsubara: but if you modified the config to only get i386 then it would fail that way [16:30] smoser: can i mark this bug as fix released? 1218530 [16:30] bug #1218530 [16:30] bug 1218530 in maas (Ubuntu) "FFE 13.10: curtin new development" [Undecided,Triaged] https://launchpad.net/bugs/1218530 === CyberJacob|Away is now known as CyberJacob [18:52] roaksoax, ping [18:52] adam_g: pong [18:53] roaksoax, do you have any curtin'd insatlled 12.04 machines up atm? [18:53] adam_g: nope [18:55] roaksoax, ah, k [18:56] adam_g: btw i have a fix for the upmi thing but havent tested yet [18:56] roaksoax, oh right i saw you pinged me [18:56] roaksoax, unfortuantely i wont be able to test today [18:56] adam_g: no worries ;) [18:57] === kentb-out is now known as kentb [19:13] roaksoax, I'm still getting the checksum failure running maas-import-pxe-files while using a proxy (after I cleared the cache). rvba mentioned he tried on a canonistack instance and the script completed successfully. Could be a bug of the script interacting with a proxy? Or perhaps I'm not cleaning the cache properly (although I do see lots of TCP_MISS in the squid logs, so it indicates the cache has been cleared fine) [19:13] uhmmm [19:13] matsubara: remove squid cache with rm -rf [19:14] and restart squid and retry? [19:14] that's what I tried [19:14] I stopped squid, removed the cache_dir and restarted [19:14] but got the error again [19:14] but I'm seeing in the squid log: 11/Oct/2013:15:10:16 -0400.124 0 10.98.0.90 TCP_MEM_HIT/200 2221 GET http://maas.ubuntu.com/images/ephemeral/releases/streams/v1/index.sjson - NONE/- text/plain [19:14] and 11/Oct/2013:15:10:16 -0400.387 163 10.98.0.90 TCP_REFRESH_UNMODIFIED/200 13497 GET http://maas.ubuntu.com/images/ephemeral/releases/streams/v1/com.ubuntu.maas:download.sjson - DIRECT/91.189.90.19 text/plain [19:15] so, maybe it's using some other cache I'm not aware of. squid's memory cache? sudo service squid3 stop should clear the cache in memory, no? [19:15] matsubara: check in /var/lib/cache/squid.... [19:15] or somewhere around there for the actual squid cache [19:16] -bash: cd: /var/lib/cache: No such file or directory [19:16] matsubara: restart squi-deb-proxy [19:16] yeah, the config is pointing to /var/spool/squid3 and that's the one I removed [19:17] squid deb proxy too? this is just regular squid, not the deb-proxy one [19:17] I mean, the maas-import-pxe-files uses the squid cache not the s-d-p one, AFAICT [19:17] s-d-p is used for the archive stuff [19:17] (I'll try restarting s-d-p in anyway) [19:17] matsubara: we dont ship anyconfig for squid3 [19:18] only for squid-deb-proxy [19:18] roaksoax, that's in the QA lab, it's not part of maas, we use a cache there to speed up maas-import-pxe-files [19:19] ok so if the logs say theres a hit on the cache then probably the cache was not cleaned [19:19] i dunno where rlse would it be [19:19] but thats weird [19:20] tzzzzt [19:42] roaksoax, do you know much about curtin in terms of its preseed data/cloud-config? [19:42] i know smoser is out [19:43] adam_g: no unfortunately [19:44] what do you need to do? [19:44] roaksoax, do you know if the /etc/maas/preseeds/curtin_userdata is userdata that is executed during provisioning? [19:44] trying to work around https://bugs.launchpad.net/bugs/1238915 by adding a curtin hook to just remove that file after install / before first boot [19:44] Ubuntu bug 1238915 in curtin (Ubuntu) "Keystone fails to start after installation: invoke-rc.d: policy-rc.d denied execution of start." [Undecided,New] [19:45] adding a final_command seems like it shoudl work [19:45] adam_g: yeah it is [19:45] so thats where you eould put lay [19:45] te_commands [19:45] err [19:45] late_commands [19:46] roaksoax, http://paste.ubuntu.com/6223940/ [19:46] roaksoax, late_commands is a DI thing right? that doc makes it look like curtin's final_command is the same thing [19:47] adam_g: correct [19:47] roaksoax, ill add that to curtin_userdata and see if it works [19:50] it should [19:51] i think stokachu did it for vlan config [21:37] Quick Question: Is there a way to set a password for commissioned nodes? [21:39] I need to access the node locally because networking doesn't come up properly after reboot [21:48] Not sure if I understand the question, but ubuntu user gets the ssh key placed in their authorized keys [21:49] No? [21:49] I understand, but I can't ssh [21:49] I have to log in locally [21:49] Nik_: hence the password [21:49] Oh i see... [21:49] If you are local, you can go into maintenance mode [21:49] and set any password [21:50] You'll get root access [21:50] maintenance mode? [21:50] Umm.. that kernel boot option [21:50] ah [21:50] in grub? [21:50] Tim, this might help: https://lists.launchpad.net/maas-devel/msg00808.html [21:51] yeah something like that [21:51] It's called single-user mode or maintennace mode [21:51] You get a shell prompt and no drives mounted [21:51] maybe this can help: http://www.cyberciti.biz/faq/linux-reset-forgotten-root-password/ [21:51] matsubara: Interesting, thanks for the link [21:51] Nik_: I'll try that as well [21:51] You got juju working or just maas? [21:52] np [21:52] Nik_: I'm tryinig to get juju working, but since networking does come up on my node after reboot juju deploy times out [21:52] *does not [21:52] Its a problem with out 10gb switch [21:53] if anyone cares [21:53] I see. Well hopefully I understand how maas works and how things happen with PXE enough for this to fork for you :) [21:54] And I didn't check mat's link. That may help too [21:54] Nik_: Lol. I wouldn't worry about, I've got the pxe covered. It sounds like I might just be able to manually set the password using your suggestion [21:54] and if not, modify the preseed file via matsubara's suggestion [21:55] It took long enough to get MAAS and juju working [21:55] it would really suck if this was what killed it [21:56] MAAS Set password for node | http://askubuntu.com/q/356936 [21:56] Yes, that was me [21:56] I asked that [22:03] Nik_: Damn, skips the GRUB screen during boot [22:12] You can try hold shift I think [22:13] though that may be a no-go on ubuntu... [22:13] http://askubuntu.com/questions/71867/grub-menu-doesnt-appear-when-pressing-shift [22:14] :( [22:20] Yeah, tried that [22:20] either it wasn't passed via KVM, or it doesn't work === kentb is now known as kentb-out === CyberJacob is now known as CyberJacob|Away [22:47] Nik_: Thanks! that worked! [23:06] Tim_: Glad to hear! Anyways, I'm off for today. Hopefully you get your all solved :)