/srv/irclogs.ubuntu.com/2013/10/11/#maas.txt

bigjoolsroaksoax: in case you are not really asleep yet, there's one more fix that needs to get in to maas, the WSGIImportScript change to fix the bson problem.  It's possibly only a small packaging change for the Apache config.00:24
bigjoolsmight have to SRU it though00:24
bigjoolseither way, it's urgent of course00:24
jpdsbigjools: Any idea how I can figure out why MAAS is no longer serving TFTP?01:44
jpdspserv.log shows: TFTP Listener started at 172.30.171.5:6901:44
jpdsBut it's not.01:45
jpdsOh, wait, might have fied it.01:46
bigjoolscan you co..01:46
jpdsSo, I chwoned pserv.log back to maas:mas.01:53
bigjools...01:53
jpdsBut it still can't do its thing: chinstrap.canonical.com/~jpds/maas.png01:53
jpdsInteresting how it doesn't say that it's starting up...01:54
bigjoolsrestart it01:55
bigjoolsit didnt shut down01:55
jpdsIf i stop it, I get another  Received SIGTERM, shutting down. line.01:56
jpdsstart doesn't being anything up.01:56
bigjoolswhen you stop it, do you see the process actually go away?01:56
jpdsYep, it's no longer listening on port 69.01:56
bigjoolsthat means nothing, is the process still there?01:57
jpdsNo, no pserv.01:57
bigjoolsok01:57
jpds$ sudo service maas-pserv start01:57
bigjoolswhat user does it run as?01:57
jpdsI have a process, port 69, user maas.01:57
jpdsNothing in logs.01:57
bigjoolsis the log file +w?01:58
jpdsYep.01:58
bigjoolsapparmor ?01:59
bigjoolshaving said that your last error was hours ago01:59
bigjoolsso once it's up, can you use a cmd line tftp client?01:59
bigjoolslooks like that part worked actually, from the bios screen02:00
bigjoolsunless the dots mean it's waiting and not downloading02:00
jpdsInterestinly, if I download from the cmd line, I actually get logs in pserv.log.02:05
jpdsSorry, suffering the _worst_ Internet connection I've seen here.02:05
davecheneyjpds: do you have negative bandwidth ?02:06
jpdsdavecheney: I wouldn't be surprised, took me 5 minutes to write that last line it kept dropping out.02:06
jpdstftp> get pxelinux.002:07
jpdsReceived 26837 bytes in 0.0 seconds02:07
jpdsWorks fine, VM on the same phyiscally host as the MAAS VM, doesn't work... :-/02:07
bigjoolssounds like your VM network setup is wrong then02:08
jpdsbigjools: They're both sitting on br0.02:08
bigjoolswell something is wrong02:09
jpdsbigjools: Can you see anything breaking if I "sudo -u postgres pg_dump maasdb" and import that into a new MAAS install?02:17
bigjoolsyes02:18
bigjoolsbut if all the nodes are Ready then it's ok02:18
bigjoolsI  think02:18
jpdsThey're ready, I don't want to re-enlist everything though.02:22
jtvDoes anybody know why bug 1232174 was reset from Fix Committed to Triaged?02:27
ubot5bug 1232174 in MAAS "maas-import-pxe-files fails because Saucy's images information is not available." [Critical,Triaged] https://launchpad.net/bugs/123217402:27
bigjoolsnfi02:29
bigjoolsdoes it work?02:29
bigjoolsbtw our sampledata seems screwed02:29
jtvI thought it did, but...  Wanted to try just now, but couldn't get a large canonistack instance.02:29
bigjoolsyou get "#### INVALID STRING ####" in the node listing for each node's name02:30
jtvI think that's hard-coded IDs in the sample data fixture.02:30
bigjoolsclicking on them gets a crash02:30
jtvYes.02:30
jtvSay the sample data creates a nodegroup, and the database assigns it ID 3.02:31
jtvUnfortunately the fixture assumes it gets ID 1.02:31
jtvI don't know how exactly that would work, given referential integrity, but ISTR that was how it happened.02:31
roaksoaxbigjools: yep that's on my radar02:53
bigjoolsroaksoax: cool, SRU?02:53
roaksoaxbigjools: i think we might be able to get in critical fixes02:53
bigjoolseven more better02:54
roaksoaxbigjools: what other critical fixes we have?03:00
bigjoolsroaksoax: that's the only one I know of03:01
bigjoolsI'm doing some doc changes now but that won't affect the release03:01
roaksoaxbigjools: this aaffects maas-import-ephemerals, and I';m filking a new one03:01
roaksoaxhttps://bugs.launchpad.net/simplestreams/+bug/123799003:01
ubot5Ubuntu bug 1237990 in simplestreams "FileStore checksumming will always fail on resumed download" [Undecided,Fix committed]03:01
roaksoaxbigjools: i think we should unify maas-import-pxe-files and maas-import-ephemerals config to be just one03:02
bigjoolsroaksoax: +103:03
bigjoolsit will happen when we convert mipf03:03
roaksoaxbigjools: https://bugs.launchpad.net/maas/+bug/123837603:07
ubot5Ubuntu bug 1238376 in MAAS "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Undecided,New]03:07
roaksoaxbigjools: is the bug clear or am I too tired?03:08
bigjoolsreading03:08
bigjoolsroaksoax: hmmm,  jtv any comments ^03:10
* jtv reads03:11
jtvIf you still have the shell-style config for maas-import-ephemerals, it includes the maas-import-pxe-files by default.03:13
jtvSo AFAICS the bug only applies to a fresh install.03:14
jtvAnd there, the behaviour seems reasonable per se — as long as we can make the configuration clear to the user.03:14
roaksoaxjtv: might also affect in upgrades I think I had issues with upgrades as well03:15
roaksoaxthat's how I think I discovered it first03:15
jtvLet me just check the old config file...03:15
roaksoaxi can't really remember though03:15
roaksoaxjtv: but either way, on a fresh install, if before we used to modify 1 file, now we need to modify 203:15
roaksoaxwhich can become painful03:15
jtvThe old /etc/maas/import_ephemerals config started with:03:15
jtv[ ! -f /etc/maas/import_pxe_files ] || . /etc/maas/import_pxe_files03:16
roaksoaxjtv: right but that upgrade should remove import_pxe_files03:16
roaksoaxerr03:16
roaksoaximport_ephemerals03:16
jtvIt moves it, but after reading the variables it sets.03:16
roaksoaxtechnically speaking, and due to packaging standards, it shold be removfed on upgrade IIRC03:16
jtvSo the resulting config _should_ include any settings from /etc/maas/import_pxe_files03:16
roaksoaxright03:17
roaksoaxjtv: so I've seen upgrades where it generated a new pserv.yaml03:17
roaksoaxwhile I've seen others that didn't03:17
roaksoaxi think it requires more corner case escenarios03:17
roaksoaxbut the thing is that should not happen in fresh installs either03:17
roaksoaxbecause we are supposed to only run maas-import-pxe-files which also runs maas-import-ephemerals03:17
jtvWhat exactly should not happen in fresh installs?03:18
jtvIf you've seen a fresh install rewrite pserv.yaml, that's a bug.03:18
roaksoaxjtv: not  fresh, an upgrade03:18
roaksoaxjtv: in fresh install.. because we run maas-import-pxe-files, which runs maas-import-pepehemrals, then it should inhering03:19
roaksoaxwhatever config for release is in /etc/maas/import_pxe_files03:19
jtvIt doesn't.  I think we should update the documentation for this.03:19
roaksoaxjtv: right, it doesn't iunherit, so that *is* a bug03:20
roaksoaxjtv: that's a "regression"03:20
roaksoaxfrom how it used to work03:20
jtvOr "change in behaviour" as we software engineers like to justify ourselves.  :)03:20
roaksoaxa regression because maas-import-pxe-files *does* run maas-import-ephemerals03:20
roaksoaxlol03:20
roaksoaxjtv: yeah unfortantely users say otherwise and I deal with that :)03:21
roaksoaxbut anytways03:21
roaksoaxsimply because maas-import-pxe-files runs maas-import-ephemerals, then it is a regression and should be fixed03:21
roaksoaxjtv: that's why i suggested bigjools to somehow unify the config between maas-import-pxe-files and maas-import-ephemerals03:21
roaksoaxbut either way that's an standing bug03:21
roaksoaxjtv: there's users been hitting that issue already (internal and in important projects that caused blockers)03:22
jtvThat's fast.03:22
jtvWe have been unifying the config, but this is a transitional issue.03:22
roaksoaxso I'm happy it wasn't just me03:22
roaksoaxjtv: yeah, transitional should mean backwards compatible03:23
roaksoaxand fresh installs are not03:23
roaksoaxand might affect upgrades03:23
roaksoaxi;m not 100% sure it doesn't cause I found the issues on an upgrade03:23
jtvIt shouldn't be too hard to preserve the old behaviour — it's just a bit ugly to keep running one shell config but moving the other into the unified config.03:23
roaksoaxyeah03:23
roaksoaxanyway03:23
roaksoaxi'm off to bed03:23
roaksoaxhave a good night03:24
jtvIt's important to know when you hit those issues.03:24
jtvGood night... dream of MAAS.  :)03:24
jtv¡Buenas!03:24
bigjoolsnn roaksoax03:24
bigjoolshttps://code.launchpad.net/~julian-edwards/maas/more-docs/+merge/190525  but no rush I am going to eat04:09
jpdsAny way I can force a rewrite of dhcpd.conf ?04:34
jpdsI tried editing eth0 on the web UI, but it's not writing dhcpd.conf for some reasons.04:37
jpdsreason*04:37
jpdsDid it by hand, and got MAAS to work.04:58
bigjoolsjpds: edit the cluster IIRC05:59
jpdsbigjools: Yeah, I tried that, it doesn't write the file.06:01
jpdsTried removing the interface too.06:02
jpdsAnd readding it.06:02
bigjoolswhy did you need to rewrite it?06:04
jpdsbigjools: I didn't have one, this is a fresh MAAS install.06:04
bigjoolssounds like your cluster's celeryd is not workig06:04
jpdsmaas-cluster-celery start/running, process 599 - hmm.06:05
bigjoolsor perhaps dhcp not set to managed?06:05
jpdsAh, /var/log/maas/celery.log as HTTPErrors.06:05
bigjoolsor the cluster is not accepted06:06
jpdsINFO/PoolWorker-4] The DHCP leases file does not exist.  This is only a problem if this cluster controller is managing its DHCP server.  If that's the case then you need to install the 'maas-dhcp' package on this cluster controller.06:07
bigjoolsif editing the cluster doesn't write it one of those conditions is not met06:07
jpdsIt is installed, so I guess the file was just missing to begin with.06:07
bigjoolsthat error is orthogonal to this06:09
jpdsAha, i seem to have a pending cluster.06:12
bigjoolsjtv: we don't have any loud announcement of pending clusters do we?06:18
jtvNo, they're just listed on the settings page.06:18
=== CyberJacob|Away is now known as CyberJacob
bigjoolsjtv: "  If multiple tags attached to a node have the kernel_opts defined, the first one ordered by name is used."06:37
bigjoolsok?06:37
jtvbigjools: yes, thanks06:39
bigjoolscheers06:39
kampyhi all07:21
kampyi have installed maas07:21
kampybut facing issues with dhcp07:21
kampywhen i am trying to get ip for  machine through dhcp07:22
kampyi am geting the ip from the range i have configured in /etc/dhcp/dhcpd.conf07:22
kampybut when i am trying the same conf through maas07:22
kampyit is not able to fetch the ip07:23
kampyplease help me in configuring dhcp with maas correctly07:23
kampyhttp://maas.ubuntu.com/docs/install.html#configure-dhcp followed the steps in this link07:24
=== CyberJacob is now known as CyberJacob|Away
jpdsbigjools: I think DOWNLOAD in masa-import-pxe-files should have a -c by default.10:11
rvbaHi smoser, I'm getting this http://paste.ubuntu.com/6221791/ when running the import script… I assume this is a transient issue… ?11:45
jamespagervba, smoser's not around today12:49
jamespagervba, I'd hoped that commits to published simplestream data might be atomic - does it resolve if you re-run straight away?12:49
rvbajamespage: I ran the script twice… but maybe this is due to the caching proxy we use in the lab… I'm investigating…12:50
jamespagervba, that could certainly created some issues depending on config12:51
nesusvetHello everyone!14:17
nesusvetI tryied to deploy the node, but after deployment it returns: 2013-10-11 07:15:54 DEBUG juju.state open.go:88 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused14:18
nesusvetI guess there a problem in different version on the mongodb14:18
nesusvetand I guess I should add the stable-juju repository to this file: /etc/apt/sources.list before installation, but I don't know how14:20
jtvnesusvet: sounds like something the Juju folks would be better able to diagnose.  It doesn't seem to be anything in MAAS per se.15:00
nesusvetthanks15:00
jtvrvba: just hit that checksum error too15:11
jtvOn a "precise" download this time.15:11
jtvBut also 20131010, strangely enough.15:12
jtvroaksoax: are you here?  If so, could you review a packaging change for me?  It's https://code.launchpad.net/~jtv/maas/pkg-bug-1238376/+merge/19065815:15
roaksoaxjtv: branch needs fixing15:17
roaksoaxhuge diff :)15:17
jtvUh-oh15:17
jtvI bet I proposed for the wrong branch!15:17
jtvroaksoax: hoping this one'll work out better... https://code.launchpad.net/~jtv/maas/pkg-bug-1238376/+merge/19069415:18
jtvDefinitely a smaller diff...15:18
jtvroaksoax: I also landed another critical fix in r1702, so maybe I should just bump the revision number to that...15:19
jtvThanks for the review.15:19
roaksoaxyep15:28
roaksoaxjtv: if you guys can make a list of critical bug fixes would be great15:29
jtvroaksoax: there's no "us guys" at the moment I'm afraid...  I think the rest of the squad's gone home, and I'm about to do the same.15:31
jtvI know there's bug 1238376 and 1238681.15:32
ubot5bug 1238681 in MAAS "Legacy import config treats arches/releases as lists of characters" [Critical,Fix committed] https://launchpad.net/bugs/123868115:32
ubot5bug 1238376 in MAAS "maas-import-ephemerals no longer inherits config from maas-import-pxe-files" [Critical,Fix committed] https://launchpad.net/bugs/123837615:32
roaksoaxjtv: next week is fine15:33
roaksoaxover email better ;)15:33
jtvroaksoax: this listing should give you a pretty good idea, on an ongoing basis: https://bugs.launchpad.net/maas/+bugs?field.searchtext=&orderby=-date_last_updated&search=Search&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.importance%3Alist=CRITICAL&field.importance%3Alist=HIGH&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&fie15:34
jtvld.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on15:34
jtvHum.  Pretty long URL.15:35
jtvCall it http://bit.ly/1bLZzBX for short.15:35
jtvroaksoax: that's critical bugs, fix committed & fix released, sorted from most recently changed to less recently changed.15:36
matsubararoaksoax, have you seen this before: http://pastebin.ubuntu.com/6222957/?15:40
matsubararoaksoax, triggered during a test run for the saucy daily package15:40
roaksoaxjtv: awesome! thanks yoiu15:42
roaksoaxmatsubara: looking15:42
roaksoaxmatsubara: yeah, it is pretty self explanatory15:42
roaksoaxmatsubara: do you have the images for the release?15:42
matsubararoaksoax, well, it did run maas-import-pxe-files15:42
roaksoaxmatsubara: make sure the directory and images exist15:42
matsubaraso I guess maas-import-pxe-files is failing quietly15:43
roaksoaxmatsubara: probably is, or maas-import-ephemerals15:43
matsubararoaksoax, there's a /var/lib/maas/ephemeral/precise/ephemeral/i386 dir but there's not a amd64 dir15:45
roaksoaxmatsubara: so you only imported i386 images15:45
roaksoaxyou need both, amd64 and i38615:45
matsubararoaksoax, the test ran maas-import-pxe-files, that should be enough to get both right?15:47
matsubaraI'm investigating why it didn't, just wondered if you had seen it before15:47
roaksoaxmatsubara: by default it should get both yes15:47
roaksoaxmatsubara: but if you modified the config to only get i386 then it would fail that way15:48
roaksoaxsmoser: can i mark this bug as fix released? 121853016:30
roaksoaxbug #121853016:30
ubot5bug 1218530 in maas (Ubuntu) "FFE 13.10: curtin new development" [Undecided,Triaged] https://launchpad.net/bugs/121853016:30
=== CyberJacob|Away is now known as CyberJacob
adam_groaksoax, ping18:52
roaksoaxadam_g: pong18:52
adam_groaksoax, do you have any curtin'd insatlled 12.04 machines up atm?18:53
roaksoaxadam_g: nope18:53
adam_groaksoax, ah, k18:55
roaksoaxadam_g: btw i have a fix for the upmi thing but havent tested yet18:56
adam_groaksoax, oh right i saw you pinged me18:56
adam_groaksoax, unfortuantely i wont be able to test today18:56
roaksoaxadam_g: no worries ;)18:56
roaksoax 18:57
=== kentb-out is now known as kentb
matsubararoaksoax, I'm still getting the checksum failure running maas-import-pxe-files while using a proxy (after I cleared the cache). rvba mentioned he tried on a canonistack instance and the script completed successfully. Could be a bug of the script interacting with a proxy? Or perhaps I'm not cleaning the cache properly (although I do see lots of TCP_MISS in the squid logs, so it indicates the cache has been cleared fine)19:13
roaksoaxuhmmm19:13
roaksoaxmatsubara: remove squid cache with rm -rf19:13
roaksoaxand restart squid and retry?19:14
matsubarathat's what I tried19:14
matsubaraI stopped squid, removed the cache_dir and restarted19:14
matsubarabut got the error again19:14
matsubarabut I'm seeing in the squid log: 11/Oct/2013:15:10:16 -0400.124      0 10.98.0.90 TCP_MEM_HIT/200 2221 GET http://maas.ubuntu.com/images/ephemeral/releases/streams/v1/index.sjson - NONE/- text/plain19:14
matsubaraand 11/Oct/2013:15:10:16 -0400.387    163 10.98.0.90 TCP_REFRESH_UNMODIFIED/200 13497 GET http://maas.ubuntu.com/images/ephemeral/releases/streams/v1/com.ubuntu.maas:download.sjson - DIRECT/91.189.90.19 text/plain19:14
matsubaraso, maybe it's using some other cache I'm not aware of. squid's memory cache? sudo service squid3 stop should clear the cache in memory, no?19:15
roaksoaxmatsubara: check in /var/lib/cache/squid....19:15
roaksoaxor somewhere around there for the actual squid cache19:15
matsubara-bash: cd: /var/lib/cache: No such file or directory19:16
roaksoaxmatsubara: restart squi-deb-proxy19:16
matsubarayeah, the config is pointing to /var/spool/squid3 and that's the one I removed19:16
matsubarasquid deb proxy too? this is just regular squid, not the deb-proxy one19:17
matsubaraI mean, the maas-import-pxe-files uses the squid cache not the s-d-p one, AFAICT19:17
matsubaras-d-p is used for the archive stuff19:17
matsubara(I'll try restarting s-d-p in anyway)19:17
roaksoaxmatsubara: we dont ship anyconfig for squid319:17
roaksoaxonly for squid-deb-proxy19:18
matsubararoaksoax, that's in the QA lab, it's not part of maas, we use a cache there to speed up maas-import-pxe-files19:18
roaksoaxok so if the logs say theres a hit on the cache then probably the cache was not cleaned19:19
roaksoaxi dunno where rlse would it be19:19
roaksoaxbut thats weird19:19
roaksoaxtzzzzt19:20
adam_groaksoax, do you know much about curtin in terms of its preseed data/cloud-config?19:42
adam_gi know smoser is out19:42
roaksoaxadam_g: no unfortunately19:43
roaksoaxwhat do you need to do?19:44
adam_groaksoax, do you know if the /etc/maas/preseeds/curtin_userdata is userdata that is executed during provisioning?19:44
adam_gtrying to work around https://bugs.launchpad.net/bugs/1238915 by adding a curtin hook to just remove that file after install / before first boot19:44
ubot5Ubuntu bug 1238915 in curtin (Ubuntu) "Keystone fails to start after installation: invoke-rc.d: policy-rc.d denied execution of start." [Undecided,New]19:44
adam_gadding a final_command seems like it shoudl work19:45
roaksoaxadam_g: yeah it is19:45
roaksoaxso thats where you eould put lay19:45
roaksoaxte_commands19:45
roaksoaxerr19:45
roaksoaxlate_commands19:45
adam_groaksoax, http://paste.ubuntu.com/6223940/19:46
adam_groaksoax, late_commands is a DI thing right?  that doc makes it look like curtin's final_command is the same thing19:46
roaksoaxadam_g: correct19:47
adam_groaksoax, ill add that to curtin_userdata and see if it works19:47
roaksoaxit should19:50
roaksoaxi think stokachu did it for vlan config19:51
TimQuick Question: Is there a way to set a password for commissioned nodes?21:37
TimI need to access the node locally because networking doesn't come up properly after reboot21:39
Nik_Not sure if I understand the question, but ubuntu user gets the ssh key placed in their authorized keys21:48
TimNo?21:49
TimI understand, but I can't ssh21:49
TimI have to log in locally21:49
TimNik_: hence the password21:49
Nik_Oh i see...21:49
Nik_If you are local, you can go into maintenance mode21:49
Nik_and set any password21:49
Nik_You'll get root access21:50
Timmaintenance mode?21:50
Nik_Umm.. that kernel boot option21:50
Timah21:50
Timin grub?21:50
matsubaraTim, this might help: https://lists.launchpad.net/maas-devel/msg00808.html21:50
Nik_yeah something like that21:51
Nik_It's called single-user mode or maintennace mode21:51
Nik_You get a shell prompt and no drives mounted21:51
Nik_maybe this can help: http://www.cyberciti.biz/faq/linux-reset-forgotten-root-password/21:51
Timmatsubara: Interesting, thanks for the link21:51
TimNik_: I'll try that as well21:51
Nik_You got juju working or just maas?21:51
matsubaranp21:52
TimNik_: I'm tryinig to get juju working, but since networking does come up on my node after reboot juju deploy times out21:52
Tim*does not21:52
TimIts a problem with out 10gb switch21:52
Timif anyone cares21:53
Nik_I see. Well hopefully I understand how maas works and how things happen with PXE enough for this to fork for you :)21:53
Nik_And I didn't check mat's link. That may help too21:54
TimNik_: Lol. I wouldn't worry about, I've got the pxe covered. It sounds like I might just be able to manually set the password using your suggestion21:54
Timand if not, modify the preseed file via matsubara's suggestion21:54
TimIt took long enough to get MAAS and juju working21:55
Timit would really suck if this was what killed it21:55
AskUbuntuMAAS Set password for node | http://askubuntu.com/q/35693621:56
TimYes, that was me21:56
TimI asked that21:56
TimNik_: Damn, skips the GRUB screen during boot22:03
Nik_You can try hold shift I think22:12
Nik_though that may be  a no-go on ubuntu...22:13
Nik_http://askubuntu.com/questions/71867/grub-menu-doesnt-appear-when-pressing-shift22:13
Nik_:(22:14
Tim_Yeah, tried that22:20
Tim_either it wasn't passed via KVM, or it doesn't work22:20
=== kentb is now known as kentb-out
=== CyberJacob is now known as CyberJacob|Away
Tim_Nik_: Thanks! that worked!22:47
Nik_Tim_: Glad to hear! Anyways, I'm off for today. Hopefully you get your all solved :)23:06

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!