[00:36] davecheney: this is for local [00:39] davecheney: they can connect as machine-0 using the password from .juju agents.conf, but nothing is working for admin. [00:42] davecheney: I will have them try that when they are back online tomorrow though .. using that admin-secret from local.jenv [00:42] thanks :) [00:47] why is the local provider so weird ?!? [00:47] it has too many edge cases [01:29] yeah [03:19] no axw today ? [05:04] I might have accidentally use the word "shittle" when doing my expenses [05:04] oh well, i'm sure they know what I mean === uru_ is now known as urulama === menn0_ is now known as menn0 [05:58] morning all [06:12] menn0: hey, you still around? [06:58] morning [06:59] hey TheMue [07:00] jam, hey, are we about to start? [07:03] dimitern: very soon [07:05] thumper, i've dialed into the conference from skype, as there's a toll free us number, and i'm also in the hangout for video [07:06] dimitern: niemeyer has just finished up with the docker demo [07:06] people are moving to rooms now [07:06] right, cool [07:57] hey dimitern [07:57] so I like the idea of "juju deploy —net??? foo,bar,endpoint=baz" or something along those lines. [07:57] Did we change anything for "netzone" vs "network" in the doc already? [07:57] I didn't get a chance to read it today [07:58] I also should be going to the review of openstack charm changes, but I tihnk James was in with mark so that one might be getting moved [08:04] jam1, just a few sentences, as I'm not sure yet how netzones and subnets will be given at deployment time [08:07] thumper: I'm here now === mjs0 is now known as menn0 [08:15] thumper: I'll be aroudn for a while working on a presentation so ping me if you like [09:16] weird, it won't let me be "jam1" but it will let me be j-a-meinel. I wonder if my machine has been connecting + disconnecting too much and it started blocking me. === jameinel is now known as jam [09:20] hello [09:22] jam: hiya :-) [09:37] hey rogpeppe1, just trying to figure out what usernames I was able to connect as [09:37] jam: i realised. just thought i'd reply anyway. === psivaa is now known as psivaa-reboot [10:00] voidspace, hey hey [10:00] voidspace, any troubles with the networker bug so far? [10:20] morning [10:26] mornig [10:26] morning [10:30] folks, I'm on call reviewer today so if there's anything you want me to take a look at give me a shout [10:38] dimitern: hey, yes sorry [10:38] dimitern: missed your message - but yes I have an issue [10:38] hi, [10:39] voidspace, we can talk about it at the standup in 5m then :) [10:39] dimitern: ok [10:39] :-) [10:39] has anyone tried deploying services on ec2 with the current trunk of juju? Deploying mongo fails (it tries to connect to itself using a public address and fails to do that) [10:47] voidspace, standup? [10:47] dimitern: on my way [10:48] dimitern: hmmm... trying again [10:50] dimitern: TheMue: my internet is awful [10:50] dimitern: TheMue: I may be in and out I'm afraid [10:50] here for now though [10:50] voidspace: ok [11:02] dimitern: TheMue: http://pastebin.ubuntu.com/7960111/ [11:06] dimitern: a.st.EnvironConfig() [11:06] entity.Jobs() [11:07] dimitern: "booting system without network configuration" [11:07] dimitern: possibly my network is now screwed in my vm :-) [11:07] dimitern: I did take a snapshot [11:07] voidspace, a.CurrentConfig().Value(agent.ProviderType) can give you the provider type [11:19] dimitern: with this change, straight away I get "agent-state: started" [11:19] dimitern: which I didn't before [11:19] unblocked! [11:19] dimitern: TheMue: thanks guys [11:20] voidspace, \o/ [11:20] dimitern: need to confirm it works and that it fixes the bug [11:20] but progress [11:38] voidspace, :) yep, as usual [11:39] voidspace: great [11:41] * dimitern needs to step out for ~45m [11:58] sinzui: ping [11:58] hi perrito666 [11:58] good morning sir [12:03] sinzui: I was toying with patches yesterday, trying to run ha test and fix the issue [12:03] I noticed two things: [12:04] one, I often get an error which looks a lot like a race condition where the test fails because it tries to do something while still upgrading [12:04] did this happen to you? [12:05] the other one is, while checking past runs on jenkins I noticed that there is a not very consistent set of blue and red dots, has this test ever been consistently passing? [12:05] perrito666, yes, and yes [12:05] sinzui: sorry If I am not being very clear, I woke up a bit dizzy [12:06] The test doesn't pass on first try every time, CI will accept a pass if it passes withing 3 tried [12:07] perrito666, It usually fails because of the upgrade error, but you maye see cases in the ha-backup-restore tests where HA could not be achieved to start the backup :( [12:07] sinzui: I got those too [12:07] perrito666, and if juju and clouds were less brittle, we would require the tests to pass in the first try [12:07] actually when running locally I get that error all the time :( [12:17] so, last test run actually passed, could we trigger a couple more just in case? [12:18] isnt there a way to determine if juju is upgrading so we can poll until its ready for ha? [12:30] dimitern, ping? [12:31] dimitern, cancel that [12:31] sinzui: ? [12:33] perrito666, Yes, I can rerun the tests for the last build-revision id [12:34] perrito666, the tests are scheduled to run in about an hour when the current build published [12:34] oh, ok [12:35] "Nokia N900 modem driver included in mainline " <- the kernel certainly is falling a bit behing on device support [12:35] perrito666, oh, sorry, the publish job has already removed the last binaries. We need to wait [12:35] perrito666, once CI goes idle, we can retest as oftena s we like [12:36] tx sinzui [12:44] discovered https://github.com/rakyll/coop (Go concurrency examples) this morning. might be an interesting read. [12:47] tasdomas: ping [12:47] voidspace, pong [12:48] tasdomas: on July 24th you had an error with local provider [12:48] FATAL: Could not load /lib/modules/3.13.0-32-generic/modules.dep: No such file or directory [12:48] tasdomas: did you discover what it was? [12:48] voidspace, no - I did not [12:48] your mention of it in the juju-dev logs is the only reference to that particular error that google can fine [12:48] http://irclogs.ubuntu.com/2014/07/24/%23juju-dev.txt [12:48] tasdomas: hah, ok [12:48] voidspace, was busy and, tbh, forgot about it [12:48] now google will have two mentions of it I guess [12:49] voidspace ;-] [12:49] tasdomas: but it went away for you? [12:50] voidspace, haven't tried since then [12:50] tasdomas: ah, ok [12:50] voidspace, I can take a look later today and ping you with the result [12:51] tasdomas: that would be cool [12:51] tasdomas: networking on my vm is a bit screwed up anyway, so it just be due to that [12:51] I'm going to revert to a snapshot [12:59] hmmm, no I took my snapshot too late [12:59] network configuration is screwed there too [12:59] time to do some digging [13:03] helping the wife out, biab [14:03] dimitern: we're doing the ipv6 in juju, maas, and openstack. Do you want us to call you on the phone, or just a google hangout? [14:14] any chance that someone might be able to review this straightforward PR? https://github.com/juju/charm/pull/35 [14:19] dimitern, mgz, voidspace, wwitzel3: ^ [14:20] natefinch: to make matters worse, the person that introduced the bug is out all week [14:20] rogpeppe1: must be nice to work on a repo that isn't blocked by the bot ;) [14:20] natefinch: lol [14:20] natefinch: our bot is down though [14:20] rogpeppe1, in a call, sorry [14:21] natefinch: and i also have a couple of branches blocked by the juju-core bot [14:21] natefinch: but yes, in theory :-) [14:22] dimitern: np [14:22] natefinch: so, given that you're blocked on the bot, you obviously have a few moments to spend on that review :-) [14:24] rogpeppe1: on the contrary, I'm trying to unblock the bot :) [14:24] natefinch: fair enough [14:28] jam, is the next call happening in the same room? [14:30] it's not the bot that's blocked... [14:30] it's that master doesn't pass our own validation [14:31] if you want to land things, you either say on the list that we need to land on a broken master, or we unregress [14:31] mm I am seeing this pop a lot 2014-08-05 13:59:02 ERROR juju.cmd supercommand.go:323 connection is shut down [14:43] gsamfira, thanks for making the changes - Hi, I'm Matt by the way [14:44] Hi Matt :) [14:44] thanks for the review [14:44] much appreciated [14:44] rick_h__: https://github.com/juju/juju/pull/468 is a WIP initial draft of the Actions API [14:44] rick_h__: if you get a chance can you let me know if this is close to what you were looking for last week? [14:45] rick_h__: s/Actions API/& documentation/ [15:43] if I can get a brief review on https://github.com/juju/juju/pull/448 it would be much appreciated, I can close the bug if I can land it [15:43] it's very simple [15:49] bodie_: is there a reason not to use SameContents checker? [15:50] rather than the obtainedIds expectedIds block [15:50] mgz, ignorance? :/ [15:50] didn't know I didn't have to roll my own [15:50] 's in github.com/juju/testings/checkers/checkers.go [15:51] -s [15:51] okay, great [15:59] mgz, I'm not sure that will work since SameContents does not regard duplicates as a problem [15:59] in this case, I'm counting appearances of each id [16:00] fair enough === viperZ28__ is now known as viperZ28 [17:05] really amazon? io timeout? [17:06] that's how they do [17:07] perrito666: I'm wondering if it's because of the journaling changes possibly overloading mongo? [17:07] wwitzel3: ec2 is being especially mean with me [17:07] perrito666: strike that I was thinking of azure not amazon [17:09] trying to help hackedbellini get his up and running again and we are seeing this error .. http://pastebin.ubuntu.com/7962740/ .. for some reason it cannot connect to the web socket api? (local provider) [17:11] it's not the machine-0 problem anymore, this error is from the other machines (the lxc containers) [17:13] wwitzel3: perhaps api server is not up, check dmesg and logs for machine 0 you might get jujud-machine-0 respawning a lot trying to get up [17:21] 29% done on memtest. so slow [17:21] so far no errors [17:23] natefinch: welcome back [17:23] natefinch: 1st pass? [17:24] yeah, doing both sticks at once to see if anything pops up. [17:25] my tablet wasn't charged which is why I couldn't get on earlier... so I got the oil changed in my car while it was charging. [17:25] natefinch: that is slow [17:25] not you, I mean the memtest [17:25] it only started a little while ago, I got your email while I was out. [17:26] natefinch: anyway, if your mem if factory dual channel, the stick not broken will break soon [17:28] * perrito666 tried to purchase 2x8sodimm today and the nearest one is 800km away and will be hard to get :p === psivaa is now known as psivaa-holiday [17:37] 44% no errors... [17:38] I think it takes longer because it only tests two gigs arts a time... so it has to do 8 passes total [17:42] you should have removed one of the banks, your result would be less certain but the test would take half the time [17:53] do we have concrete info on why our replica takes so much? [17:55] takes so much.... what? [17:55] natefinch: sorry I typed half of what I was thinking [17:55] do we have concrete information on why each replicaSet takes so long to be ready? [17:57] mongo.... other than that, no. [17:57] we should really talk to the mongo guys about it [17:58] 65% done. still no errors [17:59] natefinch: does you bug happen on heavy loads? [18:08] only on boot, seems to be. [18:09] after I reinstalled it was fine until I rebooted.... I had just installed some updates, but I don't know if that was the cause or not. [18:09] 85% done [18:13] well if the ram is broken you should see fs corruption [18:14] if you dont see fs corruption you might be stumbling into some odd update breakage alghough I guess axw and ericsnow should have also [18:14] you might want to check the SMART data for you hd [18:14] or sdd [18:15] smart was fine last time [18:15] I have RAM errors in dmesg which is why I thought it was ram.... [18:16] tests done. all tests pass [18:16] well fuck [18:17] natefinch: what is a ram error in dmesg? I am curious [18:18] * perrito666 remembers natefinch recommending that laptop yesterday [18:18] hahahahahaha [18:18] I'll get the message when it finishes booting [18:22] under a section called total RAM covered: 16334 MB [18:24] *BAD*gran_size: 64k chunk_size:32M num_reg: 10 lose cover RAM: -2M [18:26] a bunch of those with different sizes, but the bad ones all have negative numbers at the end. ones that don't say bad all have 0 or a positive number [18:27] no friggin' clue what it actually means besides "bad" and "RAM" [18:29] ahh I see [18:29] natefinch: http://velenux.wordpress.com/2014/01/ [18:29] linux [18:29] natefinch: you might find a better explanation at http://my-fuzzy-logic.de/blog/index.php?/archives/41-Solving-linux-MTRR-problems.html [18:30] hmm.. if you are running memtest, you should run multiple passes. We used to do 20h memtest on production servers, just to make sure the memory was fine :). Sometimes we would see errors after a few passes [18:36] perrito666: interesting [18:37] I wonder if that is my problem [18:37] yet, that does not seem to be the cause of your other problem [18:37] although it might [18:37] try that and then see if the rest of the machine works properly [18:38] and in that case you might want to return the ram :p [18:41] perrito666: now if I can figure out what exactly it is that site is trying to tell me to do [18:43] rogpeppe1: ping? [18:44] it says to look at /proc/config.gz ... but that doesn't exist [18:47] cat /process/mtrr says almost the whole 16 gigs is in reg00 .... that seems bad. [18:47] natefinch: you most likely need to compile a kernel with such option on [18:48] OK, there is no way the answer to this problem is compile a friggin kernel [18:48] natefinch: try the one shorter [18:48] natefinch: dude this is linux, most problems, if poked enough will end up having recompile a kernel for an answer [18:49] natefinch: http://askubuntu.com/questions/244473/how-and-why-should-i-specify-mtrr-gran-size-mtrr-chunk-size [18:49] that one seems to be more ubuntu oriented [18:50] the answer seems to be pretty explanatory [18:50] OK, how do I add that to my grub config? [18:51] natefinch: /etc/default/grub [18:51] append to GRUB_CMDLINE_LINUX_DEFAULT [18:51] also, how do I know what to specify? no one seems to answer that part [18:51] "it appears that the error you see is reported when mtrr sanitizer is unable to choose from several options of memory layout. It should print a list of possible options next to the error message. To make the message to go away, you need to specify something lik" [18:52] pretty trivial problem.... [18:54] not when there's 100 different choices and the effect is not well understood. I presume I should pick something that has a small lose cover RAM? [18:54] * perrito666 was being sarcastic [18:54] oh [18:55] apparently the error should come with a list of possible values [18:55] and if I can pick anything that doesn't say bad, why can't the computer do that? [18:55] I guess is a trial and error approach [18:56] natefinch: well I guess you would not want your computer restarting randomly to try new ram mappings [18:56] there's a bunch of gran size lines mixed in with the bad ones.... I guess those are the options? it's not at all obvious [18:56] if you wanted an OS that did that you would use another one :p [18:56] natefinch: pastebin? [18:57] if it says num_reg 10 and I only have 7, I guess that's not a valid option? [18:58] natefinch: well apparently if you eff it up you will get dmesg to tell you what would be better choices [18:58] I don't have network on the stupid machine... I can boot into a live disk... [18:59] brb === uru_ is now known as urulama [19:32] perrito666, http://pastebin.ubuntu.com/7963815/ [19:33] btw I have dmesg logs from the 29th where these things show up and i was able to reboot ok then [19:33] * perrito666 looks [19:33] (running on the live USB disk I made last time... one nice thing about this having happened so recently) [19:33] do you have an nvidia video card? [19:35] yep [19:37] you might have better results if you use the closed driver [19:37] but that is just a side note [19:37] oh, yeah, I used to beforeI reinstalled [19:37] just forgot to this time [19:40] natefinch: could you pastebin your /etc/default/grub ? [19:43] sure [19:45] http://pastebin.ubuntu.com/7963935/ [19:46] realnatefinch: you are using a live linux on that machine right? [19:46] one last, paste me cat /proc/mttr [19:49] * perrito666 doesn't like what he sees [19:50] if not you can blindly try with gran_size: 128M chunk_size: 128M num_reg: 7 lose cover RAM: 206M [19:50] and check if there is a bios upgrade for your machine === uru_ is now known as urulama [19:51] ok [19:54] can you link me to that article again? I'm on my machine, so don't have it in irc history here [19:54] perrito666, ^ [19:56] nevermind, google found it [19:56] http://my-fuzzy-logic.de/blog/index.php?/archives/41-Solving-linux-MTRR-problems.html [19:56] and the ubuntu summary http://askubuntu.com/questions/244473/how-and-why-should-i-specify-mtrr-gran-size-mtrr-chunk-size [19:59] it says to append that to the "boot parameters" ...... where is that? [19:59] realnatefinch: /etc/default/grub -> GRUB_CMDLINE_LINUX_DEFAULT [20:03] ok rebooting [20:03] gluck [20:03] ks [20:03] thanks [20:05] bah. well, let's see what dmesg says [20:06] meaning? [20:08] meaning same problem on startup. dmesg says same thing - please specify chunk size etc [20:08] natefinch: did you run update-grub? [20:09] i dont recall the exact command [20:09] it's definitely in my grub config [20:09] oh uh no? [20:09] good thing is you dont need network for that [20:12] they're update grub and update grub2. [20:13] * natefinch reboots [20:14] bah [20:14] bah means no good luck? [20:16] correct [20:17] is that factory default ram? [20:18] yep [20:18] so the mtrr error is gone [20:19] but my Ethernet and Wi-Fi still terminate with status 1 [20:19] where is that I did not see it in the dmesg you passed [20:20] near the bottom, search for eth0 [20:22] mm, I see [20:22] ifconfig -a shows something? [20:26] shows wifi and Ethernet as 0 packets and not make as UP or RUNNING [20:26] s/make/marked/ [20:35] it does have a couple virtual Ethernet adapters that I don't recognize [20:35] at least I assume veth means a virtual Ethernet adapter [20:36] natefinch: well, what you seem to have is not a problem with the interfaces but with network manager [20:36] k [20:36] which is most likely something you installed [20:36] the virtual ones are most likely juju local :) [20:36] or virtualbox [20:37] v tells me that is virtualbox, bc I think juju local is lxbrt [20:37] I do have both those things installed [20:37] so we are close [20:37] tell me what is the content of /etc/networking/interfaces [20:37] sorry [20:37] /etc/network/interfaces [20:38] I have /etc/network/interfaces.d/ which is empty but no interfaces file [20:40] natefinch: what version of ubuntu are you using? [20:40] 14.04.1 [20:41] mm, as it comes ootb you should have /etc/network/interfaces which contains [20:41] # interfaces(5) file used by ifup(8) and ifdown(8) [20:41] auto lo [20:41] iface lo inet loopback [20:41] bc, basically your problem here, if I understand correctly is that your networking service is not starting properly [20:41] and network manager seems to be breaking along [20:42] and therefore your interfaces remain down [20:42] you could ifup them by hand but sadly negotiating wpa by hand is harder than negotiating an extension with a loan shark [20:43] perrito666: you sound like you speak from experience [20:44] have you tried to buy bonds from my country lately? they are cheaper than candy [20:44] sudo ifup eth0 says unknown interface eth0=eth0 [20:45] natefinch: try adding what I told you to interfaces and sudo initctl restart networking [20:45] by sheer coincidence my /etc/network/interfaces file has vanished [20:45] and I'm attempting to repair networking too [20:45] * perrito666 smells a bug [20:46] perrito666: there is an open bug that the local provider kills networking [20:46] and I'm working on it [20:46] lol [20:46] which is probably why my network is broken [20:46] you seem to have found the problem [20:46] perrito666: can you pastebin yours [20:46] perrito666: that's the result not the cause [20:46] well, it's the current cause of my broken network yes [20:46] voidspace: mine? [20:46] perrito666: /etc/network/interfaces [20:46] I have none whatsoever [20:47] voidspace: look 20 lines up [20:47] I copied its contents here [20:47] was that the sum total? [20:47] its 3 lines long [20:47] heh [20:47] fair enough, thanks [20:47] natefinch: ^ you might be on the same page there [20:48] now to reboot the vm and see if I get a network configuration this time [20:48] voidspace: you dont need to reboot the machine for that [20:48] and I do, yay [20:48] perrito666: no, I want to [20:48] I can manully start the network service anyway [20:48] voidspace: lol [20:49] I want to make sure that when I reboot I have it [20:49] natefinch: well, apparently if you create your interfaces by hand you get network back [20:49] mm, this is getting crowded with natefinchs [20:49] hey, replacing my /etc/network/interfaces with a valid one and doing sudo ifup eth0 worked [20:49] haha [20:49] natefinch: voidspace just explained that that is actually a bug of juju [20:50] OMFG [20:50] and to be fair, one is more than enough... [20:50] we seem to be deleting interfaces from computers [20:50] well shit [20:50] natefinch__: https://bugs.launchpad.net/juju-core/+bug/1349635 [20:50] local provider kills networking on the host.... [20:51] mmm, seems to be a bug in lx [20:51] lxc [20:51] I don't think so [20:51] see dimiter's comment further down [20:51] ah, no dimiter commented something [20:51] we screw with /etc/network/interfaces [20:52] and normally that's fine, because we control the machine [20:52] fuuuu, this happens bc local provider runs with sudo [20:52] ffff [20:52] but not for the host on local provider [20:52] voidspace: that seems like a colossally stupid idea on local [20:52] voidspace: it sort of seems like a terrible idea for manual provider, too [20:52] natefinch__: so my fix, which I'm struggling to test due to screwed networking, is precisely "don't do that on local" [20:52] natefinch__: we need to do some of it for a maas fix for legacy reasons [20:53] can someone post their /etc/network/interfaces for me? [20:53] that maybe have already gone away [20:53] we should at least run local provider machine-0 on a chroot [20:53] so I can update mine to what it's supposed to look like [20:53] if we cannot in a lxc [20:53] perrito666: it has to be a chroot that can create lxc containers [20:53] not sure if that's possible [20:53] voidspace: mm, should with the right config [20:53] perrito666: that certainly seems better than letting it muck with the user's configuration [20:53] # interfaces(5) file used by ifup(8) and ifdown(8) [20:53] auto lo [20:53] iface lo inet loopback [20:54] natefinch__: ^ [20:54] those 3 lines [20:54] you can skip the first one [20:54] perrito666: I had to add stuff about eth0 to get ifup to let me turn on eth0 [20:55] natefinch__: if you add that and then initctl restart networking (or start) and then the same with network manager you will most likely get it running or you will need to reboot [20:55] natefinch__: just that one worked for me [20:55] perrito666: ok [20:55] natefinch__: for future references when the error in dmesg says exited with 1 is a service, when its the actuall device it throws an ugly memory dump [20:56] ok [20:56] rebooting === natefinch__ is now known as natefinch [20:57] ahh better [20:57]