[01:26] wallyworld: you have a moment? WOuld like help deciphering an error message I'm seeing in the failed tests. [01:26] ok [01:26] wallyworld: I'm seeing:"cannot add unit to application "memcached": getting instance types: listing VM sizes: azure.ServicePrincipalToken#Refresh:" for 3 tests, is it possible that it's the azure side that's failed or juju? [01:27] wallyworld: the logs are at: [01:27] http://reports.vapour.ws/releases/4490/job/azure-arm-deploy-bundle-lxd/attempt/687 [01:27] well, that's one of them at least [01:27] not too sure about azure, i'll look [01:27] wallyworld: thank you [01:33] veebers: the error in the logs seems to be x509: certificate signed by unknown authority [01:34] not sure off hand the root cause; azure deploys seem to work in general [01:43] wallyworld: ok thanks, I'll keep looking [01:44] veebers: andrew would know probably but he's out till thursday. seems strange though it's just one test out of many. there may be some difference in test set up or something [01:48] veebers: what u r describing may be related to an exisiting bug 1629759?... [01:48] Bug #1629759: azure subscriptions api throws x509 error "certificate signed by unknown authority" [01:50] anastasiamac: awesome thanks, I'll check that bug is similar and file the issue away etc. [01:50] veebers: \o/ [01:51] wallyworld: ack, I'll file or +1 the bug at this point [02:12] Bug #1633554 changed: juju ssh uses old/invalid known_hosts data === rmcall_ is now known as rmcall [03:36] thumper: or anyone: trivial https://github.com/juju/juju/pull/6457 [03:37] lgtm [03:48] \o/ [04:12] wallyworld: PPTAL https://github.com/juju/juju/pull/6458 ? should b trivial :D [04:13] ok [04:24] wallyworld: and another :D https://github.com/juju/juju/pull/6459 [04:26] what bug is that? [04:33] bug 1633607 [04:33] Bug #1633607: juju grant help text puts examples before valid inputs [04:33] i'll add it to the pr.. sorry.. it's in the name but not in prose :D [04:36] ah, i just didn't ee it [08:02] axw: ping [08:57] could anyone take a look on this PR https://github.com/juju/juju/pull/6414 ? [09:26] menn0: ping? [09:31] babbageclunk: pong [09:31] frobware / dimitern / voidspace / babbageclunk: https://github.com/juju/juju/pull/6460 is short and has a nice big positive impact. [09:31] menn0: hey, just going through the redirect after migration task [09:31] review please! [09:31] * dooferlad goes for coffee [09:32] dooferlad: looking [09:32] voidspace: thanks [09:32] dooferlad: looking [09:32] dooferlad: does this work with LXD on Y? [09:32] btw I still need a review on https://github.com/juju/juju/pull/6454 please << dooferlad, voidspace, babbageclunk, frobware [09:32] dooferlad: or, put another way, is the domain name still the same in LXD 2.4? [09:33] dooferlad: LGTM [09:33] menn0: So basically Login just needs to return a CodeRedirect if there's a successful migration, and RedirectInfo returns the addresses and cert from the TargetInfo? [09:33] dimitern: I will look at it in a bit [09:33] babbageclunk: that's what I was thinking === tinwood_ is now known as tinwood [09:33] menn0: Cool - seems pretty straightforward! [09:33] babbageclunk: at a hand-wavey level anyway [09:34] voidspace: cheers! [09:35] babbageclunk: yep, hopefully. I was very glad when I saw rog doing the initial work for this and what he did was very similar to what I was thinking [09:37] babbageclunk: I assigned you a card for this on a a-team board btw [09:38] menn0: Yeah, saw an email about that - I'll move it to in-progress then [09:38] babbageclunk: sweet [09:39] dimitern: taking a look [09:47] dimitern: personally I think selecting the fastest based on duration is a step too far. why? it so depends on what the machines are doing _at_ that _moment_ in time. And that may be totally different when you come to initiate the SSH connection. [09:47] dimitern: as a first pass do we not just care for something that is listening? [09:48] frobware: we do care for those we can connect to, and if one takes >100ms while another takes <10ms, we should pick the latter [09:49] dimitern: this is all noise, IMO. [09:49] dimitern: the numbers are noise 100 vs 10? [09:54] frobware: sorry, I'm not sure I follow :/ [09:55] dimitern: if you open and stare at a document for 5 mins. would it matter that it took 100ms or 10ms to open? If you initiate and do stuff over an SSH connection for 5 mins would it make any difference if the initial connection was 100ms or 10ms. IMO, nope. [09:56] frobware: ok, I see that, but I'm not sure why do you think picking the lowest-latency endpoint is wrong [09:56] dimitern: it's code that we don't need. to test. to document. or grok. [09:57] frobware: what do you suggest instead? [09:58] dimitern: probe for a listening connection. if it is "open" just record that fact. I don't think it matters how long it took. there should/would be an overall timeout on the overall probe anyway. [09:59] dimitern: in terms of latency, your sample size is 1. [09:59] dimitern: which is not huge. :) [10:00] frobware: not necessarily - you could have multiple equally good endpoints [10:00] frobware: but then it shouldn't matter I guess [10:00] dimitern: I think less is more in this case. all we want to know is if the addr/port is reachable. [10:02] frobware: so you're saying "quit trying as soon as you connect successfully" ? [10:04] dimitern: if there was more than one answer would we connect on more than one addr:port? If not, then isn't the first enough? [10:05] dimitern: that was the approach I was playing with. http://178.62.20.154/~aim/portscan.go [10:06] frobware: the way it's currently implemented, I think we'll only connect to one endpoint at a time [10:06] dimitern: right. so first open port would be good enough. true? [10:06] frobware: but otherwise, with a goroutine-per-hostport it's possible to connect to more than one endpoint [10:06] frobware: yeah [10:06] dimitern: but would we do that as part of bootstrap? [10:07] dimitern: previously we would blindly connect to the first addr:port. Now we at least verify there's something on the other end. [10:08] dimitern: my take on commits. It should be so obviously simple that I would be nuts not to accept it... [10:09] frobware: ok, fair enough - will change that and push [10:09] dimitern: appreciated. ty [10:25] oops, missed this discussion, I might have reviewed an obsolete PR. Hopefully there are still some useful comments in it dimitern! [10:34] dimitern: ping [10:34] babbageclunk: thanks! I'm changing it to simplify the logic [10:34] voidspace: pong [10:34] dimitern: :) [10:35] dimitern: do you know how to get juju bootstrap to use ssh (particularly a specific ssh key)? [10:35] dimitern: I have the juju-ci ssh key and can ssh to the machine I want access to [10:35] dimitern: but I can't bootstrap [10:35] voidspace: that's a manual bootstrap? [10:35] dimitern: is it? [10:36] frobware, dimitern: re-ping https://github.com/juju/juju/pull/6460 [10:36] dimitern: no, I just want "juju bootstrap vsphere vsphere" to work [10:36] dimitern: I have the vsphere credentials and configuration [10:36] voidspace: ah, ok [10:37] frobware: I haven't tried a public cloud. I put some notes in about detecting the correct domain in a follow up. [10:37] dimitern: mgz says I should be able to do this just using the juju-ci ssh key, but spelunking through the juju-release-tools to work out the right incantation is painful [10:37] dimitern: I can see the code paths that bootstrap, but working out exactly what arguments / environment it is using will take a pen and paper and some time... [10:37] voidspace: then I think it's up to ~/.ssh/config - if you have a Host section with IdentityFile and Username, etc. set up correctly and you can do 'ssh ' OK (without any args) [10:37] dimitern: and I thought you might "just know"... [10:37] dooferlad: yep, looking [10:37] dimitern: I have that setup === gnuoy` is now known as gnuoy [10:38] dooferlad: I think we need to try at least AWS [10:38] dimitern: so it *should* "just work" if my ssh config is correct? [10:38] dimitern: I wonder if I have some conflicting rules [10:39] voidspace: I expect so [10:39] voidspace: as juju's ssh during bootstrap tries effectively `ssh :22` [10:39] dimitern: I definitely *can* ssh to the IP I want [10:39] but the bootstrap times out [10:40] voidspace: without specifying username and key? [10:40] dimitern: correct [10:40] voidspace: hmm - well, try bootstrap --debug ? [10:41] dimitern: ah, it's trying *my* ssh key - not the juju-ci one [10:41] voidspace: is the juju-ci ssh key in ~/.ssh ? [10:41] voidspace: it should try all keys it finds (IIRC) [10:42] it is [10:42] I terminated that one when I saw the wrong key - I'll try again [10:43] nope, doesn't seem to try more than the one key [10:44] voidspace: if you just go to the box, su to jenkins [10:45] and do the steps as written in the job [10:45] that's all you need [10:45] so, set JUJU_HOME=~jenkins/cloud-city and so on [10:45] to just repo something on a ci machine, you should do exactly what ci does [10:46] mgz: I want to be able to make a custom build - bootstrap and debug - change and repeat [10:46] you don't need to do your own config unless you're trying to do something else [10:46] mgz: so I *really* want to do it from my box, not from the CI box if possible [10:46] you can just scp a new binary onto the box [10:46] mgz: but I think at this point I have to give up, and do that [10:46] ok [10:46] mgz: dimitern: thanks [10:47] frustrating, but such is life [10:49] mgz: by the way, deploy_job.py has no reason to exist (juju-ci-tools) - just add that main stanza to deploy_stack.py [10:49] yeah, there's a bunch of minor things that could be cleaned up with the older scripts [11:18] mgz: ping [11:19] voidspace: hey [11:36] dooferlad: LGTM as long as we have a sanity bootstrap test for AWS and MAAS. As you mention, should be a change to the LXD provider only. [11:37] dooferlad: I do wonder whether we should query for the domain as it is the LXD provider. [11:38] dooferlad: otherwise at some point this will bite us [11:38] dooferlad: so, a little contradictory to our call. :/ [11:56] Bug #1420996 changed: Default secgroup reset periodically to allow 0.0.0.0/0 for 22, 17070, 37017

[12:02] Bug #1559062 changed: ensure-availability not safe from adding 7+ nodes, can't remove stale ones

[12:19] hi guys! I'm trying to bootstrap a lxd environment but I'm having a problem. It seems that it bootstraps fine, the lxd container gets online, but at some point it tries to get a client, but it sends the request to my gateway machine and not the machine that I'm running juju itself [12:19] here are the logs: http://pastebin.ubuntu.com/23326644/ [12:20] this is my /etc/network/interfaces http://pastebin.ubuntu.com/23326622/ and this is my /etc/default/lxd-bridge http://pastebin.ubuntu.com/23326614/ [12:21] as you can see, the machine that I'm tying to bootstrap has an ip of 192.168.99.3. There's another machine on the network, the gateway, that has 192.168.99.4, but it has nothing to do with juju. In fact, the only place that I put its ip was on /etc/network/interfaces file. So I'm very confused on why it is trying to do that request on that machine [12:23] am I doing something wrong? I saw the same problem reported in this comment, but I don't think it is related to the bug it was reported: https://bugs.launchpad.net/juju/+bug/1547268/comments/19 [12:23] Bug #1547268: Can't bootstrap environment after latest lxd upgrade <2.0-count> [12:29] Bug # changed: 1379882, 1383492, 1384350, 1384353, 1384359, 1384363, 1384565, 1384572, 1384725, 1384803, 1385098, 1387751, 1388983, 1389656, 1390525, 1391387, 1391966, 1392340, 1393879, 1393932, 1396159, 1399310, 1399322, 1399604, 1399606, 1400193, 1400207, 1403032, 1403674, 1408307, 1412720, [12:29] 1413653, 1414021, 1414890, 1416406, 1418325, 1419715, 1423278, 1423548, 1424837, 1425137, 1425680, 1429864, 1433244, 1435543, 1435999, 1438258, 1438721, 1438798, 1441749, 1442541, 1444537, 1445053, 1447390, 1449617, 1451616, 1452393, 1452490, 1454627, 1454661, 1455284, 1455646, 1457895, 1458447, [12:29] 1458452, 1459082, 1459148, 1459761, 1461358, 1461534, 1461889, 1461954, 1466272, 1466951, 1467590, 1468187, 1468855, 1474386, 1474892, 1475341, 1475425, 1475641, 1477709, 1480942, 1483987, 1484718, 1484833, 1485017, 1485145, 1490947, 1490977, 1491451, 1492396, 1498518, 1498575, 1498859, 1499332, [12:29] 1503039, 1503449, 1508392, 1508595, 1512191, 1516077, 1524089, 1536838, 1557302, 1568654, 1593855, 1611379 [12:35] This fixes an intermittent test failure in juju; reviews appreciated, thanks: https://github.com/juju/juju/pull/6461 [12:38] Bug # changed: 1282025, 1282211, 1284167, 1284734, 1285082, 1285422, 1285685, 1287662, 1287665, 1287669, 1288302, 1288900, 1288950, 1290654, 1290727, 1292344, 1293686, 1295682, 1296515, 1297094, 1298120, 1298141, 1299027, 1300637 [13:07] hackedbellini: what's in your LXD profile? Run: lxd profile show default [13:08] Bug # changed: 833064, 862418, 945862, 998238, 1009687, 1015637, 1026422, 1029976, 1037027, 1037045, 1043076, 1050245, 1057650, 1057652, 1083558, 1086670, 1089297, 1089298, 1089304, 1096840, 1097018, 1100076, 1122135, 1122889, 1129218, 1129219, 1130771, 1131409, 1137902, 1144355, 1149889, 1156654, [13:08] 1161919, 1163983, 1164601, 1168754, 1169773, 1170419, 1172811, 1174190, 1176740, 1185143, 1191418, 1194869, 1197365, 1197372, 1199888, 1201117, 1202175, 1204851, 1206759, 1208292, 1209044, 1209155, 1209313, 1210035, 1210076, 1210449, 1210484, 1211498, 1212177, 1212936, 1214178, 1214949, 1214952, [13:08] 1214954, 1214957, 1214959, 1214961, 1214967, 1214968, 1215052, 1215252, 1215402, 1215777, 1217282, 1217508, 1217742, 1217768, 1217860, 1218167, 1218176, 1218834, 1219630, 1220260, 1220705, 1220816, 1221834, 1223325, 1224057, 1224368, 1224456, 1224492, 1224515, 1226460, 1226652, 1226786, 1227142, [13:08] 1227586, 1228241, 1228311, 1229199, 1229506, 1229883, 1230053, 1230131, 1230289, 1230370, 1230702, 1231526, 1231551, 1233371, 1233938, 1234217, 1234687, 1234715, 1237304, 1237518, 1238938, 1239368, 1239509, 1239908, 1240461, 1242237, 1243811, 1244760, 1245649, 1247688, 1248839, 1250104, 1250115, [13:08] 1250153, 1250965, 1251118, 1251697, 1252781, 1253056, 1253704, 1254237, 1255502, 1256053, 1256852, 1257758, 1257975, 1258116, 1258644, 1258889, 1259490, 1261324, 1262175, 1266476, 1266729, 1267387, 1267518, 1267541, 1268550, 1269014, 1270543, 1274455, 1277048, 1277116, 1277139, 1278734, 1279986, [13:08] 1300882, 1301352, 1302119, 1302561, 1303455, 1303942, 1304863, 1305385, 1307101, 1308088, 1308491, 1309434, 1309449, 1310453, 1311976, 1312068, 1312173, 1312201, 1313793, 1314754, 1316174, 1316593, 1316602, 1317917, 1319346, 1320080, 1322705, 1322747, 1324904, 1328151, 1329483, 1331694, 1336542, [13:08] 1340483, 1340836, 1341264, 1345848, 1347994, 1358227, 1359573, 1364847, 1364866, 1365623, 1366027, 1366793, 1368362, 1368981, 1372566, 1372759, 1374159, 1445093, 1445658, 1446159, 1446168, 1447235, 1449613, 1449633, 1450729, 1453297, 1453890, 1455089, 1455368, 1455445, 1455703, 1455840, 1456258, [13:08] 1456398, 1456703, 1456728, 1456857, 1457089, 1458758, 1459327, 1459610, 1461339, 1461605, 1461961, 1462097, 1462423, 1462874, 1464235, 1467873, 1467964, 1468359, 1468637, 1496188, 1496221, 1496975, 1497653, 1497788, 1498511, 1498577, 1499400, 1499570, 1501084, 1501093, 1501203, 1502016, 1505309, [13:08] 1505435, 1505460, 1506865, 1506866, 1506994, 1509099, 1510333, 1511543, 1511944, 1514616, 1516698, 1517391, 1517428, 1519081, 1519848, 1521017, 1523837, 1524021, 1527595, 1530992, 1533790, 1534289, 1537586, 1537731, 1540580, 1546492, 1546795, 1550033, 1554060, 1554807, 1564018, 1565196, 1567179, [13:08] 1568161, 1572022, 1583409, 1585750, 1590958, 1599129, 1620830, 1620832 [13:09] hackedbellini: also, using the defaults I'm surprised to see the .3 for LXD_IPV4_ADDR="192.168.99.3". This would normally be .1 [13:14] Bug # changed: 1391941, 1392229, 1392379, 1392407, 1392684, 1392810, 1392876, 1393452, 1393883, 1393892, 1394668, 1395900, 1396096, 1396474, 1396862, 1397171, 1398055, 1399303, 1399506, 1399722, 1399730, 1468639, 1468756, 1469318, 1469731, 1470345, 1471138, 1471237, 1474411, 1475386, 1475635, [13:14] 1477281, 1477712, 1478706, 1478762, 1478934, 1478936, 1479278, 1479653, 1481366, 1482876, 1482939, 1483525, 1483932, 1484177, 1484930, 1489217, 1489484, 1490665, 1491608, 1492232, 1492598, 1494782, 1494831, 1495952, 1496127 [13:14] wow [13:15] frobware: lxd profile? How can I see it? [13:16] hackedbellini: $ lxd profile show default [13:16] hackedbellini: lxc profile show default [13:16] hackedbellini: not lxd, lxc [13:16] frobware: ahh ok, here: [13:17] https://www.irccloud.com/pastebin/eVgPQz7R/ [13:17] hackedbellini: looks fine [13:18] and the .3 is because this machine should have this ip. We have 3 machines here, one that servers nfs, one that provides dhcp and acts as the network firewall (the .4) and this one that runs our services. We were using juju 1.x in it for some time with this bridge configuration without any problems [13:18] hackedbellini: did you run `sudo dpkg-reconfigure lxd` and accept the defaults. Note: you must not have IPv6 enabled in your lxd-bridge setup. [13:19] frobware: what do you mean by defaults? [13:19] hackedbellini: if you run `sudo dpkg-reconfigure lxd` it will ask you questions about your IPv4 (and 6) network setup [13:20] hackedbellini: in general, you can just accept the defaults with the proviso you should not accept any IPv6 setup [13:20] frobware: yeah, but the defaults means "no bridge"? Because I need the bridge on this machine [13:20] Bug # opened: 1391941, 1392229, 1392379, 1392407, 1392684, 1392810, 1392876, 1393452, 1393883, 1393892, 1394668, 1395900, 1396096, 1396474, 1396862, 1397171, 1398055, 1399303, 1399506, 1399722, 1399730, 1468639, 1468756, 1469318, 1469731, 1470345, 1471138, 1471237, 1474411, 1475386, 1475635, [13:20] 1477281, 1477712, 1478706, 1478762, 1478934, 1478936, 1479278, 1479653, 1481366, 1482876, 1482939, 1483525, 1483932, 1484177, 1484930, 1489217, 1489484, 1490665, 1491608, 1492232, 1492598, 1494782, 1494831, 1495952, 1496127 [13:20] hackedbellini: ah, good point. because you're bridged on br0. (the .3) [13:21] frobware: yeah. But answering your question, I did not accept IPv6 [13:23] hackedbellini: what's files are in /etc/network/interfaces.d/ [13:24] frobware: nothing, just /etc/network/interfaces (http://pastebin.ubuntu.com/23326622/) [13:25] hackedbellini: and the output of `ifconfig -a` and `ip route` ? [13:28] https://www.irccloud.com/pastebin/DFGS6yFn/ [13:28] frobware: ^ [13:29] hackedbellini: thanks. going to try and reproduce this. [13:30] frobware: thank you so much! Please tell me if you need more info and/or want me to test anything else [13:30] hackedbellini: one more thing: sudo lsof -i:8443 [13:31] https://www.irccloud.com/pastebin/eE1VSqxN/ [13:38] Bug # changed: 1332049, 1332221, 1332545, 1333496, 1334390, 1336313, 1336353, 1337318, 1337804, 1340077, 1340133, 1340184, 1340749, 1342738, 1343318, 1343569, 1345440, 1345541, 1349908, 1350008, 1351426, 1353239, 1353482, 1353571, 1354039, 1355216, 1356806, 1356857, 1358376, 1358474, 1359925, [13:38] 1360607, 1361365, 1361723, 1361759, 1363183, 1364013, 1365604, 1365621, 1365633, 1391941, 1392229, 1392379, 1392407, 1392684, 1392810, 1392876, 1393452, 1393883, 1393892, 1394668, 1395900, 1396096, 1396474, 1396862, 1397171, 1398055, 1399303, 1399506, 1399722, 1399730, 1400559, 1400782, 1401568, [13:38] 1402763, 1403955, 1408191, 1408467, 1408472, 1408848, 1409104, 1409381, 1409746, 1409806, 1409856, 1412020, 1412917, 1413052, 1413424, 1414027, 1414710, 1417874, 1418608, 1419864, 1421262, 1423364, 1423626, 1424901, 1425245, 1425506, 1425930, 1426127, 1426217, 1426458, 1426730, 1426940, 1427510, [13:38] 1427770, 1428893, 1429353, 1430220, 1430839, 1430943, 1431401, 1433589, 1434246, 1436766, 1436961, 1438590, 1440445, 1441899, 1441915, 1442046, 1445078, 1468639, 1468756, 1469318, 1469731, 1470345, 1471138, 1471237, 1474411, 1475386, 1475635, 1477281, 1477712, 1478706, 1478762, 1478934, 1478936, [13:38] 1479278, 1479653, 1481366, 1482876, 1482939, 1483525, 1483932, 1484177, 1484930, 1489217, 1489484, 1490665, 1491608, 1492232, 1492598, 1494782, 1494831, 1495952, 1496127 [13:46] mgz: ping [13:50] anastasiamac: I blame you for this ^ :) === coreycb` is now known as coreycb [13:53] natefinch: u blame me for closing bugs? ;D [13:53] hackedbellini: can you try the following for me: [13:53] hackedbellini: juju bootstrap lxd lxd [13:53] natefinch: believe me, i'd rather not have any (would save me pain of some hate mail) [13:54] hackedbellini: and in another window [13:54] hackedbellini: lxc list [13:54] hackedbellini: lxc exec bash [13:54] frobware: "bootstrap lxc" or "bootstrap localhost"? [13:54] hackedbellini: cat /etc/network/interfaces.d/50-cloud-init.cfg [13:55] hackedbellini: doesn't matter - whatever bootstrap command you are already using. I just want to bootstrap and then poke around in the image that lxd/juju creates. [13:55] https://www.irccloud.com/pastebin/vPi7Gpvg/ [13:56] frobware: this is the result: [13:56] $ lxc exec juju-147208-0 bash [13:56] root@juju-147208-0:~# cat /etc/network/interfaces.d/50-cloud-init.cfg [13:56] # This file is generated from information provided by [13:56] # the datasource. Changes to it will not persist across an instance. [13:56] # To disable cloud-init's network configuration capabilities, write a file [13:56] # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: [13:56] # network: {config: disabled} [13:56] auto lo [13:56] iface lo inet loopback [13:56] auto eth0 [13:56] iface eth0 inet dhcp [13:57] ops, sorry, was going to pastenbin it but selected the wrong option [13:57] hackedbellini: interesting. because mine 'inet manual' [13:57] hackedbellini: which explains why my bootstrap will never complete. (it never gets an IP address) [13:58] frobware: so you are unable to reproduce the issue? [13:58] mgz: successfully bootstrapping to vsphere from the jenkisn machine [13:58] * frobware wonders how this can be [13:58] mgz: using 2.0-beta17 which is the installed version on that box, but still bootstrapping a custom build should be straightforward from here [13:58] hackedbellini: no. but there is mostly definitely an issue where we have 'iface eth0 inet manual' [13:58] mgz: thanks for your help and patience [13:59] frobware: is there any workaround I can do here? I need to setup this to get our services up ASAP [14:00] voidspace: katco natefinch mgz ping for standup [14:00] dimitern: ^ [14:00] hey dimitern, first of all thank you for making sure local provider works! I still have a couple of questions. Can you spare some a few minutes? [14:00] rick_h_: omw [14:00] hackedbellini: in a conf. call - will work through this again in about 30 mins [14:01] frobware: ok, thanks! [14:14] rick_h_: btw, show-controller only shows the current user even for admins [14:17] natefinch: ah, ok. That's the gap. [14:17] natefinch: k, let's punt this then please [14:18] hackedbellini: is your client Mac OS X (sierra) or Linux? [14:19] frobware: Linux. Ubuntu Xenial to be more specifically [14:19] hackedbellini: thx [14:19] frobware: going to launch now, brb in about 30 mins [14:20] hackedbellini: ack [14:20] frobware: dooferlad http://jeffreifman.com/2016/10/01/fix-macos-sierra-upgrade-breaking-ssh-keys/ reads a lot more clear [14:20] natefinch: can you check that juju creates/uses RSA keys please as a quick card? [14:20] natefinch: looking to work out if that link above is an issue with Juju [14:21] natefinch: frobware dooferlad I'm going to guess we've not been using DSA keys for along time. [14:21] kjackal: hey! can you send me those questions by mail please? too many things to fix today.. sorry :/ [14:22] yeap no problem dimitern I am trying to sort them out, let hope i will not send you any email :) [14:22] kjackal: ;) np, I'll have a look later this evening [14:27] rick_h_: sure [14:53] rick_h_: so I can repro the IPv6 bug on vsphere by the way [14:54] rick_h_: preparing more logging [14:56] frobware: I'm back [14:56] hackedbellini: on the phone still [14:56] frobware: np :) [15:10] lazyPower: this is still a problem? https://bugs.launchpad.net/juju/+bug/1426728 ? [15:10] Bug #1426728: Windows Charms are unable to raise error state

[15:10] perrito666 tbh i dont know, that issue is quite old now. I haven't deployed a windows charm in some time [15:12] * perrito666 is fishing for a bug to work on [15:13] so what I'm seeing is my logging never being hit and therefore machine addresses never being set! [15:26] rick_h_: GenerateKey makes a 2048 bit RSA no-passphrase SSH capable key [15:27] rick_h_: that's what we use for the bootstrap server [15:29] rick_h_: and I verified that the comment matches the implementation ;) [15:34] natefinch: ty [15:36] rick_h_: now what? [15:40] natefinch: look at the interive add-cloud and where we left off. [15:41] natefinch: maybe chat with katco back on the schema updares, i think she was out during some of tgat. [15:41] rick_h_: ahh, cool, yeah, I'd like to get that done [16:04] perrito666 - is this expected behavior when upgrading from rc-3 to 2.0.0? http://paste.ubuntu.com/23339291/ [16:05] i don't think its a problem, just double checking that i haven't managed to botch my controller by upgrading from rc's to 2.0 proper [16:08] seems fine, disregard the question. [16:14] hackedbellini: purged my system of LXD and other stale stuff. Get different result now. [16:14] hackedbellini: in fact, I believe I see your error [16:14] hackedbellini: http://pastebin.ubuntu.com/23339436/ [16:15] natefinch: I'm trying to update juju.fail since 2.0 is out, is 2.1 now "master? [16:16] frobware: yeah it looks like my error! [16:16] marcoceppi: hasn't been updated [16:16] marcoceppi: https://github.com/juju/juju/blob/master/version/version.go#L22 [16:17] marcoceppi: https://github.com/juju/juju/blob/staging/version/version.go#L22 [16:17] marcoceppi: oh.. hmm [16:17] marcoceppi: i think staging is what you want to track [16:17] yeah [16:17] just figured that out [16:17] marcoceppi: master will remain last released version so that we can do hotfixes [16:17] marcoceppi: master lags behind staging, staging is what master used to be [16:19] so I should track staging for blocking bugs? [16:19] or is juju.fail even needed anymore? [16:20] OMG [16:20] not sure.... rick_h_ ? ^ [16:20] I think we must still have blocking bugs at some point... but we PR into development, so if anything is blocked, that's what would get blocked [16:20] hackedbellini: I think I may have found at why. [16:21] marcoceppi: i'm not sure tbh. i think so bc we can once again have a concept of "blocking" for the staging branch; i.e. changes to develop can't land into staging if it's not blessed [16:21] gotchya, I'll try to sync up with the qa/release team on how blocking bugs are decided so I can update the site [16:21] New workflow is branch from staging, PR into development. if/when development passes CI, it's merged into staging. releases are cut from staging to master [16:24] frobware: really? Why? Is there a fix/workaround? [16:24] hackedbellini: so I'm now back to no address on second bootstrap. still investigating. === dames is now known as thedac [16:30] Brb lunch [16:43] hackedbellini: OK, I see the problem now. https://github.com/juju/juju/blob/staging/provider/lxd/environ_raw.go#L155 [16:44] hackedbellini: on your bridged configuration this is going to be whatever your DNS deems the gateway (IIRC, .4 in your case). there is no LXD listening there. [16:48] hackedbellini: the only unsatisfactory solution I have right now is reconfigure and use a NAT'd lxdbr0. :( [16:48] rick_h_: ^^ [16:49] frobware: :/ can we get the list of addresses that lxd is listening on and error/note when that's not legit? [16:50] rick_h_: I think it's more fundamentally broken than that. in the case where you do not use a NAT'd lxd bridge we need to understand the network config and not return the gateway address (presumably associated with the DHCP lease). [16:53] frobware: hrm that is sad :(. What do you mean by "NAT'd lxdbr0"? [16:53] frobware: is this a case where we can say it's not supported? I mean if you don't have a NAT'd lxd bridge doesn't that have impact on getting agents/etc? [16:53] frobware: or am I mis-understanding things [16:53] frobware: oh nvm, non-nat'd as in dhcp so it's on the main netwokr [16:54] hackedbellini: if you run dpkg-reconfigure you can say, "my bridge is lxdbr0", and when asked to configure IPv4 subnet, say "yes". This will detect an unused subnet and your containers will typically end up on 10.x.x.x/24. [16:54] rick_h_: correct, it's on your local LAN [16:56] rick_h_: this mode (use my hosts network bridge) is akin to the addressable containers we have on MAAS. The containers (nicely) end up on your local LAN. [16:56] frobware: right, gotcha. why does this not go boom on maas? because they're on a maas provider host vs being the provider? [16:56] rick_h_: yep [16:57] rick_h_: juju/provider/lxd/environ_raw.go [16:57] rick_h_: this code was changed recently [16:58] rick_h_: https://github.com/juju/juju/pull/6078 [16:58] rick_h_: though it's not really clear whether it made it better/worse. I suspect many people don't use this setup by default. Clearly we don't in CI. [17:02] frobware: ping [17:02] dimitern: need to EOD. quick? [17:02] frobware: I think I did all we discussed on https://github.com/juju/juju/pull/6454 [17:02] frobware: if you can +1 it, I'll set it to land [17:03] frobware: can wait till tomorrow ofc [17:03] dimitern: I would like to cast a fresh eye in the morning. ok? [17:03] frobware: sure, np [17:10] frobware: I see. But I won't be able to access my containers on my lan, right? [17:11] hackedbellini: not directly [17:11] :( [17:11] frobware: can I at least change it later after the bootstrap? [17:11] if not this is a _huge_ problem for me [17:12] hackedbellini: thinking... you could add another NIC to your profile which DHCP's off your LAN... [17:13] frobware: hrmm, how can I do that? [17:13] hackedbellini: let me futz for 5 mins or so... :) [17:14] frobware: k =P [17:14] hackedbellini: how much manual hackery could you put up with? [17:15] frobware: as much as needed, as long as this works :) [17:15] lazyPower: you are not upgrading to 2.0 but to whatever your local working copy of the repo is seems to me [17:21] hackedbellini: my /etc/default/lxd-bridge - http://pastebin.ubuntu.com/23339686/ [17:32] frobware: you want me to try to use it? [17:33] hackedbellini: I don't have any quick solution atm [17:33] hackedbellini: I was about to paste some more stuff, but it won't work with juju [17:36] frobware: I see. Let me explain how things are setup here so maybe you can suggest an alternative solution for me: [17:37] That gateway pc that provides dhcp (the .4) has apache running on it. All accesses to our services will go into it first, and apache will do a reverse proxy to the service in the container [17:37] but the containers are running inside a second machine, the .3, the one that I'm trying to bootstrap juju [17:38] so, that is the reason that the containers need to have access to my lan [17:38] hackedbellini: the containers are doing DHCP, correct? [17:38] or more specifically, my lan has to have access to the containers [17:38] yeah [17:55] I'm hitting a strange issue with the 2.0 GA release and lxd. xenial instances come up fine, but trusty units get stuck in 'waiting for machine' even though the lxd image is up and running. If I try to 'juju ssh' to the machine number or unit name, it says that the keys for that unit are not found, if that's useful. I don't see anything regarding that unit in the debug-log [17:55] This only started after upgrading to the GA [17:55] Could someone please offer some advice? [17:57] cory_fu: can you check for anything interesting in the lxc logs? [17:57] cory_fu: do they have network interfaces that can talk to the controller? [17:58] hackedbellini: so this may work... [17:58] hackedbellini: are you still about? [17:59] frobware: so lets try it :) [17:59] hackedbellini: http://pastebin.ubuntu.com/23339861/ - a new LXD profile [17:59] rick_h_: They do have network interfaces and networking seems to be working from what I can tell [17:59] rick_h_: Any particular lxc log I should be looking in? [18:00] hackedbellini: lxd profile create fix-up [18:00] hackedbellini: use that content, ^^ substituting your bridge name (IIRC, br0) [18:00] perrito666: going to be a min late [18:00] rick_h_: no worries [18:00] It would feel very weird if one of my meetings started on time these days [18:01] perrito666: it's getting them to end on time that's hard :P [18:01] frobware: and then I try to bootstrap using that profile? [18:02] hackedbellini: not quite. a little more involved unfortunately. [18:02] hackedbellini: you need to use the lxd-bridge config I gave you earlier [18:02] hackedbellini: sudo service stop lxd [18:03]