[11:51] frankban: ping, did you see the emails from Gary last night? [11:52] rick_h_: the ones related to the charm? [11:53] frankban: yes, the charm release [11:53] rick_h_: already replied, do you see the email I sent just some minutes ago? [11:53] frankban: I've got to get the little guy off to day care but I'll start looking for the webops upgrade process docs for jujucharms after I get back. [11:53] frankban: ah no, I've got a built in delay running offlineimap [11:54] nvm then ('ll see in 2-4min ish [11:54] how did I I/) ugh, morning [11:55] hah, or offlineimap hung on the server last night and I'm missing a lot of email [12:13] frankban: awesome email, one question back at you [12:13] bac: ping, when you get around this morning it appears charmworld hasn't ingested since the 20th. Did a deploy happen monday? Can you look into it? [12:14] rick_h_: thanks. i'll look into it later [12:15] rick_h_: deploy was on thursday, the 16th [12:16] bac: thanks. Off to run the boy to day care (and a coffee run) back in a bit. [12:42] rick_h_: thanks, replied [12:42] * frankban lunches [12:43] bac: how can I help with the ValueError problem? Perhaps I should attempt to reproduce it first. [12:43] benji: morning. i have discovered the problem. the int() was in jujuclient, a piece of the chain i hadn't looked at [12:43] cool! [12:44] ok, I'll do... the other thing I was going to do, whatever that was [12:44] benji: but the actual problem lies in the deployer [12:44] interesting [12:44] easy peasy fix, though [12:44] cool [12:44] hey, maybe i'll add some tests [12:46] but, benji & rick_h_, the solution will require getting changes made upstream into the deployer and then having the charm use that new version. [12:46] i don't see that happening before noon today [12:46] probably not [12:48] oh, look, i should read my email before starting my day. [12:51] hi frankban [12:52] bac: rgr, not a problem. frankban's got the charm updated and I'll look at getting it deployed [12:53] thanks for looking into the constraints issue. it turns out the deployer has a utility function called 'parse_constraints' that turns '1G' into a proper int value. making that call in action/importer.py before calling self.env.deploy does what we want === gary_poster|away is now known as gary_poster [13:08] bac: why is the deployer parsing these at all, shouldn't it delegate that to juju? [13:09] benji: the juju cli parses them. the juju api expects a uint64 [13:09] ah [13:09] that's unfortunate (because of issues like this) [13:10] next thing you know one of them will be parsing 1M as 1,000,000 and the other as 1,048,576 [13:12] bah, and #webops is a ghost town this morning [13:12] I want a long bell pull I can yank on. "The bosses are coming the bosses are coming. Get up and deploy!" [13:13] heh [13:17] gary_poster: backup plan to push it up to AMZ for meeting? [13:18] rick_h_: comingsoon still working; s'ok. I can still announce the release was made, and jujucharms in progress [13:18] thank you [13:19] ok cool, wasn't sure what extent was required for the meeting. [13:21] hey frankban, are you familiar with local.jenv/bug 1271341? Does quickstart handle this? [13:21] #1271341 mup? [13:21] oh, no mup [13:21] mup sadness [13:22] :-) bug 1271341 come on mup [13:23] * gary_poster pretends to be mup: https://bugs.launchpad.net/bugs/1271341 [13:24] gary_poster: looking [13:24] ty [13:25] "right-click, edit search engines, scroll way down, add new, [Launchpad Bugs, b, https://bugs.launchpad.net/bugs/%s] and done" [13:25] now just b1271341 [13:25] and chrome syncs all that across browsers/installs. I forget I set it up originally. [13:26] (and the right-click is on the address bar to bring up "edit search engins") [13:30] gary_poster: AFAIK the situation is the following: if the admin-secret is included in envs.yaml, juju uses that, otherwise a random one is generated. In both cases the admin secret is included in environments/[env name].jenv. quickstart is only aware of envs.yaml, and refuses to start if the admin-secret cannot be found there. when creating environments, quickstart always requires and includes the admin-secret fie [13:30] ld. not sure about whether we can consider jenv files an internal detail or not. anyway, a fix for that should be quite easy to add [13:32] frankban: sounds great, thanks. I will file a quickstart bug and make a high maintenance card to represent "we need to investigate this" (since IIUC you are not sure yet that we actually need to *fix* this). Sound right? [13:33] gary_poster: sounds perfect, in theory we could consider the mandatory presence of the admin-secret field just one of the many quickstart opinions [13:33] heh, agreed :-) [13:39] For the curious, https://docs.google.com/a/canonical.com/presentation/d/1zJGPoNp8PTJCekluHzVy5Dr6aKViNqrg5fkBA7IIWyQ/edit#slide=id.g25d342ae6_35 has the slide I'll use to present our progress. Comments/suggestions welcome. [13:40] gary_poster: maybe note on quickstart as "fastest way to node 0 hosted gui env"? or the like? [13:41] rick_h_: maybe replace the last quickstart bullet with that? [13:41] gary_poster: +1 [13:41] cool thanks [13:42] gary_poster: I forgot to mention in my email: while "juju-quickstart --gui-charm-url cs:precise/juju-gui-82" worked the GUI icon in the topology is a broken image. I suppose this is due to the ingestion problems in charmworld [13:43] huh [13:43] frankban: you must be right about ingestion problems. 81 is still most recent [13:44] frankban: yes https://manage.jujucharms.com/api/3/charm/precise/juju-gui-82/files/icon.svg [13:44] according to charmworld [13:44] gary_poster: I've put a card up in urgent for the ingestion issue. It's not run some the 20th [13:44] :-( ok thank you [13:44] at least according to heatbeat [13:45] heatbeat, the new way to dance to summer tunes that's sweeping the nation! [13:45] gary_poster: i have a deployer fix made and tested. i'm trying it out with our charm now. [13:45] bac, great thank you. that's for the issue you were investigating yesterday, rt? [13:45] gary_poster: yes [13:45] cool [13:51] jujugui, who owns the "investigate MJC not ingesting" card? May I have a volunteer? [13:51] gary_poster: i am looking [13:51] gary_poster, rick_h_ : i suspect it is related to a problem jcsackett saw and was solving. [13:51] jcsackett: you around? [13:52] bac, ok. you already have another card active though. worth dragging benji in? maybe not: sounds like multiple cooks already [13:52] and no vanguard in webops this morning to help debug the logs [13:52] argh [13:53] Makyo: welcome back! when you get in would like to talk about subdividing relationship line project into task cards. Afterwards we'll want to talk to rick_h_ about whether it should come before debug log, since it is not landing with speed. [13:53] (in juju core) [13:53] gary_poster: sounds good to me. I can be put to work. [13:54] :-) k [13:54] benji: do you have access to staging for charmworld? [13:54] rick_h_: for logs, if we need to, I'm +1 on escalating the issue [13:54] bac: last time I tried I was getting strange errors no one could help with [13:54] gary_poster: ok, I have an ops going to do the deploy soon (non-vanguard) [13:55] rick_h_, bac: if we decide we need mjc logs, please make RT then ping #webops and ping me [13:55] I can try to charm my way into getting a little extra if possible [13:55] oh ok [13:55] ok thanks [13:55] try to relate it in together and nab his time [13:55] :-) k [13:55] sounds like a plan [13:56] gary_poster, rick_h_: after a quick look at staging, i see that ingest is failing to start. the problem is the lack of an index, which is now occuring since the number of charms has grown and exceeded a threshold. this is the problem jcsackett is fixing. it is almost certainly the same as affecting production. [13:56] rick_h_: if you do get app.log from production it'll likely show ingest attempting to start every few seconds [13:57] bac: ok good to know [13:57] what index is lacking that's causing ingest not to start? [13:57] bac: mongodb or ES? [13:57] ok, thank you bac. If not obvious, I would categorize this as urgent/critical/stop-the-line/eek. Anything we can do to help move this along (short of running around like chickens with our heads cut off) makes sense to me [13:57] mongo [13:58] bac: ok, so we just need a migration in place to add the index and a deploy then? [13:58] rick_h_: pymongo.errors.OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit [13:58] we probably should hjave a retrospective as to why this didn't get caught sooner [13:58] at the very least with some kind of operational alert [13:58] bac: orly, now that's not something I've seen. Mongo up and quits because it's too long? [13:58] I will add this card [13:58] gary_poster: i was in the discussion with jc but failed to recognize it would cause production to fall over. [13:59] ack [13:59] happens. but production has been falling over since Monday [13:59] that's what seems to me to be the most obvious failure of some sort [13:59] that we ought to address for future [13:59] make sense? [13:59] bac: ok, it shows JC as out the last two days and back today [13:59] bac: the calendar that is [14:00] gary_poster: my troublesome bundle just deployed successfully. i'll propose the deployer fix upstream to hazmat and submit the charm change after the new tgz is built [14:00] fantastic thanks bac. perhaps two candybars are needed [14:00] rick_h_: it must've been an apparition then. [14:00] bac: lol ok. I was mainly looking to see if he's around today. [14:00] rick_h_: you can review the irc logs to see if i dreampt it up. :) [14:01] woooooo ghost of charmworld past [14:01] gary_poster: should someone phone jc to check his status? [14:01] bac, ok. will do so [14:02] bac: FWIW I replied to your email [14:04] bsc, called, left message :-/ [14:04] frankban: i made the change lower in the gui env [14:04] bac [14:05] bac: sounds good [14:10] frankban: mp here https://code.launchpad.net/~bac/juju-deployer/parse-constraints/+merge/202674 [14:11] hazmat: when you have a moment could you review https://code.launchpad.net/~bac/juju-deployer/parse-constraints/+merge/202674 [14:15] bac: that branch looks great [14:15] thx [14:16] gary_poster: wait on hazmat or release the charm with a forked version of deployer? [14:16] bac, forked ( :-( ) [14:16] k [14:24] hey bac: gary_poster|away just called me to ask that i speak with you. ingest issues on charmworld? [14:24] apologies if you've now been double pinged by me; not sure i was connected the first time. [14:24] hi jcsackett. yes, the problem you saw on staging has brought down production and now we need a fix asap [14:24] jcsackett: unsure how far you got. [14:25] bac: ack; brought down production, or just production ingest? [14:25] jcsackett: just ingest. hasn't happened since the 20th. [14:25] bac: i see. [14:26] jcsackett: but we're looking to do a juju gui charm release today and so we'd like to see it on charmworld [14:26] bac: ack. do you have a traceback from production, or are we just assuming it's the same issue (which is a very good assumption, i think). [14:27] jcsackett: just assuming it is the same based on same heartbeat output [14:27] yay jcsackett and thanks [14:27] jujugui: trivial charm review https://codereview.appspot.com/49720044 [14:27] jcsackett: no vanguards today so logs difficult :-/ [14:27] bac: on it [14:29] bac: done [14:29] thx [14:30] frankban: going to set up a call tomorrow with you and william (and maybe me) to review quickstart to talk through https://bugs.launchpad.net/juju-quickstart/+bug/1271341 and any other upcomg juju changes that might change/affect quickstart (e.g. the ssh key stuff that Tim is working on). I figured you would quickly walk through what quickstart does, noting needed conversation topics as you go; and then, as time [14:30] permits, try to resolve those issues. I think the review list is our first priority though, so we know where we stand. sound ok? [14:31] bac: take a look at https://code.launchpad.net/~jcsackett/charmworld/fix-enqueue-sort-failure/+merge/202678 for me? this should fix ingest, and we can try it on staging straight away. [14:31] gary_poster: sounds good [14:31] cool ty [14:32] bac: mind you, this doesn't add indexes, so we're still going to see some issues, but i'm uncertain of the best way to do that on production and this should remove the current problem. [14:35] jcsackett: looking [14:38] jcsackett: voted approved. didn't mark the MP as 'Approved' as it'll start the landing process. do it when you're ready. [14:38] bac: and away we go. [14:39] i'll monitor staging and let you know if it does fix it. then we'll have to hound someone to do a production release. [14:40] jcsackett: ping me if you see it merged [14:40] i'm tailing app.log so should notice when it actually starts ingesting [14:41] bac: ok. === rogpeppe2 is now known as rogpeppe [14:46] jcsackett: CI rejected your branch [14:46] bac: yeah, i saw. trying to open the jenkins data to see why. [14:46] tests pass locally. :-/ [14:49] rick_h_: jc.com on pyjuju is really surprising :-/ [14:49] frankban: yes, that was quite surprising and caught us off guard a bit [14:50] frankban: I thought that was upgraded months ago with charmworld, but maybe only one got done [14:50] rick_h_: yeah [14:50] rick_h_: thanks for updating the wiki page [14:50] frankban: np, I'll file an RT to get it moved to -core and upgrade should owrk better [14:50] rick_h_: at this point I'd also remove sandbox=true from there [14:50] of course I remove pyjuju support [14:51] sandbox? /me thought it needed that [14:51] I'll look at what all it does again. [14:51] rick_h_: yes, you are right, it was staging the missing bit [14:52] rick_h_: sandbox is ok. [14:52] frankban: right, staging messed it up initially. I missed that one while editing [14:53] bac: can you open the results? i think jenkins might actually be down. http://162.213.35.27:8080/job/charmworld-autoland-lxc/105/ [14:56] jcsackett: did you connect to the vpn? [14:56] bac: zuh? this may be a thing i missed. [14:58] jcsackett: where does that jenkins run? is it on canonistack? [14:58] jcsackett: the vpn is required for the qa lab, so nm [14:58] rick_h_: do you have a moment to discuss "Auto open/close sidebar when using flyout from unit/inspector"? [14:58] bac: i believe so. but that's a floating ip. [14:59] jcsackett: ask your fellow orangies [14:59] benji: I think that's getting punted. [14:59] benji: working with webops atm though, will need a bit to chat [14:59] gary_poster: ^^ [14:59] abentley: charmworld qa is on canonistack, right? [14:59] s/qa/jenkins/ [14:59] oooo-k [14:59] jcsackett: Right. [15:00] benji, it's a reasonable job separate from the project. [15:00] benji, want to call? [15:00] gary_poster: sure; how about the daily meeting hangout? [15:00] abentley: which environment? it looks like it may be down. i can't get a response trying to look at test results or anything else (162.213.35.27:8080/job/charmworld-autoland-lxc/105/) [15:01] benji, k (though apparenty it pings some people when we use that? ) joining [15:01] jcsackett: It's looked like it was down for weeks now. [15:01] jcsackett: So maybe they've gone ahead with their plan to switch to something else. [15:01] jcsackett: bac I've got the logs from prod charmworld [15:02] abentley: it seems to be running tests; it's approved earlier branches of mine and has rejected the most recent. [15:02] rick_h_: gimme. [15:02] :-P [15:02] frankban: the juju-gui charm is failing locally with pip install. perhaps it doesn't like the name i used juju-deployer==0.3.1-prerelease [15:02] jcsackett: bac join #cloud-dev on canonical? I don't think there's info in here but don't want to put it in pub irc [15:02] frankban: it worked fine with juju-deployer==0.3.1 [15:03] bac: uhm, what's the pip error? [15:03] jcsackett: the env was juju-gui-qa [15:04] jcsackett: it looks like a github issue [15:05] jujugui lf a quick review on a slight refactor of the drag and drop code https://github.com/juju/juju-gui/pull/83 [15:05] it's mostly code reorg [15:11] abentley: thanks. [15:11] looks like juju-qui-qa has lost its floating ips. [15:13] gary_poster, rick_h_ sounds good. Let me take care of filing vacation (whoops), then I'll be all caught up. [15:15] :-) k thanks [15:18] hatch, I have a callin 12. I'll try to get review done before then [15:21] gary_poster, accidentally filed MLK as PTO instead of nat'l holiday, just deny that one. [15:21] ack [15:27] jcsackett: what's the story with charmworld CI? [15:27] bac: still can't see the results, but i think it's a lint failure. [15:28] pushing up a change now. [15:28] cool [15:28] do i need to re-approve? [15:28] tarmac used to be pissy about such things [15:29] gary_poster: juju gui charm on LP is currently broken. new version being proposed now. [15:29] Makyo: approved/rejected everything as appropriate; sent email to Sarah for 2013 days [15:29] gary_poster, cheers, thanks. [15:29] ack thanks bac. will ask for details on daily call [15:30] hatch, looked good but ran out of time. get someone else? or I will return to it after daily call [15:30] frankban: can you look at this https://codereview.appspot.com/49720044 trivial review? [15:30] gary_poster, just curious, can I claim airport parking? There's no public transit to the airport from FoCo. [15:31] Makyo: yes [15:33] FoCo....haha [15:34] ? [15:34] Fort Collins, sorry :) [15:34] FoCo, NoCo. [15:35] yeah I guessed that [15:35] we just say Saskatchewan to laugh at people who can't pronounce it [15:35] lol [15:36] bac: done [15:39] bac: branch has been merged; should be on staging shortly. [15:39] jcsackett: cool [15:42] jcsackett: new version running: http://paste.ubuntu.com/6797831/ [15:43] bac: that lock thing doesn't look good... [15:43] nope [15:44] is it just thrashing? [15:45] jcsackett: i'm not sure what is happening. if it grabbed the lock but then errored due to the lack of index it should've released the lock. so i'm not sure why it is starting again with the lock held [15:45] * jcsackett nods [15:45] the lock should release in about 10 minutes. [15:45] jcsackett: i'm going to watch it for a bit. if i need to manually restart ingest i can [15:45] based on the timestamp. [15:45] yeah, they have a 15 minute cycle [15:47] gary_poster: changes landed to juju gui charm at r155. https://code.launchpad.net/~juju-gui/charms/precise/juju-gui/trunk [15:47] ty [15:48] bac: gary_poster note that we can't release that until we get the env to juju-core [15:48] bac: so left a 'deploy jujucharms' up there. If we can do another charm release, once I get through everything we can look to deploy that updated charm release. [15:48] ...on call confused [15:48] will focus soon [15:49] rick_h_: i'm confused to by 'get env...' [15:49] bac: yea sorry. Figured I'd go through it on the call [15:49] ok [15:49] jujucharms.com is on pyjuju, the charm no longer supports it. Things go boom right now [15:49] jcsackett: we really should make those times human readable... :( [15:50] jcsackett: time outputs in the log, i mean [15:50] * rick_h_ is changing locations pre-call [15:50] jujugui call in 10 [15:54] jcsackett: stale locked removed on staging and it is now ingesting [15:54] bac: whoo! [15:54] jcsackett: http://staging.jujucharms.com/heartbeat [15:55] bac: already have it opened. [15:55] bac: so we need a charm upgrade and deploy done on production. [15:55] jcsackett: but is it the same issue on production? [15:55] bac: i can't tell. the log i got is corrupted. [15:58] jcsackett: for production we just need an update of the charmworld code, the charmworld charm is ok. correct? [15:58] bac: no, a revision i landed requires a charm update so the ini file is updated. [15:59] jcsackett: ack [15:59] bac: you're making the request of webops? [15:59] jcsackett: i will [15:59] jujugui call in 1 [16:00] ty [16:01] Makyo [16:02] Sorry, got signed out [16:02] jcsackett: the tarball didn't work I sent you? [16:02] rick_h_: no, the bunzip command says the data is corrupt. [16:03] jcsackett: it's tar bz2? [16:03] * rick_h_ looks up bunzip [16:10] jcsackett: email on the way untar'd [16:25] is anyone else getting an email about the daily call being changed every day? [16:28] benji: ping if you want to chat [16:28] hatch: if you look I think it's for dates far in the future [16:28] rick_h_: I'll be there in a sec. [16:28] like on holidays or something. At least the last one I noticed was Oct or something? [16:28] hatch, the bluefin thing? yeah/ [16:28] I could uninvite it :-) [16:28] rick_h_ yeah it just shows a slew of dates and I can't see anything different, but you're probably right [16:29] rick_h_: this log worked. [16:29] unfortunately, it makes it look less likely that my fix will sort out production, since i don't see the same error. [16:37] Vimeo tab crashed 55% of the way through uploading, argh. [16:37] I kind of hate computers. [16:42] you sure it's not Vimeo? It sometimes stops playing a video half way through too lol [16:44] heh [16:44] Nah, this was a chrome crash. [16:44] boo urns [16:49] sigh, toilet clog, overflow, upstairs, sigh [16:50] back to review [16:51] those are the worst! [16:51] yup :-) [16:52] I like the new toilets though, because they use such little water it would take like 3 flushes to overflow :) [16:52] lol [16:52] 1960s toilets up here [16:52] at least here, I don't think we can even buy a 'traditional' toilet anymore [16:52] we use such little water that they had to increase our water bills because they weren't able to support the infrastructure lol [16:52] that's cool, mostly! [16:53] downside to being 'green' I guess haha [16:54] hatch +1 on code. do you need a qa for that? I have call in 6. [16:55] gary_poster I qa'd and it worked so if you want to live on the edge....no :) [16:55] Heh ok, I think the edge will be where I live then :-) [16:56] hatch trivial comment request. shipit otherwise. If you really wanna you could push comment into next branch. :-) [16:56] sounds like a plan thanks [16:56] manage.jujucharms.com api down? [16:56] jcsackett: bac rgr, that was my impression that this looked different [16:57] marcoceppi: we are def having issues. yes, looks like it. bac, ^^^ [16:57] gary_poster: cool, as long as you guys are aware :) [16:57] not best timing in the world :-P [16:57] thanks marcoceppi [16:57] marcoceppi: gary_poster looks like deploy is going on [16:58] chat in #webops on it [16:58] rick_h_: oh ok [17:01] marcoceppi: back [17:02] \o/ [17:09] frankban you still around? [17:10] hatch: yes [17:10] so I'm following the README in juju-core which says to use `go get -v launchpad.net/juju-core` but after that go complains that there are no Go source files (which is true) so the readme I'm assuming is missing a step/ [17:10] ? [17:13] do you have $GOPATH set up? [17:13] yeah it downloaded all the files [17:14] I am trying again on a fresh instance but I'm guessing I'll have the same outcome [17:16] hatch: have you created the $GOPATH directory? [17:16] mkdir $GOPATH [17:16] yeah.... oh you know what I did? I forgot the ... after the url [17:17] I'm not really sure what that does or why it's required [17:17] go get -v launchpad.net/juju-core/... [17:17] hatch: it means get recursively everything [17:17] normally it only gets the top level? [17:18] bac, rick_h_, gary_poster: manage.jujucharms.com appears to be ingesting. [17:18] yay! [17:18] hatch, rick_h_: I also set up a bzr lightweight checkout following http://bazaar.launchpad.net/~go-bot/juju-core/trunk/view/head:/doc/bazaar-usage.txt [17:18] thank you jcsackett, bac [17:18] if you do that you can skip the cobzr instructions [17:19] frankban so is that a thing with `go get` that if you don't put the ... at the end then it won't get anything deeper than a single level? [17:22] jcsackett: what was the mjc error about? [17:22] jcsackett: and yay [17:23] hatch: see "go help packages" [17:23] hatch, rick_h_: so basically IIRC I followed http://bazaar.launchpad.net/~go-bot/juju-core/trunk/view/head:/README#L22 (from line 22) and then http://bazaar.launchpad.net/~go-bot/juju-core/trunk/view/head:/doc/bazaar-usage.txt [17:26] rick_h_: so, the problem my branch fixed was removing a sort on a query that we didn't need, b/c the collection is too big to sort without an index. [17:26] that was observably the problem with ingest on staging. [17:26] not to so much on prod, but it appears to have fixed it. ...or there was an unknown issue that updating/upgrading cleared. [17:28] rick_h_: was marco's API issue transient due to the upgrade? [17:29] bac: yes, I believe so [17:29] cool. just reading backwards after being afk [17:30] jcsackett: ok, so a restart or something might be hiding a production error? It was 'fixed' as a by-product of updating? [17:31] rick_h_: well, we know the index issue was real and present. unsure what, if anything, might still be lurking [17:31] bac: rgr, ok [17:31] rick_h_: but to reduce the variables we did the deploy and it now seems (temporarily) happy [17:37] rick_h_ just fyi, super easy to compile juju if you don't make any stupid assumptions that the docs were wrong [17:37] :P [17:38] hatch: lol [17:47] rick_h_ my vm thinks that there are branches on the remotes/origin (my fork) that don't actually exist is there any harm in deleting them as if they do? [17:47] at least from what you may have run into in the past [17:47] hatch: I'd want better details [17:47] what's the output of git branch -a [17:48] oh `git remote prune` I think I'm supposed to use [17:49] yea, if you don't clean up your branches when they're merged they hang around and that's valid [17:49] yeah the issue is that they don't actually exist but it thinks they do [17:49] this is the part that confuses me [17:49] so it must just be stale and doesn't actually update on pull [17:49] they don't exist locally? [17:49] git fetch updates [17:49] nope, or remotely [17:50] just run a git fetch and git branch -a should be good [17:50] yeah no luck, they are still there [17:50] what is one of their names? [17:51] remotes/origin/viewlet-test-scaffold [17:51] which doesn't exist in my remote fork [17:51] or locally [17:51] that shows up in git branch -a? [17:51] yup [17:51] there are a lot here [17:51] which fall into the same category of "don't exist" [17:52] so a fetch/pull must not update the remote list [17:52] git remote show origin [17:52] hatch: did you add another remote that has those branches? [17:52] nope [17:52]