=== gary_poster is now known as gary_poster|away [12:18] frankban: I'm trying to run the functional tests and getting errors on "for pe in os.environ['PATH'].split(os.pathsep):" which seems odd? Seen that before? [12:28] rick_h_: functional tests for the GUI? no, never seen it [12:30] frankban: yea, they all failed in the FF selenium stuff on PATH keyerror. Trying again [12:31] rick_h_: uhm... so in the environment the tests are run on the path is not defined, weird [12:31] wondered if I had something missing, will keep poking at it [12:32] brb daycare run [12:33] benji: jujucharms.com is updated so moved the check the etags over. Can you check the etags while I get the charm updated and hopefully we can close the book on the move/upgrade today. [12:34] rick_h_: sure [12:38] rick_h_: 100% consistent etags and they are in the right format (which is different than they were before) [12:53] benji: rock on, thanks for the check [12:54] np === gary_poster|away is now known as gary_poster [13:10] * gary_poster adds quiet yay in benji's direction :-) [13:17] can I not restart juju-gui with normal service tools? Does it need to have an upstart job to manage it? [13:18] gQuigs: the gui is run via an apache install. So you have to restart apache. The gui is just some JS that talks to Juju. Reloading the browser reloads the Gui [13:18] gQuigs: what are you trying to do? [13:19] heh [13:20] rick_h_: just curious how it starts up and is managed; sometimes it was coming up, but not ever finishing connecting to the environment [13:22] gQuigs: so yea, when you load the browser it tries to talk to the juju environment. That's purely the browser trying to setup a websocket connection to the charm and from the charm to the juju api endpoint in the environment. [13:26] gQuigs: on the charm/server side, the upstart job is named guiserver, so "service guiserver restart". logs can be found in /var/log/upstart/guiserver.log [13:27] We've never seen an issue there though, have we? Knowing more about the symptom he describes would be valuable [13:28] no issues I've heard. My first reaction is local network issues during wss setup? [13:28] rick_h_: frankban, thanks both.. this helps a lot [13:28] gary_poster: agreed. gQuigs ^^^: what browser are you using? does it happen after e.g. cleaning the cache/running in incognito mode? [13:29] gQuigs: sure thing, let us know if you hit more issues we can watch for. We've not heard of any issues, but of course want to make sure we catch anything early [13:31] we have heard of occasional issues on MaaS. those have seemed to be related to networking. Getting a test MaaS cluster might be a nice idea eventually. [13:31] frankban: was using Firefox stable on Ubuntu 13.10. I'll see if I can repdoduce reliably or if it was a one ff [13:31] cool thanks gQuigs [13:32] frankban: are you on trusty or saucy? [13:32] rick_h_: saucy [13:32] rick_h_: I have a trusty vm [13:52] * gary_poster steps out. biab [13:56] frankban: if you get a sec can you run the tests on charm trunk on both to see if you can dupe my issue on the trusty side? I'm getting it on both laptop/desktop. [13:56] I'm working on running hte tests manually so I can try to debug vs wrapped in juju-test, but a sanity check on your env would be appreciated. [13:58] rick_h_: sure, so "make ftest" on the charm in both saucy and trusty, right? [13:59] frankban: yes please [13:59] the hope is that saucy passes and trusty fails and gives me a hint that it's trusty package related [13:59] rick_h_: started tests on saucy [13:59] vs some mystery thing in my install that causes it to not work [14:04] rick_h_: frankban the issue appears once the juju-gui VM has been restarted.. it just hangs at connecting to environment (has happened 3/5ish reboots) [14:05] gQuigs: VM restarted? [14:05] gQuigs: what are you deploying to? [14:07] rick_h_: openstack [14:07] gQuigs: so you're restarting the machines then? and on restart it fails to connect at all? Does it come up eventually? [14:08] rick_h_: io've givenb it 5 mins [14:09] gQuigs: ok, so next time it fails, see if you can ssh to the machine and see if the guiserver is running via the upstart script frankban mentioned. I wonder if that's not coming back up for some reason. "service guiserver restart" [14:09] gQuigs: also appreciate if you can open firebug or the javascript console and post any error output from there when it's not connecting [14:10] gQuigs: also, does the bootstrap node ip address change between restarts? [14:11] gQuigs: between machine reboots [14:12] frankban: no, no IPs change [14:14] rick_h_: Firefox can't establish a connection to the server at wss://10.55.60.229/ws. all-yui.js:28 [14:14] 09:14:06.710 The connection to wss://10.55.60.229/ws was interrupted while the page was loading. all-yui.js:28 [14:14] gQuigs1: ok, so that is the IP of the gui unit? [14:14] yes [14:15] ok, yea so if you can juju ssh to that and check if guiserver is running "ps aux | grep guiserver" [14:15] gQuigs: and if not, try to restart it to see if that solves the issue [14:15] "service guiserver restart" [14:18] hmm... nothing running with guiserver in the name, but: [14:18] $ sudo service guiserver restart [14:18] guiserver stop/waiting [14:18] guiserver start/running, process 1844 [14:18] ubuntu@juju-openstack-machine-1:~$ ps aux | grep 1844 [14:18] root 1844 2.3 4.5 152996 23032 ? Ss 14:16 0:00 /usr/bin/python /usr/local/bin/runserver.py --logging=info --guiroot=/var/lib/juju-gui/juju-gui/build-prod --sslpath=/etc/ssl/juju-gui --charmworldurl=https://manage.jujucharms.com/ --apiurl=wss://10.55.60.220:17070 --apiversion=go [14:19] gQuigs1: ah sorry yea. [14:20] gQuigs1: is 10.55.60.220 the ip of the bootstrap node? [14:20] hmm, so that's running. and you can't connect to it. The ip of the machine is the same. [14:21] gQuigs1: anything of interest in the /var/log/upstart/guiserver.log? Especially since reboot time? [14:21] frankban: yup, and that did not get restarted [14:22] gQuigs1: just to exclude any client issues, does it work with chrome in incognito mode? [14:25] frankban: interesting.. it worked after I disabled all of my firefox add-ons [14:26] frankban: no it must have been a cache thing that required a reboot to clear... weird [14:26] (private browsing mode didn't do enough...) [14:26] thanks rick_h_ and frankban! [14:27] gQuigs1, rick_h_ : uhm... so our websocket client impl is affected in some way by ff cache... [14:28] rick_h_: we should be able to dupe that stopping and restarting a GUI lxc in a local env [14:28] frankban: something on the network? [14:28] gQuigs1: welcome [14:28] frankban: yea, should be simple to test out, I'll add a card to investigate [14:28] rick_h_: thanks [14:29] anyone know a good big data bundle offhand? [14:29] jcastro: there's one sec [14:29] or better yet, know a trick to have the GUI show me all the bundles in the store? [14:29] jcastro: under dev right now [14:29] (the second part) [14:30] ack [14:30] jcastro: http://comingsoon.jujucharms.com/sidebar/search/bundle/~gary/demo/2/instantBigDataNoSQL/?text=instantBigDataNoSQL [14:30] jcastro: is one that maarten was asking about at CT [14:30] perfect [14:35] rick_h_: charm ftests passed on saucy. I'll set up the charm on trusty in a minute [14:35] frankban: thanks [14:43] * gary_poster back [14:46] marcoceppi, jcastro, benji: the promulgation marking for bundles in charmworld is broken and i'd like to nail down the rules with you. [14:47] sure [14:47] bac: sounds good [14:47] for a charm, it is marked as promulgated if the series==distroseries in the branch data. [14:47] yesterday marcoceppi suggested any bundle owned by ~charmers should be considered promulgated [14:47] is that true? [14:49] imo, yes, but I'm biased as that's what I suggested [14:50] marcoceppi: well, nows a good time to rethink it! :) [14:50] i mean, if you need to [14:50] what says jcastro? [14:51] (thinking) [14:51] the reason we went away from ~charmers for charms was so that others could own the charm (like ~juju-gui) [14:51] so do we expect the same for bundles? [14:51] yes, eventually [14:51] right, but I dont' imagine bundles would get as much attn [14:51] or not [14:51] then let's not go through that again [14:51] like, if I was MySQL and I wanted to own the bundles for MySQL [14:51] rick_h_: i do not fully understand the series == distroseries test for charms [14:52] marcoceppi, I would argue that bundles will and should get all the attention [14:52] jcastro: right, but they're probably more important and arguably more static than charms are [14:52] true [14:52] bac: it's LP workings. I just know originally we had the rule that chrams owned by ~charmers were promulgated, but some LP mechanics allowed that to work for others to own the promulgated one [14:52] I think having them owned by ~charmers is fine [14:52] * rick_h_ never got all the series level stuff in LP [14:53] I'd argue that I can't see a circumstance in the near future where a team would want to own a bundle [14:53] marcoceppi: so no sales team or open stack team or something? [14:53] or ~jujugui [14:53] rick_h_: they can author bundles all they want, but if they want to put them in the store it needs to be vetted by charmers [14:53] I'm all for simple if we can do it, just want to avoid migrating again if we can. At least for a w2hile [14:53] yeah ~charmers will be fine [14:53] marcoceppi: hmm, that's not going to be true in 6mo [14:54] well 8+ [14:54] rick_h_: oh? [14:54] if we ever have a problem with bundle ownership it will be a good problem to have [14:54] but we can deal with it then [14:54] rick_h_: what happens in 8mos? [14:54] Rick is rewriting the store in Erlang [14:54] j/k [14:54] woot! [14:54] haha [14:55] ok so any bundle owned by ~charmers will be promulgated. the charmers will need to understand not to push any up that aren't of that quality [14:55] bac: correct, which is current policy [14:55] cool, thanks [14:55] bac: [14:55] rick_h_: [14:56] did you have more? [14:56] bac: no, sorry. Meant to be "cool thanks bac" [14:56] marcoceppi: well, it is theoretically current policy. no code yet to make it happen. :) [14:56] but I keyboard fail [14:57] and are next to each other on this kenisis [14:57] bac: well, we have that policy for charms, so the same would apply to bundles, and given how small the ~charmers are it's pretty well enforced [14:57] rt [14:57] cool [14:58] frankban: ok, got tests passing in trusty by manually bootstrapping, making sure precise is the default series, and manually running JUJU_ENV="ec2" ./tests/20-functional.test [14:58] frankban: so closer yay [15:02] rick_h_: oh, so the problem could be juju-test not propagating PATH? [15:02] frankban: right [15:02] frankban: in trusty [15:02] I'm setting up a saucy lxc to see if it works for me there [15:03] rick_h_: where are you taking juju-test from? [15:03] frankban: good question [15:04] rick_h_: mine is from bzr+ssh://bazaar.launchpad.net/+branch/juju-plugins/ [15:05] frankban: is it part of charmtools or something? [15:06] rick_h_: not sure, I just checked out the juju-plugins branch and made a symlink like /home/frankban/bin/juju-test -> /home/frankban/devel/juju-plugins/sandbox/plugins/juju_test.py [15:06] so yea, I've got it from the juju stable ppa version of charm-tools for trusty [15:06] http://bazaar.launchpad.net/~charmers/charm-tools/1.2/view/head:/setup.py [15:08] rick_h_: mine has no version :-/ [15:08] 1.2.8-0ubuntu1~ubuntu14.04.1~ppa1 here [15:09] juju-plugins hmm [15:09] that's a diff repo entirely [15:09] marcoceppi: ^ juju test from juju-plugins vs charmtools? discuss? [15:10] rick_h_: only charm-tools, juju-plugins is depreciated and if I knew how to delete a project on LP I would [15:10] frankban: ^ [15:10] marcoceppi: k [15:10] marcoceppi: so we've got juju test tests that fail because PATH isn't in os.environ on the newer stuff? [15:10] marcoceppi: sound familiar at all? [15:11] rick_h_: ok, so I'll re-run the tests on saucy using charmtools [15:11] probably not if the tests didn't hit os.environ I'd guess. [15:11] rick_h_: there's a whitelist of envs that are set during the test run [15:11] frankban: k, sounds like we'll need a card to update the tests or setup to work in the newer stuff [15:11] marcoceppi: orly, cool Sounds like we might need to add one then [15:11] or discuss why PATH is a bad one not allowed [15:12] things are making a lot more sense now :) [15:12] rick_h_: yeah, I'll expose an environment variable/flag for additional params [15:12] rick_h_: not that it's a bad one, just one that didn't have a use case [15:12] idk who put juju/charm-tools up but that's pretty out of date [15:12] marcoceppi: rgr, ok. That explains why it's not set. Seemed a really strange thing to have no PATH but having juju test kill it explains it [15:13] rick_h_: the idea behind that was to provide as sterile as an environment as possible when running the tests as to not have one users env cause a test to pass/fail [15:14] charmtools/test.py has the ENV_WHITELIST which is just hardcoded atm [15:14] frankban: ok, so I think we can kill off the debugging now [15:14] frankban: I've added a card to update the tests and possibly charm-tools [15:14] rick_h_: actually, PATH is in the whitelist [15:14] so, not sure what is breaking then [15:14] uhm... [15:15] marcoceppi: hmm, in our tests it throws a keyerror on os.environ['PATH'] [15:15] rick_h_: full trace? [15:15] bah, give me a few min to restart a test run (takes a few min in ec2 land) [15:15] rick_h_: ack, thanks! [15:16] rick_h_: OH, wait, I don't think that patch as landed yet [15:17] marcoceppi: ah, phew. [15:17] rick_h_: I can cut a release for you now if you'd like [15:17] marcoceppi: yea, I'm pulling from the stable ppa [15:17] ah, yeah, that's only in trunk atm [15:17] so I'm on 1.2.8 I think I had up there [15:17] ok cool [15:17] marcoceppi: yea, a release with that fix would be useful. [15:18] yup, this would be a 1.2.9 release, will do a check for anything else that needs to land and a release [15:18] marcoceppi: been chasing tests all morning [15:18] marcoceppi: thanks, appreciate it [15:18] ah, sorry about that! [15:19] marcoceppi: no hurry, now that I know what's up I can work around it for the moment. [15:39] hazmat: for when you are back and have time: could you please take a look at https://code.launchpad.net/~frankban/juju-deployer/unit-errors/+merge/206967 ? it's my proposed fix to bug 1279075 and bug 1252301 (the deployer part). Thanks! [15:39] <_mup_> Bug #1279075: Deployer should not stop when a unit has an error [15:39] <_mup_> Bug #1252301: guiserver reports second bundle as failing after the first fails

[15:40] hazmat: we've got this on board for thurs release so appreciate look in next day ish if you can. [15:40] hazmat: to beat freature freeze and release with new gui/charmworld this week. [15:42] hazmat: and while you're in the deployer mindset, if you could look at my branch too that would be great. [15:42] https://bugs.launchpad.net/juju-gui/+bug/1277696 [15:42] <_mup_> Bug #1277696: Bundles with revision-less charm URLs should deploy [15:53] frankban, bac, rick_h_ ack.. it will have to be tomorrow morning i've got a customer deadline i'm racing against today [15:54] hazmat: understood thanks [15:54] thanks [15:54] ty [15:54] jujugui call in 6 [15:55] * rick_h_ needs to set a 10min alarm [16:00] jujugui call in 1 [16:01] Makyo: standup ping [16:01] benji: beep [16:02] 2fa sorry :T [16:05] benji, https://github.com/juju/juju-gui/pull/130 [16:06] thanks [16:29] Makyo: review done; looks good. [16:29] benji, thanks [16:30] rick_h_, for your blog post [16:30] don't do mediawiki [16:30] too boring [16:30] do something epic like the big data one [16:30] or the mongodb cluster [16:37] jcastro: k [16:37] jcastro: for the 'explain' was that a side note? [16:38] yeah [16:38] marcoceppi: I have a charmtools branch up for review which adds a warning for bundles that include charms with revisionless URLs: https://code.launchpad.net/~benji/charm-tools/add-versionless-charm-in-bundle-warning/+merge/206986 [16:38] so like if I rerun quickstart it relaunches the browser? [16:38] yes (it rocks) [16:38] jcastro: so when you run quickstart it looks "do you have an environment? If you do, does juju status show a gui? If it does, where is that gui at?" [16:38] jcastro: +1, why i said in the blog post, it's the fastest way to get back into your env days later [16:39] jcastro: so one more reason for you to <3 and use it [16:39] benji: I figured you guys would put this in the online proof instead of the offline, but works for me. I'll review shortly [16:39] cool [16:42] jcastro: aside from that, is the blog post ok for what they're wanting? I couldn't tell if they want one, two, etc? [16:57] rick_h_ you have a blog? [16:59] hatch: yea, though don't hit it all that often [16:59] cool what's the url? I wana see this blog post :) [17:00] hatch: oh, this is a doc not submitted yet [17:00] ohh ok [17:00] hatch: https://docs.google.com/a/canonical.com/document/d/1ma0U1ZxILTh5s3NoHiuwKiclRSxJLIY-Q2MEMzBO7Dk/edit [17:00] cool I'll check it out [17:01] I used quickstart yesterday from scratch it was pretty awesome using that terminal gui thing [17:01] is that a python lib? [17:01] to make the gui that is [17:01] ascii gui :) [17:02] ncurses [17:02] urwid is the python lib to help make it usable [17:03] cool I didn't know about that [17:03] * rick_h_ shakes head at the idea of someone that never installed debian/ubuntu using the text installer [17:03] jcastro: is winning, cli deprecation goes on and on [17:05] rick_h_ Well my first version of Ubuntu was Warty so I'm pretty sure I used the text installer :) [17:06] hatch: ah ok then it was curses based cli [17:06] do a server install sometime and get to see it [17:06] but anyway, yea quickstart curses for getting going is coolness [17:07] kind of caught me off guard because I remembered the cli q/a version :) [17:07] I was like....woah what's this? haha [17:07] man that same tests fails every other run in CI. [17:08] Makyo: that test failure is spurious. heads up :/ [17:08] Boo, okay. [17:13] rick_h_ it's in a different test this time :/ [17:13] hatch: it's been that one for the last 4 or 5 times it's failed I've seen it [17:13] ohh I saw it in a few others [17:13] hatch: it's always that IE10 setHTML error [17:14] it's odd how it says there are so many failures but there is just the one [17:14] and yes it's always the setHTML error but I've seen the same error in different tests [17:16] hatch: hmm, went back through last 4 failures and it's always that test and always that error [17:16] really? I thought that it showed up in others...maybe I was mistaken [17:17] well at least then maybe we can track down the issue [17:17] hatch: yea [17:19] oo this looks cool http://www.indiegogo.com/projects/fin-wearable-ring-make-your-palm-as-numeric-keypad-and-gesture-interface [17:35] * rick_h_ runs out for lunch today, biab [18:08] Ugh, finally. [18:54] rick_h_: I have some questions about this bundle search task; do you have a minute? [19:11] benji: sure thing [19:11] rick_h_: I'll make a hangout [19:12] thanks [19:20] benji: would you be able to review charmworld promulgation fix? https://codereview.appspot.com/65550043 [19:22] bac: sure, one sec [19:32] bac: looks good [19:33] cool [20:24] benji: i also killed that annoying message from enqueue, telling you all of the branches it was skipping. hope you don't miss it. [20:24] there goes my Saturday night [20:24] * rick_h_ sends cookies to bac [20:25] we even had a test to ensure the output was sufficiently annoying [20:26] benji: do you need a review? [20:27] bac: nope (unless your name is marcoceppi ) [20:27] benji: yes, swamped, will have it reviewed today [20:27] bac: once you're ready for a new task I have one for the bundle searching things [20:28] benji: i am. what you talking about? [20:29] benji: i was thinking about quickly doing the normalizing to lowercase one [20:29] bac: well I'm trying to figure out the make-bundles-appear-in-search-results-when-one-of-their-charms-is-searched-for, so you can help me with that or... [20:29] benji: did you get a clearer idea of what to do? [20:30] benji: but i'm glad to talk about it [20:30] * bac beverage run [20:30] that's cool, another quick one (and one of the top-two in priority) is makeing "bundle(s)" return all the bundels (and "charm(s)" return all the charms if you are in a giving mood) [20:30] yeah, I know what, I'm trying to figure out how now [20:31] benji: one note, the next deploy we need to bring up a new charmworld unit to replace the bad one running in prodstack [20:32] benji: at that time, we could bring up a new unit and wait for the index to update (the 30min) and then switch over [20:32] benji: it draws out the deploy, but reduces that downtime issue without an index [20:32] ah, good thinking [20:34] jujugui tiny review and QA! https://github.com/juju/juju-gui/pull/134 [20:34] Though... hmm. [20:34] Makyo: looking ... or not [20:34] rick_h_, take a look and see if that passes. It can be incremental. [20:34] * Makyo makes follow up card. [20:35] benji: quick call? [20:36] Makyo: seems ok here. What's missing? [20:37] rick_h_, the fix works around the fact that relations in the vis and relations in the bundle have the same dom id. [20:37] Added a card to ensure unique dom ids. [20:37] oh! [20:37] benji-cito: https://plus.google.com/hangouts/_/calendar/Z2FyeS5wb3N0ZXJAY2Fub25pY2FsLmNvbQ.t3m5giuddiv9epub48d9skdaso [20:37] gotcha, cool [20:37] But this will be a good increment. [20:38] Makyo: is this something we can add a test for? or some sanity check for future reference? Or if the dom ids were unique we'd not hit this issue so that's the root cause/fix [20:41] rick_h_, the latter, yeah. I'll make that my next card, and that part will be testable. [20:42] Makyo: cool thanks. :+1:'d the current branch [20:43] rick_h_: can you join benji and me in the hangout above? [20:43] bac: joining [20:48] so benji i will start on the bundly one [20:49] bac: ok, I guess I'll do the lower-case-all-the-things one [20:49] benji: and i added your face to the pumpkin colored card [20:49] heh [20:50] i guess we can kill it now? we have cards for the required items [20:52] bac: +1 it was there to evaluate what we could/should do. [20:53] I'm out. Have a good night all. [20:56] yeah [21:13] jujugui this stupid IE10 CI thing is really biting me. Anyone have any hints? [21:34] Makyo: What is this IE10 CI thing? I've got some experience with automated browser testing thanks to my last job [21:36] Makyo: grrr, not yet. It was something hatch could not reproduce locally so his branch went on [21:36] Makyo: and now it's annoying everyone [21:36] lazyPower, "Error: Object doesn't support property or method 'setHTML'", but unable to repro locally. [21:36] Makyo: the key thing for retrying is to remove the comment in the pull request that's Status: merge request accepted [21:36] Oh, okay. [21:36] Makyo: that causes the lander to not try to load it again [21:36] so just delete that comment and it'll retry [21:37] Aha! [21:37] putting a card up for cleaning up this, must be some way to test [21:39] There we go. [21:39] rick_h_: do you have a few seconds for a hangout with regards to promulgated charms and me about to make a mess of that? [21:39] Makyo: there's some notes in the CI doc for the future [21:39] marcoceppi: sure, sec [21:40] rick_h_, cool, thanks. Will rtfm next time :) [21:41] marcoceppi: hangout? [21:43] Makyo: all good, I cheat by writing it so I know the tricks [21:43] rick_h_: https://plus.google.com/hangouts/_/7acpja2hqsviddhvokd1jo82pc?hl=en [21:44] marcoceppi: sec, fighing the 'not allowed' get your google account right mess [22:18] Makyo: yay [22:18] I watched it so of course not it passed [22:21] Woo! [22:21] Alright, dogs walked, quick errand, then I'll get the dom ids. [23:09] rick_h_: you still around? [23:11] marcoceppi: kinda [23:11] marcoceppi: what's up? [23:12] rick_h_: ah, don't mean to be a bother, re-promulgated heat to ~openstack-charmers but the API is still reporting ~charmers as the approved version. Wasn't sure how long to wait (been about 45 mins since re-promulgation) [23:12] http://manage.jujucharms.com/api/3/search?name=heat&type=approved&series=precise [23:12] marcoceppi: did you unpromulgate the other? [23:13] good question [23:13] marcoceppi: hmm, don't see the new one at all atm http://manage.jujucharms.com/api/3/search?name=heat [23:13] marcoceppi: so give it a bit more [23:13] rick_h_: will do [23:14] wasnt sure if the API was updated at every injest or not [23:14] marcoceppi: what's the full path? ~openstack-charmers/precise/heat? http://manage.jujucharms.com/api/3/charm/~openstack-charmers/precise/heat ? [23:14] since it was a re-promulgation, I didn't first unpromlugate (since all promulgate does is set the canonical branch for the source package in the distro) [23:14] es [23:14] k [23:14] yes* [23:15] https://code.launchpad.net/~openstack-charmers/charms/precise/heat/trunk [23:15] ruh roh...damn [23:15] marcoceppi: http://manage.jujucharms.com/heartbeat [23:15] it shows as mapped to lp:charms/heat [23:15] rick_h_: I don't know what that means, it says PASS to me [23:15] marcoceppi: looks like ingest has run off the rails [23:15] Ingest queue sizesPassqueued charms: 18537, queued baskets: 1740. [23:15] OH CRAP [23:15] that's a lot to ingest [23:16] heh, pass is "the queue is loading" [23:16] not that "it's not shrinking and getting caught up like it should" [23:16] marcoceppi: will have to look into the queue issue tomorrow. Sorry, will ping when we get it updated/straighted out [23:17] rick_h_: np, thanks! [23:17] marcoceppi: no vangaurd and such in webops atm. Sorry for the issue [23:17] but that explains it not loading [23:17] yeah, that makes a lot more sense [23:17] * marcoceppi has become accustomed to the 15 min turn around [23:17] yea, when it works it's kind of nice :) === TheRealMue is now known as TheMue