/srv/irclogs.ubuntu.com/2010/02/15/#ubuntuone.txt

kklimondahmm.. if I change part of file is the whole going to be reuploaded or only the part of it?03:27
ryehi duanedesign08:56
ryeok, time to write diagnostic script for .cache/ubuntuone/ file permissions09:02
ryebug 491950 - added to ubuntuone-client-diagnose09:22
ubottuLaunchpad bug 491950 in ubuntuone-client "100% cpu usage calling calcuate_hash on same file over and over (dup-of: 407762)" [High,Triaged] https://launchpad.net/bugs/49195009:22
ubottuLaunchpad bug 407762 in ubuntuone-client "ubuntuone-syncdaemon maxes out system resources when trying to hash a file with root permissions" [Medium,Triaged] https://launchpad.net/bugs/40776209:22
=== Urda_ is now known as Urda
sebi`hi, is it possible to make certain files public?12:20
beunosebi`, it will soon. The feature is mostly done, we're figuring out some deployment details12:21
beunostay tuned!12:21
sebi`oh, darn12:21
sebi`but nice to hear it's going to be implemented \o/12:22
sebi`also, it might sound silly, but will there ever be a non-linux client, as in, for windows XP, or Mac for example? (please don't hurt me for asking that :P)12:23
beunosebi`, sure, there is an on-going effort to build a windows client12:24
beunothere will be a sprint during pycon12:24
sebi`neat-o12:24
sebi`:D12:24
beunosebi`, we're anticipating all your needs12:25
beunowhat else you got?  :)12:25
sebi`nothing so far :D12:25
sebi`but thank you for your time12:25
beunoyou're welcome12:26
ryeok, I guess we'll need a script that parses the syncdaemon.log of current version to find what state syncdaemon is, since "About a month ago Ubuntu One just stopped working." requires syncdaemon.log to find possible reason.12:41
beunorye, do it!12:42
ryebeuno, thinking on how to do that atm...12:43
rye:)12:43
beunorye, verterok is always a good person to talk to when you want to do funky things, he's an ex-gentoo user  ;)12:43
ryei just hate repeating "Please upload syncdaemon log", scan it, only to find that it is in STANDOFF_WAITING_FOR_THE_END_OF_THE_WORLD12:43
* verterok looks12:47
ryeverterok, erm, no, it is just an ongoing conversation, nothing to look on so far12:48
verterokrye: there are other ways to know the state of syncdaemon :)12:50
verterokrye: e.g: dbus-send --session --print-reply --dest=com.ubuntuone.SyncDaemon --type=method_call /status com.ubuntuone.SyncDaemon.Status.current_status12:50
verterokrye: that should work in all ubuntu installs :)12:50
ryeverterok, oh, thanks! That is nice thing, I want to be able to parse log files so that there is clear sequence of events (file spotted, file uploaded, uuid is that one), state changed @ 1 AM, error occured, etc. :)12:54
ryeoh12:54
ryei know i know i know12:54
ryesyncdaemon needs some method to enable debugging right _now_.12:55
verterokrye: parsing the config is a good idea12:55
verterokrye: a method? at runtime?12:55
ryeverterok, yup12:55
ryeverterok, i.e. i have a bug. syncdaemon does not want to upload files. we instruct the user to enable debug, add more files, then package this into a nice fine tarball and send to us, if this does not help, then restart syncdaemon. it may not be thaaat useful, though.12:56
verterokrye: could you file a bug about this requirement? :)13:00
ryeverterok, which exactly? runtime debug or log parser? :)13:11
verterokrye: runtime debug :)13:11
verterokrye: regarding the log parser, maybe facundobatista alsready have some scripts :)13:12
ryeverterok, btw, end users might also want to see what's happening behind the scenes w/o enormous numbers of uuids that don't give much info about filename unless one scans through the logs with a magnifier (or grep) ...13:13
facundobatistarye, what you said is logical: people want to see what's going on with the client13:14
facundobatistarye, how many files already uploaded/downloaded, how many are still in the queue13:14
facundobatistarye, at which speed everything is working, etc13:14
ryefacundobatista, frankly speaking I'd like to know what's actually happening too :)13:14
ryeand what Blocked(filename) means... it all is in the sources... thought13:15
ryethough13:15
facundobatistarye, however, referring the user to the logs it's a poor solution: we should provide a GUI for that13:15
ryefacundobatista, yep, to show the current, aaand a log reader to show the past :)13:15
facundobatistarye, mmm13:17
ryebtw, is anybody here running syncdaemon enabled on session login?13:17
facundobatistarye, let's take for example a torrent client... how many times did you want to see the past?13:17
ryeit looks like syncdaemon eats my harddrive when it starts leading to an increased waiting time...13:17
facundobatistarye, in any case... I don't have any scripts for the logs13:18
ryefacundobatista, ok, but if the torrent client removed the entries for already downloaded file then there should have been some kind of logs13:18
ryeerm13:18
facundobatistarye, or it shouldn't remove the entries for already downloaded files ;)13:19
ryetorrents not apply here - they stay seeding forever so the only thing that is important is the speed charts, to show that to ISP and convince them that they really have a shaper in place :)13:19
ryebut files are uploading and we forget about them13:19
ryefacundobatista, in any way, runtime debug switch seems to be easier to implement13:19
facundobatistarye, yeap13:20
beunofacundobatista, rye, I hear quickly is great for building quick UIs13:20
facundobatistarye, anyway... for GUI enhancements you should talk with Chipaca or beuno13:20
facundobatistarye, I just say that logs shouldn't be the way to see what's going on for final users13:20
ryefacundobatista, beuno, verterok, ok so back to syncdaemon running LR on login - have you experienced any elevated hdd activity and gdm-login-to-usable-session time?13:23
verterokrye: it's a known issue that LR takes a lot of time in some cases13:23
beunorye, I showed ubuntu devs my bootchart graph, and they all cursed at u113:23
ryegrrr13:24
ryenot good13:24
ryemy session login takes up to 1 minute13:24
ryew/ syncdaemon eating ( "D" ) my drive.13:24
beunooh, my boot time is about 11 seconds from power on to desktop  :)13:25
verterokbeuno, rye: that's the applet/nautilus starting syncdaemon earlier than they should13:25
ryeverterok, ok, what about trunk?13:26
verterokrye: you can't start syncdaemon on login using trunk :)13:26
verterokrye: there is no applet13:26
ryeverterok, I can :)13:26
ryeverterok, so what?13:26
verterokrye: hehe, yes you can.13:26
ryeverterok, applet is no longer the launcher :)13:26
verterokrye: who is launching it now?13:27
beunoverterok, rye, https://devpad.canonical.com/~beuno/beuno-laptop-lucid-20100212-2.png13:27
ryeverterok, u1sdtool can launch it but even when syncdaemon itself is launched it starts doing LR which greatly impacts the ability to do something before it calms down13:27
verterokrye: yes13:28
* rye becomes sad because he has 75 seconds of bootcharting13:28
verterokrye: ATM, there is no way to avoid the LR at startup13:28
ryeverterok, but that can be read in chunks, much like js calls are chained so that browser does not get too sad about executing all that 12 billion lines at once13:29
ryeverterok, freeing the hd for some time to perform the task...13:30
verterokrye: it's being done in chunks :/13:32
verterokrye: one of the options is to change the filesystem manager layer to make it thread safe, so we can do the LR in a different thread13:33
ryeverterok, i trapped on bug 409972 - could that improve anything? By reading one file instead of tons of small open()s and read()s ?13:34
ubottuLaunchpad bug 409972 in ubuntuone-client "Use cPickle hack to speed up syncdaemon start" [Medium,Triaged] https://launchpad.net/bugs/40997213:34
verterokrye: no, that's a different issue13:34
ryeverterok, LR is local rescan, not metadata loading....13:34
ryegrrr13:35
ryei got it13:35
verterokrye: on startup, syncdaemon builds a data structure that holds all the sync states/transitions13:35
ryelocal rescan is traversal of all the files and folder to say "hi, file!"13:35
verterokrye: that bug is to pickle that data structure on the first startup, to speedup the next one13:35
verterokrye: actually local rescan traverse all files *and* loads metadata13:35
verterokrye: the issue with the startup isn't local rescan actually, it's the metadata loading of filesyetem manager13:36
verterokrye: FilesystemManager builds an index on startup, and needs to read all the metadata to do it :(13:36
ryeverterok, With all the buzz around improving startup time we forget that there's still time-to-usable-desktop and that is heavily impacted by running anything disk-intensive right at startup...13:37
verterokrye: that's the only block-until-finish bit on syncdaemon startup13:39
ryeverterok, one seems to be enough :)13:39
verterokrye: indeed13:40
verterokrye: and we are awere of this problem13:40
verterokrye: I did a little experiment, and replaced the current metadata storage with sqlite13:40
verterokrye: the result wasn't a big improvement :(13:40
ryebeuno, was that lucid install mentioned in bootchart a clean one or upgrade from Karmic ?13:44
beunorye, upgrade from karmic to lucid13:45
beunoafter a tweak13:45
facundobatistarye, #476711 could improve LR timings13:45
beunoI got it 10 seconds down13:45
beunohttp://launchpadlibrarian.net/39214382/beuno-laptop-lucid-20100213-2.png13:45
ryebeuno, any problems w/ ureadahead not generating pack files?13:45
beunorye, not atm, no13:45
beunoalso, I have an SSD HD13:45
ryefacundobatista, ah, stat() on every file found?13:46
ryebeuno, still, my upgrade went not that smooth, i don't have plymouth displaying nice things, ureadahead is broken and tmp is not cleaned on start (IT SHOULD BE TMPFS!)13:47
facundobatistarye, yeap13:47
beunorye, wierd. It was flawless for me13:48
ryebeuno, thanks for SSD hint, is your /tmp in memory or on disk?13:49
beunorye, disk13:50
aquariushey mattgriffin13:55
beunorodrigo_, ping13:57
beunoI'm working on the notes branch13:58
beunoand I have a questionfor you when you get back13:58
beunorodrigo_, ok, so notes are saving now. Just need to make the flow nice and fix quirks: lp:~beuno/ubuntuone-servers/dont-autosave14:06
beunoI'm interested in something you said about revisions and not sending them creating problems14:07
beunoaquarius, maybe you can shed some light on it  ^14:16
rodrigo_beuno, cool!14:33
rodrigo_beuno, the problem is that when saving notes yto couchdb, you need to specify the _rev you are updating14:34
rodrigo_beuno, and the autosave code didn't take them into account14:35
rodrigo_beuno, so if there is no autosave, it should be ok, just sending the POST when the user pressed 'Save' should do it correctly14:35
beunorodrigo_, could you enlighten me on why the manual save won't cause this?14:38
rodrigo_beuno, because the autosave code doesn't know the ID of the document AFAIR14:39
rodrigo_the _rev, sorry14:39
beunorodrigo_, so this is what I did to make it work: https://pastebin.canonical.com/27865/14:40
rodrigo_beuno, not sure if you need the format=json part? but if it works, it looks good to me :-)14:41
rodrigo_beuno, have you submitted the branch for merging?14:41
beunorodrigo_, not yet, it needs a lot of clean up14:41
beunorodrigo_, the json bit14:41
beunois because otherwise the response is an HTML14:41
rodrigo_ah, ok14:41
beunoand everything goes BOOM14:41
beunoI need to let you go into edit mode, make the buttons pretty, and maybe a spinner while it saves  :)14:42
beunohow urgent is this?14:42
beunoI can split it into 2 branches14:42
rodrigo_if you can, that would be great14:42
beunowill do14:43
rodrigo_it's a bit urgent, because the web UI is broken14:43
rodrigo_but if you want to have all in 1 branch, that's ok with me14:43
beunorodrigo_, I'll see how time consuming it is14:43
beunostart with the basics14:43
beunoand cut off when I get to a reasonable amount of time spent  :)14:43
rodrigo_ok :)14:43
rodrigo_thanks!14:43
beunode nada!14:44
beunogood thing its a holiday in the US today14:44
beunogave me an extra day before going knee-deep into phone sync14:44
rodrigo_hehe14:44
beunorodrigo_, is this the traceback we're trying to fix with this?  https://pastebin.canonical.com/27866/14:53
rodrigo_beuno, no, but if you get that, I guess we should fix it14:54
beunorodrigo_, I get that with the new implementation  :)14:54
rodrigo_beuno, I can fix it if you want, it needs to 'if "Tomboy" in annotations14:54
rodrigo_beuno, right, I changed the code to use dc's get_application_annotations, which doesn't have the has_key14:55
beunorodrigo_, if you can throw a quick patch at me, it would be great14:55
rodrigo_beuno, but since that part wasn't working, I didn't change it it seems14:55
beunoor change in your branch and I'll merge14:55
rodrigo_beuno, ok, let me see14:55
dobeyhttp://www.youtube.com/results?search_query=ubuntuone&search_type=&aq=f14:58
rodrigo_beuno, ok, patched (but not tested) in my branch, so merge it14:58
dobeyinteresting14:58
beunorodrigo_, merging, will loet you know in a sec14:58
rodrigo_beuno, ok14:59
beunorodrigo_, new error!  https://pastebin.canonical.com/27868/15:00
rodrigo_hmm15:01
beunorodrigo_, that's missing stuff15:02
beunoone sec15:02
rodrigo_ah15:03
beunorodrigo_, https://pastebin.canonical.com/27869/15:03
rodrigo_hmm, I'm lost, that should work :-(15:05
rodrigo_beuno, see line 426, that works, so not sure what the difference is??15:06
beunohrm15:07
beunorodrigo_, that does: if existing_doc is not None and "_rev" in existing_doc._data:15:09
beunoI wonder if that check makes the diff?15:10
rodrigo_that just adds the _rev field if it's an update15:11
rodrigo_but the error says a DC.Record is not serializable15:11
beunoright15:11
beunoI get that error when editing and saving15:11
rodrigo_aquarius or thisfred should know better what the error means, since it's in DC code15:11
beunoso it probably should add the rev as well, no?15:11
beuno(even if its unrelated)15:12
rodrigo_but the 2 calls to .put_record use a python list15:12
rodrigo_beuno, if it's a new document, no, you can't make up a revision15:12
rodrigo_couchdb will assign it automatically15:12
beunoaha15:12
beunook, good to know15:12
=== jamlata-afk is now known as jamalta
rodrigo_so, to put_record you pass a Record(list values) and internally DC converts that to a Record, and so it shouldn't be serializing it if it can't be serialized15:17
rodrigo_not sure what's going on, sorry :-(15:17
rodrigo_the 2 calls to put_record seem to use the same format15:17
beunoI can't figure it out either  :/15:17
rodrigo_aquarius, ^^15:18
* aquarius reads the backscroll15:18
beunoaquarius, the meat of it is: https://pastebin.canonical.com/27869/15:18
aquariuswtf?15:18
beunoon saving an edited note15:18
aquariushow have you created a nonserialisable dictionary? :)15:18
rodrigo_aquarius, https://pastebin.canonical.com/27869/15:19
beunoaquarius, I'm special that way  :)15:19
rodrigo_aquarius, see lib/u1/web/notes/views.py#16115:19
aquariusbeuno, can you pastebin the value of doc just before line 161 in web/notes/views.py?15:19
beunosure15:20
rye_beuno, you stuffed mergeable list that you've received originally into the record?15:20
beunoaquarius, where would print statements be outputted to?15:21
beunorye_, I haven't changed anything in python15:22
aquariusbeuno, no idea. I always do raise Exception(str(doc)) :-)15:22
rye_beuno, aquarius - the same at client side: http://paste.ubuntu.com/376910/15:22
rye_so it IS possible, i have filed a bug, searching...15:22
aquariusrye_, hang on, what's that paste?15:23
aquariusrye_, ah, that replicates the problem. cheers.15:23
beunoaquarius, nice trick: https://pastebin.canonical.com/27871/15:23
rye_aquarius, bug 510223, though you are already using the server code...15:24
ubottuLaunchpad bug 510223 in desktopcouch "The new version of update_fields() should be backported from the server code" [Medium,Fix committed] https://launchpad.net/bugs/51022315:24
aquariusweird15:24
aquariusthisfred?15:24
aquariusCardinalFang?15:24
aquariusno thisfred15:24
aquariusCardinalFang, ping?15:24
aquariusI don't know much about update_fields15:24
rodrigo_aquarius, hmm, so the problem is the doc["application_annotations"] = record.application_annotations?15:25
rodrigo_aquarius, I guess that needs to be converted to a list?15:25
rye_thisfred has told me to do... wait, searching...15:25
aquariussomething odd about that. It might not be possible to ship a_a around that way15:26
CardinalFangaquarius, hi.  USians have holiday today.  I've seen that bug, and I don't understand it yet.  I want to compare the code in py couchdb mod.15:26
rye_old_record.application_annotations["Tomboy"] = application_annotations15:26
rye_record_id = CouchDbNote.notes_db.put_record(old_record)15:26
rye_so no update_fields but my application_annotations is just a regular python dict15:27
aquariusCardinalFang, oh, sorry, pal, didn't realise it was a holiday!15:27
* beuno fixes js while the experts figure this out15:29
rye_Jan 20 20:11:54 <thisfred>      so it needs to be a dictionary :(15:30
rye_Jan 20 20:12:02 <thisfred>      you can do this:15:30
rye_Jan 20 20:12:05 <rtgz>  sorry, http://paste.ubuntu.com/359656/15:30
rye_Jan 20 20:12:20 <thisfred>      db.update_fields(record_id, {15:30
rye_Jan 20 20:12:20 <thisfred>                   "application_annotations": application_annotations._data })15:30
rye_Jan 20 20:12:40 <thisfred>      but boy that method stinks15:30
beunohrm15:30
rodrigo_I guess it's better then to do doc["applications_annotations"] = record.application_annotations._data ?15:31
rodrigo_and then just the put_record(Record(data=doc)) ?15:31
aquariusdb.update_fields(record_id, { "application_annotations": application_annotations._data }) should work15:32
aquariushaving just tested it15:32
rodrigo_aquarius, but we'll have to list all the fields15:32
aquariusmark it with a big UGH THIS IS A WORKAROUND nastiness :)15:32
rodrigo_and wouldn't doc["applications_annotations"] = record.application_annotations._data work also?15:33
* beuno is prepared to merge15:33
rodrigo_go beuno go! :D15:33
beunomerging!15:35
beunoman15:35
beunoworking on U1 is *so* much more fun than launchpad...15:35
beunoyou can actually get something done in an hour!15:35
beunorodrigo_, nothing to merge!15:35
rodrigo_beuno, ah, you meant merging from my branch?15:36
rodrigo_let me change that then15:36
beunorodrigo_, anywhere!15:36
beunoas long as it has the fix  :)15:36
rye_is it possible for syncdaemon to be running as root?15:37
rodrigo_beuno, ok, merge now15:38
rye_i see the config entry im_ok_with_being_root_pretty_please_let_me_be_root.default = False and i think whether it does anything...15:38
beunorodrigo_, works15:39
beuno!15:39
rodrigo_beuno, cool!15:39
beunothank you rodrigo_, aquarius, rye_15:39
beunoon with it15:39
=== rye_ is now known as rye
rodrigo_beuno, let me know when you propose the branch, and I'll review it15:40
beunorodrigo_, will do, I need to fix a few things on the javascript side first so people don't hav to use firebug to edit notes  :)15:42
ryere "UGH THIS IS A WORKAROUND". I remember having seen "This should not be done this way, it needs to be rewritten ASAP!" in a file that had this since 2004. Nobody knew what was actually wrong there so it was left as is :)15:42
rodrigo_rye, yeah, comments you never revisit don't work :-)15:42
beunook, I'm off to pay the rent. I'll be back in 30-40'15:43
aquariusrye, if you put "FIXME" then pylint bitches at you about it, which is why I do it :)15:43
=== beuno is now known as beuno-lunch
kjollerHi all. I posted quite a lot of messages on a bug this weekend. I would like some feedback/advice on it.15:44
kjoller(https://bugs.launchpad.net/bugs/497143)15:44
ubottuUbuntu bug 497143 in desktopcouch "CouchDatabase.execute_view should take an optional extra parameters dictionary" [Low,In progress]15:44
kjollerIt is basically a minor correction to a solution by aquarius15:45
beuno-lunchrodrigo_, I will finish this today, I need to be free tomorrow for phone sync  ;)15:45
rodrigo_beuno-lunch, ok, if you finish the js, I can continue on the othjer issues, if you haven't done them15:51
dobeyaquarius: we don't use pylint on the client, and pyflakes doesn't complain15:57
dobeywhy oh why rhythmbox, did you stick some of my quite old mp3s in "recently added"?16:02
=== beuno-lunch is now known as beuno
beunorodrigo_, sure, I think it's just js now16:53
beunowill let you know as I progress16:53
rodrigo_ok16:54
beunorodrigo_, bug 494322 and 501020 feel like dupes17:28
ubottuLaunchpad bug 494322 in ubuntuone-servers "editing notes in ubuntu one web interface doesn't have a "Save" button" [Low,In progress] https://launchpad.net/bugs/49432217:28
ubottuLaunchpad bug 501020 in ubuntuone-servers "Tomboy note sync and web-ui for notes are broken" [High,Confirmed] https://launchpad.net/bugs/50102017:28
beunowhat's your take?17:28
rodrigo_beuno, they are indeed17:29
beunoI will do the honors17:29
* rye is slow today, will shutdown now to prevent damage to the source code18:08
beunorodrigo_, getting closer18:14
beunoneed to figure out why the save call returns HTML18:14
beunoand I think we have a working notes web UI which saves18:14
beunorodrigo_, got a minute?18:50
beunoI've traced the remaining issue down to a URL not returning json when it should18:51
beunoaquarius, maybe you can help as well  :)18:52
=== teknico is now known as teknico_away
beunorodrigo_, branch proposed!19:24
dobeyhttp://www.facebook.com/album.php?aid=157659&id=122313196726&saved#!/photo.php?pid=4039110&id=12231319672620:29
dobeywhoot.20:29
beunodobey, look at that!  one step closer from uninstalling Firefox!20:31
dobeytoo bad I'm not working on a facebook app20:32
beunoone step at a time20:32
dobeyWell, with FB, I don't also have to build the API. It's already mostly all there. :)20:33
rodrigo_beuno, cool, reviewing it now20:51
beunorodrigo_, thanks. I'm unsure about a specific part of it, I mentioned it in the MP20:51
beunorodrigo_, the branch went deeper than I expected, I can tell you that20:53
rodrigo_:)20:54
beunoit also made me realize I need to spend 2 or 3 days cleaning up CSS, javascript files and templates20:55
beunothey're all clustered into sections20:55
beunothe app is still small enough to fix it in a reasonable time frame20:55
beunoin 6 months, no way20:55
* beuno eyeballs jblount to pair with him in the neat future20:56
rodrigo_beuno, approved!21:08
beunorodrigo_, woooo!  thanks, on its way to PQM21:09
rodrigo_beuno, ping some losa when it gets merged, so that it gets deployed asap21:10
beunorodrigo_, this will need to get cherrypicked into production?21:10
rodrigo_beuno, not sure what the process is, it depends what other revisions there are before21:11
rodrigo_beuno, so better ask a losa :-)21:11
beunorodrigo_, will do21:11
beunorodrigo_, what if we let it roll out to edge first, test it, then cherry pick?21:15
rodrigo_beuno, yeah, sounds good21:15
beunoor is production broken enough that whatever will be better?21:15
beunosuper21:15
rodrigo_well, web ui editing of notes is completely broken21:15
rodrigo_but going through edge should be ok, it gets rolled every 4 hours there, I think21:16
beunoyeah, I'll let all the moving parts settle, we should have it on edge tomorrow, and have statik and other around to help us with the cherrypick21:16
rodrigo_ok21:18
=== jamalta is now known as jamalta-afk
=== jamalta-afk is now known as jamalta
=== jamalta is now known as jamalta-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!