lots of disk writes

Message boards : Number crunching : lots of disk writes

Author	Message
Nasicus Volunteer tester Joined: Sep 13 06 Posts: 13 ID: 35 Credit: 666,725 RAC: 0	Message 80 - Posted 13 Sep 2006 18:22:46 UTC
	IMO there are really many disk writes, roughly estimated at least every second. In my "general preferences" the value for "Write to disk at most every" is 60 seconds. Could you please check that? Perhaps this is also a feature to optimize the performance?
	ID: 80 \| Rating: 0 \| rate: /

Andre Kerstens Forum moderator Project tester Volunteer tester Joined: Sep 11 06 Posts: 749 ID: 1 Credit: 15,199 RAC: 0	Message 93 - Posted 13 Sep 2006 19:50:30 UTC - in response to Message ID 80 .
	I'm pretty sure that's our app. Will check. Andre IMO there are really many disk writes, roughly estimated at least every second. In my "general preferences" the value for "Write to disk at most every" is 60 seconds. Could you please check that? Perhaps this is also a feature to optimize the performance?
	ID: 93 \| Rating: 0 \| rate: /

Nasicus Volunteer tester Joined: Sep 13 06 Posts: 13 ID: 35 Credit: 666,725 RAC: 0	Message 105 - Posted 13 Sep 2006 21:02:20 UTC - in response to Message ID 93 .
	I'm pretty sure that's our app. Will check. Andre Thanks in advance, you're anwering all quickly, great job.
	ID: 105 \| Rating: 0 \| rate: /

Test User Volunteer tester Joined: Sep 13 06 Posts: 5 ID: 70 Credit: 0 RAC: 0	Message 131 - Posted 14 Sep 2006 4:38:22 UTC - in response to Message ID 105 .
	Found that our app is actually not using the boinc function boinc_time_to_checkpoint() that makes sure that checkpoints are only done according to the user pref "Write to disk at most every". Will build that in soon. Our app also writes to a logfile though and the frequency of log writing will be hard to control. Have to see if we can do anything about this. Andre I'm pretty sure that's our app. Will check. Andre Thanks in advance, you're anwering all quickly, great job.
	ID: 131 \| Rating: 0 \| rate: /

Billy Volunteer tester Joined: Sep 28 06 Posts: 27 ID: 123 Credit: 3,279 RAC: 0	Message 771 - Posted 3 Oct 2006 12:59:05 UTC - in response to Message ID 131 . Last modified: 3 Oct 2006 13:00:13 UTC
	I just downloaded the new app 5.2 for the Intel Mac. The constant disk activity is still there. On my activity monitor it shows as a constant minimum 4 read/writes per second up to a maximum of over 400 per second. I don't want to wear out my disk prematurely and I don't have a backup yet for this new computer.
	ID: 771 \| Rating: 0 \| rate: /

Rebirther Volunteer tester Joined: Sep 13 06 Posts: 63 ID: 52 Credit: 69,033 RAC: 0	Message 804 - Posted 4 Oct 2006 9:01:11 UTC
	Any news to fix this in later windows apps?
	ID: 804 \| Rating: 0 \| rate: /

Andre Kerstens Forum moderator Project tester Volunteer tester Joined: Sep 11 06 Posts: 749 ID: 1 Credit: 15,199 RAC: 0	Message 805 - Posted 4 Oct 2006 14:56:23 UTC - in response to Message ID 804 .
	Yes, for sure. We have started on this yesterday and will hopefully finish by next week monday or tuesday. Beware that this will make sure the checkpointing only takes place according to the disk write preference that the users sets. Other writes the app does, of course won't (we would loose important data if we can't write our results to file). This is a fix for all platforms by the way. Thanks Andre Any news to fix this in later windows apps? ____________ D@H the greatest project in the world... a while from now!
	ID: 805 \| Rating: 0 \| rate: /

[BAT]bigbee Volunteer tester Joined: Sep 19 06 Posts: 2 ID: 118 Credit: 256 RAC: 0	Message 809 - Posted 4 Oct 2006 19:25:29 UTC - in response to Message ID 805 . Last modified: 4 Oct 2006 19:28:15 UTC
	Other writes the app does, of course won't (we would loose important data if we can't write our results to file). This is a fix for all platforms by the way. Does the app writes the results to a file once after the wu has finished or does it continuously writes results to the hard disk or are there some intermediate results during the crunching of a wu? Can you specify (or give an estimation) of the amount of diskwrites your app needs to store its results(without checkpointing)? ____________ boinc.be , crunching belgian style...
	ID: 809 \| Rating: 0 \| rate: /

Andre Kerstens Forum moderator Project tester Volunteer tester Joined: Sep 11 06 Posts: 749 ID: 1 Credit: 15,199 RAC: 0	Message 812 - Posted 4 Oct 2006 21:47:56 UTC - in response to Message ID 809 .
	It continuously updates a logfile (charmm.out), the minimum energy found (minenergy.pdb), the minimum rmsd found (minrmsd.pdb) and the percentage done (percentdone.str). This should not be more than once or twice a second though. Somebody was mentioning 400 writes/sec in another thread, which we have never seen and been able to reproduce. Thanks Andre Other writes the app does, of course won't (we would loose important data if we can't write our results to file). This is a fix for all platforms by the way. Does the app writes the results to a file once after the wu has finished or does it continuously writes results to the hard disk or are there some intermediate results during the crunching of a wu? Can you specify (or give an estimation) of the amount of diskwrites your app needs to store its results(without checkpointing)? ____________ D@H the greatest project in the world... a while from now!
	ID: 812 \| Rating: 0 \| rate: /

John McLeod VII Volunteer tester Joined: Oct 3 06 Posts: 9 ID: 179 Credit: 240,291 RAC: 0	Message 822 - Posted 5 Oct 2006 1:31:57 UTC - in response to Message ID 812 .
	It continuously updates a logfile (charmm.out), the minimum energy found (minenergy.pdb), the minimum rmsd found (minrmsd.pdb) and the percentage done (percentdone.str). This should not be more than once or twice a second though. Somebody was mentioning 400 writes/sec in another thread, which we have never seen and been able to reproduce. Thanks Andre Other writes the app does, of course won't (we would loose important data if we can't write our results to file). This is a fix for all platforms by the way. Does the app writes the results to a file once after the wu has finished or does it continuously writes results to the hard disk or are there some intermediate results during the crunching of a wu? Can you specify (or give an estimation) of the amount of diskwrites your app needs to store its results(without checkpointing)? Since the app will be restarted at the checkpoint, you should be able to stop the writes to the log file at that point as well - otherwise, there would be an overlap in the log as the same section gets run twice - and logged twice.
	ID: 822 \| Rating: 0 \| rate: /

[BAT]bigbee Volunteer tester Joined: Sep 19 06 Posts: 2 ID: 118 Credit: 256 RAC: 0	Message 836 - Posted 5 Oct 2006 9:34:34 UTC - in response to Message ID 812 . Last modified: 5 Oct 2006 9:38:31 UTC
	It continuously updates a logfile (charmm.out), the minimum energy found (minenergy.pdb), the minimum rmsd found (minrmsd.pdb) and the percentage done (percentdone.str). This should not be more than once or twice a second though. Somebody was mentioning 400 writes/sec in another thread, which we have never seen and been able to reproduce. Does that file contains a list of the evolution of the minimal energy? Or does it only contains the minimal value? (same for rmsd) There are several solutions to only write results to files at checkpoints. Eg. keeping a string/array variable with logfile entries and result progress to write away at checkpointing. Like mentioned before after checkpointing there is a possibility to enter the same logdata twice when boinc gets restarted in between. 2 writes/sec (I have a dual processor -> 4 writes/sec) is still a huge load on your hd and should be avoided at any cost ps: is it the final result you are interested in or rather the evolution? Because the evolution can quickly be reproduced for interesting final results ____________ boinc.be , crunching belgian style...
	ID: 836 \| Rating: 0 \| rate: /

Michela Forum moderator Project administrator Project developer Project tester Project scientist Joined: Sep 13 06 Posts: 163 ID: 10 Credit: 97,083 RAC: 0	Message 886 - Posted 6 Oct 2006 2:31:54 UTC - in response to Message ID 836 .
	Hi, each task includes several docking attempts at the end of each we store the protein-ligand conformation if its energy or rmsd is the lowest found so far in the minenergy.pdb or minrmsd.pdb respectively (so we overwrite the previous found conformations). The summary allows us to rebuild other results starting from the initial conditions stored in the summary (in case this is needed). The current charmm app does a lot of checkpointing (we did not want people to loose computation) and for each docking attempt we save the starting point so that if the computation is broken exactly during that attempt, we can restart from its beginning reducing the amount of lost computation time. Please note that the complex that we are testing is very small and we are now testing larger proteins and ligands so the checkpointing will be done less frequently. Moreover we are working on a new version of charmm that will include the boinc_time_to_checkpoint. We will also review our checkpointing policy. We'll keep you posted! Thanks, Michela Does that file contains a list of the evolution of the minimal energy? Or does it only contains the minimal value? (same for rmsd) There are several solutions to only write results to files at checkpoints. Eg. keeping a string/array variable with logfile entries and result progress to write away at checkpointing. Like mentioned before after checkpointing there is a possibility to enter the same logdata twice when boinc gets restarted in between. 2 writes/sec (I have a dual processor -> 4 writes/sec) is still a huge load on your hd and should be avoided at any cost ps: is it the final result you are interested in or rather the evolution? Because the evolution can quickly be reproduced for interesting final results
	ID: 886 \| Rating: 0 \| rate: /

Nicolas Volunteer tester Joined: Sep 13 06 Posts: 66 ID: 17 Credit: 29,510 RAC: 0	Message 923 - Posted 8 Oct 2006 1:56:45 UTC
	As long as the file isn't opened/written/closed all the time, and you don't flush() it, all the cache systems should help here. It won't really be 2 physical writes/sec.
	ID: 923 \| Rating: 0 \| rate: /

Atomic Booty Volunteer tester Joined: Sep 13 06 Posts: 59 ID: 23 Credit: 30,370 RAC: 0	Message 926 - Posted 8 Oct 2006 3:03:11 UTC - in response to Message ID 886 .
	I appreciate the update Michela, I had been wondering about the progress on this issue myself. Thanks! ~Atomic Hi, each task includes several docking attempts at the end of each we store the protein-ligand conformation if its energy or rmsd is the lowest found so far in the minenergy.pdb or minrmsd.pdb respectively (so we overwrite the previous found conformations). The summary allows us to rebuild other results starting from the initial conditions stored in the summary (in case this is needed). The current charmm app does a lot of checkpointing (we did not want people to loose computation) and for each docking attempt we save the starting point so that if the computation is broken exactly during that attempt, we can restart from its beginning reducing the amount of lost computation time. Please note that the complex that we are testing is very small and we are now testing larger proteins and ligands so the checkpointing will be done less frequently. Moreover we are working on a new version of charmm that will include the boinc_time_to_checkpoint. We will also review our checkpointing policy. We'll keep you posted! Thanks, Michela
	ID: 926 \| Rating: 0 \| rate: /

Message boards : Number crunching : lots of disk writes

Database Error
: The MySQL server is running with the --read-only option so it cannot execute this statement

array(3) {
  [0]=>
  array(7) {
    ["file"]=>
    string(47) "/boinc/projects/docking/html_v2/inc/db_conn.inc"
    ["line"]=>
    int(97)
    ["function"]=>
    string(8) "do_query"
    ["class"]=>
    string(6) "DbConn"
    ["object"]=>
    object(DbConn)#19 (2) {
      ["db_conn"]=>
      resource(108) of type (mysql link persistent)
      ["db_name"]=>
      string(7) "docking"
    }
    ["type"]=>
    string(2) "->"
    ["args"]=>
    array(1) {
      [0]=>
      &string(50) "update DBNAME.thread set views=views+1 where id=21"
    }
  }
  [1]=>
  array(7) {
    ["file"]=>
    string(48) "/boinc/projects/docking/html_v2/inc/forum_db.inc"
    ["line"]=>
    int(60)
    ["function"]=>
    string(6) "update"
    ["class"]=>
    string(6) "DbConn"
    ["object"]=>
    object(DbConn)#19 (2) {
      ["db_conn"]=>
      resource(108) of type (mysql link persistent)
      ["db_name"]=>
      string(7) "docking"
    }
    ["type"]=>
    string(2) "->"
    ["args"]=>
    array(3) {
      [0]=>
      object(BoincThread)#3 (16) {
        ["id"]=>
        string(2) "21"
        ["forum"]=>
        string(1) "2"
        ["owner"]=>
        string(2) "35"
        ["status"]=>
        string(1) "0"
        ["title"]=>
        string(19) "lots of disk writes"
        ["timestamp"]=>
        string(10) "1160276591"
        ["views"]=>
        string(4) "1484"
        ["replies"]=>
        string(2) "13"
        ["activity"]=>
        string(23) "1.1334874920065999e-129"
        ["sufferers"]=>
        string(1) "0"
        ["score"]=>
        string(1) "0"
        ["votes"]=>
        string(1) "0"
        ["create_time"]=>
        string(10) "1158171766"
        ["hidden"]=>
        string(1) "0"
        ["sticky"]=>
        string(1) "0"
        ["locked"]=>
        string(1) "0"
      }
      [1]=>
      &string(6) "thread"
      [2]=>
      &string(13) "views=views+1"
    }
  }
  [2]=>
  array(7) {
    ["file"]=>
    string(63) "/boinc/projects/docking/html_v2/user/community/forum/thread.php"
    ["line"]=>
    int(184)
    ["function"]=>
    string(6) "update"
    ["class"]=>
    string(11) "BoincThread"
    ["object"]=>
    object(BoincThread)#3 (16) {
      ["id"]=>
      string(2) "21"
      ["forum"]=>
      string(1) "2"
      ["owner"]=>
      string(2) "35"
      ["status"]=>
      string(1) "0"
      ["title"]=>
      string(19) "lots of disk writes"
      ["timestamp"]=>
      string(10) "1160276591"
      ["views"]=>
      string(4) "1484"
      ["replies"]=>
      string(2) "13"
      ["activity"]=>
      string(23) "1.1334874920065999e-129"
      ["sufferers"]=>
      string(1) "0"
      ["score"]=>
      string(1) "0"
      ["votes"]=>
      string(1) "0"
      ["create_time"]=>
      string(10) "1158171766"
      ["hidden"]=>
      string(1) "0"
      ["sticky"]=>
      string(1) "0"
      ["locked"]=>
      string(1) "0"
    }
    ["type"]=>
    string(2) "->"
    ["args"]=>
    array(1) {
      [0]=>
      &string(13) "views=views+1"
    }
  }
}

query: update docking.thread set views=views+1 where id=21