How to prevent the project from over-granting credits?

As many of you may be awared of, overgranting / overclaiming the credit can be regarded by some participants as a great concern, unless a BOINC project relies on a specific credit policy, giving up the very standard framework. What I'd like to come out here is, how the policy of credit granting should be. Up to now under the standard framework the mean value of credits claimed from 3 hosts is adopted. But several projects such as S@H and R@H use another kind of credit system. They award the fixed amount of credits. Especially R@H is remarkable in that with its system hosts are given credits, whose value is based on the real works, not on benchmark of boinc. A credit system which no longer depends on benchmark would benefit not only honests participants but also the project owners, since it enables the devs to grasp the real amount of work done.

Anyway probably an important thing is this: to award 'fair' amount of credits. Indeed honesty does pay, but what seems 'dishonesy' also pay. What makes me to say such a thing is, in World Community Grid there has been a new kind of credit granting system introduced, as the first post of this thread (which I've taken up many times:) mentions like this:

2) About 5% of results claim points significantly in excess of the average for their platform.

...

In order to address issue #2 we are going to implementing a new algorithim for awarding credit granted. In the new algorithim we will no longer automatically discard the high and low values during validation in order to compute the credit granted. Instead we will exclude any claimed credits that are statistical outliers and then average the remaining values. Additionally, in cases where the claimed credit is significantly higher then the granted credit, the member will only be awarded half of the granted credit. Finally, invalid results will only be granted half credit as well.

But although there is a clear fact that modified core client with SIMD Streaming Extensions never speeds up calculation of science application like charmm (what speeds up with the extensions is calculation of science application itself), there is even a little possibility that some of the users simply don't know the fact.

Hi Yoda,

But although there is a clear fact that modified core client with SIMD Streaming Extensions never speeds up calculation of science application like charmm (what speeds up with the extensions is calculation of science application itself), there is even a little possibility that some of the users simply don't know the fact.

In my opinion, most people who are using these optimised clients would be well aware of what they are doing. The average participant just downloads their BOINC Client from Berkeley and doesn't even know about optimised applications. There may of course be exceptions.

It is my understanding that Docking work units will be variable in size, so fixing credits may be too difficult. Proteins@home use fixed credits, but there are huge fluctuations between batches. Pick any computer and you'll see that the amount of credit per hour's CPU time varies by as much as 100%.

Perhaps the SETI example would work best in this project - count the flops, if that's possible. I understand all current work units have the same flops count, but future work units will have different sizes. Count the flops for each new batch on a reference machine and issue credit accordingly?

A fixed credit system works best with a project that has standard length WUs. Einstein's credits are pretty standard no matter who or what crunches them. As for projects whose WUs vary greatly I am not sure what the best way to award points would be. I'm not sure how flops are counted either so I do not know if that would be a good way either. I'm OK with the way things are counted now.

My opinion and if someone has a better way I would consider it as well.
____________

I've always run standard BOINC clients and never got into the SETI credit argument but, If I understand the history of this, here's what started the credit mess.

- The SETI client software is open source. Someone took the source code and ran it through a better compiler + possibly hand optimizing it.

- They felt that since using this SETI client made them run a work unit faster, they should modify the BOINC client to represent the fact that their machines were actually running the work units faster. I can sort of see this being fair, if the BOINC client mods didn't have an effect on other projects and the BOINC client and SETI client were a matched pair. Unfortunately, that's not the case, and once people learned to easily mess with the BOINC clients benchmark results, those results became useless.

As long as the BOINC client is completely open source, there's going to be a way to cheat with granted credits unless each project has some way to calculate the credits independent of what the BOINC client tells them.

While the BOINC client is open source, as far as I know, some project clients are closed source. I believe Charmm is something they have to pay for and I've never heard of the source being available to anyone outside of the people running the project. For closed source projects like this, I would suggest some changes.

1. Have the closed source project client run it's own benchmark and inform the BOINC client of the benchmark results so that the BOINC client can properly calculate, and display to the user, the time remaining for the result to finish. The BOINC client would then have to track benchmark results on a per project or per work unit basis. This also takes care of the problem where the stock BOINC client benchmarks don't reflect the project accurately because the project client was compiled with a different compiler or different compiler switches.

2. Have the closed source client append a signed - encrypted benchmark result, work unit CPU runtime, result checksum, and possibly some other data to the end of the results file and invalidate the result if this doesn't decrypt properly. Use this to calculate credit for the work unit. Change the key with each project client version.

3. You might want to incorporate some new things into the benchmark such as allocating a large amount of memory (estimated to be how much the work unit will actually use), touch the memory block one byte per virtual page size increment before the benchmark so any needed virtual disk will be allocated and that one-time penalty will not be included in the benchmark timing, and use random sections of this memory block during the benchmark process so that memory latency and swapping are taken into account in the benchmark. Free the memory after the benchmark so it can be used by the work unit. I'd also have some of the functions called in the benchmark allocate some big blocks on the stack (using a part of them in some way so that the compiler doesn't optimize them out of existence) to more accurately reflect the CPU cache thrashing with real function calls. Unless the real application is one big source file, break the benchmark into multiple source files since this usually prevents certain compiler optimizations.

These are just some ideas on the subject. I use stock clients and choose projects on the basis of their scientific value and whether the results are going to be freely available to the scientific community. Credit is nice, but I take what a project gives and basically just use it for personal milestones. Cross-project credit parity would be nice though.

-- David

Hello david,

While the BOINC client is open source, as far as I know, some project clients are closed source. I believe Charmm is something they have to pay for and I've never heard of the source being available to anyone outside of the people running the project.

For closed source projects like this, I would suggest some changes.

1. Have the closed source project client run it's own benchmark and inform the BOINC client of the benchmark results so that the BOINC client can properly calculate, and display to the user, the time remaining for the result to finish. The BOINC client would then have to track benchmark results on a per project or per work unit basis. This also takes care of the problem where the stock BOINC client benchmarks don't reflect the project accurately because the project client was compiled with a different compiler or different compiler switches.

2. Have the closed source client append a signed - encrypted benchmark result, work unit CPU runtime, result checksum, and possibly some other data to the end of the results file and invalidate the result if this doesn't decrypt properly. Use this to calculate credit for the work unit. Change the key with each project client version.

3. You might want to incorporate some new things into the benchmark such as allocating a large amount of memory (estimated to be how much the work unit will actually use), touch the memory block one byte per virtual page size increment before the benchmark so any needed virtual disk will be allocated and that one-time penalty will not be included in the benchmark timing, and use random sections of this memory block during the benchmark process so that memory latency and swapping are taken into account in the benchmark. Free the memory after the benchmark so it can be used by the work unit. I'd also have some of the functions called in the benchmark allocate some big blocks on the stack (using a part of them in some way so that the compiler doesn't optimize them out of existence) to more accurately reflect the CPU cache thrashing with real function calls. Unless the real application is one big source file, break the benchmark into multiple source files since this usually prevents certain compiler optimizations.

These are just some ideas on the subject. I use stock clients and choose projects on the basis of their scientific value and whether the results are going to be freely available to the scientific community. Credit is nice, but I take what a project gives and basically just use it for personal milestones. Cross-project credit parity would be nice though.

Hi Suguru,

With respect to the three things I mentioned for benchmarks:

#1 I think that this one would require a change to the BOINC client.

#2 This is probably what is needed for fairness and to prevent cheating.

#3 Probably the best compromise would be to do a benchmark that actually calls some of the more commonly used functions in the Charmm program with some standard constant data. Get this benchmark to run about 30 to 60 seconds on the standard machine. Figure out how much credit you want to award per hour on the standard machine for cross project compatibility.

Credit Correction factor = Seconds to run benchmark on standard machine / Seconds to run benchmark on user client machine;

WU credit = Seconds to run WU on client machine * (standard credit per hour / 3600.0) * Credit Correction Factor;

This benchmark has nothing to do with SpecInt or Flops. It's based on how long the client computer takes to run the Charmm code relative to a standard machine. If the benchmark runs 30 seconds on the standard machine and only 15 seconds on the user machine then CCF (Credit Correction Factor) = 30/15 = 2. The user gets twice the standard credit per hour since they run the benchmark in half the time. If the user machine takes 90 seconds to run the benchmark then CCF = 30/90 = 0.33333 so the user only gets 1/3 of the standard credit per hour since they take 3 times as long to run the benchmark.

On the question of building this into the standard BOINC client: I would only allow the project client to tell the BOINC client percentage complete, estimated time to completion, and (optionally) points earned so far on the WU. These are for displaying to the user. Since the BOINC client is open source, letting it have anything to do with calculating the benchmark or encrypting the benchmark result data allows someone to easily compile an Optimized BOINC that cheats.

Regards,

-- David

Note: I just woke up so hopefully I got the math right :-)

#3 Probably the best compromise would be to do a benchmark that actually calls some of the more commonly used functions in the Charmm program with some standard constant data. Get this benchmark to run about 30 to 60 seconds on the standard machine. Figure out how much credit you want to award per hour on the standard machine for cross project compatibility.

Thanks for detailing, dave:)
Also,

#3 Probably the best compromise would be to do a benchmark that actually calls some of the more commonly used functions in the Charmm program with some standard constant data. Get this benchmark to run about 30 to 60 seconds on the standard machine. Figure out how much credit you want to award per hour on the standard machine for cross project compatibility.

We are actually looking into doing this. BOINC supplies an API for custom benchmarks. I don't think any other projects are using this yet though...

Andre

tampaku has created a custom benchmark by timing a bit of known code similar to the code in their wu. unfortunatly this app is only released to windozers and now they all claim approx 6 times what my linux machines there do.

am still working on attaching 'node' machines to here with restricted access and command line only
____________

am still working on attaching 'node' machines to here with restricted access and command line only

am still working on attaching 'node' machines to here with restricted access and command line only

doh on the main page there is a button to send me acct. key

and this project does attach ./boinc -attach_project <url> <account key>
most projects won't attach that way anymore, of the 12 that I have the keys for only tanpaku did thats why they got them machines :(

:) my troubles were

• I didn't know My account key and missed the link to send it to me on front page
  
• pico only allows input of a 25 character line
  

creating the .pbs script on my windows machine and uploading it to the primary-node, worked.

./boinc_cmd gives me an error "cannot execute binary file"
./boinc -attach_project <url> <key> worked

getting the key from account_docking.utep.edu.xml is a trick i will remember :)
[edited to try to get the second * to turn into a diamond ]
____________

./boinc_cmd gives me an error "cannot execute binary file"