Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 35
Posts: 35   Pages: 4   [ 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 37564 times and has 34 replies Next Thread
wcgridmember
Advanced Cruncher
Joined: Mar 30, 2005
Post Count: 110
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
confused By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

... from any sub-projects at a given moment?
[Jul 5, 2020 10:08:07 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

Weekend thought experiment? Cannot download a WU first time, second time, never? The techs wrote in prepping for OPN1 launch they were scaled up for an increase by a factor 8 or something, so 'any' means from 2 million to 16 million daily I suppose.
[Jul 6, 2020 6:11:05 AM]   Link   Report threatening or abusive post: please login first  Go to top 
wcgridmember
Advanced Cruncher
Joined: Mar 30, 2005
Post Count: 110
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

If the goal was that a client would never be able to download a WU, then the capacity would have to multiply by infinity, of course.

How did you estimate those numbers?
[Jul 6, 2020 9:06:53 AM]   Link   Report threatening or abusive post: please login first  Go to top 
wcgridmember
Advanced Cruncher
Joined: Mar 30, 2005
Post Count: 110
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

In the limit:

Say the number is X. It would mean that if in the next few years crunchers would on average invite X as enthusiastic crunchers as them, researchers would bearly wait (I guess only the runtime of the heaviest WU) after placing a sub-project online before getting all its output. In other words, it would be the highest computing capacity that WCGrid would ever need.

Yes, I understand this is utopic, but it could give us a little bit of insight into where we are now.

By the way, don't think of it as a thought experiment, rather a reality check.
----------------------------------------
[Edit 2 times, last edit by wcgridmember at Jul 8, 2020 8:55:37 PM]
[Jul 6, 2020 10:10:21 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

Answer to your question: How long is a piece of rope?
[Jul 6, 2020 3:00:32 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Bill F
Advanced Cruncher
USA
Joined: Jan 16, 2008
Post Count: 52
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

You Folks have too much time on your hands.... Go pick up some cheap used systems with obsolete parts that are hard to find. Start running more WCG WU's until they break the systems and then occupy yourself constructively fixing your Geezer systems. That will keep you busy enough to not be asking / answering questions like this.

Bill F
----------------------------------------


[Jul 7, 2020 5:16:12 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7242
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

You Folks have too much time on your hands.... Go pick up some cheap used systems with obsolete parts that are hard to find. Start running more WCG WU's until they break the systems and then occupy yourself constructively fixing your Geezer systems. That will keep you busy enough to not be asking / answering questions like this.
Bill F

Hey, try to be nice. Curious minds always ask questions. That is sometimes that is how we learn stuff. This is merely a theoretical question concerning limits. A mental exercise. Something to help keep the brain nimble. Oh, by the way some of those geezer systems run for years without problems.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Jul 7, 2020 4:15:19 PM]   Link   Report threatening or abusive post: please login first  Go to top 
yoerik
Senior Cruncher
Canada
Joined: Mar 24, 2020
Post Count: 413
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

You Folks have too much time on your hands.... Go pick up some cheap used systems with obsolete parts that are hard to find. Start running more WCG WU's until they break the systems and then occupy yourself constructively fixing your Geezer systems. That will keep you busy enough to not be asking / answering questions like this.

Bill F

There's nothing wrong with having a discussion. You have no right to judge how people use their free time.
----------------------------------------

[Jul 7, 2020 8:52:57 PM]   Link   Report threatening or abusive post: please login first  Go to top 
wcgridmember
Advanced Cruncher
Joined: Mar 30, 2005
Post Count: 110
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

Let me try a formulation that perhaps could help answer the question I was presenting:

For the theoretical maximum capacity to be achieved the multiplying factor would be:

current estimated (previous and future) runtime of the longest sub-project, given current capacity
/
average WU runtime.

Is there anything wrong with this formulation?

I guess it should not be too hard to estimate these numbers, or is it?
----------------------------------------
[Edit 2 times, last edit by wcgridmember at Jul 8, 2020 11:23:18 PM]
[Jul 8, 2020 9:19:49 PM]   Link   Report threatening or abusive post: please login first  Go to top 
hchc
Veteran Cruncher
USA
Joined: Aug 15, 2006
Post Count: 735
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: By how much would you need to multiply current WCGrid computing capacity so that a Client cannot download a WU

@wcgridmember, seems like the answer to your original question is basically the same as looking for the weakest link in WCG's IT systems. aka "Find the bottleneck."

In other words, you're simply asking for the breaking point in WCG's ability to distribute work units, so it makes sense to take a look at each link in the chain of work unit distribution. From this recent thread, armstrdj says:
We use Apache Mesos and Apache Aurora to manage the backend jobs that manage the flow of input data to and from the researchers (i.e., downloading them from the researchers systems, preparing them to be sent to volunteers, validating results as they come back from volunteers and packaging sets of result data and sending those sets of data back to the researchers). These two software packages allow us to treat our backend server infrastructure as a single entity--even though it runs across multiple servers--and makes it easier for us to distribute these backend tasks.

So we'd need more details on how much capacity WCG has in their server farm that runs Apache Mesos and Aurora as well as any load balancers, routers, and switches in their infrastructure. I think they moved to IBM Cloud a few years ago.

Overload any of those links in the chain, and it would prevent members from downloading more work units until the overcapacity is fixed.

To really answer your question, we'd have to measure the percent capacity for each of those links by maybe looking at system logs for the last several weeks/months and comparing it to the maximum speed or capacity.
---------------------
Or maybe your original question isn't related to bottlenecks at all. Maybe you were asking how many more factors of the current active members would be needed to complete all projects immediately (without taking into account infrastructure bottlenecks and assuming unlimited capacity)? As in, "Let's throw in 10 million more volunteers at the same time and assume their IT infrastructure doesn't break....does that crunch through all the work in a day?"
----------------------------------------
  • i3-8100 (Coffee Lake, 4C/4T) @ 3.6 GHz
  • i5-4590 (Haswell, 4C/4T) @ 3.3 GHz
  • E5800 (Wolfdale, 2C/2T) @ 3.2 GHz

----------------------------------------
[Edit 2 times, last edit by hchc at Jul 9, 2020 6:55:48 AM]
[Jul 9, 2020 1:53:28 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 35   Pages: 4   [ 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread