c# 4.0 - How to have many consumer threads using BlockingCollection -

i using producer / consumer pattern backed blockingcollection read data off file, parse/convert , insert database. code have similar can found here: http://dhruba.name/2012/10/09/concurrent-producer-consumer-pattern-using-csharp-4-0-blockingcollection-tasks/

however, main difference consumer threads not parse data insert database. bit slow, , think causing threads block.

in example, there 2 consumer threads. wondering if there way have number of threads increase in intelligent way? had thought threadpool this, can't seem grasp how done.

alternatively, how go choosing number of consumer threads? 2 not seem correct me, i'm not sure best # be. thoughts on best way choose # of consumer threads?

the best way choose number of consumer threads math: figure out how many packets per minute coming in producers, divide how many packets per minute single consumer can handle, , have pretty idea of how many consumers need.

i solved blocking output problem (consumers blocking when trying update database) adding blockingcollection consumers put completed packets in. separate thread reads queue , updates database. looks like:

input thread(s) => input queue => consumer(s) => output queue => output thread 

this has added benefit of divorcing consumers output, meaning can optimize output or change output method without affecting consumer. might allow you, example, batch database updates rather making 1 database call per record, update dozen or hundred (or more) records single call.

i show simple example of (using single consumer) in article simple multithreading, part 2. works text file filter, concepts same.